From 1ce44bedebc66acde3579425ba16bc318c9b9270 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Wed, 28 Oct 2026 21:05:01 +0000 Subject: [PATCH 001/175] docs(chord): slice 8 S2 and S3 wording as refined --- packages/chord/docs/next/decisions.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index c437875fe..f425d5bbf 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -1763,12 +1763,14 @@ validation and copies the v2 kernel added around Delta. back: `host.serve(R, { compose, accepts: { [old]: { contribution?: (old) => current, value?: (current) => old } } })`; a reader bundling an older token gets the composed value mapped to its version (computed once per composition and version, only while such a reader exists); without a `value` mapping, or for a reader newer than the provider, its - value is failed `contract_mismatch` (reported). + value is failed `contract_mismatch` (reported); such a reader fails `broken`, not `transient` (nothing heals it before + a reload, which retries broken objects; build review 2026-10-27). - **Slice 8 S3: `self.registry` is a dependency, as `self.get` (owner, 2026-10-27: "isn't self.registry meant to wait like self.get?"; my spec had it as a plain read, a mistake against design §6, §7 and §17):** activation waits until - the registry has a value (served and composed once; an empty table composes too, at the serve), bounded by - `readyTimeout`; without one in time the object fails `transient` (`unavailable/undeclared`) and is retried when the - registry gets its first value (R18); once live, a later failure of the registry changes nothing (last good value). + the registry has a value (served and composed once; an empty table composes too, at the serve); after `load`'s + setups a registry without a value cannot get one before a reload, so a reader then fails at once `transient` + (`unavailable/undeclared`) and is retried when the registry gets its first value (R18), as `self.get` of a type not + served here fails at once (refined 2026-10-27 after the second spec review: no `readyTimeout` wait at start); once live, a later failure of the registry changes nothing (last good value). - **Design audit of slices 1-6a (owner, 2026-10-27; `/tmp/design-audit-{opus,sol}.md`):** the built code follows the design except for five bugs, fixed after 6b's commit with the auditors' reproductions as tests: on a failed activation references are released after the natives closed (§6, §13); a dependency going live while the dependent From 790a98b9ab39653def35100463fc24408a3346dc Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Wed, 28 Oct 2026 21:07:05 +0000 Subject: [PATCH 002/175] =?UTF-8?q?docs(chord):=20design=20=C2=A717=20comp?= =?UTF-8?q?ose=20failure=20and=20helper=20names=20as=20built?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- packages/chord/docs/next/design.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/packages/chord/docs/next/design.md b/packages/chord/docs/next/design.md index 7429a66e9..ba7b8378b 100644 --- a/packages/chord/docs/next/design.md +++ b/packages/chord/docs/next/design.md @@ -1518,9 +1518,10 @@ const router = self.registry(Http).value; // live (host.registry(Http) outsi `compose(table, { report })` is pure and runs whenever the table changes (after a commit, when a contributor goes live or ends, when a dynamic contribution changes). It always delivers a value with its fallback built in (e.g. the first by plugin name wins) and reports conflicts through `report` (owner notices and telemetry events); readers get - a plain value and never handle conflicts. Only if `compose` throws is the registry in an error state, and readers - get an error when they use it. Work that started earlier keeps the value it read (a pinned Durable phase keeps its - old table). Helpers: `keyed(table, entriesOf)`, `ordered(table, itemsOf, orderOf?)`. + a plain value and never handle conflicts. If `compose` throws, the failure is reported and its + watches get a `failed` call, while `.value` keeps the last good composition (slice 8); only before any good + composition do readers get the error. Work that started earlier keeps the value it read (a pinned Durable phase keeps + its old table). Helpers: `keyedEntries(table, entriesOf, report)`, `orderedEntries(table, itemsOf, orderOf?)`. - **Who contributes (R21):** only facets and host-lifetime objects (`local`, empty params); their factory's `self` has `contribute`, a referenced object's `self` does not (a type error). Reading a registry's composed value stays open to every object. - **A contribution is in the table exactly while its contributor is live:** a facet's while its plugin is loaded; an From edab3efa834709a5ecfa191db751003e21e0a596 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Wed, 28 Oct 2026 21:07:42 +0000 Subject: [PATCH 003/175] docs(chord): slice 8: only Chord watchables are followed as contributions --- packages/chord/docs/next/slice-8.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/packages/chord/docs/next/slice-8.md b/packages/chord/docs/next/slice-8.md index 2bd988d7a..64c682534 100644 --- a/packages/chord/docs/next/slice-8.md +++ b/packages/chord/docs/next/slice-8.md @@ -58,8 +58,9 @@ await host.root(async (ctx) => (await ctx.get(Notes, { session })).add(note, ctx when the table is built, an older one without an adapter or a newer one is left out and reported (`HostError` `contribution_refused { registry, plugin, version }`). - **Contributions:** a plain value, or a Chord watchable (a `writable`, a derive, a document member; another - registry's value too, a cycle of registries reading each other refused when the second one serves or contributes), followed as a dependent of the table (its change recomposes); a foreign watchable (a Durable watch) - is refused at `contribute` (declaration policy). The table's entry is `{ plugin, type?, version, value }` (`type` for + registry's value too, a cycle of registries reading each other refused when the second one serves or contributes), followed as a dependent of the table (its change recomposes), recognised by Chord's own mark; anything + else is a plain value, whatever methods it has (a harness, a conversation handle or a runtime is contributed as a plain + value, not followed; build review 2026-10-27, replacing "a foreign watchable is refused"). The table's entry is `{ plugin, type?, version, value }` (`type` for an object contributor; served types carry their plugin's name). - **`compose(table, { report })`:** pure, lazy as a derive: the registry's value is marked when its table changes (a contributor goes live or ends, a watchable contribution changes) and composed when read or, while watched, in the From 6eb99ad65ba38d4e058f69ee573c390c40abe277 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Wed, 28 Oct 2026 21:33:59 +0000 Subject: [PATCH 004/175] feat(chord): Chord v2 slice 8, registries and host.root Registries: shared tokens (defineRegistry), one provider per host (host.serve(R, { compose, accepts })), contributions from facets and host-lifetime objects, plain values or followed Chord watchables, a table sorted by plugin, lazy compose in the batch that changed the table, failures left out and reported with the last good value kept; self.registry as a declared dependency, host.registry outside objects, older readers mapped by accepts[old].value; keyedEntries and orderedEntries. Native sources: self.writable, registry values and ref.state. self.retryOn wakes failed objects on registry values. host.root gives code outside objects a call scope sealed at settle, abort and host.close. Durable: registryReader(value, context); a Session close ends synchronous observers failed ("session closed"), watches still end ended. --- packages/chord/CHANGELOG.md | 1 + packages/chord/docs/next/benchmarks.md | 50 ++ packages/chord/docs/next/design.md | 23 +- packages/chord/docs/next/slice-8.md | 3 +- packages/chord/docs/next/slices.md | 24 +- packages/chord/src/next/data/errors.ts | 3 +- packages/chord/src/next/host/README.md | 59 +- packages/chord/src/next/host/attempt.ts | 217 +++++- packages/chord/src/next/host/ctx.ts | 8 +- packages/chord/src/next/host/facet.ts | 89 ++- packages/chord/src/next/host/host.ts | 121 +++- packages/chord/src/next/host/index.ts | 5 +- packages/chord/src/next/host/members.ts | 43 +- packages/chord/src/next/host/object.ts | 70 +- packages/chord/src/next/host/ref-state.ts | 69 +- packages/chord/src/next/host/reference.ts | 21 +- packages/chord/src/next/host/retry-on.ts | 55 ++ packages/chord/src/next/host/root.ts | 76 ++ packages/chord/src/next/host/routines.ts | 13 +- packages/chord/src/next/index.ts | 1 + packages/chord/src/next/registries/README.md | 81 +++ .../chord/src/next/registries/contribution.ts | 158 +++++ packages/chord/src/next/registries/entries.ts | 54 ++ packages/chord/src/next/registries/index.ts | 16 + .../chord/src/next/registries/internal.ts | 5 + packages/chord/src/next/registries/record.ts | 301 ++++++++ .../chord/src/next/registries/registry.ts | 89 +++ packages/chord/src/next/registries/value.ts | 177 +++++ packages/chord/src/next/state/README.md | 31 +- packages/chord/src/next/state/batch.ts | 43 +- packages/chord/src/next/state/derive.ts | 24 +- packages/chord/src/next/state/index.ts | 4 +- packages/chord/src/next/state/internal.ts | 15 +- .../chord/src/next/state/native-source.ts | 291 ++++++++ packages/chord/src/next/state/writable.ts | 78 +++ packages/chord/test/next/bench/calls.ts | 90 +++ packages/chord/test/next/bench/registries.ts | 108 +++ packages/chord/test/next/data/errors.test.ts | 6 +- .../chord/test/next/examples/29-registries.ts | 115 ++++ .../chord/test/next/examples/30-host-root.ts | 85 +++ packages/chord/test/next/host/attempt.test.ts | 33 +- packages/chord/test/next/host/ctx.test.ts | 6 +- packages/chord/test/next/host/facet.test.ts | 28 +- packages/chord/test/next/host/members.test.ts | 4 +- .../chord/test/next/host/retry-on.test.ts | 204 ++++++ packages/chord/test/next/host/root.test.ts | 175 +++++ .../test/next/registries/entries.test.ts | 49 ++ .../chord/test/next/registries/record.test.ts | 649 ++++++++++++++++++ .../test/next/registries/registry.test.ts | 67 ++ .../chord/test/next/registries/value.test.ts | 398 +++++++++++ .../test/next/state/native-source.test.ts | 152 ++++ .../chord/test/next/state/writable.test.ts | 210 ++++++ packages/chord/test/next/support.ts | 18 +- packages/durable/CHANGELOG.md | 4 + packages/durable/docs/spec.md | 8 +- .../durable/src/harness/registry-reader.ts | 34 + packages/durable/src/harness/task-graph.ts | 17 +- packages/durable/src/harness/view.ts | 8 +- packages/durable/src/index.ts | 1 + packages/durable/src/session/session.ts | 28 +- packages/durable/src/types.ts | 7 +- .../test/examples/37-harness-registry.ts | 172 +++++ .../durable/test/harness-observers.test.ts | 5 +- .../test/harness-registry-reader.test.ts | 198 ++++++ .../durable/test/session-observers.test.ts | 30 +- scripts/check-entry-graphs.mjs | 5 +- 66 files changed, 5036 insertions(+), 196 deletions(-) create mode 100644 packages/chord/src/next/host/retry-on.ts create mode 100644 packages/chord/src/next/host/root.ts create mode 100644 packages/chord/src/next/registries/README.md create mode 100644 packages/chord/src/next/registries/contribution.ts create mode 100644 packages/chord/src/next/registries/entries.ts create mode 100644 packages/chord/src/next/registries/index.ts create mode 100644 packages/chord/src/next/registries/internal.ts create mode 100644 packages/chord/src/next/registries/record.ts create mode 100644 packages/chord/src/next/registries/registry.ts create mode 100644 packages/chord/src/next/registries/value.ts create mode 100644 packages/chord/src/next/state/native-source.ts create mode 100644 packages/chord/src/next/state/writable.ts create mode 100644 packages/chord/test/next/bench/calls.ts create mode 100644 packages/chord/test/next/bench/registries.ts create mode 100644 packages/chord/test/next/examples/29-registries.ts create mode 100644 packages/chord/test/next/examples/30-host-root.ts create mode 100644 packages/chord/test/next/host/retry-on.test.ts create mode 100644 packages/chord/test/next/host/root.test.ts create mode 100644 packages/chord/test/next/registries/entries.test.ts create mode 100644 packages/chord/test/next/registries/record.test.ts create mode 100644 packages/chord/test/next/registries/registry.test.ts create mode 100644 packages/chord/test/next/registries/value.test.ts create mode 100644 packages/chord/test/next/state/native-source.test.ts create mode 100644 packages/chord/test/next/state/writable.test.ts create mode 100644 packages/durable/src/harness/registry-reader.ts create mode 100644 packages/durable/test/examples/37-harness-registry.ts create mode 100644 packages/durable/test/harness-registry-reader.test.ts diff --git a/packages/chord/CHANGELOG.md b/packages/chord/CHANGELOG.md index f475c4c8a..d0c9d0eff 100644 --- a/packages/chord/CHANGELOG.md +++ b/packages/chord/CHANGELOG.md @@ -14,6 +14,7 @@ ### Added - Chord v2 has a package export again, `@earendil-works/chord/next` (experimental): every layer's API for apps (contracts, data, documents, hosts, jobs, links, state, the store), and, marked internal, the store's surface for adapters (`StoreOwnerTables`, `OwnerTable`, the documents engine's reads and writes, the commit step, `applyBatch`). v2's store gains one id allocator for every id in it, claims per table and per document namespace, Durable's Storage semantics for a refused commit (it rejects that commit only; an unknown outcome stops the store), document histories `none`, `latest` and `rewindable` with reads at a point, archives carrying document revisions, and batch apply; v2 objects gain `self.follow` (another runtime's synchronous observers as sources) and persisted jobs (`persist: { resume }`, a stored `abort(id, ctx)`, deferred state changes). +- Chord v2 (`@earendil-works/chord/next`) gains registries, writables, wake signals and root scopes (slice 8): `defineRegistry(name, { version })` tokens shared by plugins, served by the defining plugin's facet with `host.serve(R, { compose, accepts? })`; contributions from facets (`defineFacet((host, self) => self.contribute(R, value))`) and host-lifetime objects (`StoreSelf.contribute`), plain values or Chord watchables, in a table sorted by plugin name and composed lazily (once at the end of `load`'s setups, then when read or followed, in the batch that changed the table); readers through `self.registry(R)` (a declared dependency: an object activates once the registry has a value, and fails `transient` with the new reason `unavailable/undeclared` when nobody serves it) and `host.registry(R, ctx)`, a `RegistryValue` with `observe(observer, ctx)`; older contributions and readers adapted by `accepts`; the helpers `keyedEntries` and `orderedEntries`; the `onError` kinds `registry_conflict`, `registry_failed`, `contribution_failed` and `contribution_refused`. `self.writable(initial)` is an in-process value an attempt sets (identity compared); native sources (writables, registry values, `ref.state`) are derive sources. `self.retryOn(...sources)` retries a failed object when a registry value changes. `host.root(fn, context, { principal? })` runs code outside objects with a handler's `ctx`, sealed when `fn` settles, `context` aborts or the host closes. - Chord v2, experimental and not exported yet: the next object runtime in `src/next` (contracts, hosts and objects, documents, derives, links and relays, routines and jobs, the store), with its tests and runnable examples in `test/next` and its design in `docs/next` (`design.md`, the decision log `decisions.md`, the slice plan `slices.md`). Its API changes with every slice until it gets an entry point. - `NativeWatchable`: an in-process value that changes and can be watched (`value`, `watch(context)` returning a `NativeWatchHandle`), the in-process counterpart of `Watchable`. - `WatchHandle` and `WatchEnd`, the one watch API, moved from Durable: a serialized observation with `value`, `start(listener)`, `stop()`, and `closed`; the listener gets a `WatchChange` (`{ value, ops, retired }`) and the context. `WatchEnd` reasons are `stopped`, `cancelled`, `ended`, `listener_error`, and `failed`. diff --git a/packages/chord/docs/next/benchmarks.md b/packages/chord/docs/next/benchmarks.md index 8e63d8d36..916c70838 100644 --- a/packages/chord/docs/next/benchmarks.md +++ b/packages/chord/docs/next/benchmarks.md @@ -583,3 +583,53 @@ each; the head lookup and scans under it are at parity or faster in the micro-be memory, 55.5 µs persisted (two commits: the record and state created, then the reservation); a state change inside `run` 2.7 µs in memory, 21.3 µs persisted (one delta commit, as the gate's (c)), 3.2 µs deferred (stored with the outcome). + +## Slice 8: registries and `host.root` (2026-10-28) + +Node 24.4.1, one CPU (`taskset -c 6`), runs alternating. HEAD is `391e83fb1`, extracted at a path of the same length +(`/home/badlogic/pi-h8`); the build was measured in the worktree and, where the worktree itself skewed a result (below), +in an extract of the same sources at `/home/badlogic/pi-x8`. + +**Paths under every object** (`test/next/bench/calls.ts`, new: runs on HEAD as it is), median per operation over nine +bursts, heap allocated per operation, three alternating runs each, HEAD → build: + +| Case | HEAD | build | +| --- | --- | --- | +| local method call (its ctx) | 2.80-2.81 µs, 6,894 B | 2.81-2.83 µs, 6,894 B | +| commit through a watched derive (`batch.compute`) | 10.34-10.43 µs, 15,312 B | 10.46-10.51 µs, 15,310 B | +| activation churn: get, live, release (an object with params) | 57.5-59.3 µs, 43,696-43,717 B | 58.1-58.9 µs, 43,782-43,787 B | +| retained per live object (10,000 kept) | 6,586 B | 6,602 B | + +The attempt keeps its registry reads, contributions, wake signals and writables in one record made by the first such +declaration, and iterates nothing when there is none: the first build allocated 1,020 B more per activation (six +arrays per attempt, an array iterator per `for … of` over an empty list, a copy from `splice`) and retained 72 B more +per live object; now +70 B allocated and +16 B retained (one field on the attempt, one on the object). + +**Existing benchmarks**, HEAD → build: the derive diamond (`derives.ts`, three runs) 9.4-9.8 → 9.5-9.9 µs median; its +pooled p95 16.5-17.1 → 18.4-20.8 µs comes from the run in which V8 tiers the commit path up (per run, after it: HEAD +10.7-11.3 µs, build 10.1-11.3 µs; the first measured runs 15.0-25.6 µs on either side, in another run each), the same +with HEAD's `batch.ts` and `derive.ts` in the build. The store gate (`store.ts`, three runs): every median within +0.4 µs (Chord store 13.0-13.9 → 13.0-14.2 µs, Chord host 17.3-18.2 → 17.3-17.8 µs, in-memory document 2.1 → 2.2 µs), +heap 3,498 → 3,144 KiB. Jobs (`jobs.ts`): start 8.4 → 8.5 µs in memory, 55.4 → 56.2 µs persisted; state change 2.7 → +2.7, 21.3 → 21.8, 3.4 → 3.5 µs. Links (`links.ts`, three runs): wire sizes, encodes, decodes and heap per message +unchanged; its round trips are one cold burst each and vary as much between HEAD runs (commit to frame, text codec, +0.055-0.076 ms). A warmed remote call (2,000 calls per burst, nine bursts, a memory link): binary 7.16-7.24 → 7.17-7.29 +µs, text 7.66-7.72 → 7.70-7.83 µs in the extracts; the worktree itself runs the same sources at 7.8-7.9 µs (binary), +an effect of the tree, not the code. + +**Registries and writables** (`test/next/bench/registries.ts`, new), median over nine bursts: `writable.set` delivered +to one watch 0.81-0.91 µs (2.9 KB allocated: the batch and the watch's call); a set of the same value 0.008 µs, 0 B +(one comparison: its batch work is a method of its own, so `set` allocates no closure context); one of 10 contributions +changed while the registry is watched 0.74-0.77 µs, of 100 1.52-1.59 µs (the table built again over every entry, one +compose per set: the counted `chord.registry.composes` is 1.00 per set). + +**Fix round 1** (2026-10-28), HEAD extract → build extract (`/home/badlogic/pi-x8`), CPU 6. `calls.ts`, two runs: local +method call 2.80 → 2.80-2.81 µs, 6,894 B both; commit through a watched derive 10.51-10.65 → 10.72 µs; activation churn +57.9-59.0 → 59.3-60.0 µs, 43,714 → 43,769-43,787 B. The derive diamond (five runs): median 9.4-9.8 → 9.5 µs, pooled p95 +16.4-16.7 → 19.0-19.2 µs. Per run, both have one slow run while V8 tiers the commit path up (HEAD its second, median/p95 +15.5/22.6-22.9 µs; the build its first, 16.4/26.7-27.2 µs) and equal runs after it (HEAD 8.8-9.6/9.6-11.2 µs, build +8.9-9.8/9.7-10.6 µs). `batch.compute` is not the cause: HEAD's `batch.ts` and `derive.ts` in the build give the same +p95, and swapping host files moves it without a pattern (code layout, not work on the path). `registries.ts`: +`writable.set` 0.87 µs, the same value 0.011-0.016 µs and 0 B (the seal check now comes first: a set of an ended +attempt's writable throws whatever its value); one of 10 contributions set 0.71-0.78 µs, of 100 1.43-1.61 µs (one run +in six 3.4 µs on either variant tried: JIT). diff --git a/packages/chord/docs/next/design.md b/packages/chord/docs/next/design.md index ba7b8378b..b3ec994d1 100644 --- a/packages/chord/docs/next/design.md +++ b/packages/chord/docs/next/design.md @@ -207,7 +207,8 @@ parent link closes. `resolve` returns the link before the worker has loaded (the | `hold(Type, params, context, { readyTimeout? })` | a reference at once that follows its object until released (section 7) | | `close(context)` | closes the links (children, routes, then the parent), then ends every object at once and tears them down, the last to go live first (reason `close`); rejects with an `AggregateError` of the native closes that threw | | `revoke(r, context)` | revocation (section 4; slice 4b) | -| `root(fn, …)` | a call scope for code outside objects (a Durable hook, a CLI); built in slice 8 (**decision needed** on its signature, section 26) | +| `root(fn, context, { principal? })` | a call scope for code outside objects (a Durable hook, a CLI): `fn` gets a handler's `ctx`, sealed when `fn` settles, `context` aborts or the host closes (slice 8 D3, section 8) | +| `registry(R, context)` | registry `R`'s composed value for code outside objects; waits for `load`, rejects when it has none (slice 8, section 17) | **Options of `createHost`:** `address`, `principal`, `principalKey` (section 4; the same function on every host), `telemetry` (section 29), `onError`, `closeTimeout`, `readyTimeout`, `heartbeat: { ping?, close? }`; later `revokedBy` @@ -216,7 +217,7 @@ and closed by the app with `store.close(ctx)` after `host.close`; the host never background, as a tagged `HostError`: `listener_error` (a watch listener threw), `delivery_failed` (a frame delivery threw), `activation_failed { object, failure }`, `close_failed { object, native, error }`, `close_timeout { object, native }`, `drain_timeout { object }`, `protocol_error { error }`, `resolve_failed { address, error }`; later routine -errors (slice 5), `derive_failed { object, member?, error }` (a derive's compute threw, once per failure episode, slice 6a), `object_failed { object, failure }` (a live object failing, slice 6a, `broken`; `transient` when a `self.follow` source failed, slice 6b R4; `failure` where the value is a `FailureCause`, `error` otherwise, 2026-10-27), `job_interrupted { object, job, id, restarts }` (a persisted run ended at the restart limit, slice 6b D6) and `storage_failed { error }` +errors (slice 5), `derive_failed { object, member?, error }` (a derive's compute threw, once per failure episode, slice 6a), the registries' `registry_conflict { registry, message, plugins }`, `registry_failed { registry, error }`, `contribution_failed { registry, plugin, type?, error }` and `contribution_refused { registry, plugin, type?, version }` (each once per episode, slice 8), `object_failed { object, failure }` (a live object failing, slice 6a, `broken`; `transient` when a `self.follow` source failed, slice 6b R4; `failure` where the value is a `FailureCause`, `error` otherwise, 2026-10-27), `job_interrupted { object, job, id, restarts }` (a persisted run ended at the restart limit, slice 6b D6) and `storage_failed { error }` (slice 6a), which every app must handle: Chord never ends the process, the app exits with its own code for its supervisor or reopens. @@ -546,7 +547,7 @@ host.serve(Notes, ({ session }, principal, self) => link protocol (adding one bumps the protocol version; a receiver rejects unknown reasons; a test pins the list): `invalid` (`args`, `params`, `member`, `placement`, `reentrant`, `read_after_bulk`, `cycle`), `contract_mismatch` (`updating`), `forbidden` (`revoked`), `gone`, `not_ready`, `released` (`ended`, `closed`, `settled`, `version`, `discarded`; none for a holder's own release), `too_large` (`part`), `unavailable` (`locked`, - `park_timeout`, `busy`, `storage_failed`), `aborted`. A failed check names TypeBox's instance path ("unknown field", "missing", else TypeBox's + `park_timeout`, `busy`, `storage_failed`, `undeclared`: a registry no plugin serves here, slice 8), `aborted`. A failed check names TypeBox's instance path ("unknown field", "missing", else TypeBox's message; keys containing `/` or `~` make the path ambiguous, documented). Plugins throw `PluginError(code, message, details?)` for codes of their own; anything else crosses a link as the plugin code `internal` with its message, never a stack or a cause; plugin errors never use Chord's codes. An error rebuilt from the wire is not copied and only its @@ -1505,7 +1506,7 @@ host.serve(Http, { // the defining p accepts: { 1: { contribution: (old: HttpContributionV1) => upgrade(old), value: (router) => routerV1(router) } }, }); self.contribute(Http, { routes: { … }, wraps: [ … ] }); -const router = self.registry(Http).value; // live (host.registry(Http) outside objects) +const router = self.registry(Http); // a declared dependency; read router.value at use (await host.registry(Http, ctx) outside objects) ``` - Each contributor (a facet or an object) contributes **at most one value per registry**; the table is an immutable, @@ -1521,7 +1522,9 @@ const router = self.registry(Http).value; // live (host.registry(Http) outsi a plain value and never handle conflicts. If `compose` throws, the failure is reported and its watches get a `failed` call, while `.value` keeps the last good composition (slice 8); only before any good composition do readers get the error. Work that started earlier keeps the value it read (a pinned Durable phase keeps - its old table). Helpers: `keyedEntries(table, entriesOf, report)`, `orderedEntries(table, itemsOf, orderOf?)`. + its old table). Helpers: `keyedEntries(table, entriesOf, report)` (a map, the first by plugin name wins, a duplicate + reported), `orderedEntries(table, itemsOf, orderOf?)`; both keep each item's plugin (named so beside the contracts' + `keyed`, slice 8 build). - **Who contributes (R21):** only facets and host-lifetime objects (`local`, empty params); their factory's `self` has `contribute`, a referenced object's `self` does not (a type error). Reading a registry's composed value stays open to every object. - **A contribution is in the table exactly while its contributor is live:** a facet's while its plugin is loaded; an @@ -1962,14 +1965,20 @@ loop. Chord has no stream type (it would need its own rules for drops, cancellat loading object (refines R17). A missing tool gives its call an error result the model sees; a missing model makes the generation report "model unavailable" and wait (not fail) until a model contribution appears; a task whose definition is missing stays blocked (`missing_task`) until the plugin is installed or the task is aborted, as Durable does. +- **Durable's views from other objects** (slice 8 §5): the harness object contributes its `Native` to a + registry of p3's; another object declares that registry (`self.registry`), wakes on it (`self.retryOn`) and follows + a view through its value (`self.follow`). A Session close ends Durable's synchronous observers `failed` ("session + closed"; decision "Slice 8 S1"; app watches still end `ended`), so the follower fails `transient` and follows the + next Harness once its contribution arrives. `registryReader`'s listener runs inside Chord's batch: it only wakes + Durable's scheduler (a Chord call there throws `invalid/reentrant`). - `HarnessModels` and `HarnessEnvironments` start with one contribution each, from the models and envs plugins' objects that own the natives; several providers or environment kinds are their `compose`. After a models reload the next request uses the new runtime; a running stream finishes on the old one. - **Chord from Durable code (Q6):** tools, hooks and sections get Durable's `ToolExecutionApi` and a `Context`, never a Chord `ctx`. p3's core opens the harness with a base context holding the host under a typed key (`withContextValue(ChordHostKey, host, …)`); every phase context inherits it. A core helper opens a call scope from - any Durable context: `chord(async (ctx) => …, context)` = `host.root(fn, { signal: context.abortSignal, telemetry: - context.telemetry })` (host principal; attribution as data; references released when `fn` settles). No wrapping of + any Durable context: `chord(async (ctx) => …, context)` = `host.root(fn, context)` (host principal; attribution as + data; references released when `fn` settles; slice 8 D3). No wrapping of tools and no p3 `defineTool`. Clients are never reachable this way (they are not addressable): a tool that needs the user goes through an owner's state, e.g. the Questions service (`apps/p2/docs/questions.md`): the model-facing `ask_user` tool and any other tool call `Questions.ask`/`wait`, tabs render the open questions and `answer` them. diff --git a/packages/chord/docs/next/slice-8.md b/packages/chord/docs/next/slice-8.md index 64c682534..65f1c494f 100644 --- a/packages/chord/docs/next/slice-8.md +++ b/packages/chord/docs/next/slice-8.md @@ -1,6 +1,7 @@ # Slice 8: registries and `host.root` (spec) -Status: **ready to build** (2026-10-27; D1-D3, the registry failures and S1-S3 decided by the owner; two spec reviews, +Status: **built** (2026-10-28; build review and two fix rounds by the same four reviewers, +`/tmp/s8-build-review-*.md`, `/tmp/s8-build-rereview-*.md`). Before: ready to build (2026-10-27; D1-D3, the registry failures and S1-S3 decided by the owner; two spec reviews, `/tmp/s8-review{,2}-{astra,fable,opus,sol}.md`, found no remaining blocker; S3 refined after the second: with no value after `load`, a reader fails at once, as `self.get` of a type not served here) diff --git a/packages/chord/docs/next/slices.md b/packages/chord/docs/next/slices.md index 87d395336..cf346eb9e 100644 --- a/packages/chord/docs/next/slices.md +++ b/packages/chord/docs/next/slices.md @@ -27,15 +27,16 @@ slice spec's section citations are for orientation, not a reading limit (owner, - **L1** (`src/next/contracts`, `src/next/data`): contracts, params and identity, argument checks, principal keys, errors. `index.ts` for apps, `internal.ts` for Chord's own code. -- **Slices 1, 2, 3a, 3b, 4a, 4b, 5, 6a, 6b** (committed; 5 `4a5b45c45`, 6a and 6b 2026-10-27/28): `src/next/documents`, `state`, `host`, `jobs`, `link`, - `remote`, `store`; examples 01-25; benchmarks in `benchmarks.md`. +- **Slices 1, 2, 3a, 3b, 4a, 4b, 5, 6a, 6b, 8** (committed; 5 `4a5b45c45`, 6a and 6b 2026-10-27/28, 8 2026-10-28): `src/next/documents`, `state`, `host`, `jobs`, `link`, + `remote`, `store`, `registries`; examples 01-30 (Durable's 35-37); benchmarks in `benchmarks.md`. - **Chord's shared code** (upstream pi): `src/watch.ts` (`Watch`, `NativeWatch`, `WatchHandle`, `WatchEnd`, `NativeWatchable`), Delta (`track`, `diffRevisions`, `applyImmutable`, encoders, `PendingOps`, `appendText`), context, JSON, output, SQLite drivers. ## The slices -Order (owner, 2026-10-22: routines and jobs before the store and reload). +Order (owner, 2026-10-22: routines and jobs before the store and reload; 2026-10-27: slice 8 before slice 7, whose +reload has to reload real registries). | # | Slice | In short | | --- | --- | --- | @@ -48,8 +49,8 @@ Order (owner, 2026-10-22: routines and jobs before the store and reload). | 5 | **Routines and jobs** | `self.routines`, `self.job` | | 6a | **The store** | the store (backends, tables, queries, writes, commit queue, outbox, documents engine, migrations, export), persisted documents, `self.table`, keyed watches and retirement | | 6b | **Durable and jobs on the store** | persisted jobs, Durable's `StoreStorage`, the Chord source over Durable's views (`self.follow`) | +| 8 | **Registries and `host.root`** | registries, `self.writable`, `retryOn`, Durable's `registryReader`, `host.root` (`chord(fn, context)` and the harness object are p3's base code) | | 7 | **Reload and rollout** | plugin loading, staging, swaps and restarts, revisions, `accepts` in use | -| 8 | **Registries and `host.root`** | registries, `self.writable`, `retryOn`, a `RegistryReader` for Durable, `host.root` (`chord(fn, context)` and the harness object are p3's base code) | | 9 | **Telemetry** | spans, events, metrics across hosts | The playground (`apps/chord-playground`) moves to the new code once slice 4 exists. Round-3 cuts (C3-C16) and @@ -211,14 +212,13 @@ deleted (the slice's spec and code are the record). Moved here from `carry-over. internal ones are rebuilt; add matching only if a real need appears. (W1.) - **Retrying broken failures:** reload retries a broken object as `ref.retry()` does (`ChordObject.retry`: the backoff skipped, an attempt now; holders are told `loading`). (Slices 2, 3b.) - -### Slice 8: registries and `host.root` - -- **Durable's views from other objects** (from slice 6b): the harness carried in a registry entry, so an object other - than the one that opened Durable can `self.follow` its views. - -- Specified in `slice-8.md` (decided 2026-10-27): `host.root` (D3), `retryOn` (D2), registries recomposing in the - owner's flush (W4); `host.activity` on the facet's host dropped (nothing needs it). +- **Registries on reload** (from slice 8 §6): a reload replaces a plugin's contributions and a provider atomically, in + one batch (staging composes nothing, decision D1 of "Review round 1"); the record keeps the provider, facet + contributions by plugin and object contributions by object apart for it (`RegistryRecord.serve` refuses a second + serve today). Older registry readers across a reload of the provider (their mapped values follow the new + `accepts[old].value`). A `writable` across a pure swap: the new factory makes a new `self.writable(initial)`, so a + contribution goes back to its initial value until the restarted routine sets it (MCP's tools would vanish briefly). + `retryOn` watches across a reload (a reload retries failed objects anyway). (Slice 8 spec.) ### Slice 9: telemetry diff --git a/packages/chord/src/next/data/errors.ts b/packages/chord/src/next/data/errors.ts index 697f3b578..623c05fe2 100644 --- a/packages/chord/src/next/data/errors.ts +++ b/packages/chord/src/next/data/errors.ts @@ -18,7 +18,8 @@ export const CHORD_ERROR_REASONS = { // `discarded`: a transaction cut at the seal. released: ["ended", "closed", "settled", "version", "discarded"], too_large: ["part"], - unavailable: ["locked", "park_timeout", "busy", "storage_failed"], + // `undeclared`: a registry no plugin serves on this host. + unavailable: ["locked", "park_timeout", "busy", "storage_failed", "undeclared"], aborted: [], } as const; diff --git a/packages/chord/src/next/host/README.md b/packages/chord/src/next/host/README.md index 2474ca16a..da13258bc 100644 --- a/packages/chord/src/next/host/README.md +++ b/packages/chord/src/next/host/README.md @@ -4,8 +4,10 @@ Hosts, facets, objects, natives and references: hosts serving `local` and `remot principals and variants, links to children, to a parent and over routes to addressable hosts (with relays between them), the heartbeat, the `ctx` of method handlers, (slice 4b) revocation, cycles across hosts, presence, keep-alive counts, size limits per part and compression at the origin, (slice 5) routines and the objects' jobs -(declared here, built in `jobs/`), and (slice 6a) the host's store: tables, persisted documents and eager table -migrations at activation (built in `store/` and `documents/`), and live objects that fail. +(declared here, built in `jobs/`), (slice 6a) the host's store: tables, persisted documents and eager table +migrations at activation (built in `store/` and `documents/`), and live objects that fail, and (slice 8) registries +(served, contributed and read here, built in `registries/`), writables, wake signals (`retryOn`) and root scopes +(`host.root`). ## For apps (`index.ts`) @@ -189,8 +191,41 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded: route's `resolve` rejected; the address waits its backoff), `routine_failed` (`object`, `routine`: its position, `error`: a routine call threw, or its watch could not be taken; a source's failure calls never reach a routine), `job_failed` (`object`, `job`, `id`, `error`: a run threw, not after an abort), `job_finish_failed` (the same: a `finish` threw; it is retried) and - `job_dropped` (`object`, `job`, `id`: a blocked run dropped without `finish`, by `abort` or its object's end). A - throwing `onError` is ignored. + `job_dropped` (`object`, `job`, `id`: a blocked run dropped without `finish`, by `abort` or its object's end), and + the registries' (slice 8, each once per episode) `registry_conflict` (`registry`, `message`, `plugins`: a conflict + `compose` resolved), `registry_failed` (`registry`, `error`: `compose` or an older version's `value` adapter threw), + `contribution_failed` (`registry`, `plugin`, `type?`, `error`: a watchable contribution failed or ended, or its + adapter threw) and `contribution_refused` (`registry`, `plugin`, `type?`, `version`). A throwing `onError` is + ignored. +- **Registries** (slice 8, `registries/`): a facet serves a registry it defines with `host.serve(R, { compose, + accepts? })` (one provider per host; a second serve throws at `load`); a facet's `self.contribute(R, value)` (its + setup's second argument, `FacetSelf`) is in the table while the plugin is loaded, a host-lifetime object's (`local`, + empty params: `StoreSelf.contribute`; any other factory throws, R21) from `live` until it fails, ends or restarts, + entering before anyone sees it live and leaving at its seal. One value per registry each; a Chord watchable (a + source, a constant, a document handle, a reference member, a writable, a registry value) is followed through the + batch, anything else is a plain value whatever methods it has (registries carry natives); a registry value closing + a cycle of registries (through the tables and the factory's own earlier contributions) is refused `invalid/cycle` + at `contribute` (an edge lives as long as its contribution is in a table). Every served registry composes once at the end of `load`'s setups, before any object activates. + `self.registry(R)` is a declared dependency: with no value at activation (no provider, a first `compose` that threw, + a version not served here) the attempt fails `dependency_failed` (`transient`, the registry's error as the cause, + the references kept) at once, and the registry's first value retries it (R18); a version the provider does not + serve fails it `broken` (`contract_mismatch`: only a reload helps). `host.registry(R, ctx)` waits for `load` and + rejects without a value. +- **Wake signals** (`self.retryOn(...sources)`, D2): registry values (constants are accepted and never wake; anything + else throws: a reference's member, the attempt's writable, `ref.state`). The object follows them from the factory's + return (before hydration, references and natives, so a failed open still has them) until the next attempt's + declaration replaces them or it ends. A new value retries a failed object (`transient` or `broken`) as `ref.retry()` + does, after the batch; one while it activates is remembered and retries it at once if that attempt fails; one while + live is ignored; failure calls wake nothing; a composition made by the object's own contributions leaving (it + failed live) does not wake it. There is no backoff between wakes: wake an object on sources that change rarely + (models, a harness), not on every event (a writable set per message retries a failing object per message). +- `host.root(fn, ctx, { principal? })` (slice 8 D3): `fn(ctx)` with a handler's `Ctx

`: the host's principal or the + given one, this host's incarnation, `ctx`'s telemetry, a signal linked to `ctx`'s, `ctx.get`. When `fn` settles, + `ctx` aborts or the host closes, the scope is sealed: its references released (their calls reject `released`), a + waiting `ctx.get` gives up, a later one is refused `released/settled`, its signal aborted. An abort rejects + `host.root` at once (an uncooperative `fn` runs on and fails at its next use of Chord); an aborted `ctx` runs no + `fn`; `host.close` rejects it at once `released/closed`, as a method's caller when its object ends; after `close` + it rejects `released/closed`. Not counted in `host.activity`. - `host.load([{ name, facet }], ctx)`: once per host; runs every setup, then activates every type with empty params and resolves when each has had its first outcome, live or failed (the start barrier; later retries never hold it). A setup that throws rejects it with that error and closes the host as `close` does, its links included. Gets wait @@ -281,7 +316,8 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded: `broken`; a live object whose read fails so fails while live (`object_failed`), keeping its documents for the next attempt. - A factory `(params, self) => implementation` declares with `self.state`, `self.derive`, `self.get`, `self.open`, - `self.on`, `self.onLive` and `self.job` (`jobs/`), and writes with `self.tx`. It runs once per attempt. The implementation has exactly the contract's members: state + `self.on`, `self.onLive`, `self.job` (`jobs/`), `self.registry`, `self.writable` and `self.retryOn` (below), and + writes with `self.tx`. It runs once per attempt. The implementation has exactly the contract's members: state members are JSON watchables (document handles, derives, reference members, constants), methods `(args, ctx) => Promise`, or `(ctx) => Promise` for a method without arguments (the reference omits them too: `notes.clear(ctx)`). A remote factory `(params, principal, self) => ({ variant, members })` gets the principal its @@ -456,8 +492,9 @@ caller), `Placement` (where a reference or an arriving hold goes: here, a route' served here is placed here), `Route` (one per routed kind: per address its `Path`, at most one running `resolve`, the backoff per address; an address is kept while it has holders, a link or a running `resolve`), `ChordObject` (one per identity: its state across attempts, holders, retries, cycle checks with its instance id and -the chain it was created for, lingering, ending and revocation, and the count of its running and finishing job runs, -which keeps it like a reference: `countRuns`), `Activity` (`host.activity`: counters, one +the chain it was created for, lingering, ending and revocation, the count of its running and finishing job runs, +which keeps it like a reference: `countRuns`, its contributions while live and its wake signals), `RetryWakes` (an +object's `retryOn` subscriptions), `RootScopes` (the open `host.root` scopes, sealed once each), `Activity` (`host.activity`: counters, one publication per microtask turn), `References` (one type's presence list), `RevocationKeys` (each principal's keys, in a `WeakMap`), `Attempt` (one activation: it is the @@ -471,9 +508,9 @@ native: opened once, closed once, a late value closed at once; `within` bounds a windows per identity and when each ends, holding no object), `Binding` (a reference's link to its object or remote object: its state, the object that declared it, its per-reference watches, calls, parked calls and the waits for a state; released, it keeps only its name), `ParkedCalls` (a reference's queue of calls made while loading, a timer -each), `ReferenceState` (`ref.state`: its native watches, told of each change), `CallContext` (a handler's -`ctx`), `withLinkedAbortSignal` (a call's or a get's context aborted by the caller and by a long-lived signal, its +each), `ReferenceState` (`ref.state`, a native source: its native watches, told of each change; derives over it +marked), `CallContext` (a handler's or a root scope's `ctx`), `withLinkedAbortSignal` (a call's or a get's context aborted by the caller and by a long-lived signal, its reference's or its attempt's, linked only until it settles: `AbortSignal.any` keeps cleanup entries per call while its -sources live before Node 24.16.0), `checkImplementation` (the variant and member check), `setUp` (runs one facet's setup) and -`resolveWatchable` (the member check, Q14), and `nextDelay` (the 1 s to 60 s doubling every backoff shares). Cycle checks follow the waits of activating objects with a visited set, +sources live before Node 24.16.0), `checkImplementation` (the variant and member check), `setUp` (runs one facet's setup), +`resolveWatchable` (the member check, Q14), `contributionOf` (a contribution as Chord follows it, the cycle check), and `nextDelay` (the 1 s to 60 s doubling every backoff shares). Cycle checks follow the waits of activating objects with a visited set, linear in the waits. diff --git a/packages/chord/src/next/host/attempt.ts b/packages/chord/src/next/host/attempt.ts index b6d04a898..259b6a0cf 100644 --- a/packages/chord/src/next/host/attempt.ts +++ b/packages/chord/src/next/host/attempt.ts @@ -8,7 +8,7 @@ import type { JsonValue } from "../../json.ts"; import type { NativeWatchable, Watchable, WatchEnd } from "../../watch.ts"; import type { Contract, Implementation, ParamsOf } from "../contracts/index.ts"; import { contractData, type ShapeOf } from "../contracts/internal.ts"; -import { ChordError, type FailureCause, type Principal } from "../data/index.ts"; +import { ChordError, type FailureCause, isChordError, type Principal } from "../data/index.ts"; import { asChordError, checkArgs, checkParams, invalid, openFailure } from "../data/internal.ts"; import type { HandleOf, JsonObject, StateDefinition, Tx } from "../documents/index.ts"; import { @@ -21,7 +21,10 @@ import { } from "../documents/internal.ts"; import type { JobDefinition, JobHandle, JobImplementation } from "../jobs/index.ts"; import { type AnyJobImplementation, Job } from "../jobs/internal.ts"; -import { Derive, type DeriveSource } from "../state/internal.ts"; +import type { Contribution, Registry, RegistryValue } from "../registries/index.ts"; +import { type RegistryContribution, type RegistryEdge, RegistryValueSource } from "../registries/internal.ts"; +import type { Writable } from "../state/index.ts"; +import { Derive, type DeriveSource, WritableSource } from "../state/internal.ts"; import type { Columns } from "../store/columns.ts"; import { tableSqlName } from "../store/sql.ts"; import type { ChordStore } from "../store/store.ts"; @@ -32,7 +35,7 @@ import { CallContext, type Caller, type Ctx, withLinkedAbortSignal } from "./ctx import { Deadline } from "./deadline.ts"; import { type FollowAttach, FollowSource } from "./follow.ts"; import type { HostError } from "./host.ts"; -import { attachDerives, checkImplementation, type KeyedBinding, type Members } from "./members.ts"; +import { attachDerives, checkImplementation, contributionOf, type KeyedBinding, type Members } from "./members.ts"; import { type CloseFailed, drained, @@ -44,6 +47,7 @@ import { } from "./native.ts"; import type { ChordObject } from "./object.ts"; import type { Binding, CheckedParams, Reference, Target } from "./reference.ts"; +import { type RetrySource, retrySource } from "./retry-on.ts"; import { type OnHandler, type OnLiveHandler, @@ -53,19 +57,48 @@ import { } from "./routines.ts"; /** - * What a factory declares with. Declarations (`state`, `derive`, `get`, `open`, `on`, `onLive`, `job`) are valid only - * while it runs. `P` is the app's principal type, which the ctx of routine calls and job runs carry. + * What a factory declares with. Declarations (`state`, `derive`, `get`, `registry`, `writable`, `retryOn`, `open`, + * `on`, `onLive`, `job`) are valid only while it runs. `P` is the app's principal type, which the ctx of routine calls + * and job runs carry. */ export interface Self

extends RoutineDeclarations

{ /** This object's documents of `definition`, in memory for its life (across its attempts). */ state>(definition: D): HandleOf; - /** A value computed from `sources` by the pure, synchronous `compute`. */ - derive[], T extends JsonValue>( + /** + * A value computed from `sources` (JSON watchables and native ones: a `writable`, a registry value, `ref.state`) by + * the pure, synchronous `compute`; always JSON. + */ + derive | NativeWatchable)[], T extends JsonValue>( sources: S, - compute: (values: { readonly [I in keyof S]: S[I] extends Watchable ? V : never }) => T, + compute: ( + values: { + readonly [I in keyof S]: S[I] extends Watchable + ? V + : S[I] extends NativeWatchable + ? V + : never; + }, + ) => T, ): Watchable; /** A reference to another object, live once this object is, released when it ends. */ get(Type: C, params: ParamsOf): Reference; + /** + * Registry `R`'s composed value as this plugin's version of `R` sees it: a declared dependency, as `get` is + * (activation waits until it has a value; with none it fails `transient` and is retried when one arrives, R18). + * Read `.value` at use; once live, a later failure of the registry keeps its last good value. + */ + registry(R: Registry): RegistryValue; + /** + * An in-process value this attempt sets (`set`, identity compared): a contribution, a derive source and an `on` + * source, never a state member; its watches end with the attempt, and `set` after its end throws `released`. + */ + writable(initial: T): Writable; + /** + * Wake signals: a new value of any of `sources` (registry values; constants never change) retries this object + * while it is failed, as `ref.retry()` does, and once more at once if its current activation fails. Followed from + * the factory's return until the next attempt's declaration or the object's end; failures wake nothing. + */ + retryOn(...sources: readonly (Watchable | NativeWatchable)[]): void; /** * A native, opened during activation after the references are live and the earlier natives are open, and closed * in reverse when the attempt ends. `open` gets a context aborted when the attempt is abandoned; `close` one aborted @@ -101,6 +134,12 @@ export interface StoreSelf

extends Self

{ state>(definition: D): HandleOf; /** This object's table `definition` in the host's store, once per table name. */ table(definition: TableDefinition): Table; + /** + * Contributes `value` to registry `R` from the moment this object is live until it ends, fails or restarts (R21: a + * host-lifetime object): a plain value, or a Chord watchable of one (followed: its change recomposes). At most one + * per registry. + */ + contribute(R: Registry, value: Contribution): void; } /** @@ -111,6 +150,27 @@ export type Factory = Shape ? (params: ParamsOf, self: keyof ParamsOf extends never ? StoreSelf

: Self

) => Implementation> : (params: ParamsOf, principal: P, self: Self

) => Implementation>; +/** How an activation fails on a declared registry without a value (`#readRegistries`). */ +type UnreadRegistry = Extract; + +/** No declarations of a kind: the shared empty list. */ +const NONE: readonly never[] = []; + +/** An attempt's registry reads, contributions and wake signals, and its writables (slice 8). */ +type RegistryDeclarations = { + /** The registry values it declared (`self.registry`), registered as read during activation (`reads`). */ + readonly registries: RegistryValueSource[]; + /** Releases its registry reads (with its references). */ + readonly reads: (() => void)[]; + /** Its wake signals (`self.retryOn`), handed to its object right after the factory. */ + readonly retrySources: RetrySource[]; + /** Its writables, whose watches end with it. */ + readonly writables: { detach(error: Error): void }[]; + /** Its contributions as declared, resolved during activation into `contributions`. */ + readonly declared: { readonly registry: Registry; readonly value: unknown }[]; + contributions: readonly RegistryContribution[]; +}; + /** How an attempt's activation ended. `dependency_failed` (R18) keeps the attempt's references. */ type Activation = | { readonly kind: "live" } @@ -174,6 +234,8 @@ export class Attempt implements StoreSelf, KeyDeclarer { #abandoned: Exclude | undefined; /** A followed source's end while this attempt activated: it fails as it goes live (if its activation returned). */ #endedEarly: ChordError | undefined; + /** Its registry reads, contributions, wake signals and writables, made by the first: most attempts have none. */ + #registryDeclarations: RegistryDeclarations | undefined; constructor(object: ChordObject, previous: Attempt | undefined, context: Context) { this.#object = object; @@ -221,13 +283,21 @@ export class Attempt implements StoreSelf, KeyDeclarer { return handle as unknown as Table; } - derive[], T extends JsonValue>( + derive | NativeWatchable)[], T extends JsonValue>( sources: S, - compute: (values: { readonly [I in keyof S]: S[I] extends Watchable ? V : never }) => T, + compute: ( + values: { + readonly [I in keyof S]: S[I] extends Watchable + ? V + : S[I] extends NativeWatchable + ? V + : never; + }, + ) => T, ): Watchable { this.#assertDeclaring("derive"); // The values are the sources' values, in order; the result is `T`. - const derive = new Derive(sources, compute as (values: readonly JsonValue[]) => T, this.#object.host.batch); + const derive = new Derive(sources, compute as (values: readonly unknown[]) => T, this.#object.host.batch); this.#derives.push(derive); return derive as unknown as Watchable; } @@ -242,6 +312,50 @@ export class Attempt implements StoreSelf, KeyDeclarer { return binding.reference as Reference; } + registry(R: Registry): RegistryValue { + this.#assertDeclaring("registry"); + const value = this.#object.host.registryRecord(R.name).value(R.version); + this.#declarations().registries.push(value); + // Its values are the composition as `R`'s version has it, `V`. + return value as RegistryValue; + } + + writable(initial: T): Writable { + this.#assertDeclaring("writable"); + const owner = { + context: this.#context, + assertUsable: () => { + if (this.#token.sealed) throw this.#object.ended(); + }, + }; + const writable = new WritableSource(initial, owner, this.#object.host.batch); + this.#declarations().writables.push(writable); + return writable; + } + + retryOn(...sources: readonly (Watchable | NativeWatchable)[]): void { + this.#assertDeclaring("retryOn"); + const { retrySources } = this.#declarations(); + for (const source of sources) retrySources.push(retrySource(source, this.#object.label)); + } + + contribute(R: Registry, value: Contribution): void { + this.#assertDeclaring("contribute"); + const { served, label, host } = this.#object; + // R21: the types allow it only to host-lifetime objects; plugin code is bundled without type checks. + if (served.kind !== "local" || !served.hostLifetime) { + throw new Error(`${label}: self.contribute is valid only in a local type with empty params`); + } + const { declared } = this.#declarations(); + if (declared.some(({ registry }) => registry.name === R.name)) { + throw new Error(`${label}: registry ${R.name} is contributed to twice`); + } + // Refused at `contribute` (declaration policy), also a cycle within this factory's own declarations; the edges + // live with the contributions in the tables, from live on. + if (value instanceof RegistryValueSource) host.registryRecord(R.name).assertNoCycle(value, this.#pendingEdges()); + declared.push({ registry: R as Registry, value }); + } + open( open: (context: Context) => Promise, close: (value: T, reason: NativeCloseReason, context: Context) => void | Promise, @@ -384,6 +498,11 @@ export class Attempt implements StoreSelf, KeyDeclarer { }); } + /** Its contributions, resolved by its activation: in the registries' tables while its object is live. */ + get contributions(): readonly RegistryContribution[] { + return this.#registryDeclarations?.contributions ?? NONE; + } + /** The variant a remote object's factory chose; `undefined` for a local object. */ get variant(): string | undefined { return this.#variant; @@ -494,27 +613,34 @@ export class Attempt implements StoreSelf, KeyDeclarer { } /** - * Detaches its derives (once their object no longer needs them: reads throw `released`, their watches end) and its - * followed sources (their observers detached, their watches ended). + * Detaches its derives (once their object no longer needs them: reads throw `released`, their watches end), its + * followed sources (their observers detached, their watches ended) and its writables (their watches ended). */ detach(): void { const ended = this.#object.ended(); for (const derive of this.#derives) derive.detach(ended); for (const follow of this.#follows) follow.detach(); + const declarations = this.#registryDeclarations; + if (declarations !== undefined) for (const writable of declarations.writables) writable.detach(ended); } - /** Releases its references, and the previous attempt's if this one never bound its own. */ + /** Releases its references and registry reads, and the previous attempt's if this one never bound its own. */ releaseReferences(): void { this.#releasePrevious(); for (const { binding } of this.#references) binding.release(); + const declarations = this.#registryDeclarations; + if (declarations !== undefined) for (const release of declarations.reads.splice(0)) release(); } /** The activation steps; each catches its own failures with their kind (the table in slice 2's spec). */ async #activate(deadline: Deadline): Promise { let implementation: unknown; let dependencies: Target[]; + let unread: UnreadRegistry | undefined; try { implementation = this.#declare(); + // Right after the factory, before anything can fail: a failed attempt keeps its wake signals. + this.#object.retryOn(this.#registryDeclarations?.retrySources ?? NONE); const { documents } = this.#object; // The object holds the pins from now on. documents.hydrate(this.#declared, this.#pins.splice(0), this.#opening); @@ -528,12 +654,15 @@ export class Attempt implements StoreSelf, KeyDeclarer { host.bind(binding, contract, params, this.#opening), ); for (const dependency of dependencies) this.#object.assertNoCycle(dependency, this.#opening); + if (this.#registryDeclarations !== undefined) unread = this.#readRegistries(this.#registryDeclarations); } catch (error) { return this.#broken(error); } finally { // Bound first: a dependency held only by the previous attempt never ends in between. this.#releasePrevious(); } + // No value after `load`'s setups: none comes before a first composition (which retries it, R18) or a reload. + if (unread !== undefined) return this.#abandoned ?? unread; try { await Promise.all(dependencies.map((dependency) => this.#object.waitFor(dependency, this.#opening))); } catch (error) { @@ -586,6 +715,8 @@ export class Attempt implements StoreSelf, KeyDeclarer { const checked = checkImplementation(this.#object.served.contract, implementation); this.#members = checked.members; this.#variant = checked.variant; + if (this.#registryDeclarations !== undefined) + this.#registryDeclarations.contributions = this.#resolveContributions(this.#registryDeclarations); const { host, label } = this.#object; attachDerives(this.#derives, this.#members.state, (member) => (error) => { // During activation a throwing derive fails the attempt, which `activation_failed` reports alone. @@ -607,6 +738,41 @@ export class Attempt implements StoreSelf, KeyDeclarer { return this.#abandoned ?? { kind: "live" }; } + /** + * Registers this attempt's object as a reader of every registry it declared (woken by a first value, R18, until its + * references are released); returns how the activation fails on the first with no value, `undefined` when every + * one has a value. A version the provider does not serve (S2) is final until a reload: broken, the + * `contract_mismatch` itself the cause. Otherwise (no provider, a first `compose` that threw) a value may still come: + * `transient` (R18, the references kept), naming the registry, its error as the cause. + */ + #readRegistries({ registries, reads }: RegistryDeclarations): UnreadRegistry | undefined { + let unread: UnreadRegistry | undefined; + for (const value of registries) { + reads.push(value.read(this.#object)); + const start = value.start(); + if (unread !== undefined || start.kind === "value" || start.last !== undefined) continue; + const { error } = start; + if (isChordError(error, "contract_mismatch")) { + unread = { kind: "failed", cause: { kind: "broken", error } }; + continue; + } + const message = `${this.#object.label}: registry ${value.name} has no value`; + const cause = new ChordError({ code: "unavailable", message }, { cause: error }); + unread = { kind: "dependency_failed", cause: { kind: "transient", error: cause } }; + } + return unread; + } + + /** Its declared contributions, as Chord follows them (its references are live now), by its plugin and type. */ + #resolveContributions({ declared }: RegistryDeclarations): readonly RegistryContribution[] { + const { served, host } = this.#object; + const contributor = { plugin: served.plugin, type: served.contract.name }; + const pending = this.#pendingEdges(); + return declared.map(({ registry, value }) => + contributionOf(host.registryRecord(registry.name), contributor, registry.version, value, pending), + ); + } + /** * Brings the declared tables to their definitions before the references are bound (slice 6a §9): created fresh or * migrated eagerly, with the `readyTimeout` deadline paused meanwhile (resumed with the budget it had left). @@ -686,6 +852,29 @@ export class Attempt implements StoreSelf, KeyDeclarer { this.#previous = undefined; } + /** The registry edges its declared contributions would add (another registry's value contributed). */ + #pendingEdges(): RegistryEdge[] { + const edges: RegistryEdge[] = []; + const { host } = this.#object; + for (const { registry, value } of this.#registryDeclarations?.declared ?? NONE) { + const to = value instanceof RegistryValueSource ? value.record : undefined; + if (to !== undefined) edges.push({ from: host.registryRecord(registry.name), to }); + } + return edges; + } + + #declarations(): RegistryDeclarations { + this.#registryDeclarations ??= { + registries: [], + reads: [], + retrySources: [], + writables: [], + declared: [], + contributions: NONE, + }; + return this.#registryDeclarations; + } + #assertDeclaring(declaration: string): void { if (!this.#declaring) { throw new Error(`${this.#object.label}: self.${declaration} is valid only in the factory`); diff --git a/packages/chord/src/next/host/ctx.ts b/packages/chord/src/next/host/ctx.ts index 4eb44020b..d14d3470c 100644 --- a/packages/chord/src/next/host/ctx.ts +++ b/packages/chord/src/next/host/ctx.ts @@ -23,7 +23,7 @@ export interface Ctx

extends Context { /** Who makes a call: the principal it acts as and its host's incarnation. */ export type Caller = { readonly principal: Principal | undefined; readonly incarnation: string }; -/** The ctx of one handler run: its references are released by `settle` when the handler settled. */ +/** The ctx of one handler run (or root scope): its references are released by `settle` when it settled. */ export class CallContext implements Ctx { readonly principal: Principal | undefined; readonly incarnation: string; @@ -58,9 +58,9 @@ export class CallContext implements Ctx { } async get(Type: C, params: ParamsOf, options: GetOptions = {}): Promise> { - // An ended handler creates nothing: its call's abort signal is checked before anything else. - this.#context.abortSignal?.throwIfAborted(); + // A settled scope refuses `released` whatever aborted it; an aborted call creates nothing. if (this.#references === undefined) throw settled(Type); + this.#context.abortSignal?.throwIfAborted(); // Made for the caller: its principal and its origin's incarnation travel with the reference's holds. const caller = { principal: this.principal, incarnation: this.incarnation }; this.#settling ??= new AbortController(); @@ -96,7 +96,7 @@ function settled(Type: Contract): ChordError { return new ChordError({ code: "released", reason: "settled", - message: `${Type.name}: ctx.get after its handler settled`, + message: `${Type.name}: ctx.get after its call settled`, }); } diff --git a/packages/chord/src/next/host/facet.ts b/packages/chord/src/next/host/facet.ts index 0055285ff..4e9615852 100644 --- a/packages/chord/src/next/host/facet.ts +++ b/packages/chord/src/next/host/facet.ts @@ -1,16 +1,23 @@ -// Facets (§16): a plugin's setup for one host, typed by the app's principal type. A setup only declares: the types it -// serves and the host's presence lists; it has no documents, routines or natives, so a facet cannot fail after `load`. +// Facets (§16): a plugin's setup for one host, typed by the app's principal type. A setup only declares: the types and +// registries it serves, its contributions to registries (slice 8) and the host's presence lists; it has no documents, +// routines or natives, so a facet cannot fail after `load`. import type { Watchable } from "../../watch.ts"; import type { Contract } from "../contracts/index.ts"; import type { Principal } from "../data/index.ts"; +import type { Contribution, Registry, RegistryProvider } from "../registries/index.ts"; import type { Factory } from "./attempt.ts"; import type { Presence } from "./presence.ts"; -/** What a setup serves types with (valid only while the setup runs), and the host's presence lists (always). */ +/** What a setup serves types and registries with (valid only while the setup runs), and the host's presence lists. */ export interface FacetHost

{ /** Serves a type on this host with `factory`; a remote type's factory gets the principal its object is for. */ serve(Type: C, factory: Factory, options?: ServeOptions): void; + /** + * Serves registry `R` on this host (its defining plugin's facet, the one provider, D1): `compose` builds the value + * from the table, `accepts` adapts older versions. A second serve of a registry is refused at `load`. + */ + serve(R: Registry, provider: RegistryProvider): void; /** `host.references(Type)`: the references to objects of `Type` on this host (§19), for factories to derive from. */ references(Type: C): Watchable[]>; } @@ -24,48 +31,92 @@ export type ServeOptions = { readonly linger?: number; }; -/** A type a setup served: its contract, its factory and its options. */ -export type ServedEntry = { - readonly contract: Contract; - readonly factory: ServedFactory; - readonly options: ServeOptions; -}; +/** What a facet's setup declares with besides the host: its contributions to registries (R21). */ +export interface FacetSelf { + /** + * Contributes `value` to registry `R` while this plugin is loaded: a plain value, or a Chord watchable of one + * (followed: its change recomposes). At most one per registry. + */ + contribute(R: Registry, value: Contribution): void; +} + +/** What a setup served, by kind: a type (its contract, factory and options), or a registry (its provider). */ +export type ServedEntry = + | { + readonly kind: "type"; + readonly contract: Contract; + readonly factory: ServedFactory; + readonly options: ServeOptions; + } + | { + readonly kind: "registry"; + readonly registry: Registry; + readonly provider: RegistryProvider; + }; + +/** A contribution a setup declared. */ +export type FacetContribution = { readonly registry: Registry; readonly value: unknown }; /** A factory as a setup served it, of either kind (the host dispatches on its contract's kind). */ type ServedFactory = (params: never, principalOrSelf: never, self: never) => unknown; /** A plugin's facet for hosts of the app's principal type `P` (type only: nothing is checked at runtime). */ export type Facet

= { - readonly setup: (host: FacetHost

) => void; + readonly setup: (host: FacetHost

, self: FacetSelf) => void; /** Type only: the principal type, invariant, so a host takes only facets typed for exactly its `P`. */ readonly "~principal"?: (principal: P) => P; }; /** Defines a facet; base code binds `P` once, next to its principal type. */ -export function defineFacet

(setup: (host: FacetHost

) => void): Facet

{ +export function defineFacet

(setup: (host: FacetHost

, self: FacetSelf) => void): Facet

{ return { setup }; } -/** Runs a facet's setup and returns the types it served, in order; `references` is the host's. */ +/** + * Runs a facet's setup and returns the types and registries it served and its contributions, in order; `references` + * is the host's. + */ export function setUp

( name: string, facet: Facet

, references: (Type: C) => Watchable[]>, -): readonly ServedEntry[] { +): { readonly served: readonly ServedEntry[]; readonly contributions: readonly FacetContribution[] } { const served: ServedEntry[] = []; + const contributions: FacetContribution[] = []; let open = true; + const assertOpen = (what: string): void => { + if (!open) throw new Error(`${name}: ${what} is valid only in the facet's setup`); + }; const host: FacetHost

= { - serve: (Type, factory, options = {}) => { - if (!open) throw new Error(`${name}: host.serve is valid only in the facet's setup`); - // Served types are dispatched by name and kind; the factory matches its own contract. - served.push({ contract: Type, factory: factory as ServedFactory, options }); + serve: (Type: Contract | Registry, implementation: unknown, options: ServeOptions = {}) => { + assertOpen("host.serve"); + // Served types and registries are dispatched by their kind; a factory matches its own contract, a provider + // its own registry. + if (Type.kind === "registry") { + served.push({ + kind: "registry", + registry: Type, + provider: implementation as RegistryProvider, + }); + } else { + served.push({ kind: "type", contract: Type, factory: implementation as ServedFactory, options }); + } }, references, }; + const self: FacetSelf = { + contribute: (registry, value) => { + assertOpen("self.contribute"); + if (contributions.some((each) => each.registry.name === registry.name)) { + throw new Error(`${name}: registry ${registry.name} is contributed to twice`); + } + contributions.push({ registry: registry as Registry, value }); + }, + }; try { - facet.setup(host); + facet.setup(host, self); } finally { open = false; } - return served; + return { served, contributions }; } diff --git a/packages/chord/src/next/host/host.ts b/packages/chord/src/next/host/host.ts index cbafd799b..5b360e4ba 100644 --- a/packages/chord/src/next/host/host.ts +++ b/packages/chord/src/next/host/host.ts @@ -1,10 +1,16 @@ // Hosts (§3): `createHost`, loading facets and the start barrier, links to children, the parent (connected again after // a drop) and routes, the origins it relays holds for, getting and holding references as a principal, the heartbeat -// of every link, revocation (§4), presence (§19) and keep-alive counts, closing; one batch per host, the backoff map -// and the endings new instances wait for. +// of every link, revocation (§4), presence (§19) and keep-alive counts, registries and root scopes (slice 8), closing; +// one batch per host, the backoff map and the endings new instances wait for. import type { TelemetryContext } from "@earendil-works/pi-telemetry"; -import { BACKGROUND_CONTEXT, type Context, withAbortSignal, withoutAbortSignal } from "../../context/index.ts"; +import { + awaitWithContext, + BACKGROUND_CONTEXT, + type Context, + withAbortSignal, + withoutAbortSignal, +} from "../../context/index.ts"; import type { JsonValue } from "../../json.ts"; import type { Watchable } from "../../watch.ts"; import type { Address, Contract, HostKind, ParamsOf } from "../contracts/index.ts"; @@ -21,6 +27,13 @@ import { } from "../data/internal.ts"; import { binaryCodec, type Codec, type Compression, type Link, type PartLimits } from "../link/index.ts"; import { linkLimits } from "../link/internal.ts"; +import type { Registry, RegistryValue } from "../registries/index.ts"; +import { + type RegistryContribution, + type RegistryEdge, + type RegistryError, + RegistryRecord, +} from "../registries/internal.ts"; import { type HoldRequest, HostLink, @@ -34,14 +47,16 @@ import type { ChordStore, Store } from "../store/store.ts"; import { Activity, type ActivityCounts } from "./activity.ts"; import { notLive } from "./attempt.ts"; import { Backoff } from "./backoff.ts"; -import { type Caller, withLinkedAbortSignal } from "./ctx.ts"; -import { type Facet, type ServedEntry, setUp } from "./facet.ts"; +import { type Caller, type Ctx, withLinkedAbortSignal } from "./ctx.ts"; +import { type Facet, type FacetContribution, type ServedEntry, setUp } from "./facet.ts"; +import { contributionOf } from "./members.ts"; import type { CloseFailed } from "./native.ts"; import { ChordObject, type Holder, type LocalContract, type RemoteContract, type Served } from "./object.ts"; import { type Placed, Placement } from "./placement.ts"; import { type Presence, References } from "./presence.ts"; import { Binding, type CheckedParams, type Held, type Reference, type Target } from "./reference.ts"; import { RevocationKeys, type RevokedBy } from "./revocation.ts"; +import { type RootOptions, RootScopes } from "./root.ts"; import { Route, type RouteOptions } from "./route.ts"; import { uuidv7 } from "./uuid.ts"; @@ -93,6 +108,7 @@ export type Position = { readonly kind: "on" | "onLive" | "native"; readonly pos */ export type HostError = | BatchError + | RegistryError | { readonly kind: "activation_failed"; readonly object: string; readonly failure: FailureCause } /** A live object failed at runtime (a `migrate` failing on access, a stored version newer than the code). */ | { readonly kind: "object_failed"; readonly object: string; readonly failure: FailureCause } @@ -163,7 +179,7 @@ export type GetOptions = { * A host whatever its principal type, as Chord's own parts use it: `ChordHost

` is invariant in `P` (`load` takes * facets typed for exactly `P`). */ -export type AnyHost = Omit; +export type AnyHost = Omit; /** A host. `P` is the app's principal type, `R` its revocation type. */ export interface Host

{ @@ -228,8 +244,21 @@ export interface Host

; /** - * Closes its links, then ends every object at once, then drains and closes them one by one, the last to go live - * first (reason `close`). Rejects with an `AggregateError` of the closes that threw. + * Registry `R`'s composed value as `R`'s version sees it, for code outside objects; waits for `load` first. Rejects + * when it has no value then (`unavailable/undeclared` without a provider, a first `compose` that threw, a version + * not served here: `contract_mismatch`); none can come before a reload. + */ + registry(R: Registry, context: Context): Promise>; + /** + * Runs `fn` with a call context for code outside objects (a Durable tool, a CLI): the host's principal (or + * `options.principal`), this host's incarnation, `context`'s telemetry and signal, `ctx.get`. When `fn` settles or + * `context` aborts, the context is sealed: its signal aborted, `ctx.get` refused `released`, every reference it took + * released; an abort rejects at once. `host.close` seals the open ones; after it, `root` rejects `released`. + */ + root(fn: (ctx: Ctx

) => Promise, context: Context, options?: RootOptions

): Promise; + /** + * Closes its links, seals its root scopes, then ends every object at once, then drains and closes them one by one, + * the last to go live first (reason `close`). Rejects with an `AggregateError` of the closes that threw. */ close(context: Context): Promise; } @@ -241,6 +270,9 @@ export function createHost

(options); } +/** No registry edges pending (a facet's contributions). */ +const NO_EDGES: readonly RegistryEdge[] = []; + /** No activation chain (Q5). */ const NO_CHAIN: readonly string[] = []; @@ -291,6 +323,9 @@ export class ChordHost

(); + readonly #roots = new RootScopes(this); /** The one heartbeat timer, running while the host has a link. */ #ticking: ReturnType | undefined; #state: "new" | "loading" | "loaded" | "closed" = "new"; @@ -321,7 +356,13 @@ export class ChordHost

{ + for (const contribution of contributions) contribution.record.add(contribution); + for (const record of this.#registries.values()) record.composeServed(); + }, context); } catch (error) { // The host closes as `close` would: its links too. Nothing was activated, so no close can fail. this.#closing ??= this.#close(); @@ -422,6 +463,35 @@ export class ChordHost

[]>; } + async registry(R: Registry, context: Context): Promise> { + this.batch.assertOutside("registry"); + const loading = this.pendingLoad; + if (loading !== undefined) await awaitWithContext(loading, context); + this.#assertOpen(); + const value = this.registryRecord(R.name).value(R.version); + const start = value.start(); + if (start.kind === "failed" && start.last === undefined) throw start.error; + // Its values are the composition as `R`'s version has it, `V`. + return value as RegistryValue; + } + + async root(fn: (ctx: Ctx

) => Promise, context: Context, options: RootOptions

= {}): Promise { + this.batch.assertOutside("root"); + this.#assertOpen(); + // The scope's ctx acts as `principal`, a `P`. + return this.#roots.run(fn as (ctx: Ctx) => Promise, options.principal ?? this.principal, context); + } + + /** The record of registry `name` on this host, made by its first use. */ + registryRecord(name: string): RegistryRecord { + let record = this.#registries.get(name); + if (record === undefined) { + record = new RegistryRecord(name, this.batch, (error) => this.report(error)); + this.#registries.set(name, record); + } + return record; + } + async close(_context: Context): Promise { this.batch.assertOutside("close"); this.#closing ??= this.#close(); @@ -782,6 +852,8 @@ export class ChordHost

}[]): void { + /** + * Runs every facet's setup, checking the plugin names (npm names, each once), and serves what they declared; returns + * their contributions, which enter the registries' tables once the setups ran. + */ + #setUp(facets: readonly { readonly name: string; readonly facet: Facet

}[]): RegistryContribution[] { const names = new Set(); + const made: RegistryContribution[] = []; for (const { name, facet } of facets) { if (!PLUGIN_NAME.test(name)) throw new Error(`"${name}" is not a valid plugin name`); if (names.has(name)) throw new Error(`plugin "${name}" is loaded twice`); names.add(name); - for (const entry of setUp(name, facet, (Type) => this.references(Type))) this.#serve(entry); + const { served, contributions } = setUp(name, facet, (Type) => this.references(Type)); + for (const entry of served) this.#serve(name, entry); + for (const contribution of contributions) made.push(this.#contribution(name, contribution)); } + return made; } - #serve({ contract, factory, options }: ServedEntry): void { + /** A facet's contribution, checked: in its registry's table while the host lives (its plugin stays loaded). */ + #contribution(plugin: string, { registry, value }: FacetContribution): RegistryContribution { + // A facet contributes no registry's value (it reads none): no edges of its own. + const record = this.registryRecord(registry.name); + return contributionOf(record, { plugin, type: undefined }, registry.version, value, NO_EDGES); + } + + #serve(plugin: string, entry: ServedEntry): void { + if (entry.kind === "registry") { + this.registryRecord(entry.registry.name).serve(entry.registry.version, entry.provider); + return; + } + const { contract, factory, options } = entry; if (this.#types.has(contract.name)) throw new Error(`${contract.name} is served twice`); // A client serves only local types (§3): a remote object is owned by a principal of the host serving it. if (contract.kind === "remote" && this.principal === undefined) { throw new Error(`${contract.name}: a host without a principal serves only local types`); } - const common = { provider: bindProvider(contract, {}), linger: options.linger ?? 0 }; + const common = { plugin, provider: bindProvider(contract, {}), linger: options.linger ?? 0 }; // The contract's kind decides the factory's signature and the contract's shape. const served: Served = contract.kind === "local" diff --git a/packages/chord/src/next/host/index.ts b/packages/chord/src/next/host/index.ts index 3d7611c5d..79b8a2634 100644 --- a/packages/chord/src/next/host/index.ts +++ b/packages/chord/src/next/host/index.ts @@ -1,10 +1,10 @@ // Hosts for apps: creating a host, facets, factories, natives, references and the ctx of method handlers, routines and -// their waiting helpers, revocation, presence and keep-alive counts. +// their waiting helpers, revocation, presence and keep-alive counts, root scopes. export type { ActivityCounts } from "./activity.ts"; export type { Factory, Self, StoreSelf } from "./attempt.ts"; export type { Ctx } from "./ctx.ts"; -export { defineFacet, type Facet, type FacetHost, type ServeOptions } from "./facet.ts"; +export { defineFacet, type Facet, type FacetHost, type FacetSelf, type ServeOptions } from "./facet.ts"; export type { FollowAttach, FollowAttachment, FollowObserver } from "./follow.ts"; export { createHost, @@ -20,6 +20,7 @@ export type { Presence } from "./presence.ts"; export type { RefState } from "./ref-state.ts"; export type { Held, Reference, RefIn } from "./reference.ts"; export type { RevokedBy } from "./revocation.ts"; +export type { RootOptions } from "./root.ts"; export type { RouteOptions } from "./route.ts"; export type { OnHandler, OnLiveHandler, OnNativeHandler, RoutineDeclarations } from "./routines.ts"; export { sleep, untilAborted } from "./sleep.ts"; diff --git a/packages/chord/src/next/host/members.ts b/packages/chord/src/next/host/members.ts index 7eaf1fc73..ed2222cd9 100644 --- a/packages/chord/src/next/host/members.ts +++ b/packages/chord/src/next/host/members.ts @@ -1,5 +1,6 @@ // The member check (Q14): an implementation is one variant with exactly that variant's members, methods being -// functions and state members JSON watchables; and the resolution of every derive's declared sources. +// functions and state members JSON watchables; and the resolution of what Chord follows behind every derive's declared +// sources (JSON and native) and every contribution to a registry (slice 8). // // The rule for watches across attempts (slice 6a §1.2, §1.4): references and other objects' derives follow a local // object's state member through its member source (`member-source.ts`), and a key of a keyed member they hold (`at`, @@ -17,7 +18,15 @@ import type { Context } from "../../context/index.ts"; import type { Contract } from "../contracts/index.ts"; import { currentVariants, LOCAL, takesNoArgs, type VariantData } from "../contracts/internal.ts"; import type { KeyedSlot } from "../documents/internal.ts"; -import type { Derive, DeriveSource } from "../state/internal.ts"; +import { + type ContributionContent, + type Contributor, + RegistryContribution, + type RegistryEdge, + type RegistryRecord, + RegistryValueSource, +} from "../registries/internal.ts"; +import { type Derive, type DeriveInput, type DeriveSource, NativeSource } from "../state/internal.ts"; import { type Binding, resolveKeyed, resolveWatchable } from "./reference.ts"; /** A checked implementation, as Chord calls and follows it. */ @@ -114,11 +123,35 @@ export function attachDerives( derives.forEach((derive, position) => { const member = exposed.get(derive); const name = member ?? `derive #${position + 1}`; - const resolve = (source: unknown, index: number): DeriveSource => { - const resolved = resolveWatchable(source); - if (resolved === undefined) throw new Error(`${name}: source ${index} is not a JSON watchable`); + const resolve = (source: unknown, index: number): DeriveInput => { + const resolved = resolveWatchable(source) ?? (source instanceof NativeSource ? source : undefined); + if (resolved === undefined) throw new Error(`${name}: source ${index} is not a Chord watchable`); return resolved; }; derive.attach(resolve, report(member)); }); } + +/** + * A contribution of `contributor` at `version` to `record`, as Chord follows it: a Chord watchable (a source, a + * constant, a document handle, a reference member, a native source) is followed, anything else is a plain value, + * whatever methods it has (registries carry natives). Refused (`invalid/cycle`) when it is a registry's value that + * closes a cycle of registries, through the tables and the contributor's `pending` edges. + */ +export function contributionOf( + record: RegistryRecord, + contributor: Contributor, + version: number, + value: unknown, + pending: readonly RegistryEdge[], +): RegistryContribution { + if (value instanceof RegistryValueSource) record.assertNoCycle(value, pending); + return new RegistryContribution(record, contributor, version, contentOf(value)); +} + +/** What Chord follows behind a contribution: its JSON or native source when it is a Chord watchable. */ +function contentOf(value: unknown): ContributionContent { + if (value instanceof NativeSource) return { kind: "native", source: value }; + const source = resolveWatchable(value); + return source === undefined ? { kind: "plain", value } : { kind: "json", source }; +} diff --git a/packages/chord/src/next/host/object.ts b/packages/chord/src/next/host/object.ts index f422a58fb..91e0bd2b8 100644 --- a/packages/chord/src/next/host/object.ts +++ b/packages/chord/src/next/host/object.ts @@ -2,7 +2,8 @@ // and job runs, its member sources, slice 6a §1.2), its holders, retries after failures (backoff, dependency recovery // R18, `ref.retry()`), a live object's failure (slice 6a §1.4), the waits and cycle checks of activations (Q5, its // instance id and the activation chain it was created for), lingering after its last reference (§3, D8), the job runs -// that keep it alive (§15), and its ending, revocation included (§4, R11). +// that keep it alive (§15), its contributions to registries while live and its wake signals (`retryOn`, slice 8), and +// its ending, revocation included (§4, R11). import { awaitWithContext, type Context } from "../../context/index.ts"; import type { Contract, ParamsOf } from "../contracts/index.ts"; @@ -23,6 +24,7 @@ import type { KeyedBinding } from "./members.ts"; import type { CloseFailed, NativeCloseReason } from "./native.ts"; import { type Ended, type Failed, LIVE, LOADING, type RefState } from "./ref-state.ts"; import type { Target } from "./reference.ts"; +import { type RetrySource, RetryWakes } from "./retry-on.ts"; import { uuidv7 } from "./uuid.ts"; /** A `local` contract, as a served type has it. */ @@ -48,6 +50,8 @@ export type Served = }); type ServedType = { + /** The plugin whose facet served it: its objects contribute to registries by it. */ + readonly plugin: string; readonly contract: C; readonly provider: Provider; /** How long a live object stays live after its last reference, in milliseconds (`serve`'s `linger`, 0 by default). */ @@ -146,6 +150,8 @@ export class ChordObject implements Target { readonly #keys: KeyedMembers; /** The one timer storing its jobs' held state changes by their earliest deadline (D8), while one is held. */ #flush: { readonly deadline: number; readonly timer: ReturnType } | undefined; + /** Its wake signals (`retryOn`), the last attempt's declaration; made by the first. */ + #wakes: RetryWakes | undefined; constructor( host: AnyHost, @@ -264,6 +270,17 @@ export class ChordObject implements Target { this.attempt(); } + /** + * Follows `sources` as its wake signals instead of the last attempt's (`self.retryOn`, right after a factory + * returned): a new value retries it while failed, as `ref.retry()` does; one while it activates retries it at once + * if that attempt fails; one while live does nothing. + */ + retryOn(sources: readonly RetrySource[]): void { + if (this.#wakes === undefined && sources.length === 0) return; + this.#wakes ??= new RetryWakes(); + this.#wakes.replace(sources, () => this.#wake()); + } + /** Calls a method of the live object for `caller`. */ async call(method: string, args: unknown, context: Context, caller: Caller): Promise { return this.#live().call(method, args, context, caller); @@ -383,6 +400,7 @@ export class ChordObject implements Target { this.#lastFailure = error; this.#outcome = settlement(); this.#outcome.reject(error); + this.#withdraw(attempt); attempt.seal(error); // Members following another object's sources freeze now: no commit of this object changes them. this.#freeze(attempt, "foreign"); @@ -530,10 +548,18 @@ export class ChordObject implements Target { #wentLive(attempt: Attempt): void { this.#woken = undefined; + if (this.#wakes !== undefined) this.#wakes.during = undefined; this.#state = { kind: "live", attempt }; this.#exposed = attempt; if (this.#members.size > 0 || !this.#keys.empty) this.host.batch.within(() => this.#rebind(attempt), this.#context); + const { contributions } = attempt; + // In the registries' tables before anyone sees it live (a getter released now reads them recomposed). + if (contributions.length > 0) { + this.host.batch.within(() => { + for (const contribution of contributions) contribution.record.add(contribution); + }, this.#context); + } this.#lastFailure = undefined; this.host.wentLive(this); this.#outcome.resolve(); @@ -551,6 +577,8 @@ export class ChordObject implements Target { #failed(attempt: Attempt, cause: FailureCause, holding: Attempt | undefined): void { const woken = this.#woken === attempt; this.#woken = undefined; + const wokenBy = this.#wakes?.during === attempt; + if (this.#wakes !== undefined) this.#wakes.during = undefined; // What its activation loaded for the pinned keys goes; they stay frozen. this.#keys.discard(); if (cause.kind === "transient") this.host.backoff.failed(this.identity); @@ -573,6 +601,44 @@ export class ChordObject implements Target { this.#idle(); // A dependency went live while this attempt activated: the wake it missed retries it now (R18). if (holding !== undefined && woken) this.dependencyLive(); + // A wake signal arrived while it activated: it retries it now. + else if (wokenBy) this.retry(); + } + + /** + * A wake signal's new value (inside its batch): retries a failed object (after the batch: a factory may not run in + * one), remembered while it activates, ignored while live and when its own contributions' leaving made it. + */ + #wake(): void { + const state = this.#state; + // Called by its wakes only. + const wakes = this.#wakes!; + if (state.kind === "activating") { + wakes.during = state.attempt; + return; + } + if (state.kind !== "failed" || wakes.leaving) return; + queueMicrotask(() => this.retry()); + } + + /** + * The live attempt `attempt` leaves the registries' tables (it fails or ends), inside a batch; a composition this + * leaving makes wakes no wake signal of this object. + */ + #withdraw(attempt: Attempt): void { + const { contributions } = attempt; + if (contributions.length === 0) return; + const { batch } = this.host; + const wakes = this.#wakes; + batch.within(() => { + if (wakes !== undefined) { + wakes.leaving = true; + batch.flushing(() => { + wakes.leaving = false; + }); + } + for (const contribution of contributions) contribution.record.remove(contribution); + }, this.#context); } /** The member source of `member`, made bound to `exposed`'s member by its first use. */ @@ -637,6 +703,7 @@ export class ChordObject implements Target { this.#flush = undefined; this.documents.seal(); this.#stopLingering(); + this.#wakes?.stop(); this.host.forget(this); this.#outcome.reject(error); // In a batch: the derives over its members are marked, so none keeps a cached value. @@ -656,6 +723,7 @@ export class ChordObject implements Target { const { holding } = state; return holding === undefined ? { kind: "end", previous } : { kind: "release", attempt: holding, previous }; } + if (state.kind === "live") this.#withdraw(state.attempt); state.attempt.seal(error); return { kind: "close", attempt: state.attempt, previous }; } diff --git a/packages/chord/src/next/host/ref-state.ts b/packages/chord/src/next/host/ref-state.ts index 75f9fa42e..9e5720405 100644 --- a/packages/chord/src/next/host/ref-state.ts +++ b/packages/chord/src/next/host/ref-state.ts @@ -1,9 +1,10 @@ // The state of a reference (§7): loading, live, failed or ended; and `ref.state`, the native watchable that follows // it. -import type { Context } from "../../context/index.ts"; -import { NativeWatch, type NativeWatchable } from "../../watch.ts"; +import { BACKGROUND_CONTEXT, type Context } from "../../context/index.ts"; +import { NativeWatch } from "../../watch.ts"; import type { ChordError, FailureCause } from "../data/index.ts"; +import { type Batch, NativeSource } from "../state/internal.ts"; /** * Where a reference stands. `loading`: its object activates or retries, it waits for `load` or a parent link, or its @@ -29,27 +30,21 @@ export const LOADING: RefState = { kind: "loading" }; export const LIVE: RefState = { kind: "live" }; -const IGNORE = (): void => {}; - /** - * `ref.state`: its value is the reference's state, read through `read`; each watch gets every new state, in its own - * context, and ends `ended` when the reference is released (the end is the notification). + * `ref.state`, a native source: its value is the reference's state, read through `read`; each watch gets every new + * state, in its own context (not one call per batch, as a followed source's), and ends `ended` when the reference is + * released (the end is the notification). Derives reading it are marked in a batch of its own when it changes. */ -export class ReferenceState implements NativeWatchable { +export class ReferenceState extends NativeSource { readonly #read: () => RefState; - /** Where a listener's throw goes (the host); dropped on release, so a released reference keeps no host. */ - #report: (error: unknown) => void; - /** Throws `invalid/reentrant` inside the host's batch (a watch taken in a derive compute); dropped on release. */ - #assertOutside: () => void; /** The watches with the contexts they were taken in, which their frames carry. */ readonly #watches = new Map, Context>(); /** The state the watches have: a change is announced only when the state is another object. */ #last: RefState; - constructor(read: () => RefState, report: (error: unknown) => void, assertOutside: () => void) { + constructor(read: () => RefState, batch: Batch | undefined) { + super(batch); this.#read = read; - this.#report = report; - this.#assertOutside = assertOutside; this.#last = read(); } @@ -57,35 +52,57 @@ export class ReferenceState implements NativeWatchable { return this.#read(); } + /** Every state in order: a watch sees each step it went through. */ async watch(context: Context): Promise> { - this.#assertOutside(); const state = this.#read(); - const watch: NativeWatch = new NativeWatch(state, () => this.#watches.delete(watch), this.#report); - // An ended reference has nothing more to announce. - if (state.kind === "ended") { - watch.end({ kind: "ended" }); - return watch; + const batch = this.batch; + // An ended reference has nothing more to announce, and keeps no host to report to. + if (batch === undefined || state.kind === "ended") { + const ended = new NativeWatch(state, IGNORE, IGNORE); + ended.end({ kind: "ended" }); + return ended; } + batch.assertOutside("watch"); + const watch = new NativeWatch(state, () => this.#watches.delete(watch), batch.reportListener); this.#watches.set(watch, context); if (context.abortSignal !== undefined) watch.observeAbort(context.abortSignal); return watch; } - /** The reference's state may have changed: every watch gets it when it is another state. */ - changed(): void { + /** The reference's state may have changed: every watch gets it when it is another state, derives are marked. */ + moved(): void { const state = this.#read(); if (state === this.#last) return; this.#last = state; for (const [watch, context] of this.#watches) watch.advance(state, context); + if (this.dependents.size > 0) this.#mark(); } - /** The reference ended (`ended`, the reference's state): every watch ends `ended`. */ - end(ended: Ended): void { + /** The reference ended (`ended`, the reference's state): every watch ends `ended`, derives are marked a last time. */ + released(ended: Ended): void { // An ended reference keeps nothing of its object: not the last failure's error, not the host. this.#last = ended; - this.#report = IGNORE; - this.#assertOutside = IGNORE; // Ending detaches, which deletes the current entry (safe while iterating) and runs no listener. for (const watch of this.#watches.keys()) watch.end({ kind: "ended" }); + const batch = this.batch; + if (batch === undefined || this.dependents.size === 0) { + this.end(); + return; + } + // Ended where the marks run: a release while a batch's flushes send marks in the next commit, after this. + batch.within(() => { + this.changed(); + this.end(); + }, BACKGROUND_CONTEXT); + } + + /** + * Marks the derives reading it, in the batch running now or one of its own (apart from `moved`: a closure there + * would allocate on every state change). A reference has no context of its own: the background context. + */ + #mark(): void { + this.batch?.within(() => this.changed(), BACKGROUND_CONTEXT); } } + +const IGNORE = (): void => {}; diff --git a/packages/chord/src/next/host/reference.ts b/packages/chord/src/next/host/reference.ts index 4b9ed1ff8..65da7ab33 100644 --- a/packages/chord/src/next/host/reference.ts +++ b/packages/chord/src/next/host/reference.ts @@ -136,8 +136,6 @@ const WAIT: Decision = { kind: "wait" }; const DONE: Decision = { kind: "done" }; -const IGNORE = (): void => {}; - /** A wait for a state: checked at every change, rejected with the reference's end. */ type Waiter = { readonly check: () => void; readonly end: (error: ChordError) => void }; @@ -257,7 +255,7 @@ export class Binding implements Holder { * rejects them with the cause; waits for a state decide. */ moved(): void { - this.#refState?.changed(); + this.#refState?.moved(); const state = this.refState; if (state.kind === "live") { this.#parked?.flush( @@ -430,7 +428,7 @@ export class Binding implements Holder { this.#parked?.reject(error); this.#parked = undefined; for (const watch of this.#watches) watch.end({ kind: "ended" }); - this.#refState?.end(ended); + this.#refState?.released(ended); // Each removes itself; promise reactions run later. for (const waiter of this.#waiters) waiter.end(error); this.#ended?.resolve(); @@ -495,13 +493,14 @@ export class Binding implements Holder { #watchableState(): ReferenceState { if (this.#refState !== undefined) return this.#refState; const state = this.#state; - // A released reference's state never changes: its watches end at once, with no listener to report. Bound, not a - // closure: `read` would share the closure's scope and keep the batch after release. - const [report, assertOutside] = - state.kind === "released" - ? [IGNORE, IGNORE] - : [state.batch.reportListener, state.batch.assertOutside.bind(state.batch, "watch")]; - this.#refState = new ReferenceState(() => this.refState, report, assertOutside); + // A released reference's state never changes: it is made ended, keeping no batch. + if (state.kind === "released") { + const made = new ReferenceState(() => this.refState, undefined); + made.released(state.state); + this.#refState = made; + return made; + } + this.#refState = new ReferenceState(() => this.refState, state.batch); return this.#refState; } diff --git a/packages/chord/src/next/host/retry-on.ts b/packages/chord/src/next/host/retry-on.ts new file mode 100644 index 000000000..84f997a11 --- /dev/null +++ b/packages/chord/src/next/host/retry-on.ts @@ -0,0 +1,55 @@ +// Wake signals (slice 8 §3, D2): `self.retryOn(...sources)`. The object follows each declared source while it lives, +// replaced by each attempt's declaration; a new value of a source wakes it (the object decides what a wake does: a +// retry while failed, remembered while it activates, nothing while live). Failures of a source wake nothing. + +import { RegistryValueSource } from "../registries/internal.ts"; +import { Constant, type NativeObserver } from "../state/internal.ts"; + +/** What `retryOn` takes: sources that live independently of the attempt (a registry value; a constant never wakes). */ +export type RetrySource = RegistryValueSource | Constant; + +/** + * Checks a `retryOn` source (declaration policy: TypeScript cannot tell a reference's member or this attempt's + * `writable` from a registry value or a constant). + */ +export function retrySource(source: unknown, label: string): RetrySource { + if (source instanceof RegistryValueSource || source instanceof Constant) return source; + throw new Error(`${label}: self.retryOn takes registry values and constants (sources that outlive the attempt)`); +} + +/** + * The wake signals of one object: its subscriptions (one set at a time, the last attempt's declaration) and what its + * object decides a wake by. + */ +export class RetryWakes { + /** The attempt a wake arrived during, while it activates: if it fails, it is retried at once. */ + during: object | undefined; + /** + * Set from its object's contributions leaving until that batch delivered: a composition its own leaving made does + * not wake it (an object failing live would retry itself without backoff). + */ + leaving = false; + #subscribed: { readonly source: RegistryValueSource; readonly observer: NativeObserver }[] = []; + + /** Follows `sources` instead of the previous set; `wake` runs inside the batch of each new value (it only records). */ + replace(sources: readonly RetrySource[], wake: () => void): void { + this.stop(); + const observer: NativeObserver = { + advance: (call) => { + if (call.kind === "value") wake(); + }, + end: () => {}, + }; + for (const source of sources) { + if (!(source instanceof RegistryValueSource)) continue; + source.subscribe(observer); + this.#subscribed.push({ source, observer }); + } + } + + /** Stops following (the object ended, or a new declaration replaces the set). */ + stop(): void { + for (const { source, observer } of this.#subscribed) source.unsubscribe(observer); + this.#subscribed = []; + } +} diff --git a/packages/chord/src/next/host/root.ts b/packages/chord/src/next/host/root.ts new file mode 100644 index 000000000..3ebdf3d4f --- /dev/null +++ b/packages/chord/src/next/host/root.ts @@ -0,0 +1,76 @@ +// Root scopes (slice 8 §4, D3, design §8): `host.root(fn, context, options?)`, a call context for code outside objects +// (a Durable tool, a CLI), sealed when `fn` settles, when `context` aborts and when the host closes: its signal aborted, +// its `ctx.get` refused `released`, every reference it took released. + +import { awaitWithContext, type Context } from "../../context/index.ts"; +import { ChordError, type Principal } from "../data/index.ts"; +import { CallContext, type Ctx, withLinkedAbortSignal } from "./ctx.ts"; +import type { AnyHost } from "./host.ts"; + +/** What `host.root` takes besides `fn` and the context. */ +export type RootOptions

= { + /** Whom the scope acts as: the host's principal by default; a trusted same-host caller's choice (base code). */ + readonly principal?: P; +}; + +/** An open root scope: its seal, and the rejection its caller gets when the host closes. */ +type OpenScope = { readonly seal: (reason: unknown) => void; readonly closed: PromiseWithResolvers }; + +/** The open root scopes of a host, each sealed once; `close` seals the rest and rejects their callers. */ +export class RootScopes { + readonly #host: AnyHost; + /** Each open scope: how it is sealed with a reason, and its caller's rejection when the host closes. */ + readonly #open = new Set(); + + constructor(host: AnyHost) { + this.#host = host; + } + + /** + * Runs `fn` with a call context like a handler's (`principal`, this host's incarnation, `context`'s telemetry, a + * signal linked to `context`'s), sealed when `fn` settles, `context` aborts or the host closes. Resolves as `fn` + * does; rejects at once when `context` aborts or the host closes (an uncooperative `fn` runs on and fails at its + * next use of Chord), or when `context` was aborted already (`fn` does not run). + */ + run(fn: (ctx: Ctx) => Promise, principal: Principal | undefined, context: Context): Promise { + context.abortSignal?.throwIfAborted(); + const host = this.#host; + const controller = new AbortController(); + const linked = withLinkedAbortSignal(controller.signal, context); + const scope = linked.context; + const ctx = new CallContext(scope, { principal, incarnation: host.incarnation }, host); + const open: OpenScope = { + seal: (reason) => { + if (!this.#open.delete(open)) return; + scope.abortSignal?.removeEventListener("abort", aborted); + // References first: a `get` still waiting gives up with `released`, not with the abort's reason. + ctx.settle(); + // While still linked, so the scope's own signal (a caller's context has one) aborts too. + controller.abort(reason); + linked.release(); + }, + closed: Promise.withResolvers(), + }; + const aborted = (): void => open.seal(scope.abortSignal?.reason); + this.#open.add(open); + scope.abortSignal?.addEventListener("abort", aborted, { once: true }); + // Plugin code: a synchronous throw becomes the rejection. + const running = (async () => fn(ctx))(); + const settled = (): void => open.seal(SETTLED); + running.then(settled, settled); + // A method's caller gets `released` when its object ends, whatever the handler still does: so does this one. One + // wait on the caller's signal around both, so the close removes its listener too. + return awaitWithContext(Promise.race([running, open.closed.promise]), context); + } + + /** The host closes: every open scope is sealed with `reason`, its caller rejected with it. */ + close(reason: unknown): void { + for (const open of [...this.#open]) { + open.seal(reason); + open.closed.reject(reason); + } + } +} + +/** The abort reason of a scope whose `fn` settled. */ +const SETTLED = new ChordError({ code: "released", reason: "settled", message: "host.root: its fn settled" }); diff --git a/packages/chord/src/next/host/routines.ts b/packages/chord/src/next/host/routines.ts index ea924ec7b..837551b9e 100644 --- a/packages/chord/src/next/host/routines.ts +++ b/packages/chord/src/next/host/routines.ts @@ -14,7 +14,7 @@ import { type WatchEnd, } from "../../watch.ts"; import type { Principal } from "../data/index.ts"; -import { Source } from "../state/internal.ts"; +import { NativeSource, Source } from "../state/internal.ts"; import { nextDelay } from "./backoff.ts"; import { CallContext, type Caller, type Ctx } from "./ctx.ts"; import type { AnyHost, Position } from "./host.ts"; @@ -94,6 +94,9 @@ type Called = { readonly kind: "done" } | { readonly kind: "failed"; readonly er const DONE: Called = { kind: "done" }; +/** What `#failing` says of a source failed now. */ +const FAILING = { kind: "failed" } as const; + /** One run of an `onLive` that lasted this long resets its backoff. */ const LONG_RUN = 60_000; @@ -267,10 +270,14 @@ class OnRoutine { } /** - * Whether the source is a Chord source failed now (`failed`), or one with no value yet (`no_value`, with the source to - * follow); `undefined` otherwise (natives and Durable's views tell their failures through their watches). + * Whether the source is a Chord source failed now (`failed`), or a JSON one with no value yet (`no_value`, with the + * source to follow); `undefined` otherwise (Durable's views tell their failures through their watches). */ #failing(): { readonly kind: "failed" } | { readonly kind: "no_value"; readonly source: Source } | undefined { + const declared = this.#declaration.source; + // A native source opens its watch at its last good value while failed (a registry whose `compose` threw): its + // first call is the failure, not that stale value. + if (declared instanceof NativeSource) return declared.start().kind === "failed" ? FAILING : undefined; let source: unknown; try { source = resolveWatchable(this.#declaration.source); diff --git a/packages/chord/src/next/index.ts b/packages/chord/src/next/index.ts index 8e759f345..1055a2cf6 100644 --- a/packages/chord/src/next/index.ts +++ b/packages/chord/src/next/index.ts @@ -7,6 +7,7 @@ export * from "./documents/index.ts"; export * from "./host/index.ts"; export * from "./jobs/index.ts"; export * from "./link/index.ts"; +export * from "./registries/index.ts"; export * from "./state/index.ts"; /** @internal The store's surface for adapters (slice 6b §2.1). */ export { type DecodedRow, OwnerTable, StoreOwnerTables } from "./store/adapter.ts"; diff --git a/packages/chord/src/next/registries/README.md b/packages/chord/src/next/registries/README.md new file mode 100644 index 000000000..070c98877 --- /dev/null +++ b/packages/chord/src/next/registries/README.md @@ -0,0 +1,81 @@ +# registries + +Registries (design §17, slice 8): an extension point between plugins. The defining plugin shares a token and serves +it with `compose`; facets and host-lifetime objects contribute; readers follow the composed value. + +## For apps (`index.ts`) + +```ts +import { defineRegistry, keyedEntries } from "./index.ts"; + +// Shared by every plugin, as a contract: identity and types only. +export const Http = defineRegistry("chord.http", { version: 2 }); + +defineFacet((host, self) => { + host.serve(Http, { // the defining plugin: the one provider + compose: (table, { report }) => routerOf(keyedEntries(table, (c) => c.routes, report)), + accepts: { 1: { contribution: (old: HttpV1) => upgrade(old), value: (router) => routerV1(router) } }, + }); + self.contribute(Http, { routes: { "GET /notes": exportNotes } }); // while this plugin is loaded +}); +host.serve(Mcp, (_, self) => { // local, empty params: may contribute (R21) + const ext = self.writable(mcpExtension([])); + self.contribute(Extensions, ext); // from live until it fails, ends or restarts + self.retryOn(self.registry(Models)); // a new model wakes it while failed (rare changes) + return {}; +}); +host.serve(Server, (_, self) => { + const router = self.registry(Http); // a dependency; `.value` at use + return { request: async (req, ctx) => router.value.route(req, ctx) }; +}); +``` + +- `defineRegistry(name, { version })`: a token. `name` follows the type names' rule (`[a-z0-9@/._~-]`), + `version` is the version this plugin is built against. `C` is what contributors give, `V` what readers get. +- `host.serve(R, { compose, accepts? })` in the defining plugin's facet (`host/`). `compose(table, { report })` is pure + and synchronous and always returns a value (a fallback built in); `report({ message, plugins })` reports a conflict + it resolved, as `registry_conflict`, once per episode (a conflict reported by the last good composition is not + reported again). It runs at the end of `load`'s setups (every served registry once, an empty table too), then lazily: + a change of the table marks it, and it composes when read or, while someone watches, observes or waits for it, in + the batch that changed the table (after the commit's adoptions); an unchanged table (every entry the same) composes + nothing. A `compose` that throws is the owner's bug: `registry_failed` once per episode, watches get one `failed` + call, `.value` keeps the last good composition (before any, it throws the error); the next good composition is a + value call, also when its value is the same. +- The table (`RegistryTable`): one `RegistryEntry { plugin, type?, version, value }` per live contributor, sorted by + plugin name, then type (a facet's entry first): never load order. A contribution of an older version is adapted with + `accepts[version].contribution` once per contribution value; a newer one, or an older one without an adapter, is left + out and reported `contribution_refused` once. A Chord watchable contribution (a writable, a derive, a document, a + reference member, another registry's value) is followed: its change marks the table; one that fails or ends (a + registry whose `compose` threw too, though its readers keep its last good value), or whose adapter throws, is left + out and reported `contribution_failed` once per episode, and is back when it heals. Anything else is a plain value, + whatever methods it has (`undefined` included). +- Readers: `self.registry(R)` (a declared dependency, `host/`) and `await host.registry(R, ctx)` give a + `RegistryValue`: a native watchable (`.value`, `watch`) with `observe(observer, ctx)`, a synchronous observer of + its compositions (Durable's shape, for `registryReader`). A reader whose token is older than the provider's gets the + composition mapped back with `accepts[version].value`, once per composition, only while read or followed; a mapping + that throws fails that version only (`registry_failed`, its last good mapping kept). A reader without a mapping, or + newer than the provider, gets `contract_mismatch` (its object fails `broken`: only a reload helps). The mapping is + kept only while a reader or follower is there. Without a provider a registry has no value + (`unavailable/undeclared`); its contributions wait in the table. +- `keyedEntries(table, entriesOf, report)`: a map of every entry's keyed items, the first by plugin name keeping a key, + a duplicate reported through `compose`'s `report`; `orderedEntries(table, itemsOf, orderOf?)`: every entry's items in + table order, or by `orderOf`. Both keep each item's `plugin` (and `type`): a contribution is another plugin's code, so + a use site catches its throws and rejections, reports the plugin, and goes on. +- Registries carry natives: an entry lives exactly as long as its contributor. A last good composition kept after a + throwing `compose` may still name a contributor that left: use sites look up at each use and keep nothing. + +## For Chord's layers (`internal.ts`) + +- `RegistryRecord`: one per registry and host (`ChordHost.registryRecord(name)`, made by its first use): the provider + (`serve`), the contributions in table order (`add`, `remove`, inside a batch), the last table and composition + (`start()`: composed now when dirty; a read while composing is a cycle, `invalid/cycle`), the episode reports, the + direct edges between registries whose values are contributions in its table (counted, kept while those + contributions are in it; `assertNoCycle` refuses a cycle at `contribute`, through those edges and the + contributing attempt's own earlier declarations), and its values by reader version + (`value(version)`); `close()` at the host's close. +- `RegistryContribution` (`contribution.ts`): one contributor's contribution, a `Dependent` of its source; + `entry(version, provider)` is its table entry (the same object while its value is), or `undefined` when left out. +- `RegistryValueSource`: a `FollowedSource` per reader version; `read(waiter)` registers an activation's object + (`RegistryWaiter`), woken with `dependencyLive()` (on a microtask) when the value gets its first good value; readers + waiting for one count as followers, so a change composes for them. At the host's close it keeps its last start and + lets the record (and the host) go. diff --git a/packages/chord/src/next/registries/contribution.ts b/packages/chord/src/next/registries/contribution.ts new file mode 100644 index 000000000..7572a40a7 --- /dev/null +++ b/packages/chord/src/next/registries/contribution.ts @@ -0,0 +1,158 @@ +// One contributor's contribution to a registry (slice 8 §1): its table entry, made once per value (an older version's +// adapted once per value), followed when it is a Chord watchable, left out and reported when it fails or is refused. + +import { type Dependent, type DeriveSource, handedOut, type NativeSource, Source } from "../state/internal.ts"; +import type { RegistryRecord } from "./record.ts"; +import type { RegistryEntry, RegistryProvider } from "./registry.ts"; +import { RegistryValueSource } from "./value.ts"; + +/** Who contributes: a facet (by plugin) or a host-lifetime object (by plugin and type). */ +export type Contributor = { readonly plugin: string; readonly type: string | undefined }; + +/** + * What a contribution is: a plain value, a Chord JSON watchable (a source or a constant), or a native source; the + * watchables followed for their value. + */ +export type ContributionContent = + | { readonly kind: "plain"; readonly value: unknown } + | { readonly kind: "json"; readonly source: DeriveSource } + | { readonly kind: "native"; readonly source: NativeSource }; + +/** + * One contributor's contribution to a registry: a plain value, or a Chord source it follows as a dependent (a change + * marks the registry's table). Its table entry is made once per value (an older version's adapted once per value); + * a failure or refusal leaves it out, reported once per episode. + */ +export class RegistryContribution implements Dependent { + /** The registry it contributes to. */ + readonly record: RegistryRecord; + readonly contributor: Contributor; + /** The registry version its contributor was built against. */ + readonly version: number; + readonly #content: ContributionContent; + /** The last adaptation of an older contribution: the value it was made from and its result. */ + #adapted: { readonly from: unknown; readonly value: unknown } | undefined; + #entry: RegistryEntry | undefined; + /** Left out since its last entry: failing (reported once), or refused (reported once). */ + #out: "none" | "failed" | "refused" = "none"; + + constructor(record: RegistryRecord, contributor: Contributor, version: number, content: ContributionContent) { + this.record = record; + this.contributor = contributor; + this.version = version; + this.#content = content; + } + + /** Its source changed (inside a batch): the registry's table is marked. */ + invalidate(): void { + this.record.contributionChanged(); + } + + /** A retirement of its source's document changes nothing beyond the value. */ + sourceRetired(): void {} + + /** Starts following its source, if it has one. */ + follow(): void { + const source = this.#followed(); + source?.dependents.add(this); + } + + /** Stops following its source. */ + unfollow(): void { + this.#followed()?.dependents.delete(this); + } + + /** The registry whose value it is, when it is one (its edge in the cycle check, while it is in the table). */ + get reads(): RegistryRecord | undefined { + const content = this.#content; + return content.kind === "native" && content.source instanceof RegistryValueSource + ? content.source.record + : undefined; + } + + /** Its source when it changes (a constant never does). */ + #followed(): Source | NativeSource | undefined { + const content = this.#content; + if (content.kind === "native") return content.source; + return content.kind === "json" && content.source instanceof Source ? content.source : undefined; + } + + /** + * Its entry in a table of the provider at `version`: the same object while its value is the same; `undefined` when + * left out (a refused version, a failing source, a throwing adapter). + */ + entry(version: number, provider: RegistryProvider): RegistryEntry | undefined { + const adapter = this.version < version ? provider.accepts?.[this.version]?.contribution : undefined; + if (this.version > version || (this.version < version && adapter === undefined)) { + if (this.#out !== "refused") this.#refused(); + return undefined; + } + let value: unknown; + try { + const raw = this.#raw(); + value = adapter === undefined ? raw : this.#adapt(raw, adapter); + } catch (error) { + if (this.#out !== "failed") this.#failed(error); + return undefined; + } + this.#out = "none"; + const entry = this.#entry; + // `undefined` is a value: the entry, not its field, says whether there was one. + if (entry !== undefined && entry.value === value) return entry; + const { plugin, type } = this.contributor; + this.#entry = { plugin, ...(type === undefined ? {} : { type }), version: this.version, value }; + return this.#entry; + } + + /** + * The contributed value now: a JSON source's value throws while it is failed or ended; a native source that failed + * (a registry whose `compose` threw) keeps a last good value for its readers, but is left out here as failed. + */ + #raw(): unknown { + const content = this.#content; + switch (content.kind) { + case "plain": + return content.value; + case "json": + return content.source.value; + case "native": + return content.source.current(); + } + } + + #adapt(raw: unknown, adapter: (old: never) => unknown): unknown { + const adapted = this.#adapted; + if (adapted !== undefined && adapted.from === raw) return adapted.value; + const value = this.record.adapt(() => adapter(raw as never)); + this.#adapted = { from: raw, value }; + return value; + } + + #refused(): void { + this.#out = "refused"; + const { plugin, type } = this.contributor; + const registry = this.record.name; + const where = type === undefined ? {} : { type }; + this.record.reportContribution({ + kind: "contribution_refused", + registry, + plugin, + ...where, + version: this.version, + }); + } + + #failed(error: unknown): void { + this.#out = "failed"; + const { plugin, type } = this.contributor; + const registry = this.record.name; + const where = type === undefined ? {} : { type }; + this.record.reportContribution({ + kind: "contribution_failed", + registry, + plugin, + ...where, + error: handedOut(error), + }); + } +} diff --git a/packages/chord/src/next/registries/entries.ts b/packages/chord/src/next/registries/entries.ts new file mode 100644 index 000000000..069363ec8 --- /dev/null +++ b/packages/chord/src/next/registries/entries.ts @@ -0,0 +1,54 @@ +// Table helpers for `compose` (slice 8 §1, Q8): a map by key (the first by plugin name wins, duplicates reported) and +// an ordered list; both keep each item's plugin, so a use site reports the contributor of an entry that throws. + +import type { ComposeContext, RegistryTable } from "./registry.ts"; + +/** An item taken from a table, with the contributor it came from. */ +export type TableItem = { readonly value: I; readonly plugin: string; readonly type?: string }; + +/** + * A map of every entry's keyed items (`entriesOf`, e.g. its routes), in table order: a key taken already is reported + * through `report` (`compose`'s) naming both plugins, and the first by plugin name keeps it. + */ +export function keyedEntries( + table: RegistryTable, + entriesOf: (value: C) => Readonly> | undefined, + report: ComposeContext["report"], +): ReadonlyMap> { + const map = new Map>(); + for (const entry of table) { + const items = entriesOf(entry.value); + if (items === undefined) continue; + for (const [key, value] of Object.entries(items)) { + const taken = map.get(key); + if (taken !== undefined) { + report({ message: `"${key}" is contributed twice`, plugins: [taken.plugin, entry.plugin] }); + continue; + } + map.set(key, item(value, entry.plugin, entry.type)); + } + } + return map; +} + +/** + * Every entry's items (`itemsOf`) in one list: in table order, or sorted by `orderOf` (a phase's index, for example), + * table order kept among equals. + */ +export function orderedEntries( + table: RegistryTable, + itemsOf: (value: C) => readonly I[] | undefined, + orderOf?: (item: I) => number, +): readonly TableItem[] { + const items: TableItem[] = []; + for (const entry of table) { + for (const value of itemsOf(entry.value) ?? []) items.push(item(value, entry.plugin, entry.type)); + } + // `sort` is stable: equal orders keep table order. + if (orderOf !== undefined) items.sort((a, b) => orderOf(a.value) - orderOf(b.value)); + return items; +} + +function item(value: I, plugin: string, type: string | undefined): TableItem { + return type === undefined ? { value, plugin } : { value, plugin, type }; +} diff --git a/packages/chord/src/next/registries/index.ts b/packages/chord/src/next/registries/index.ts new file mode 100644 index 000000000..8c9d8154c --- /dev/null +++ b/packages/chord/src/next/registries/index.ts @@ -0,0 +1,16 @@ +// Registries for apps: tokens, what a provider serves them with, the table `compose` gets and its helpers, and the +// composed value readers follow. + +export { keyedEntries, orderedEntries, type TableItem } from "./entries.ts"; +export { + type ComposeContext, + type Contribution, + defineRegistry, + type Registry, + type RegistryAccepts, + type RegistryConflict, + type RegistryEntry, + type RegistryProvider, + type RegistryTable, + type RegistryValue, +} from "./registry.ts"; diff --git a/packages/chord/src/next/registries/internal.ts b/packages/chord/src/next/registries/internal.ts new file mode 100644 index 000000000..1a6969089 --- /dev/null +++ b/packages/chord/src/next/registries/internal.ts @@ -0,0 +1,5 @@ +// Registries for Chord's layers: a host's record of each registry, its contributions and its values by version. + +export { type ContributionContent, type Contributor, RegistryContribution } from "./contribution.ts"; +export { type RegistryEdge, type RegistryError, RegistryRecord } from "./record.ts"; +export { RegistryValueSource, type RegistryWaiter } from "./value.ts"; diff --git a/packages/chord/src/next/registries/record.ts b/packages/chord/src/next/registries/record.ts new file mode 100644 index 000000000..f879132f9 --- /dev/null +++ b/packages/chord/src/next/registries/record.ts @@ -0,0 +1,301 @@ +// A registry on a host (slice 8 §1): its provider, its contributions in table order (`contribution.ts`), the table +// and its lazy composition (marked when the table changes, composed when read or followed, in the batch's +// deliveries), the reports of conflicts and failures (once per episode), and the values its readers follow by +// version. + +import { ChordError } from "../data/index.ts"; +import { invalid } from "../data/internal.ts"; +import { type Batch, handedOut, type NativeStart } from "../state/internal.ts"; +import type { RegistryContribution } from "./contribution.ts"; +import type { ComposeContext, RegistryConflict, RegistryEntry, RegistryProvider, RegistryTable } from "./registry.ts"; +import { RegistryValueSource } from "./value.ts"; + +/** What a registry reports to its host's `onError`, by kind; each once per episode. */ +export type RegistryError = + /** `compose` reported a conflict it resolved. */ + | { + readonly kind: "registry_conflict"; + readonly registry: string; + readonly message: string; + readonly plugins: readonly string[]; + } + /** `compose` (or an older version's `value` adapter) threw: the last good composition stays. */ + | { readonly kind: "registry_failed"; readonly registry: string; readonly error: unknown } + /** A watchable contribution failed or ended, or its `contribution` adapter threw: it is left out until it heals. */ + | { + readonly kind: "contribution_failed"; + readonly registry: string; + readonly plugin: string; + readonly type?: string; + readonly error: unknown; + } + /** A contribution of a version the provider does not accept (newer, or older without an adapter) is left out. */ + | { + readonly kind: "contribution_refused"; + readonly registry: string; + readonly plugin: string; + readonly type?: string; + readonly version: number; + }; + +/** One registry's value contributed to another: `from`'s table reads `to`. */ +export type RegistryEdge = { readonly from: RegistryRecord; readonly to: RegistryRecord }; + +/** The provider as served: the defining plugin's version, `compose` and adapters. */ +type Provider = { readonly version: number; readonly provider: RegistryProvider }; + +/** Where a composition stands: never composed, its last good value, and its failure while `compose` throws. */ +type Composition = { + readonly good: { readonly value: unknown } | undefined; + readonly failure: { readonly error: unknown } | undefined; +}; + +const NEVER: Composition = { good: undefined, failure: undefined }; + +/** One registry of a host. */ +export class RegistryRecord { + readonly name: string; + readonly #batch: Batch; + readonly #report: (error: RegistryError) => void; + #provider: Provider | undefined; + /** The contributions of live contributors, sorted by plugin, then type (a facet's first). */ + readonly #contributions: RegistryContribution[] = []; + /** The last table composed, kept so an unchanged table composes nothing. */ + #table: RegistryTable | undefined; + #dirty = true; + #composition: Composition = NEVER; + /** Set while its table is built and composed: a read then is a cycle of registries. */ + #composing = false; + /** The conflicts the last good composition reported, by key: a conflict is reported once per episode. */ + #conflicts = new Set(); + /** + * Registries whose values are contributions in this one's table, each with how many: a contribution that would + * close a cycle of them is refused. An edge lives as long as its contribution is in the table. + */ + readonly #reads = new Map(); + /** The values readers follow, by version; made by the first reader of a version. */ + readonly #values = new Map(); + + constructor(name: string, batch: Batch, report: (error: RegistryError) => void) { + this.name = name; + this.#batch = batch; + this.#report = report; + } + + /** The provider's version; `undefined` while none serves it. */ + get providedVersion(): number | undefined { + return this.#provider?.version; + } + + /** The value readers of `version` follow. */ + value(version: number): RegistryValueSource { + let value = this.#values.get(version); + if (value === undefined) { + value = new RegistryValueSource(this, version, this.#batch); + this.#values.set(version, value); + } + return value; + } + + /** Serves the registry with `provider` at `version` (the defining plugin's facet); a second serve is refused. */ + serve(version: number, provider: RegistryProvider): void { + if (this.#provider !== undefined) throw new Error(`registry ${this.name} is served twice`); + // Declaration policy: plugin code is bundled without type checks. + if (typeof provider?.compose !== "function") throw new Error(`registry ${this.name}: compose is a function`); + this.#provider = { version, provider }; + } + + /** The first composition, at the end of `load`'s setups, inside one batch for every served registry. */ + composeServed(): void { + if (this.#provider === undefined) return; + this.#invalidate(); + this.start(); + } + + /** + * Refuses a contribution that is `source`'s value when it would close a cycle (declaration policy, at + * `contribute`): through the registries in the tables now and the edges `pending` (the contributing attempt's + * earlier declarations, which enter the tables only once it is live). + */ + assertNoCycle(source: RegistryValueSource, pending: readonly RegistryEdge[]): void { + const { record } = source; + if (record === undefined) return; + if (record.#reaches(this, new Set(), pending)) { + throw invalid("cycle", `registries ${this.name} and ${record.name} read each other`); + } + } + + /** Adds a live contributor's contribution, inside a batch: the table changes. */ + add(contribution: RegistryContribution): void { + const at = this.#contributions.findIndex((other) => compareContributors(contribution, other) < 0); + this.#contributions.splice(at < 0 ? this.#contributions.length : at, 0, contribution); + const { reads } = contribution; + if (reads !== undefined) this.#reads.set(reads, (this.#reads.get(reads) ?? 0) + 1); + contribution.follow(); + this.#invalidate(); + } + + /** Removes an ending contributor's contribution, inside a batch: the table changes. */ + remove(contribution: RegistryContribution): void { + const at = this.#contributions.indexOf(contribution); + if (at < 0) return; + this.#contributions.splice(at, 1); + const { reads } = contribution; + if (reads !== undefined) { + const left = this.#reads.get(reads)! - 1; + if (left === 0) this.#reads.delete(reads); + else this.#reads.set(reads, left); + } + contribution.unfollow(); + this.#invalidate(); + } + + /** A followed contribution changed, inside a batch: the table is marked, composed when read or followed. */ + contributionChanged(): void { + this.#invalidate(); + } + + /** + * The current composition, composed now when the table changed: the value, or `compose`'s failure with the last + * good one; `unavailable/undeclared` while no provider serves it. A read while it composes is a cycle. + */ + start(): NativeStart { + if (this.#composing) throw invalid("cycle", `registry ${this.name} is read while it composes`); + if (this.#provider === undefined) { + const error = new ChordError({ + code: "unavailable", + reason: "undeclared", + message: `registry ${this.name}: no plugin serves it here`, + }); + return { kind: "failed", error, last: undefined }; + } + if (this.#dirty) this.#recompose(this.#provider); + const { good, failure } = this.#composition; + if (failure !== undefined) return { kind: "failed", error: failure.error, last: good }; + // A composition without a failure has a value. + return { kind: "value", value: good!.value }; + } + + /** The adapter mapping the composed value back for readers of `version`; `undefined` when there is none. */ + valueAdapter(version: number): ((current: never) => unknown) | undefined { + return this.#provider?.provider.accepts?.[version]?.value; + } + + /** Runs an adapter of the provider as a pure compute, counted. */ + adapt(adapter: () => T): T { + this.#batch.count("chord.registry.adapts"); + return this.#batch.compute(adapter); + } + + /** Reports a contribution left out (`contribution_failed`, `contribution_refused`; the caller keeps the episode). */ + reportContribution(error: RegistryError): void { + this.#report(error); + } + + /** Reports a throw of `compose` or of a value adapter (`registry_failed`; the caller keeps the episode). */ + failed(error: unknown): void { + this.#report({ kind: "registry_failed", registry: this.name, error: handedOut(error) }); + } + + /** + * The host closed: the contributions stop following their sources, the values end (a value kept by a reader still + * reads the last composition). + */ + close(): void { + for (const contribution of this.#contributions) contribution.unfollow(); + this.#contributions.length = 0; + this.#reads.clear(); + this.#table = undefined; + this.#dirty = false; + for (const value of this.#values.values()) value.close(); + this.#values.clear(); + } + + /** + * The table changed: every version's value is marked and listed (each lists itself once per batch); while it is + * not composed since an earlier change, only listed (its dependents are marked already). + */ + #invalidate(): void { + if (this.#dirty) { + for (const value of this.#values.values()) value.relisted(); + return; + } + this.#dirty = true; + for (const value of this.#values.values()) value.recomposed(); + } + + /** + * Builds the table and composes it, unless it is the last one composed, entry for entry (`compose` is pure). A + * throw is a failure episode: reported once, the last good composition kept; conflicts are reported once per + * episode, as their message and plugins. + */ + #recompose({ version, provider }: Provider): void { + this.#dirty = false; + this.#composing = true; + try { + const table = this.#build(version, provider); + if (table === this.#table && this.#composition !== NEVER) return; + this.#table = table; + const conflicts = new Map(); + const context: ComposeContext = { + report: (conflict) => + conflicts.set(`${conflict.message}\u0000${conflict.plugins.join("\u0000")}`, conflict), + }; + let value: unknown; + try { + this.#batch.count("chord.registry.composes"); + value = this.#batch.compute(() => provider.compose(table, context)); + } catch (error) { + if (this.#composition.failure === undefined) this.failed(error); + this.#composition = { good: this.#composition.good, failure: { error } }; + return; + } + this.#composition = { good: { value }, failure: undefined }; + for (const [key, conflict] of conflicts) { + if (this.#conflicts.has(key)) continue; + this.#report({ kind: "registry_conflict", registry: this.name, ...conflict }); + } + this.#conflicts = new Set(conflicts.keys()); + } finally { + this.#composing = false; + } + } + + /** The table of the live contributions now; the last one itself when every entry is the same object. */ + #build(version: number, provider: RegistryProvider): RegistryTable { + const previous = this.#table; + const table: RegistryEntry[] = []; + let same = previous !== undefined; + for (const contribution of this.#contributions) { + const entry = contribution.entry(version, provider); + if (entry === undefined) continue; + if (same && previous![table.length] !== entry) same = false; + table.push(entry); + } + return same && previous!.length === table.length ? previous! : table; + } + + /** Whether `target` is reached from here through the tables' edges and `pending`. */ + #reaches(target: RegistryRecord, visited: Set, pending: readonly RegistryEdge[]): boolean { + if (this === target) return true; + visited.add(this); + for (const next of this.#reads.keys()) { + if (!visited.has(next) && next.#reaches(target, visited, pending)) return true; + } + for (const { from, to } of pending) { + if (from === this && !visited.has(to) && to.#reaches(target, visited, pending)) return true; + } + return false; + } +} + +/** Table order: by plugin name, then type, a facet (no type) first; code point order, never load order. */ +function compareContributors(a: RegistryContribution, b: RegistryContribution): number { + const left = a.contributor; + const right = b.contributor; + if (left.plugin !== right.plugin) return left.plugin < right.plugin ? -1 : 1; + const leftType = left.type ?? ""; + const rightType = right.type ?? ""; + if (leftType === rightType) return 0; + return leftType < rightType ? -1 : 1; +} diff --git a/packages/chord/src/next/registries/registry.ts b/packages/chord/src/next/registries/registry.ts new file mode 100644 index 000000000..6b79109b8 --- /dev/null +++ b/packages/chord/src/next/registries/registry.ts @@ -0,0 +1,89 @@ +// Registry tokens (design §17, slice 8 D1): `defineRegistry`, shared by every plugin as a contract is (identity and +// types only), and what the defining plugin's facet serves it with (`compose`, `accepts`); the table `compose` gets and +// the composed value readers follow. + +import type { Context } from "../../context/index.ts"; +import type { NativeWatchable, Watchable } from "../../watch.ts"; +import { checkName } from "../contracts/names.ts"; +import type { NativeObservation, NativeObserver } from "../state/index.ts"; + +/** + * A registry token: its name (as type names, `[a-z0-9@/._~-]`), the version the bundling plugin was built against, the + * contribution type `C` and the composed value type `V`. Shared: every contributor and reader bundles its own copy. + */ +export type Registry = { + readonly kind: "registry"; + readonly name: string; + readonly version: number; + /** Type only: `C` in, `V` out. */ + readonly "~types"?: (contribution: C) => V; +}; + +/** One contribution in a registry's table: its plugin, its object type for an object contributor, its version. */ +export type RegistryEntry = { + readonly plugin: string; + /** The contributing object's type; absent for a facet. */ + readonly type?: string; + /** The registry version its contributor was built against (an older one adapted to `C`). */ + readonly version: number; + readonly value: C; +}; + +/** A registry's table: one entry per live contributor, sorted by plugin name, then type (a facet's first). */ +export type RegistryTable = readonly RegistryEntry[]; + +/** A conflict `compose` resolved with its fallback: reported as `registry_conflict`, once per conflict episode. */ +export type RegistryConflict = { readonly message: string; readonly plugins: readonly string[] }; + +/** What `compose` gets besides the table. */ +export type ComposeContext = { + /** Reports a conflict `compose` resolved (the first by plugin name wins, for example). */ + report(conflict: RegistryConflict): void; +}; + +/** + * Adapters of older versions, by version, the provider's code: `contribution` maps an older contribution to `C` (once + * per contribution value), `value` maps the composed value back for a reader of that version (once per composition). + * A plugin annotates the old contribution's type (`(old: HttpV1) => upgrade(old)`). + */ +export type RegistryAccepts = { + readonly [version: number]: { + readonly contribution?: (old: never) => C; + readonly value?: (current: V) => unknown; + }; +}; + +/** What the defining plugin's facet serves a registry with (`host.serve(R, provider)`). */ +export type RegistryProvider = { + /** + * Composes the table: pure and synchronous, always a value (a fallback built in), conflicts reported. Runs when the + * table changed and someone reads or follows the value; a throw is the owner's bug (`registry_failed`, the last + * good composition stays). + */ + readonly compose: (table: RegistryTable, context: ComposeContext) => V; + readonly accepts?: RegistryAccepts; +}; + +/** What a contributor contributes: a plain value, or a Chord watchable of one (followed: its change recomposes). */ +export type Contribution = C | Watchable | NativeWatchable; + +/** + * A registry's composed value as a reader of one version sees it (`self.registry`, `host.registry`): a native + * watchable; `.value` is the last good composition (a later throwing `compose` keeps it), a watch gets each new one. + */ +export interface RegistryValue extends NativeWatchable { + /** + * A synchronous observer of its compositions (as Durable's `observeXXX`): `advance` runs inside the batch that + * composed, so it only records; an attached observer counts as watching, so the composition runs for it. + */ + observe(observer: NativeObserver, context: Context): NativeObservation; +} + +/** Declares a registry token: `name` and the `version` this plugin is built against. */ +export function defineRegistry(name: string, options: { readonly version: number }): Registry { + checkName(name, "registry"); + const { version } = options; + // Declaration policy: JavaScript callers reach this check. + if (!Number.isSafeInteger(version) || version < 1) throw new Error(`${name}.version: expected a positive integer`); + return { kind: "registry", name, version }; +} diff --git a/packages/chord/src/next/registries/value.ts b/packages/chord/src/next/registries/value.ts new file mode 100644 index 000000000..7b3b8e1db --- /dev/null +++ b/packages/chord/src/next/registries/value.ts @@ -0,0 +1,177 @@ +// Registry values (slice 8 §1 "Reading", S2, S3): a registry's composed value as the readers of one version see it, a +// native source. The provider's version reads the composition itself; an older one reads it mapped back with +// `accepts[old].value`, once per composition, kept only while someone reads or follows it; a version the provider +// cannot serve fails `contract_mismatch`. Readers declared by attempts are woken (R18) when the value gets its first +// good value. + +import { ChordError } from "../data/index.ts"; +import { type Batch, FollowedSource, type NativeStart } from "../state/internal.ts"; +import type { RegistryRecord } from "./record.ts"; + +/** An object waiting for a registry's value (`self.registry`): retried at once when its first value arrives (R18). */ +export interface RegistryWaiter { + dependencyLive(): void; +} + +/** The last good mapping for an older version, and the composition it was made from. */ +type Mapped = { readonly from: unknown; readonly value: unknown }; + +/** A failed mapping of composition `from`, until another composition maps. */ +type MapFailure = { readonly from: unknown; readonly error: unknown }; + +/** + * Where a value's registry stands for it: on the host, or closed with the host (its last start kept, the record let + * go, so a value an app keeps keeps no closed host). + */ +type Registered = + | { readonly kind: "open"; readonly record: RegistryRecord } + | { readonly kind: "closed"; readonly last: NativeStart }; + +/** One version's view of a registry's composed value. */ +export class RegistryValueSource extends FollowedSource { + readonly name: string; + readonly version: number; + #registered: Registered; + /** Readers declared by attempts, each a token of its own (one object may read twice), until released. */ + readonly #readers = new Set<{ readonly waiter: RegistryWaiter }>(); + /** Whether it had a good value: the first one wakes the readers. */ + #valued = false; + #mapped: Mapped | undefined; + #mapFailure: MapFailure | undefined; + + constructor(record: RegistryRecord, version: number, batch: Batch) { + super(batch); + this.name = record.name; + this.version = version; + this.#registered = { kind: "open", record }; + } + + /** Its registry's record while the host is open. */ + get record(): RegistryRecord | undefined { + const registered = this.#registered; + return registered.kind === "open" ? registered.record : undefined; + } + + /** The composition (or its mapping) at its last good value; throws the failure when there is none. */ + get value(): unknown { + const start = this.start(); + if (start.kind === "value") return start.value; + if (start.last === undefined) throw start.error; + return start.last.value; + } + + /** Its composition (mapped for an older version), composed now if its table changed; a first value wakes readers. */ + override start(): NativeStart { + const registered = this.#registered; + if (registered.kind === "closed") return registered.last; + const start = this.#start(registered.record); + if (!this.#valued && (start.kind === "value" || start.last !== undefined)) { + this.#valued = true; + // Not inside the batch that composed: a retry runs a factory, which may not run in a batch. + for (const { waiter } of this.#readers) queueMicrotask(() => waiter.dependencyLive()); + } + return start; + } + + /** Its composition (or mapping) now; throws while it is failed, though `.value` keeps the last good one. */ + override current(): unknown { + const start = this.start(); + if (start.kind === "failed") throw start.error; + return start.value; + } + + /** Followed while someone watches or observes it, and while readers wait for its first value. */ + override get watched(): boolean { + return super.watched || (this.#readers.size > 0 && !this.#valued); + } + + /** Registers `waiter` (an activation's declared dependency) until the returned release. */ + read(waiter: RegistryWaiter): () => void { + const token = { waiter }; + this.#readers.add(token); + return () => { + this.#readers.delete(token); + if (this.#readers.size === 0 && !super.watched) this.#dropMapping(); + }; + } + + /** The registry's table changed (inside a batch): its dependents are marked, its followers get the next frame. */ + recomposed(): void { + this.changed(); + } + + /** + * The registry's table changed again while not composed since (inside a batch): its dependents are marked already + * (none read it meanwhile, or it would be composed); its followers get the next frame. Marking stops here, so a + * cycle of registries through their values ends. + */ + relisted(): void { + this.list(); + } + + /** The host closed: its watches and observers end, its readers and mapping go; it keeps its last start only. */ + close(): void { + let last: NativeStart; + try { + last = this.start(); + } catch (error) { + last = { kind: "failed", error, last: undefined }; + } + this.#registered = { kind: "closed", last }; + this.#readers.clear(); + this.#dropMapping(); + this.end(); + } + + /** Nobody follows it: without readers, its mapping goes (computed again on the next read). */ + protected override unfollowed(): void { + if (this.#readers.size === 0) this.#dropMapping(); + } + + #dropMapping(): void { + this.#mapped = undefined; + this.#mapFailure = undefined; + } + + #start(record: RegistryRecord): NativeStart { + const provided = record.providedVersion; + if (provided === undefined || provided === this.version) return record.start(); + const map = provided > this.version ? record.valueAdapter(this.version) : undefined; + if (map === undefined) { + const message = `registry ${this.name}: version ${this.version} is not served here (version ${provided} is)`; + return { kind: "failed", error: new ChordError({ code: "contract_mismatch", message }), last: undefined }; + } + return this.#mappedStart(record, record.start(), map); + } + + /** + * The composition `current` mapped back with `map`, once per composition; a throwing mapping fails this version + * only (reported once per episode), keeping its last good mapping; a composition mapped before heals it. + */ + #mappedStart( + record: RegistryRecord, + current: NativeStart, + map: (current: never) => unknown, + ): NativeStart { + if (current.kind === "failed" && current.last === undefined) return current; + const from = current.kind === "value" ? current.value : current.last!.value; + // `undefined` is a composition too: the records, not their fields, say whether there is one. + const mapped = this.#mapped; + const failure = this.#mapFailure; + if (mapped !== undefined && mapped.from === from) { + this.#mapFailure = undefined; + } else if (failure === undefined || failure.from !== from) { + try { + this.#mapped = { from, value: record.adapt(() => map(from as never)) }; + this.#mapFailure = undefined; + } catch (error) { + if (failure === undefined) record.failed(error); + this.#mapFailure = { from, error }; + } + } + const last = this.#mapped === undefined ? undefined : { value: this.#mapped.value }; + if (this.#mapFailure !== undefined) return { kind: "failed", error: this.#mapFailure.error, last }; + if (current.kind === "failed") return { kind: "failed", error: current.error, last }; + return { kind: "value", value: last!.value }; + } +} diff --git a/packages/chord/src/next/state/README.md b/packages/chord/src/next/state/README.md index b386d167b..c10600a71 100644 --- a/packages/chord/src/next/state/README.md +++ b/packages/chord/src/next/state/README.md @@ -1,6 +1,7 @@ # state -Watchables, derives and the batch: how values change, are computed from each other and reach their watches. +Watchables (JSON and native), derives, writables and the batch: how values change, are computed from each other and +reach their watches. ## For apps (`index.ts`) @@ -25,6 +26,17 @@ watch.start(async (call, ctx) => { `[["r", value]]`). Values are JSON. A watch ends only for good (its owner ended, `stop()`, an aborted context, a throwing listener). - `constant(value)`: a watchable that never changes; a state member or a derive source. +- Native sources (`NativeWatchable`, W4): in-process values compared by identity, never ops, never state members (a + factory returning one fails `broken`, Q14): `self.writable(initial)`, a registry's composed value + (`self.registry`, `registries/`), and `ref.state`. They are derive sources (`self.derive([writable, doc], …)` takes + both kinds; the result is JSON) and `on` sources. A watch gets one call per batch that changed the value, each new + value in order; a failure (a registry whose `compose` threw) is one `failed` call per episode, the watch stays open. + `observe(observer, ctx)` attaches a synchronous observer (Durable's shape: `advance(call, ctx)` inside the batch, + record only; `end(end)`), which counts as watching. `ref.state` gives every state to its watches, not one per batch. +- `Writable` (`self.writable(initial)`): `set(value)`; the same value (by identity) notifies nobody and costs one + comparison; inside a batch it is marked there, outside it opens its own; inside a derive compute or a `compose` it + throws `invalid/reentrant`. It belongs to its attempt: `set` after the attempt's end throws `released`, its watches + end `ended`, its value stays readable. - `mapShared(f)`: maps an array, calling `f` once per object element (by identity) and reusing its result while the element stays the same object, so a derive over a list keeps unchanged parts identical and its diff cheap. @@ -55,8 +67,10 @@ Rules: the flushes listed with `flushing(flush)` during it (a link sends one state message per batch, W1); `within(adopt, ctx)` adopts a change inside the commit running now, or as a commit of its own, or, while a commit's flushes send, as the next commit (presence and keep-alive counts: a departure found while sending lands there); - `run(work)` sets the reentrancy flag (nested); `assertOutside(operation)`; `count(counter)` counts derive work on - the host's metrics (`chord.derive.computes`, `chord.derive.diffs`). + `run(work)` sets the reentrancy flag (nested); `compute(work)` also marks a pure compute (derives, `compose`), in + which `assertNotComputing(operation)` refuses changes; `assertOutside(operation)`; `count(counter)` counts derive and + registry work on the host's metrics (`chord.derive.computes`, `chord.derive.diffs`, `chord.registry.composes`, + `chord.registry.adapts`). `changed(source)` takes any `Delivered` (JSON or native source). - `Source`: a changing value on a host (a singleton document, a derive, a replica) with its watches, subscribers and dependents (`Dependent`: derives, and member sources passing changes on to theirs). Subclasses call `changed()` when a commit changes them and implement `deliver(ctx)` (`publish` a value, `failWatches` a recoverable failure, @@ -67,4 +81,13 @@ Rules: - `Derive`: declared with its raw sources; `attach(resolve, report)` resolves them at activation (and drops the raw list; `report` gets a failure of its compute, once per episode), `detach(error)` when its object ends (later reads throw `error` for good), `invalidate()` from its sources (a failed derive too: it recomputes on its next read). -- `Constant`: the class behind `constant()`. `DeriveSource` is what a derive reads: a `Source` or a `Constant`. +- `Constant`: the class behind `constant()`. `DeriveSource` is a JSON watchable as Chord follows it (a state member): + a `Source` or a `Constant`; `DeriveInput` is what a derive reads: those and `NativeSource`s. +- `NativeSource`: the base of native sources, what derives and registry tables read: `dependents`, `start()` + (`NativeStart`: a value, or a failure with its last good value), `changed()` (inside a batch: marks the dependents) + and `end()` (drops them and the batch). `FollowedSource` adds followers through the batch: `openWatch`, `observe` + (its abort listener removed at the end), `subscribe(observer)` (Chord's own followers, also without a value: + `retryOn`), `deliver(ctx)` (the new value when its followers have another, a failure once per episode; nothing is + kept once nobody follows). `WritableSource` is the writable, its owner (`WritableOwner`) the attempt's context and + seal check, let go when the attempt ends; `ref.state` (`host/`) is a plain `NativeSource` giving each state to its + own watches. diff --git a/packages/chord/src/next/state/batch.ts b/packages/chord/src/next/state/batch.ts index 66dbd5d5a..38eeb0901 100644 --- a/packages/chord/src/next/state/batch.ts +++ b/packages/chord/src/next/state/batch.ts @@ -4,13 +4,18 @@ import type { TelemetryMetrics } from "@earendil-works/pi-telemetry"; import type { Context } from "../../context/index.ts"; import { invalid } from "../data/internal.ts"; -import type { Source } from "./watchable.ts"; /** What a batch reports: a watch listener that threw, or a frame delivery that threw (a derive's value is no JSON). */ export type BatchError = | { readonly kind: "listener_error"; readonly error: unknown } | { readonly kind: "delivery_failed"; readonly error: unknown }; +/** What a commit lists to get its frame delivered: a watched source it changed (JSON or native). */ +export interface Delivered { + /** Delivers this batch's frame to its watches and subscribers; called once per batch that listed it. */ + deliver(context: Context): void; +} + /** A host's batch: the reentrancy flag, the watched sources a commit changed, and where background errors go. */ export class Batch { /** Reports a throwing watch listener's error (to the host's `onError`); what watches get as their `report`. */ @@ -19,8 +24,10 @@ export class Batch { readonly #metrics: TelemetryMetrics | undefined; /** Set while adopting, delivering or computing a derive; one flag per host. */ #running = false; + /** Set while a pure compute runs (a derive's, a registry's `compose`): nothing may change a source then. */ + #computing = false; /** The watched sources the current commit changed, each listed once by its `changed`. */ - #changed: Source[] = []; + #changed: Delivered[] = []; /** What runs once the current commit's frames were delivered, each listed once by its `flushing`. */ #flushes: ((context: Context) => void)[] = []; /** Set while the current commit's flushes run (its links send). */ @@ -46,6 +53,28 @@ export class Batch { if (this.#running) throw invalid("reentrant", `${operation} inside a commit or a derive compute`); } + /** + * Throws `invalid/reentrant` inside a pure compute (a derive's, a registry's `compose`): `operation` would change a + * source while one is computed from it. Changes inside an adoption are fine (they land in its batch). + */ + assertNotComputing(operation: string): void { + if (this.#computing) throw invalid("reentrant", `${operation} inside a derive compute or a compose`); + } + + /** Runs the pure compute `work` (`run`, and changes refused: `assertNotComputing`); nests. */ + compute(work: () => T): T { + const running = this.#running; + const computing = this.#computing; + this.#running = true; + this.#computing = true; + try { + return work(); + } finally { + this.#running = running; + this.#computing = computing; + } + } + /** Runs `work` with reentrant calls refused; nests (a derive compute inside a commit). */ run(work: () => T): T { const outer = this.#running; @@ -129,7 +158,7 @@ export class Batch { } /** Lists a watched source the current commit changed. */ - changed(source: Source): void { + changed(source: Delivered): void { this.#changed.push(source); } @@ -138,8 +167,12 @@ export class Batch { this.#flushes.push(flush); } - /** Counts one unit of derive work on the host's metrics (a lazy read has no context to count through). */ - count(counter: "chord.derive.computes" | "chord.derive.diffs"): void { + /** + * Counts one unit of derive or registry work on the host's metrics (a lazy read has no context to count through). + */ + count( + counter: "chord.derive.computes" | "chord.derive.diffs" | "chord.registry.composes" | "chord.registry.adapts", + ): void { this.#metrics?.counter(counter).add(1); } } diff --git a/packages/chord/src/next/state/derive.ts b/packages/chord/src/next/state/derive.ts index 6bb86f617..df364f10d 100644 --- a/packages/chord/src/next/state/derive.ts +++ b/packages/chord/src/next/state/derive.ts @@ -5,11 +5,15 @@ import type { Context } from "../../context/index.ts"; import { diffRevisions, type Op } from "../../delta/index.ts"; import type { JsonValue } from "../../json.ts"; import type { Batch } from "./batch.ts"; +import { NativeSource } from "./native-source.ts"; import { type Constant, type Dependent, handedOut, Source, type WatchStart } from "./watchable.ts"; -/** What a derive reads: a source it depends on, or a constant. */ +/** A JSON watchable as Chord follows it (a state member, a JSON derive source): a source, or a constant. */ export type DeriveSource = Source | Constant; +/** What a derive reads: a JSON source or constant, or a native source (a `writable`, a registry value, `ref.state`). */ +export type DeriveInput = DeriveSource | NativeSource; + /** * Where a derive stands. `dirty`: a source changed since the last compute (or there was none). A failed derive (its * compute threw) throws its error until a source it reads changes or is re-bound, then recomputes (self-healing, design @@ -34,10 +38,10 @@ const NO_VALUE: unique symbol = Symbol("no value"); * it. */ export class Derive extends Source implements Dependent { - readonly #compute: (values: readonly JsonValue[]) => JsonValue; + readonly #compute: (values: readonly unknown[]) => JsonValue; /** The sources as declared, until `attach` resolves them. */ #declared: readonly unknown[] | undefined; - #sources: readonly DeriveSource[] = []; + #sources: readonly DeriveInput[] = []; #state: DeriveState = { kind: "dirty" }; /** The value this derive's watches have, kept from its marking to its frame in the batch. */ #framed: JsonValue | typeof NO_VALUE | undefined; @@ -49,7 +53,7 @@ export class Derive extends Source implements Dependent { /** Its last good value, which a watch opened while it is failed starts at (`NO_VALUE` before the first). */ #good: JsonValue | typeof NO_VALUE = NO_VALUE; - constructor(sources: readonly unknown[], compute: (values: readonly JsonValue[]) => JsonValue, batch: Batch) { + constructor(sources: readonly unknown[], compute: (values: readonly unknown[]) => JsonValue, batch: Batch) { super(batch); this.#declared = sources; this.#compute = compute; @@ -77,16 +81,20 @@ export class Derive extends Source implements Dependent { * `report` gets a throw of its compute, once per failure episode (a value ends it); a failing source's error is its * source's to report. */ - attach(resolve: (source: unknown, index: number) => DeriveSource, report: (error: unknown) => void): void { + attach(resolve: (source: unknown, index: number) => DeriveInput, report: (error: unknown) => void): void { this.#report = report; this.#sources = this.#declared!.map(resolve); this.#declared = undefined; - for (const source of this.#sources) if (source instanceof Source) source.dependents.add(this); + for (const source of this.#sources) { + if (source instanceof Source || source instanceof NativeSource) source.dependents.add(this); + } } /** Its object ended: no source keeps it, its watches end `ended`, reads throw `error` (the object's). */ detach(error: Error): void { - for (const source of this.#sources) if (source instanceof Source) source.dependents.delete(this); + for (const source of this.#sources) { + if (source instanceof Source || source instanceof NativeSource) source.dependents.delete(this); + } this.#sources = []; this.#declared = undefined; this.#state = { kind: "detached", error }; @@ -146,7 +154,7 @@ export class Derive extends Source implements Dependent { } #recompute(): JsonValue { - return this.batch.run(() => { + return this.batch.compute(() => { try { this.batch.count("chord.derive.computes"); // A failing source throws its own error here, so a derive over a failed derive fails with the same error. diff --git a/packages/chord/src/next/state/index.ts b/packages/chord/src/next/state/index.ts index 8fbd9ad1e..b6ffb24db 100644 --- a/packages/chord/src/next/state/index.ts +++ b/packages/chord/src/next/state/index.ts @@ -1,5 +1,7 @@ -// State for apps: watchables, constants, and the derive helper `mapShared`. +// State for apps: watchables, constants, writables, observers of native sources, and the derive helper `mapShared`. export type { Watchable } from "../../watch.ts"; export { mapShared } from "./derive.ts"; +export type { NativeObservation, NativeObserver } from "./native-source.ts"; export { constant } from "./watchable.ts"; +export type { Writable } from "./writable.ts"; diff --git a/packages/chord/src/next/state/internal.ts b/packages/chord/src/next/state/internal.ts index 5ecd3641f..f4c0475cc 100644 --- a/packages/chord/src/next/state/internal.ts +++ b/packages/chord/src/next/state/internal.ts @@ -1,5 +1,14 @@ -// State for Chord's layers: the batch, sources, derives and constants behind the app's watchables. +// State for Chord's layers: the batch, sources (JSON and native), derives, constants and writables behind the +// app's watchables. -export { Batch, type BatchError } from "./batch.ts"; -export { Derive, type DeriveSource } from "./derive.ts"; +export { Batch, type BatchError, type Delivered } from "./batch.ts"; +export { Derive, type DeriveInput, type DeriveSource } from "./derive.ts"; +export { + FollowedSource, + type NativeObservation, + type NativeObserver, + NativeSource, + type NativeStart, +} from "./native-source.ts"; export { Constant, type Dependent, handedOut, Source, type Subscriber, type WatchStart } from "./watchable.ts"; +export { type WritableOwner, WritableSource } from "./writable.ts"; diff --git a/packages/chord/src/next/state/native-source.ts b/packages/chord/src/next/state/native-source.ts new file mode 100644 index 000000000..9d6308801 --- /dev/null +++ b/packages/chord/src/next/state/native-source.ts @@ -0,0 +1,291 @@ +// Native sources (W4, slice 8 §2): in-process values that change, compared by identity (a `writable`, a registry's +// composed value, `ref.state`). Derives and registry tables depend on them and the batch marks them; the followed +// ones (writables, registry values) give their watches and observers one call per batch that changed them. + +import type { Context } from "../../context/index.ts"; +import { + NativeWatch, + type NativeWatchable, + type NativeWatchCall, + type NativeWatchHandle, + type WatchEnd, +} from "../../watch.ts"; +import type { Batch } from "./batch.ts"; +import { type Dependent, handedOut } from "./watchable.ts"; + +/** + * A synchronous observer of a native source (Durable's observer shape): `advance` is called inside the batch that + * changed the source, so it only records; `end` once, when the source ended (`ended`) or the observer's context was + * aborted (`aborted`). + */ +export interface NativeObserver { + /** The source's call of this batch: its new value, or a failure it may recover from. */ + advance(call: NativeWatchCall, context: Context): void; + /** The observer is detached for good with `end`. */ + end(end: WatchEnd): void; +} + +/** What `observe` gives: the value the observer starts at, and how to detach it (Durable's `Observation`). */ +export type NativeObservation = { readonly value: T; detach(): void }; + +/** + * What a new watch of a native source starts at: its value, or while it is failed, the failure and its last good value + * (`last` undefined when it never had one: a watch cannot start then). + */ +export type NativeStart = + | { readonly kind: "value"; readonly value: T } + | { readonly kind: "failed"; readonly error: unknown; readonly last: { readonly value: T } | undefined }; + +/** + * A native source as derives and registry tables read it: its value (`start()` while it may fail), the dependents a + * change marks (`changed()`, inside a batch), and the host's batch until it ended. How its watches get their calls is + * the subclass's (`FollowedSource` for writables and registry values; `ref.state` gives each state to its watches). + */ +export abstract class NativeSource implements NativeWatchable { + /** Derives and registry tables reading this source; each removes itself when it stops. */ + readonly dependents = new Set(); + /** The host's batch; dropped when the source ended, so an ended source keeps no host. */ + #batch: Batch | undefined; + + /** `batch` undefined: made ended (a released reference's state). */ + constructor(batch: Batch | undefined) { + this.#batch = batch; + } + + abstract get value(): T; + + abstract watch(context: Context): Promise>; + + /** What a new watch starts at; a source that can fail and recover tells its failure. */ + start(): NativeStart { + return { kind: "value", value: this.value }; + } + + /** + * Its value now as a contribution reads it: throws its failure while it is failed, where `.value` keeps the last good + * value for readers (a registry whose `compose` threw). + */ + current(): T { + return this.value; + } + + /** The host's batch, `undefined` once the source ended. */ + protected get batch(): Batch | undefined { + return this.#batch; + } + + /** Records a change inside the current batch: every dependent marked. */ + protected changed(): void { + for (const dependent of this.dependents) dependent.invalidate(); + } + + /** Ends the source for good: its dependents dropped, the batch let go. */ + protected end(): void { + this.#batch = undefined; + this.dependents.clear(); + } +} + +/** + * What the watches and observers were last given: a value, a failure (one call per failure episode), or nothing yet + * (a source with no value, followed for its first one). + */ +type Shown = { readonly kind: "value"; readonly value: T } | { readonly kind: "failed" } | { readonly kind: "none" }; + +const ENDED: WatchEnd = { kind: "ended" }; + +const ABORTED: WatchEnd = { kind: "aborted" }; + +const FAILED: Shown = { kind: "failed" }; + +const NONE = { kind: "none" } as const; + +const NOTHING = (): void => {}; + +/** + * A native source its followers get through the batch (a writable, a registry value): a change lists it for one frame + * when someone follows it; the frame is its value then, delivered only when it is another value (identity) than the + * one its followers have, or after a failure, whose call comes once per episode. + */ +export abstract class FollowedSource extends NativeSource { + readonly #watches = new Set>(); + /** Synchronous observers and Chord's own subscribers (`retryOn`), fed inside the batch, each with its cleanup. */ + readonly #observers = new Map, () => void>(); + /** What the followers have; `undefined` while nobody follows (so no old value is kept). */ + #shown: Shown | undefined; + /** Listed for a frame in the current batch. */ + #listed = false; + + /** Whether a watch or an observer follows it (a registry value also while readers wait for its first value). */ + get watched(): boolean { + return this.#watches.size > 0 || this.#observers.size > 0; + } + + async watch(context: Context): Promise> { + return this.openWatch(context); + } + + /** + * `watch` without the Promise. On a failed source it opens at the last good value and its first call is the + * failure; with no value to start at it throws the failure. An ended source gives a watch that has ended. + */ + openWatch(context: Context): NativeWatch { + const batch = this.batch; + if (batch === undefined) { + const ended = new NativeWatch(this.value, NOTHING, NOTHING); + ended.end(ENDED); + return ended; + } + batch.assertOutside("watch"); + const start = this.start(); + if (start.kind === "failed" && start.last === undefined) throw start.error; + const value = start.kind === "value" ? start.value : start.last!.value; + this.#follow(start); + const watch: NativeWatch = new NativeWatch(value, () => this.#unfollow(watch), batch.reportListener); + this.#watches.add(watch); + if (context.abortSignal !== undefined) watch.observeAbort(context.abortSignal); + if (start.kind === "failed") watch.fail(handedOut(start.error), context); + return watch; + } + + /** + * Attaches a synchronous observer at the current value (Durable's `observeXXX`); it counts as following, so a lazy + * value is computed in the batch that changed it. Throws the failure when there is no value to start at; aborting + * `context` detaches it (`aborted`); the source's end detaches it too (`ended`), its abort listener removed. + */ + observe(observer: NativeObserver, context: Context): NativeObservation { + this.batch?.assertOutside("observe"); + const start = this.start(); + if (start.kind === "failed" && start.last === undefined) throw start.error; + const value = start.kind === "value" ? start.value : start.last!.value; + if (this.batch === undefined) { + observer.end(ENDED); + return { value, detach: NOTHING }; + } + const signal = context.abortSignal; + if (signal?.aborted) { + observer.end(ABORTED); + return { value, detach: NOTHING }; + } + const aborted = (): void => { + if (!this.#observers.has(observer)) return; + this.unsubscribe(observer); + observer.end(ABORTED); + }; + const cleanup = signal === undefined ? NOTHING : () => signal.removeEventListener("abort", aborted); + signal?.addEventListener("abort", aborted, { once: true }); + this.#follow(start); + this.#observers.set(observer, cleanup); + return { value, detach: () => this.unsubscribe(observer) }; + } + + /** + * Feeds `observer` every call from now on, inside each batch, also while the source has no value (Chord's own + * followers: `retryOn`). Nothing is delivered for the current value. + */ + subscribe(observer: NativeObserver): void { + if (this.batch === undefined) return; + this.#follow(this.#startOrNone()); + this.#observers.set(observer, NOTHING); + } + + unsubscribe(observer: NativeObserver): void { + const cleanup = this.#observers.get(observer); + if (cleanup === undefined) return; + this.#observers.delete(observer); + cleanup(); + this.#settleFollowers(); + } + + /** Delivers this batch's frame: the new value when the followers have another, or a failure once per episode. */ + deliver(context: Context): void { + this.#listed = false; + const shown = this.#shown; + if (shown === undefined) return; + this.#deliver(shown, context); + // Readers waiting for a first value pulled it through this delivery: nothing is kept for them. + this.#settleFollowers(); + } + + /** + * Records a change inside the current batch: every dependent marked, and the source listed for a frame when someone + * follows it (or `watched` says so). + */ + protected override changed(): void { + super.changed(); + this.list(); + } + + /** + * Lists the source for a frame in the current batch when someone follows it (or `watched` says so), its dependents + * left as they are (marked already). + */ + protected list(): void { + if (this.#listed || !this.watched || this.batch === undefined) return; + this.#listed = true; + // Only readers waiting for a first value follow it without a follower's record: they have none. Not computed + // here: marking runs inside adoptions, the value is pulled in the delivery. + this.#shown ??= NONE; + this.batch.changed(this); + } + + /** Nothing follows it any more (no watch, no observer): a subclass drops what it kept for followers. */ + protected unfollowed(): void {} + + /** Ends the source for good: watches and observers end `ended` (their abort listeners removed). */ + protected override end(): void { + super.end(); + this.#shown = undefined; + // Ending detaches, which deletes the current entry (safe while iterating) and runs no listener. + for (const watch of this.#watches) watch.end(ENDED); + const observers = [...this.#observers]; + this.#observers.clear(); + for (const [observer, cleanup] of observers) { + cleanup(); + observer.end(ENDED); + } + } + + #deliver(shown: Shown, context: Context): void { + const start = this.#startOrNone(); + if (start.kind === "failed") { + // A failure reaches followers that had a value: those without one wait for the first. + if (shown.kind !== "value") return; + this.#shown = FAILED; + const error = handedOut(start.error); + for (const watch of this.#watches) watch.fail(error, context); + for (const observer of [...this.#observers.keys()]) observer.advance({ kind: "failed", error }, context); + return; + } + if (start.kind !== "value" || (shown.kind === "value" && shown.value === start.value)) return; + const { value } = start; + this.#shown = start; + for (const watch of this.#watches) watch.advance(value, context); + // A copy: an observer may detach itself in its call. + for (const observer of [...this.#observers.keys()]) observer.advance({ kind: "value", value }, context); + } + + /** Its start, `none` while it has no value. */ + #startOrNone(): NativeStart | { readonly kind: "none" } { + const start = this.start(); + return start.kind === "failed" && start.last === undefined ? NONE : start; + } + + /** A follower comes: the first one records what followers have (a later one opened at the same current value). */ + #follow(start: NativeStart | { readonly kind: "none" }): void { + if (this.#shown !== undefined) return; + this.#shown = start.kind === "failed" ? FAILED : start; + } + + #unfollow(watch: NativeWatch): void { + this.#watches.delete(watch); + this.#settleFollowers(); + } + + /** No follower left: nothing is shown any more, so no old value is kept. */ + #settleFollowers(): void { + if (this.#watches.size > 0 || this.#observers.size > 0) return; + this.#shown = undefined; + this.unfollowed(); + } +} diff --git a/packages/chord/src/next/state/writable.ts b/packages/chord/src/next/state/writable.ts new file mode 100644 index 000000000..e8760b409 --- /dev/null +++ b/packages/chord/src/next/state/writable.ts @@ -0,0 +1,78 @@ +// Writables (slice 8 §2, design §17): `self.writable(initial)`, an in-process value its attempt sets, identity +// compared; a contribution, a derive source and an `on` source, never a state member. + +import type { Context } from "../../context/index.ts"; +import type { NativeWatchable } from "../../watch.ts"; +import type { Batch } from "./batch.ts"; +import { FollowedSource } from "./native-source.ts"; + +/** An in-process value that changes when its owner sets it; `set` of the same value notifies nobody. */ +export interface Writable extends NativeWatchable { + /** + * Sets the value: a new value (by identity) is a new frame for its watches and marks the derives and registry + * tables reading it, in the batch running now or in one of its own. Throws `released` once its attempt ended, and + * `invalid/reentrant` inside a derive compute or a `compose`. + */ + set(value: T): void; +} + +/** Who owns a writable: the context its own batches run in, and the check that its attempt still runs. */ +export type WritableOwner = { + readonly context: Context; + /** Throws `released` once the attempt that made it was sealed. */ + assertUsable(): void; +}; + +/** Whom a writable belongs to: its running attempt, or nothing once that attempt ended (its error kept). */ +type Ownership = + | { readonly kind: "owned"; readonly owner: WritableOwner } + | { readonly kind: "ended"; readonly error: Error }; + +/** A writable: its value, set by its owner's code, delivered through the host's batch. */ +export class WritableSource extends FollowedSource implements Writable { + #value: T; + #ownership: Ownership; + + constructor(initial: T, owner: WritableOwner, batch: Batch) { + super(batch); + this.#value = initial; + this.#ownership = { kind: "owned", owner }; + } + + get value(): T { + return this.#value; + } + + set(value: T): void { + const ownership = this.#ownership; + if (ownership.kind === "ended") throw ownership.error; + ownership.owner.assertUsable(); + // The cost guard: setting the value it has costs one comparison past the seal check. + if (value === this.#value) return; + const batch = this.batch!; + batch.assertNotComputing("writable.set"); + this.#change(value, batch, ownership.owner.context); + } + + /** + * Adopts `value` in the batch running now, or in one of its own. Apart from `set`: a closure there would make every + * `set` allocate its context, the same value's too. + */ + #change(value: T, batch: Batch, context: Context): void { + batch.within(() => { + this.#value = value; + this.changed(); + }, context); + } + + /** + * Its attempt ended with `error`: its watches end `ended`, `set` throws `error`, its value stays readable; it keeps + * neither its attempt nor the host. + */ + detach(error: Error): void { + // Kept by `set`: its stack is made text now, so it holds none of the frames' objects. + void error.stack; + this.#ownership = { kind: "ended", error }; + this.end(); + } +} diff --git a/packages/chord/test/next/bench/calls.ts b/packages/chord/test/next/bench/calls.ts new file mode 100644 index 000000000..d8e5dc0f7 --- /dev/null +++ b/packages/chord/test/next/bench/calls.ts @@ -0,0 +1,90 @@ +// Benchmark of the paths slice 8 touched under every object (no registries involved): a local method call (its ctx), +// a commit through a watched derive (the batch, a derive's compute) and activation churn (an object with params made, +// live and ended per get: its attempt's declarations). Median µs per operation over bursts, and heap allocated per +// operation (growth plus what the collections during the burst freed). Runs on HEAD as it is (copy it there). +// Run by hand from packages/chord (not in CI), pinned for comparisons: +// taskset -c 6 node --conditions=source --experimental-strip-types --expose-gc test/next/bench/calls.ts +import { performance } from "node:perf_hooks"; +import { GCProfiler } from "node:v8"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { int, local, method, obj } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; + +const BURSTS = 9; + +const gc = (globalThis as { gc?: () => void }).gc!; + +const Counter = defineState("bench.counter", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); +const Store = local("bench.store", { + version: 1, + params: {}, + members: { doubled: int(), bump: method(), set: method(int()) }, +}); +const Item = local("bench.item", { version: 1, params: { id: int() }, members: { n: int() } }); + +const host = createHost({ onError: (error) => console.log("onError", error.kind) }); +await host.load( + [ + { + name: "@bench/calls", + facet: defineFacet((h) => { + h.serve(Store, (_, self) => { + const counter = self.state(Counter); + return { + doubled: self.derive([counter], ([c]) => c.n * 2), + bump: async () => null, + set: async (n, ctx) => { + await counter.change((draft) => { + draft.n = n; + }, ctx); + return null; + }, + }; + }); + h.serve(Item, ({ id }, self) => ({ n: self.derive([self.get(Store, {}).doubled], ([d]) => d + id) })); + }), + }, + ], + context, +); + +const store = await host.get(Store, {}, context); +const watch = await store.doubled.watch(context); +watch.start(async () => {}); + +/** Median µs per operation over the bursts (after one warm-up burst), and heap bytes allocated per operation. */ +async function measure(runs: number, operation: (i: number) => Promise): Promise { + const times: number[] = []; + let bytes = 0; + let counted = 0; + for (let burst = -1; burst < BURSTS; burst++) { + gc(); + const before = process.memoryUsage().heapUsed; + const profiler = new GCProfiler(); + profiler.start(); + const started = performance.now(); + for (let i = 0; i < runs; i++) await operation(i); + const elapsed = performance.now() - started; + const after = process.memoryUsage().heapUsed; + let freed = 0; + for (const { beforeGC, afterGC } of profiler.stop().statistics) { + freed += beforeGC.heapStatistics.usedHeapSize - afterGC.heapStatistics.usedHeapSize; + } + if (burst < 0) continue; + times.push((elapsed * 1000) / runs); + bytes += after - before + freed; + counted += runs; + } + times.sort((a, b) => a - b); + return `${times[Math.floor(times.length / 2)]!.toFixed(2)} µs, ${Math.round(bytes / counted)} B allocated`; +} + +console.log("local method call:", await measure(20_000, () => store.bump(context))); +console.log("commit through a watched derive:", await measure(10_000, (i) => store.set(i + 1, context))); +console.log( + "activation churn (get, live, release):", + await measure(2_000, async (i) => (await host.get(Item, { id: i }, context)).release()), +); +await watch.stop(); +await host.close(context); diff --git a/packages/chord/test/next/bench/registries.ts b/packages/chord/test/next/bench/registries.ts new file mode 100644 index 000000000..5eb75504b --- /dev/null +++ b/packages/chord/test/next/bench/registries.ts @@ -0,0 +1,108 @@ +// Benchmark of slice 8's registries and writables: a `writable.set` delivered to one watch, a set of the same value +// (one comparison), and one contribution of N (10, 100) changed while the registry is watched (one table build and one +// compose per set, over N entries). Median µs per operation over bursts, heap allocated per operation, and the +// registry's compose count per set (the cost guard: one). +// Run by hand from packages/chord (not in CI), pinned for comparisons: +// taskset -c 6 node --conditions=source --experimental-strip-types --expose-gc test/next/bench/registries.ts +import { performance } from "node:perf_hooks"; +import { GCProfiler } from "node:v8"; +import { InMemoryTelemetryContext } from "@earendil-works/pi-telemetry"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { int, local } from "../../../src/next/contracts/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { defineRegistry } from "../../../src/next/registries/index.ts"; +import type { Writable } from "../../../src/next/state/index.ts"; + +const BURSTS = 9; + +const gc = (globalThis as { gc?: () => void }).gc!; + +/** Median µs per operation over the bursts (after one warm-up burst), and heap bytes allocated per operation. */ +async function measure(runs: number, operation: (i: number) => void): Promise { + const times: number[] = []; + let bytes = 0; + let counted = 0; + for (let burst = -1; burst < BURSTS; burst++) { + gc(); + const before = process.memoryUsage().heapUsed; + const profiler = new GCProfiler(); + profiler.start(); + const started = performance.now(); + for (let i = 0; i < runs; i++) operation(i); + // The watches' listeners run on microtasks: their work counts too. + await new Promise((resolve) => setImmediate(resolve)); + const elapsed = performance.now() - started; + const after = process.memoryUsage().heapUsed; + let freed = 0; + for (const { beforeGC, afterGC } of profiler.stop().statistics) { + freed += beforeGC.heapStatistics.usedHeapSize - afterGC.heapStatistics.usedHeapSize; + } + if (burst < 0) continue; + times.push((elapsed * 1000) / runs); + bytes += after - before + freed; + counted += runs; + } + times.sort((a, b) => a - b); + return `${times[Math.floor(times.length / 2)]!.toFixed(3)} µs, ${Math.round(bytes / counted)} B allocated`; +} + +const Sum = defineRegistry("bench.sum", { version: 1 }); + +/** A host with `contributors` host-lifetime objects each contributing a writable number to `Sum`, watched. */ +async function registryOf(contributors: number) { + const telemetry = new InMemoryTelemetryContext(); + const writables: Writable[] = []; + const types = Array.from({ length: contributors }, (_, i) => + local(`bench.contributor-${i}`, { version: 1, params: {}, members: { n: int() } }), + ); + const host = createHost({ telemetry, onError: (error) => console.log("onError", error.kind) }); + const facet = defineFacet((h) => { + h.serve(Sum, { compose: (table) => table.reduce((sum, { value }) => sum + value, 0) }); + for (const Type of types) { + h.serve(Type, (_, self) => { + const value = self.writable(0); + writables.push(value); + self.contribute(Sum, value); + return { n: self.derive([value], ([n]) => n) }; + }); + } + }); + await host.load([{ name: "@bench/registries", facet }], context); + const sum = await host.registry(Sum, context); + const watch = await sum.watch(context); + watch.start(async () => {}); + const composes = (): number => + telemetry + .getMetrics() + .filter(({ name }) => name === "chord.registry.composes") + .reduce((n, { value }) => n + value, 0); + return { host, writables, watch, composes }; +} + +const one = await registryOf(1); +const writable = one.writables[0]!; +const watch = await writable.watch(context); +watch.start(async () => {}); +console.log("writable.set, one watch:", await measure(20_000, (i) => writable.set(i + 1))); +// The listeners' backlogs dropped first: what follows measures the set alone. +await watch.stop(); +await one.watch.stop(); +writable.set(-1); +console.log("writable.set of the same value:", await measure(1_000_000, () => writable.set(-1))); +console.log("an empty operation (the loop's own cost):", await measure(1_000_000, () => {})); +await one.host.close(context); + +for (const contributors of [10, 100]) { + const { host, writables, watch, composes } = await registryOf(contributors); + const before = composes(); + const target = writables[Math.floor(contributors / 2)]!; + const sets = 5_000; + const result = await measure(sets, (i) => target.set(i + 1)); + console.log( + `one of ${contributors} contributions set, registry watched:`, + result, + `${((composes() - before) / (sets * (BURSTS + 1))).toFixed(2)} composes per set`, + ); + await watch.stop(); + await host.close(context); +} diff --git a/packages/chord/test/next/data/errors.test.ts b/packages/chord/test/next/data/errors.test.ts index 65dee086a..c5d955a99 100644 --- a/packages/chord/test/next/data/errors.test.ts +++ b/packages/chord/test/next/data/errors.test.ts @@ -25,7 +25,9 @@ describe("ChordError", () => { const error = new ChordError({ code: "too_large", reason: "part", message: "m" }) as ChordError; if (error.code === "invalid") expectTypeOf(error.reason).toEqualTypeOf>(); if (error.code === "unavailable") - expectTypeOf(error.reason).toEqualTypeOf<"locked" | "park_timeout" | "busy" | "storage_failed" | undefined>(); + expectTypeOf(error.reason).toEqualTypeOf< + "locked" | "park_timeout" | "busy" | "storage_failed" | "undeclared" | undefined + >(); if (error.code === "gone") expectTypeOf(error.reason).toEqualTypeOf(); expect(error).toBeInstanceOf(Error); expect(error.name).toBe("ChordError"); @@ -83,7 +85,7 @@ describe("protocol table", () => { not_ready: [], released: ["ended", "closed", "settled", "version", "discarded"], too_large: ["part"], - unavailable: ["locked", "park_timeout", "busy", "storage_failed"], + unavailable: ["locked", "park_timeout", "busy", "storage_failed", "undeclared"], aborted: [], }, }; diff --git a/packages/chord/test/next/examples/29-registries.ts b/packages/chord/test/next/examples/29-registries.ts new file mode 100644 index 000000000..462bd20b0 --- /dev/null +++ b/packages/chord/test/next/examples/29-registries.ts @@ -0,0 +1,115 @@ +// Registries: an extension point between plugins. The defining plugin shares a token (`defineRegistry`: a name, a +// version, the contribution and value types) and serves it with `compose`, a pure function from the table of +// contributions (sorted by plugin name, never load order) to one value, conflicts resolved by a fallback and reported +// (`registry_conflict`). Facets and host-lifetime objects contribute one value each: a plain value, or a watchable +// (`self.writable`) whose changes recompose. Readers follow the composed value (`self.registry`, a dependency like +// `self.get`; `host.registry` outside objects) and guard each contribution where they call it. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/29-registries.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { local, method, obj, str } from "../../../src/next/contracts/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { defineRegistry, keyedEntries, type TableItem } from "../../../src/next/registries/index.ts"; + +const context = BACKGROUND_CONTEXT; + +// 1. The token, shared by every plugin (bundled like a contract): routes in, a router out. +type Handler = (path: string) => string; +type Routes = { readonly routes: Readonly> }; +type Router = ReadonlyMap>; +const Http = defineRegistry("example.http", { version: 1 }); + +/** The HTTP server: it reads the router at each request and guards the contribution it calls. */ +const Server = local("example.server", { + version: 1, + params: {}, + members: { request: method(obj({ path: str() }), str()) }, +}); + +/** A plugin whose routes are known only at runtime: it contributes a writable it sets. */ +const Mounts = local("example.mounts", { version: 1, params: {}, members: { mount: method(obj({ path: str() })) } }); + +// 2. The defining plugin serves the registry: the first route by plugin name wins, a duplicate is reported. +const httpPlugin = defineFacet((host) => { + host.serve(Http, { compose: (table, { report }) => keyedEntries(table, (entry) => entry.routes, report) }); + host.serve(Server, (_, self) => { + const router = self.registry(Http); + return { + request: async ({ path }) => { + const route = router.value.get(path); + if (route === undefined) return "404"; + // A contribution is another plugin's code: catch, report its plugin, go on. + try { + return route.value(path); + } catch (error) { + console.log(`route ${path} of ${route.plugin} threw:`, (error as Error).message); + return "500"; + } + }, + }; + }); +}); + +// 3. Two plugins contribute from their facets: plain values, in the table while the plugin is loaded. +const notesPlugin = defineFacet((_, self) => { + self.contribute(Http, { routes: { "/notes": () => "notes", "/broken": () => JSON.parse("{") } }); +}); +const searchPlugin = defineFacet((_, self) => { + self.contribute(Http, { routes: { "/search": () => "results", "/notes": () => "search's notes" } }); +}); + +// 4. A host-lifetime object contributes a writable: its changes recompose the router. +const mountsPlugin = defineFacet((host) => { + host.serve(Mounts, (_, self) => { + const routes = self.writable({ routes: {} }); + self.contribute(Http, routes); + return { + mount: async ({ path }) => { + routes.set({ routes: { ...routes.value.routes, [path]: () => `mounted ${path}` } }); + return null; + }, + }; + }); +}); + +const host = createHost({ + onError: (error) => { + if (error.kind === "registry_conflict") console.log("conflict:", error.message, error.plugins); + else console.log("onError:", error.kind); + }, +}); + +// 5. A reader before `load` waits: the provider comes with `load`, which composes every registry before any object. +const early = host.registry(Http, context); +await host.load( + [ + { name: "@example/http", facet: httpPlugin }, + { name: "@example/notes", facet: notesPlugin }, + { name: "@example/search", facet: searchPlugin }, + { name: "@example/mounts", facet: mountsPlugin }, + ], + context, +); +const router = await early; +console.log( + "routes:", + [...router.value].map(([path, { plugin }]) => `${path} (${plugin})`), +); + +// 6. Requests: a duplicate went to the first plugin by name; a throwing contribution is guarded at the use site. +const server = await host.get(Server, {}, context); +console.log("/notes ->", await server.request({ path: "/notes" }, context)); +console.log("/broken ->", await server.request({ path: "/broken" }, context)); + +// 7. A watch of the composed value gets each new router; mounting recomposes it. +const watch = await router.watch(context); +watch.start(async (call) => { + if (call.kind === "value") console.log("router now has:", [...call.value.keys()]); +}); +const mounts = await host.get(Mounts, {}, context); +await mounts.mount({ path: "/files" }, context); +await new Promise((resolve) => setTimeout(resolve, 0)); +console.log("/files ->", await server.request({ path: "/files" }, context)); + +await watch.stop(); +await host.close(context); diff --git a/packages/chord/test/next/examples/30-host-root.ts b/packages/chord/test/next/examples/30-host-root.ts new file mode 100644 index 000000000..885ab4414 --- /dev/null +++ b/packages/chord/test/next/examples/30-host-root.ts @@ -0,0 +1,85 @@ +// `host.root(fn, context)`: a call scope for code outside objects (a Durable tool, a CLI command). `fn` gets a ctx like a +// method handler's: the host's principal (or one a trusted caller names), this host's incarnation, the context's +// telemetry and signal, and `ctx.get` for references. When `fn` settles, or the context aborts, or the host closes, the +// scope is sealed: its signal aborts, its references are released, and a late `ctx.get` is refused `released`; so code +// that ignores its signal fails at its next use of Chord. (p3 wraps it as `chord(fn, context)` for Durable's tools.) +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/30-host-root.ts +import { BACKGROUND_CONTEXT, withCancel } from "../../../src/context/index.ts"; +import { arr, local, method, obj, str } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { type Ctx, createHost, defineFacet, sleep } from "../../../src/next/host/index.ts"; + +const context = BACKGROUND_CONTEXT; + +type Principal = { readonly kind: "server" } | { readonly kind: "owner"; readonly name: string }; + +const Notes = local("example.notes", { + version: 1, + params: { session: str() }, + members: { list: arr(str()), add: method(obj({ text: str() })) }, +}); + +const List = defineState("example.list", { + schema: obj({ list: arr(str()) }), + version: 1, + initial: () => ({ list: [] }), +}); + +const facet = defineFacet((host) => { + host.serve(Notes, (_, self) => { + const notes = self.state(List); + return { + list: self.derive([notes], ([n]) => n.list), + add: async ({ text }, ctx) => { + await notes.change((draft) => { + draft.list.push(`${text} (by ${ctx.principal?.kind})`); + }, ctx); + return null; + }, + }; + }); +}); + +const host = createHost({ principal: { kind: "server" }, onError: (error) => console.log(error.kind) }); + +// 1. A scope made before `load` resolved: its `ctx.get` parks until the host is loaded. +const early = host.root(async (ctx) => (await ctx.get(Notes, { session: "s1" })).list.value, context); +await host.load([{ name: "@example/notes", facet }], context); +console.log("before load, read after it:", await early); + +// 2. A tool's call: get, call, return plain data. The reference is released when fn settles. +const addNote = (text: string) => + host.root(async (ctx) => { + const notes = await ctx.get(Notes, { session: "s1" }); + await notes.add({ text }, ctx); + return notes.list.value; + }, context); +console.log("after a tool call:", await addNote("first")); + +// 3. Base code acting for an authenticated user names the principal. +const owner: Principal = { kind: "owner", name: "ada" }; +console.log("as the owner:", await host.root(async (ctx) => ctx.principal, context, { principal: owner })); + +// 4. A kept ctx is sealed once fn settled. +let kept: Ctx | undefined; +await host.root(async (ctx) => { + kept = ctx; +}, context); +console.log("kept ctx aborted:", kept!.abortSignal?.aborted); +await kept!.get(Notes, { session: "s1" }).catch((error: Error) => console.log("kept ctx.get:", error.message)); + +// 5. Cancelling the context rejects at once and seals the scope; an fn that ignores its signal fails at its next +// use of Chord. +const { context: cancellable, cancel } = withCancel(context); +const stubborn = host.root(async (ctx) => { + await new Promise((resolve) => setTimeout(resolve, 20)); + await ctx.get(Notes, { session: "s1" }); +}, cancellable); +cancel(new Error("user pressed stop")); +await stubborn.catch((error: Error) => console.log("cancelled:", error.message)); + +// 6. Closing the host seals the scopes still open and rejects their callers at once: an fn sleeping on its ctx ends. +const sleeping = host.root((ctx) => sleep(60_000, ctx), context).catch((error: Error) => `closed: ${error.message}`); +await host.close(context); +console.log(await sleeping); diff --git a/packages/chord/test/next/host/attempt.test.ts b/packages/chord/test/next/host/attempt.test.ts index 867502320..4066748e7 100644 --- a/packages/chord/test/next/host/attempt.test.ts +++ b/packages/chord/test/next/host/attempt.test.ts @@ -1,5 +1,5 @@ import { createHook } from "node:async_hooks"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { afterEach, beforeEach, describe, expect, expectTypeOf, it, vi } from "vitest"; import { type Context, BACKGROUND_CONTEXT as context, withCancel } from "../../../src/context/index.ts"; import { arr, int, local, method, obj, opt, remote, str } from "../../../src/next/contracts/index.ts"; import { broken, isChordError } from "../../../src/next/data/index.ts"; @@ -12,6 +12,7 @@ import { type Self, type StoreSelf, } from "../../../src/next/host/index.ts"; +import { defineRegistry } from "../../../src/next/registries/index.ts"; import { constant } from "../../../src/next/state/index.ts"; import type { StagedWrite } from "../../../src/next/store/step.ts"; import type { ChordStore } from "../../../src/next/store/store.ts"; @@ -425,6 +426,36 @@ describe("declarations", () => { () => {}, ), ).toThrow("self.open is valid only in the factory"); + const Names = defineRegistry("app.names", { version: 1 }); + expect(() => later!.registry(Names)).toThrow("self.registry is valid only in the factory"); + expect(() => later!.writable(1)).toThrow("self.writable is valid only in the factory"); + expect(() => later!.retryOn()).toThrow("self.retryOn is valid only in the factory"); + }); + + it("contribute only in host-lifetime objects (R21): `Self` has no `contribute`, a declaration error at runtime", async () => { + expectTypeOf().not.toHaveProperty("contribute"); + expectTypeOf().toHaveProperty("contribute"); + const Names = defineRegistry("app.names", { version: 1 }); + const Store = local("app.store", { version: 1, params: {}, members: {} }); + const { host } = await loaded((h) => { + h.serve(Names, { compose: (table) => table.length }); + h.serve(Page, (_, self) => { + (self as unknown as StoreSelf).contribute(Names, "page"); + return { label: constant(""), slow: async () => null }; + }); + h.serve(Store, (_, self) => { + self.contribute(Names, "a"); + self.contribute(Names, "b"); + return {}; + }); + }); + await expect(host.get(Page, { id: 1 }, context)).rejects.toThrow( + "app.page(1): self.contribute is valid only in a local type with empty params", + ); + await expect(host.get(Store, {}, context)).rejects.toThrow( + "app.store: registry app.names is contributed to twice", + ); + await host.close(context); }); it("refuse a document definition declared twice", async () => { diff --git a/packages/chord/test/next/host/ctx.test.ts b/packages/chord/test/next/host/ctx.test.ts index 732ed2d3a..df259c227 100644 --- a/packages/chord/test/next/host/ctx.test.ts +++ b/packages/chord/test/next/host/ctx.test.ts @@ -61,7 +61,7 @@ describe("ctx", () => { expect(await rejection(seen.ctx!.get(Item, { id: 1 }))).toEqual({ code: "released", reason: "settled", - message: "app.item: ctx.get after its handler settled", + message: "app.item: ctx.get after its call settled", }); }); @@ -103,7 +103,7 @@ describe("ctx", () => { expect(await seen.late).toMatchObject({ code: "released", reason: "settled", - message: "app.item: ctx.get after its handler settled", + message: "app.item: ctx.get after its call settled", }); await flushed(); // The late reference's object ended with it. @@ -130,7 +130,7 @@ describe("ctx", () => { expect(await pending).toMatchObject({ code: "released", reason: "settled", - message: "app.far: ctx.get after its handler settled", + message: "app.far: ctx.get after its call settled", }); }); diff --git a/packages/chord/test/next/host/facet.test.ts b/packages/chord/test/next/host/facet.test.ts index 54fe1113d..0dd05e818 100644 --- a/packages/chord/test/next/host/facet.test.ts +++ b/packages/chord/test/next/host/facet.test.ts @@ -1,6 +1,7 @@ import { describe, expect, expectTypeOf, it } from "vitest"; import { int, local } from "../../../src/next/contracts/index.ts"; -import { defineFacet, type Facet, type FacetHost, type Host } from "../../../src/next/host/index.ts"; +import { defineFacet, type Facet, type FacetHost, type FacetSelf, type Host } from "../../../src/next/host/index.ts"; +import { defineRegistry } from "../../../src/next/registries/index.ts"; import { constant } from "../../../src/next/state/index.ts"; import { loaded } from "../support.ts"; @@ -17,13 +18,34 @@ describe("facets", () => { ); }); - it("only declare: a setup takes the host alone, which has no documents, routines or natives", () => { - expectTypeOf["setup"]>>().toEqualTypeOf<[FacetHost<{ kind: "owner" }>]>(); + it("only declare: a setup takes the host and a self that only contributes; no documents, routines or natives", () => { + expectTypeOf["setup"]>>().toEqualTypeOf< + [FacetHost<{ kind: "owner" }>, FacetSelf] + >(); + expectTypeOf().toEqualTypeOf<"contribute">(); expectTypeOf().not.toHaveProperty("state"); expectTypeOf().not.toHaveProperty("on"); expectTypeOf().not.toHaveProperty("onLive"); }); + it("contribute at most one value per registry, only while their setup runs", async () => { + const Names = defineRegistry("app.names", { version: 1 }); + await expect( + loaded((_, self) => { + self.contribute(Names, "a"); + self.contribute(Names, "b"); + }), + ).rejects.toThrow("@test/app: registry app.names is contributed to twice"); + let later: FacetSelf | undefined; + await loaded((host, self) => { + host.serve(Names, { compose: (table) => table.length }); + later = self; + }); + expect(() => later!.contribute(Names, "late")).toThrow( + "@test/app: self.contribute is valid only in the facet's setup", + ); + }); + it("are typed by exactly the app's principal type", () => { type App = { kind: "owner" } | { kind: "guest"; session: string }; const facet = defineFacet(() => {}); diff --git a/packages/chord/test/next/host/members.test.ts b/packages/chord/test/next/host/members.test.ts index 2efc060e0..9c5e40793 100644 --- a/packages/chord/test/next/host/members.test.ts +++ b/packages/chord/test/next/host/members.test.ts @@ -22,10 +22,10 @@ describe("the member check (Q14)", () => { }), ); await expect(host.get(Named, { n: 1 }, context)).rejects.toThrow( - "app.named(1): label: source 1 is not a JSON watchable", + "app.named(1): label: source 1 is not a Chord watchable", ); await expect(host.get(Named, { n: 2 }, context)).rejects.toThrow( - "app.named(2): derive #1: source 0 is not a JSON watchable", + "app.named(2): derive #1: source 0 is not a Chord watchable", ); }); }); diff --git a/packages/chord/test/next/host/retry-on.test.ts b/packages/chord/test/next/host/retry-on.test.ts new file mode 100644 index 000000000..5677b6c28 --- /dev/null +++ b/packages/chord/test/next/host/retry-on.test.ts @@ -0,0 +1,204 @@ +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { int, local, method } from "../../../src/next/contracts/index.ts"; +import { broken, ChordError } from "../../../src/next/data/index.ts"; +import type { FacetHost, Self, StoreSelf } from "../../../src/next/host/index.ts"; +import { ChordObject } from "../../../src/next/host/object.ts"; +import { defineRegistry } from "../../../src/next/registries/index.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import { expectCollected, failLive, flushed, instancesOf, loaded } from "../support.ts"; + +const objects = instancesOf(ChordObject, "attempt"); + +/** The models registry: the number of models contributed; it throws when told to. */ +const Models = defineRegistry("app.models", { version: 1 }); + +const Runtime = local("app.runtime", { version: 1, params: {}, members: { set: method(int()) } }); + +/** Serves `Models` (throwing for a negative sum) and `Runtime`, contributing a writable count of models. */ +function serveModels(h: FacetHost): void { + h.serve(Models, { + compose: (table) => { + const sum = table.reduce((n, { value }) => n + value, 0); + if (sum < 0) throw new Error("negative"); + return sum; + }, + }); + h.serve(Runtime, (_, self) => { + const count = self.writable(0); + self.contribute(Models, count); + return { + set: async (n: number) => { + count.set(n); + return null; + }, + }; + }); +} + +const User = local("app.user", { version: 1, params: { id: int() }, members: {} }); + +/** + * Serves `User`, whose open reads the models first and fails broken when there were none (after `wait`, if given); + * `opens` counts its attempts. + */ +function serveUser(h: FacetHost, opens: { n: number }, wait?: () => Promise): void { + h.serve(User, (_, self) => { + const models = self.registry(Models); + self.retryOn(models); + self.open( + async () => { + const seen = models.value; + opens.n++; + await wait?.(); + if (seen === 0) throw broken(new ChordError({ code: "unavailable", message: "no model" })); + }, + () => {}, + ); + return {}; + }); +} + +describe("retryOn", () => { + it("retries a failed object (a broken open) when a source gets a new value: the watches were installed before the open", async () => { + const opens = { n: 0 }; + const { host } = await loaded((h) => { + serveModels(h); + serveUser(h, opens); + }); + const user = host.hold(User, { id: 1 }, context); + await flushed(); + expect(user.state.value).toMatchObject({ kind: "failed", cause: { kind: "broken" } }); + await (await host.get(Runtime, {}, context)).set(1, context); + await flushed(); + expect(user.state.value).toEqual({ kind: "live" }); + expect(opens.n).toBe(2); + await host.close(context); + }); + + it("wakes on value calls only: a throwing compose wakes nothing, its next good composition does", async () => { + const opens = { n: 0 }; + const { host } = await loaded((h) => { + serveModels(h); + serveUser(h, opens); + }); + const user = host.hold(User, { id: 1 }, context); + await flushed(); + const runtime = await host.get(Runtime, {}, context); + await runtime.set(-1, context); + await flushed(); + expect(opens.n).toBe(1); + await runtime.set(2, context); + await flushed(); + expect(opens.n).toBe(2); + expect(user.state.value).toEqual({ kind: "live" }); + await host.close(context); + }); + + it("remembers a wake while it activates: that attempt failing is retried at once", async () => { + const opens = { n: 0 }; + const gate = Promise.withResolvers(); + const { host } = await loaded((h) => { + serveModels(h); + serveUser(h, opens, async () => { + if (opens.n === 1) await gate.promise; + }); + }); + const runtime = await host.get(Runtime, {}, context); + const user = host.hold(User, { id: 1 }, context); + await flushed(); + // The first attempt read no model and waits; a model arrives meanwhile; then that attempt fails broken. + await runtime.set(1, context); + expect(user.state.value).toEqual({ kind: "loading" }); + gate.resolve(); + await flushed(); + expect(opens.n).toBe(2); + expect(user.state.value).toEqual({ kind: "live" }); + await host.close(context); + }); + + it("ignores wakes while live", async () => { + const opens = { n: 0 }; + const { host } = await loaded((h) => { + serveModels(h); + h.serve(User, (_, self) => { + self.retryOn(self.registry(Models)); + self.open( + async () => void opens.n++, + () => {}, + ); + return {}; + }); + }); + const user = host.hold(User, { id: 1 }, context); + await flushed(); + await (await host.get(Runtime, {}, context)).set(3, context); + await flushed(); + expect(opens.n).toBe(1); + expect(user.state.value).toEqual({ kind: "live" }); + await host.close(context); + }); + + it("refuses sources that end with an attempt: a reference's member, its own writable, `ref.state`", async () => { + const Other = local("app.other", { version: 1, params: {}, members: { n: int() } }); + const Picky = local("app.picky", { version: 1, params: { which: int() }, members: {} }); + const { host } = await loaded((h) => { + h.serve(Other, () => ({ n: constant(1) })); + h.serve(Picky, ({ which }, self) => { + const other = self.get(Other, {}); + const sources = [other.n, self.writable(1), other.state, constant(1)]; + self.retryOn(sources[which]!); + return {}; + }); + }); + for (const which of [0, 1, 2]) { + await expect(host.get(Picky, { which }, context)).rejects.toThrow( + `app.picky(${which}): self.retryOn takes registry values and constants (sources that outlive the attempt)`, + ); + } + (await host.get(Picky, { which: 3 }, context)).release(); + await host.close(context); + }); + + it("are the object's: a failed object that ends releases them, and is collected while the registry lives", async () => { + const opens = { n: 0 }; + const { host } = await loaded((h) => { + serveModels(h); + serveUser(h, opens); + }); + const refs = await (async () => { + const user = host.hold(User, { id: 1 }, context); + await flushed(); + user.release(); + await flushed(); + return objects.filter((ref) => ref.deref()?.label === "app.user(1)"); + })(); + expect(refs.length).toBeGreaterThan(0); + await expectCollected(...refs); + await host.close(context); + }); + + it("does not wake an object by its own contributions leaving (it failed live: no retry without backoff)", async () => { + const Contributor = local("app.contributor", { version: 1, params: {}, members: {} }); + let contributor: StoreSelf | undefined; + const { host } = await loaded((h) => { + serveModels(h); + h.serve(Contributor, (_, self) => { + contributor = self; + self.contribute(Models, 1); + self.retryOn(self.registry(Models)); + return {}; + }); + }); + const models = await host.registry(Models, context); + const watched: number[] = []; + (await models.watch(context)).start(async (call) => void (call.kind === "value" && watched.push(call.value))); + failLive(objects, contributor as Self, new ChordError({ code: "unavailable", message: "boom" })); + await flushed(); + const reference = host.hold(Contributor, {}, context); + await flushed(); + expect(reference.state.value).toMatchObject({ kind: "failed", cause: { kind: "broken" } }); + expect(watched).toEqual([0]); + await host.close(context); + }); +}); diff --git a/packages/chord/test/next/host/root.test.ts b/packages/chord/test/next/host/root.test.ts new file mode 100644 index 000000000..76c1c2b7e --- /dev/null +++ b/packages/chord/test/next/host/root.test.ts @@ -0,0 +1,175 @@ +import { getEventListeners } from "node:events"; +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context, withAbortSignal, withCancel } from "../../../src/context/index.ts"; +import { int, local, method } from "../../../src/next/contracts/index.ts"; +import { type Ctx, createHost, defineFacet, sleep } from "../../../src/next/host/index.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import { expectCollected, flushed, loaded, rejection, type TestPrincipal } from "../support.ts"; + +const Info = local("app.info", { version: 1, params: { id: int() }, members: { id: int(), wait: method() } }); + +/** Serves `Info`; its `wait` waits until the caller's context aborts. */ +function serveInfo(h: Parameters[0]>[0]): void { + h.serve(Info, ({ id }) => ({ + id: constant(id), + wait: async (ctx) => { + await sleep(60_000, ctx); + return null; + }, + })); +} + +describe("host.root", () => { + it("gives fn a call context: the host's principal (or the given one), this host's incarnation, `ctx.get`", async () => { + const host = createHost({ principal: { kind: "server" }, onError: () => {} }); + await host.load([{ name: "@app/info", facet: defineFacet(serveInfo) }], context); + const seen = await host.root(async (ctx) => { + const info = await ctx.get(Info, { id: 7 }); + return { principal: ctx.principal, incarnation: ctx.incarnation, id: info.id.value }; + }, context); + expect(seen).toMatchObject({ principal: { kind: "server" }, id: 7 }); + const owner = { kind: "owner", access: "local" } as const; + expect(await host.root(async (ctx) => ctx.principal, context, { principal: owner })).toEqual(owner); + await host.close(context); + }); + + it("seals the context when fn settles: its references released, its signal aborted, `ctx.get` refused `released`", async () => { + const { host } = await loaded(serveInfo); + let kept: Ctx | undefined; + const info = await host.root(async (ctx) => { + kept = ctx; + return ctx.get(Info, { id: 1 }); + }, context); + expect(info.state.value).toMatchObject({ kind: "ended", error: { code: "released" } }); + expect(kept!.abortSignal?.aborted).toBe(true); + expect(await rejection(kept!.get(Info, { id: 1 }))).toMatchObject({ code: "released", reason: "settled" }); + // A throwing fn seals too, and rejects with its error. + await expect( + host.root(async () => { + throw new Error("fn failed"); + }, context), + ).rejects.toThrow("fn failed"); + await host.close(context); + }); + + it("an abort rejects at once and seals: a running call through its reference rejects, a later get is `released`", async () => { + const { host } = await loaded(serveInfo); + const { context: cancellable, cancel } = withCancel(context); + let kept: Ctx | undefined; + let call: Promise | undefined; + const running = host.root(async (ctx) => { + kept = ctx; + const info = await ctx.get(Info, { id: 1 }); + call = info.wait(context); + // Uncooperative: it waits on nothing of Chord's. + await new Promise(() => {}); + }, cancellable); + await flushed(); + cancel(new Error("stop")); + expect(await rejection(running)).toEqual(new Error("stop")); + expect(await rejection(call!)).toMatchObject({ code: "released" }); + expect(await rejection(kept!.get(Info, { id: 2 }))).toMatchObject({ code: "released", reason: "settled" }); + await host.close(context); + }); + + it("an aborted context rejects before fn runs; a synchronous throw is a rejection", async () => { + const { host } = await loaded(serveInfo); + const { context: aborted, cancel } = withCancel(context); + cancel(new Error("gone")); + let ran = false; + const fn = async (): Promise => { + ran = true; + }; + await expect(host.root(fn, aborted)).rejects.toThrow("gone"); + expect(ran).toBe(false); + await expect( + host.root( + (() => { + throw new Error("sync"); + }) as never, + context, + ), + ).rejects.toThrow("sync"); + await host.close(context); + }); + + it("a `ctx.get` before `load` resolved parks until it did", async () => { + const host = createHost({ onError: () => {} }); + const result = host.root(async (ctx) => (await ctx.get(Info, { id: 3 })).id.value, context); + await flushed(); + await host.load([{ name: "@app/info", facet: defineFacet(serveInfo) }], context); + expect(await result).toBe(3); + await host.close(context); + }); + + it("`host.close` seals every open scope (an fn sleeping on its ctx ends); after it, `root` rejects `released`", async () => { + const { host } = await loaded(serveInfo); + const sleeping = rejection(host.root((ctx) => sleep(60_000, ctx), context)); + await flushed(); + await host.close(context); + expect(await sleeping).toMatchObject({ code: "released", reason: "closed" }); + expect(await rejection(host.root(async () => 1, context))).toMatchObject({ code: "released", reason: "closed" }); + }); + + it("aborts the scope's own signal when fn settles and at `host.close`, the caller's context having one (Fable, Opus, Sol)", async () => { + const { host } = await loaded(serveInfo); + const settled = withCancel(context); + let kept: Ctx | undefined; + await host.root(async (ctx) => { + kept = ctx; + }, settled.context); + expect(kept!.abortSignal?.aborted).toBe(true); + expect(settled.context.abortSignal?.aborted).toBe(false); + const closing = withCancel(context); + let slept: "running" | "ended" = "running"; + const sleeping = host.root(async (ctx) => { + try { + await sleep(60_000, ctx); + } finally { + slept = "ended"; + } + }, closing.context); + const outcome = rejection(sleeping); + await flushed(); + await host.close(context); + await flushed(); + expect(slept).toBe("ended"); + expect(closing.context.abortSignal?.aborted).toBe(false); + expect(await outcome).toMatchObject({ code: "released", reason: "closed" }); + }); + + it("`host.close` rejects the caller at once, whatever an uncooperative fn still does (Sol)", async () => { + const { host } = await loaded(serveInfo); + const running = host.root(() => new Promise(() => {}), context); + const outcome = running.then( + () => "resolved", + (error: unknown) => error, + ); + await host.close(context); + expect(await outcome).toMatchObject({ code: "released", reason: "closed" }); + }); + + it("leaves no listener on the caller's signal once `host.close` rejected an fn that never settles (Astra)", async () => { + const controller = new AbortController(); + const host = createHost({ onError: () => {} }); + const running = host.root(async () => new Promise(() => {}), withAbortSignal(controller.signal, context)); + const rejected = running.catch((error: unknown) => error); + await host.close(context); + expect(await rejected).toMatchObject({ code: "released", reason: "closed" }); + expect(getEventListeners(controller.signal, "abort")).toHaveLength(0); + }); + + it("keeps nothing once sealed: its scope and references are collected while the caller's signal lives on", async () => { + const { host } = await loaded(serveInfo); + const { context: long } = withCancel(context); + const refs = await (async () => { + const made: WeakRef[] = []; + await host.root(async (ctx) => { + made.push(new WeakRef(ctx), new WeakRef(await ctx.get(Info, { id: 1 }))); + }, long); + return made; + })(); + await expectCollected(...refs); + await host.close(context); + }); +}); diff --git a/packages/chord/test/next/registries/entries.test.ts b/packages/chord/test/next/registries/entries.test.ts new file mode 100644 index 000000000..88bd61003 --- /dev/null +++ b/packages/chord/test/next/registries/entries.test.ts @@ -0,0 +1,49 @@ +import { describe, expect, it } from "vitest"; +import { + keyedEntries, + orderedEntries, + type RegistryConflict, + type RegistryTable, +} from "../../../src/next/registries/index.ts"; + +const table: RegistryTable<{ readonly routes?: Readonly>; readonly items?: readonly string[] }> = + [ + { plugin: "@a/one", version: 1, value: { routes: { "/x": "one", "/y": "one" }, items: ["late", "first"] } }, + { plugin: "@b/two", type: "b.store", version: 1, value: { routes: { "/x": "two" }, items: ["second"] } }, + { plugin: "@c/three", version: 1, value: {} }, + ]; + +describe("keyedEntries", () => { + it("maps every entry's keys with their plugin; a duplicate is reported and the first by plugin name keeps it", () => { + const conflicts: RegistryConflict[] = []; + const map = keyedEntries( + table, + (value) => value.routes, + (conflict) => conflicts.push(conflict), + ); + expect([...map]).toEqual([ + ["/x", { value: "one", plugin: "@a/one" }], + ["/y", { value: "one", plugin: "@a/one" }], + ]); + expect(conflicts).toEqual([{ message: '"/x" is contributed twice', plugins: ["@a/one", "@b/two"] }]); + }); +}); + +describe("orderedEntries", () => { + it("lists every entry's items in table order, with their plugin and type", () => { + expect(orderedEntries(table, (value) => value.items)).toEqual([ + { value: "late", plugin: "@a/one" }, + { value: "first", plugin: "@a/one" }, + { value: "second", plugin: "@b/two", type: "b.store" }, + ]); + }); + + it("sorts by `orderOf`, table order kept among equals", () => { + const order = (item: string) => (item === "first" ? 0 : 1); + expect(orderedEntries(table, (value) => value.items, order).map(({ value }) => value)).toEqual([ + "first", + "late", + "second", + ]); + }); +}); diff --git a/packages/chord/test/next/registries/record.test.ts b/packages/chord/test/next/registries/record.test.ts new file mode 100644 index 000000000..c25b27fcf --- /dev/null +++ b/packages/chord/test/next/registries/record.test.ts @@ -0,0 +1,649 @@ +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { bool, int, local, method, obj } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import type { HostError } from "../../../src/next/host/index.ts"; +import { ChordObject } from "../../../src/next/host/object.ts"; +import { + defineRegistry, + keyedEntries, + type Registry, + type RegistryTable, + type TableItem, +} from "../../../src/next/registries/index.ts"; +import type { NativeWatchCall } from "../../../src/watch.ts"; +import { + delivered, + expectCollected, + failures, + flushed, + instancesOf, + loadedPlugins, + metered, + storeRestartable, +} from "../support.ts"; + +const objects = instancesOf(ChordObject, "attempt"); + +/** A registry whose composition is its table as `plugin:type=value` strings. */ +const Names = defineRegistry("app.names", { version: 1 }); + +const listed = (table: RegistryTable): readonly string[] => + table.map(({ plugin, type, value }) => `${plugin}:${type ?? ""}=${value}`); + +/** A host-lifetime object contributing a writable it sets with `set`. */ +const Setter = local("app.setter", { version: 1, params: {}, members: { set: method(int()) } }); + +/** Serves `Setter` contributing a writable of `R`'s contribution type, `initial` first, `set(n)` makes `of(n)`. */ +function setter(R: Registry, initial: C, of: (n: number) => C) { + return (host: Parameters[0][string]>[0]) => + host.serve(Setter, (_, self) => { + const value = self.writable(initial); + self.contribute(R, value); + return { + set: async (n: number) => { + value.set(of(n)); + return null; + }, + }; + }); +} + +function kinds(errors: readonly HostError[]): string[] { + return errors.map((error) => error.kind); +} + +describe("the table", () => { + it("is sorted by plugin name, then type, a facet first: never load order", async () => { + const { host } = await loadedPlugins({ + "@z/late": (_, self) => self.contribute(Names, "z"), + "@a/early": (host, self) => { + self.contribute(Names, "a"); + setter(Names, "store", String)(host); + }, + "@m/provider": (host) => host.serve(Names, { compose: listed }), + }); + const value = await host.registry(Names, context); + expect(value.value).toEqual(["@a/early:=a", "@a/early:app.setter=store", "@z/late:=z"]); + await host.close(context); + }); + + it("adapts an older contribution once per value; a newer one, or an older one without an adapter, is left out and reported once", async () => { + const V3 = defineRegistry("app.versions", { version: 3 }); + const V2 = defineRegistry<{ old: string }, unknown>("app.versions", { version: 2 }); + const V1 = defineRegistry("app.versions", { version: 1 }); + const V4 = defineRegistry("app.versions", { version: 4 }); + const adapted: string[] = []; + const { host, errors } = await loadedPlugins({ + "@app/provider": (host) => + host.serve(V3, { + compose: listed, + accepts: { + 2: { + contribution: (old: { old: string }) => { + adapted.push(old.old); + return `adapted ${old.old}`; + }, + }, + }, + }), + "@app/one": (_, self) => self.contribute(V1, "one"), + "@app/two": setter(V2, { old: "first" }, (n) => ({ old: `${n}` })), + "@app/three": (_, self) => self.contribute(V3, "three"), + "@app/four": (_, self) => self.contribute(V4, "four"), + }); + const value = await host.registry(V3, context); + expect(value.value).toEqual(["@app/three:=three", "@app/two:app.setter=adapted first"]); + expect(errors).toEqual([ + { kind: "contribution_refused", registry: "app.versions", plugin: "@app/four", version: 4 }, + { kind: "contribution_refused", registry: "app.versions", plugin: "@app/one", version: 1 }, + ]); + const setters = await host.get(Setter, {}, context); + await setters.set(5, context); + expect(value.value).toEqual(["@app/three:=three", "@app/two:app.setter=adapted 5"]); + // Each value adapted once, though the table was built again; no refusal reported twice. + expect(adapted).toEqual(["first", "5"]); + expect(kinds(errors)).toEqual(["contribution_refused", "contribution_refused"]); + await host.close(context); + }); + + it("takes a contribution declared before its provider's setup into the first composition", async () => { + let composed: readonly string[] | undefined; + const { host } = await loadedPlugins({ + "@app/a": (_, self) => self.contribute(Names, "first"), + "@app/provider": (host) => + host.serve(Names, { + compose: (table) => { + composed = listed(table); + return composed; + }, + }), + }); + expect(composed).toEqual(["@app/a:=first"]); + await host.close(context); + }); +}); + +describe("compose", () => { + it("composes every served registry once at the end of load's setups, an empty table too, before any object activates", async () => { + const { telemetry, counts } = await metered(); + const Other = defineRegistry("app.other", { version: 1 }); + const Reader = local("app.reader", { version: 1, params: {}, members: { seen: int() } }); + let seen: readonly string[] | undefined; + const { host } = await loadedPlugins( + { + "@app/provider": (host, self) => { + host.serve(Names, { compose: listed }); + host.serve(Other, { compose: (table) => table.length }); + self.contribute(Names, "a"); + host.serve(Reader, (_, self) => { + const names = self.registry(Names); + self.open( + async () => { + seen = names.value; + }, + () => {}, + ); + return { seen: self.derive([self.registry(Other)], ([n]) => n) }; + }); + }, + }, + { telemetry }, + ); + expect(seen).toEqual(["@app/provider:=a"]); + expect((await host.get(Reader, {}, context)).seen.value).toBe(0); + expect(counts()["chord.registry.composes"]).toBe(2); + await host.close(context); + }); + + it("is lazy: a change marks the table, a read composes it once; no compose for an unchanged table", async () => { + const { telemetry, counts } = await metered(); + const Flags = defineRegistry("app.flags", { version: 1 }); + const Doc = defineState("app.flag", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 1 }) }); + const Flagger = local("app.flagger", { version: 1, params: {}, members: { set: method(int()) } }); + const { host } = await loadedPlugins( + { + "@app/flags": (host) => { + host.serve(Flags, { compose: (table) => table.filter(({ value }) => value).length }); + host.serve(Flagger, (_, self) => { + const doc = self.state(Doc); + self.contribute( + Flags, + self.derive([doc], ([d]) => d.n > 0), + ); + return { + set: async (n: number, ctx) => { + await doc.change((d) => { + d.n = n; + }, ctx); + return null; + }, + }; + }); + }, + }, + { telemetry }, + ); + const flags = await host.registry(Flags, context); + const flagger = await host.get(Flagger, {}, context); + expect(flags.value).toBe(1); + const base = counts()["chord.registry.composes"]!; + expect(flags.value).toBe(1); + expect(counts()["chord.registry.composes"]).toBe(base); + // The derive changes, its value does not: the table stays, nothing composes. + await flagger.set(2, context); + await flagger.set(3, context); + expect(flags.value).toBe(1); + expect(counts()["chord.registry.composes"]).toBe(base); + // Nobody reads it: nothing composes until a read. + await flagger.set(-1, context); + await flagger.set(-2, context); + expect(counts()["chord.registry.composes"]).toBe(base); + expect(flags.value).toBe(0); + expect(counts()["chord.registry.composes"]).toBe(base + 1); + await host.close(context); + }); + + it("composes a watched registry in the batch of the commit that changed a contribution, once per batch", async () => { + const { telemetry, counts } = await metered(); + const Doc = defineState("app.names", { + schema: obj({ a: int(), b: int() }), + version: 1, + initial: () => ({ a: 0, b: 0 }), + }); + const Two = local("app.two", { version: 1, params: {}, members: { both: method(int()), sum: int() } }); + const Sum = defineRegistry<{ readonly a: number; readonly b: number }, number>("app.sum", { version: 1 }); + const { host } = await loadedPlugins( + { + "@app/sum": (host) => { + host.serve(Sum, { compose: (table) => table.reduce((n, { value }) => n + value.a + value.b, 0) }); + host.serve(Two, (_, self) => { + const doc = self.state(Doc); + self.contribute(Sum, doc); + return { + sum: self.derive([self.registry(Sum)], ([n]) => n), + both: async (n: number, ctx) => { + await self.tx(async (tx) => { + const d = await tx.draft(doc); + d.a = n; + d.b = n; + }, ctx); + return null; + }, + }; + }); + }, + }, + { telemetry }, + ); + const two = await host.get(Two, {}, context); + const sums: number[] = []; + const watch = await two.sum.watch(context); + watch.start(async (call) => { + if (call.kind === "value") sums.push(call.value); + }); + const base = counts()["chord.registry.composes"]!; + await two.both(2, context); + await delivered(); + // One commit, both fields: one compose in its batch, one frame of the derive over it. + expect(sums).toEqual([4]); + expect(counts()["chord.registry.composes"]).toBe(base + 1); + await host.close(context); + }); + + it("reports a conflict once per episode", async () => { + const Routes = defineRegistry>, ReadonlyMap>>( + "app.routes", + { version: 1 }, + ); + const { host, errors } = await loadedPlugins({ + "@app/a": (_, self) => self.contribute(Routes, { "/x": "a" }), + "@app/b": setter(Routes, { "/x": "b" } as Readonly>, (n) => ({ + "/x": "b", + [`/${n}`]: "b", + })), + "@app/http": (host) => + host.serve(Routes, { compose: (table, { report }) => keyedEntries(table, (c) => c, report) }), + }); + const routes = await host.registry(Routes, context); + expect(routes.value.get("/x")).toEqual({ value: "a", plugin: "@app/a" }); + const setters = await host.get(Setter, {}, context); + await setters.set(1, context); + void routes.value; + await setters.set(2, context); + void routes.value; + expect(errors).toEqual([ + { + kind: "registry_conflict", + registry: "app.routes", + message: '"/x" is contributed twice', + plugins: ["@app/a", "@app/b"], + }, + ]); + await host.close(context); + }); + + it("a throwing compose: reported once, a failed call, `.value` keeps the last good composition; the next good one heals", async () => { + const Count = defineRegistry("app.count", { version: 1 }); + const { host, errors } = await loadedPlugins({ + "@app/count": (host) => { + host.serve(Count, { + compose: (table) => { + const sum = table.reduce((n, { value }) => n + value, 0); + if (sum < 0) throw new Error(`negative: ${sum}`); + return sum; + }, + }); + setter(Count, 1, (n) => n)(host); + }, + }); + const count = await host.registry(Count, context); + const calls: NativeWatchCall[] = []; + const watch = await count.watch(context); + watch.start(async (call) => void calls.push(call)); + const setters = await host.get(Setter, {}, context); + await setters.set(-1, context); + await setters.set(-2, context); + await delivered(); + expect(count.value).toBe(1); + expect(errors).toEqual([{ kind: "registry_failed", registry: "app.count", error: new Error("negative: -1") }]); + await setters.set(1, context); + await delivered(); + // The heal is a value call, also when its value is the last good one. + expect(calls).toEqual([ + { kind: "failed", error: new Error("negative: -1") }, + { kind: "value", value: 1 }, + ]); + await setters.set(3, context); + await delivered(); + expect(calls.at(-1)).toEqual({ kind: "value", value: 3 }); + await host.close(context); + }); + + it("before any good composition, `.value` and `host.registry` throw the error", async () => { + const Broken = defineRegistry("app.broken", { version: 1 }); + const { host, errors } = await loadedPlugins({ + "@app/broken": (host) => + host.serve(Broken, { + compose: () => { + throw new Error("bug"); + }, + }), + }); + await expect(host.registry(Broken, context)).rejects.toThrow("bug"); + expect(kinds(errors)).toEqual(["registry_failed"]); + await host.close(context); + }); +}); + +describe("contributions", () => { + it("a watchable contribution that fails is left out and reported once, and is back once it heals", async () => { + const Doc = defineState("app.n", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 1 }) }); + const Failing = local("app.failing", { version: 1, params: {}, members: { set: method(int()) } }); + const { host, errors } = await loadedPlugins({ + "@app/names": (host, self) => { + host.serve(Names, { compose: listed }); + self.contribute(Names, "facet"); + host.serve(Failing, (_, self) => { + const doc = self.state(Doc); + self.contribute( + Names, + self.derive([doc], ([d]) => { + if (d.n < 0) throw new Error("negative"); + return `n${d.n}`; + }), + ); + return { + set: async (n: number, ctx) => { + await doc.change((d) => { + d.n = n; + }, ctx); + return null; + }, + }; + }); + }, + }); + const names = await host.registry(Names, context); + const failing = await host.get(Failing, {}, context); + expect(names.value).toEqual(["@app/names:=facet", "@app/names:app.failing=n1"]); + await failing.set(-1, context); + expect(names.value).toEqual(["@app/names:=facet"]); + await failing.set(-2, context); + expect(names.value).toEqual(["@app/names:=facet"]); + await failing.set(2, context); + expect(names.value).toEqual(["@app/names:=facet", "@app/names:app.failing=n2"]); + expect(errors.filter(({ kind }) => kind === "contribution_failed")).toEqual([ + { + kind: "contribution_failed", + registry: "app.names", + plugin: "@app/names", + type: "app.failing", + error: new Error("negative"), + }, + ]); + await host.close(context); + }); + + it("a contribution adapter that throws leaves that contribution out (`contribution_failed`)", async () => { + const V2 = defineRegistry("app.adapted", { version: 2 }); + const V1 = defineRegistry("app.adapted", { version: 1 }); + const { host, errors } = await loadedPlugins({ + "@app/provider": (host, self) => { + host.serve(V2, { + compose: listed, + accepts: { + 1: { + contribution: () => { + throw new Error("cannot adapt"); + }, + }, + }, + }); + self.contribute(V2, "current"); + }, + "@app/old": (_, self) => self.contribute(V1, "old"), + }); + expect((await host.registry(V2, context)).value).toEqual(["@app/provider:=current"]); + expect(errors).toEqual([ + { kind: "contribution_failed", registry: "app.adapted", plugin: "@app/old", error: new Error("cannot adapt") }, + ]); + await host.close(context); + }); + + it("refuses a contribution closing a cycle with the registries' tables; one through a derive is cut and reported", async () => { + const A = defineRegistry("app.a", { version: 1 }); + const B = defineRegistry("app.b", { version: 1 }); + const Direct = local("app.direct", { version: 1, params: {}, members: {} }); + const Late = local("app.late", { version: 1, params: {}, members: {} }); + const Derived = local("app.derived", { version: 1, params: {}, members: {} }); + const Back = local("app.back", { version: 1, params: {}, members: {} }); + let lateAttempts = 0; + const { host, errors } = await loadedPlugins({ + "@app/cycles": (host) => { + host.serve(A, { compose: (table) => table.length }); + host.serve(B, { compose: (table) => table.length }); + host.serve(Direct, (_, self) => { + self.contribute(A, self.registry(B)); + return {}; + }); + // Its first attempt is discarded (the factory throws), then, once `Direct` is live, it closes a cycle. + host.serve(Late, (_, self) => { + self.contribute(B, self.registry(A)); + if (++lateAttempts === 1) throw new Error("first attempt"); + return {}; + }); + host.serve(Derived, (_, self) => { + self.contribute( + A, + self.derive([self.registry(B)], ([n]) => n as number), + ); + return {}; + }); + }, + "@app/back": (host) => + host.serve(Back, (_, self) => { + self.contribute( + B, + self.derive([self.registry(A)], ([n]) => n as number), + ); + return {}; + }), + }); + const late = host.hold(Late, {}, context); + late.retry(); + await flushed(); + expect(late.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "broken", error: { code: "invalid", reason: "cycle" } }, + }); + // A reads B's derive, which reads A while A composes: that contribution is cut, A composes without the cycle. + expect((await host.registry(A, context)).value).toBe(2); + expect(errors.find((error) => error.kind === "contribution_failed")).toMatchObject({ + registry: "app.b", + plugin: "@app/back", + error: { code: "invalid", reason: "cycle" }, + }); + late.release(); + await host.close(context); + }); + + it("refuses a cycle declared within one factory: broken, invalid/cycle (Sol)", async () => { + const A = defineRegistry("app.fix-a", { version: 1 }); + const B = defineRegistry("app.fix-b", { version: 1 }); + const Both = local("app.direct-cycle", { version: 1, params: {}, members: {} }); + const { host, errors } = await loadedPlugins({ + "@app/cycle": (h) => { + h.serve(A, { compose: (table) => table.length }); + h.serve(B, { compose: (table) => table.length }); + h.serve(Both, (_, self) => { + self.contribute(A, self.registry(B)); + self.contribute(B, self.registry(A)); + return {}; + }); + }, + }); + expect( + errors.find((error) => error.kind === "activation_failed" && error.object === "app.direct-cycle"), + ).toMatchObject({ failure: { kind: "broken", error: { code: "invalid", reason: "cycle" } } }); + await host.close(context); + }); + + it("leaves no cycle edge from a contribution that never entered a table (Sol)", async () => { + const A = defineRegistry("app.edge-a", { version: 1 }); + const B = defineRegistry("app.edge-b", { version: 1 }); + const Failed = local("app.failed", { version: 1, params: {}, members: {} }); + const Good = local("app.good", { version: 1, params: {}, members: {} }); + const { host, errors } = await loadedPlugins({ + "@app/a": (h) => { + h.serve(A, { compose: (table) => table.length }); + h.serve(B, { compose: (table) => table.length }); + h.serve(Failed, (_, self) => { + self.contribute(A, self.registry(B)); + throw new Error("factory discarded"); + }); + h.serve(Good, (_, self) => { + self.contribute(B, self.registry(A)); + return {}; + }); + }, + }); + expect(failures(errors)).toEqual(["app.failed: broken"]); + await host.close(context); + }); + + it("follows only Chord watchables: anything else is a plain value, whatever methods it has (Sol)", async () => { + const Natives = defineRegistry<{ watch(): string; readonly value: string }, readonly string[]>("app.natives", { + version: 1, + }); + const native = { value: "a handle", watch: () => "a native operation" }; + const { host } = await loadedPlugins({ + "@app/x": (h, self) => { + h.serve(Natives, { compose: (table) => table.map(({ value }) => value.watch()) }); + self.contribute(Natives, native); + }, + }); + expect((await host.registry(Natives, context)).value).toEqual(["a native operation"]); + await host.close(context); + }); + + it("keeps an `undefined` contribution in the table (Sol, Astra)", async () => { + const Nothing = defineRegistry("app.undefined", { version: 1 }); + const { host } = await loadedPlugins({ + "@app/a": (h, self) => { + h.serve(Nothing, { compose: (table) => table.length }); + self.contribute(Nothing, undefined); + }, + }); + expect((await host.registry(Nothing, context)).value).toBe(1); + await host.close(context); + }); + + it("leaves out a registry's value contributed to another while that registry's compose throws (Sol, Astra)", async () => { + const Source = defineRegistry("app.source", { version: 1 }); + const Target = defineRegistry("app.target", { version: 1 }); + const Bridge = local("app.bridge", { version: 1, params: {}, members: {} }); + const { host, errors } = await loadedPlugins({ + "@app/a": (h) => { + h.serve(Source, { + compose: (table) => { + const n = table.reduce((sum, { value }) => sum + value, 0); + if (n < 0) throw new Error("negative"); + return n; + }, + }); + h.serve(Target, { compose: (table) => table.map(({ value }) => value) }); + setter(Source, 1, (n) => n)(h); + h.serve(Bridge, (_, self) => { + self.contribute(Target, self.registry(Source)); + return {}; + }); + }, + }); + const target = await host.registry(Target, context); + expect(target.value).toEqual([1]); + const setters = await host.get(Setter, {}, context); + await setters.set(-1, context); + // The source's readers keep its last good value; as a contribution it is failed: left out, reported. + expect((await host.registry(Source, context)).value).toBe(1); + expect(target.value).toEqual([]); + expect(errors.find((error) => error.kind === "contribution_failed")).toMatchObject({ + registry: "app.target", + plugin: "@app/a", + type: "app.bridge", + error: new Error("negative"), + }); + await setters.set(3, context); + expect(target.value).toEqual([3]); + await host.close(context); + }); + + it("a followed contribution whose runtime ended its observer stays with its last value (R4: the source keeps it)", async () => { + let end: (() => void) | undefined; + const Follower = local("app.follower", { version: 1, params: {}, members: {} }); + const { host, errors } = await loadedPlugins({ + "@app/a": (h) => { + h.serve(Names, { compose: listed }); + h.serve(Follower, (_, self) => { + self.contribute( + Names, + self.follow(async (observer) => { + end = () => observer.end({ kind: "ended" }); + return { value: "followed", detach: () => {} }; + }), + ); + return {}; + }); + }, + }); + const names = await host.registry(Names, context); + expect(names.value).toEqual(["@app/a:app.follower=followed"]); + end!(); + await flushed(); + expect(names.value).toEqual(["@app/a:app.follower=followed"]); + expect(errors.filter(({ kind }) => kind === "contribution_failed")).toEqual([]); + await host.close(context); + }); + + it("an object's contribution is in the table from live until it fails, again once retried, until it ends", async () => { + const { host, fail, retry } = await storeRestartable( + objects, + (attempt, self) => self.contribute(Names, `attempt ${attempt}`), + { serve: (host) => host.serve(Names, { compose: listed }) }, + ); + const names = await host.registry(Names, context); + expect(names.value).toEqual(["@test/app:app.store=attempt 1"]); + fail(); + expect(names.value).toEqual([]); + await retry(); + expect(names.value).toEqual(["@test/app:app.store=attempt 2"]); + await host.close(context); + // Closed: nothing composes any more; a value a reader keeps reads the last composition. + expect(names.value).toEqual(["@test/app:app.store=attempt 2"]); + }); +}); + +describe("the table's memory", () => { + it("keeps nothing of a closed host: records, tables and followed contributions go with it", async () => { + const Doc = defineState("app.kept", { schema: obj({ on: bool() }), version: 1, initial: () => ({ on: true }) }); + const Kept = local("app.kept", { version: 1, params: {}, members: {} }); + const refs = await (async () => { + const { host } = await loadedPlugins({ + "@app/kept": (host) => { + host.serve(Names, { compose: listed }); + host.serve(Kept, (_, self) => { + self.contribute( + Names, + self.derive([self.state(Doc)], ([d]) => `${d.on}`), + ); + return {}; + }); + }, + }); + const names = new WeakRef(await host.registry(Names, context)); + await host.close(context); + return [names, new WeakRef(host)]; + })(); + await expectCollected(...refs); + }); +}); diff --git a/packages/chord/test/next/registries/registry.test.ts b/packages/chord/test/next/registries/registry.test.ts new file mode 100644 index 000000000..61db2cd14 --- /dev/null +++ b/packages/chord/test/next/registries/registry.test.ts @@ -0,0 +1,67 @@ +import { describe, expect, expectTypeOf, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { defineRegistry, type Registry, type RegistryValue } from "../../../src/next/registries/index.ts"; +import { loadedPlugins } from "../support.ts"; + +describe("tokens", () => { + it("carry a name, a version and the types only; names and versions are checked at declaration", () => { + const Http = defineRegistry<{ readonly path: string }, readonly string[]>("chord.http", { version: 2 }); + expect(Http).toEqual({ kind: "registry", name: "chord.http", version: 2 }); + expectTypeOf(Http).toEqualTypeOf>(); + expect(() => defineRegistry("Chord.Http", { version: 1 })).toThrow('registry "Chord.Http": expected lowercase'); + expect(() => defineRegistry("chord.http", { version: 0 })).toThrow( + "chord.http.version: expected a positive integer", + ); + }); +}); + +describe("providers", () => { + const Names = defineRegistry("app.names", { version: 1 }); + + it("one per host: a second serve is refused at load", async () => { + const host = createHost({ onError: () => {} }); + const serve = defineFacet((h) => h.serve(Names, { compose: (table) => table.length })); + await expect( + host.load( + [ + { name: "@app/a", facet: serve }, + { name: "@app/b", facet: serve }, + ], + context, + ), + ).rejects.toThrow("registry app.names is served twice"); + }); + + it("a provider without compose is refused at load (plugin code is not type-checked)", async () => { + await expect(loadedPlugins({ "@app/a": (h) => h.serve(Names, {} as never) })).rejects.toThrow( + "registry app.names: compose is a function", + ); + }); + + it("a registry whose provider comes with `load` heals a `host.registry` made before it", async () => { + const host = createHost({ onError: () => {} }); + const waiting = host.registry(Names, context); + await host.load( + [{ name: "@app/a", facet: defineFacet((h) => h.serve(Names, { compose: (t) => t.length })) }], + context, + ); + const value = await waiting; + expectTypeOf(value).toEqualTypeOf>(); + expect(value.value).toBe(0); + await host.close(context); + }); + + it("are served only while the setup runs", async () => { + let later: Parameters[0]>[0] | undefined; + const { host } = await loadedPlugins({ + "@app/a": (h) => { + later = h; + }, + }); + expect(() => later!.serve(Names, { compose: (t) => t.length })).toThrow( + "@app/a: host.serve is valid only in the facet's setup", + ); + await host.close(context); + }); +}); diff --git a/packages/chord/test/next/registries/value.test.ts b/packages/chord/test/next/registries/value.test.ts new file mode 100644 index 000000000..e23722948 --- /dev/null +++ b/packages/chord/test/next/registries/value.test.ts @@ -0,0 +1,398 @@ +import { getEventListeners } from "node:events"; +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context, withAbortSignal, withCancel } from "../../../src/context/index.ts"; +import { int, local, method, str } from "../../../src/next/contracts/index.ts"; +import type { defineFacet } from "../../../src/next/host/index.ts"; +import { ChordObject } from "../../../src/next/host/object.ts"; +import { defineRegistry, type Registry } from "../../../src/next/registries/index.ts"; +import type { NativeWatchCall, WatchEnd } from "../../../src/watch.ts"; +import { + expectCollected, + failures, + flushed, + instancesOf, + loadedPlugins, + metered, + rejection, + settledHeap, +} from "../support.ts"; + +const objects = instancesOf(ChordObject, "attempt"); + +/** The provider's version 2: contributions are numbers, the value their sum. */ +const Sum = defineRegistry("app.sum", { version: 2 }); + +/** A reader built against version 1: its value was a plain number. */ +const SumV1 = defineRegistry("app.sum", { version: 1 }); + +/** A reader built against a version the provider does not know yet. */ +const SumV3 = defineRegistry("app.sum", { version: 3 }); + +const Setter = local("app.setter", { version: 1, params: {}, members: { set: method(int()) } }); + +/** Serves `Setter`, contributing a writable number to `Sum`. */ +function setter(host: Parameters[0]>[0], initial = 1): void { + host.serve(Setter, (_, self) => { + const value = self.writable(initial); + self.contribute(Sum, value); + return { + set: async (n: number) => { + value.set(n); + return null; + }, + }; + }); +} + +const total = (table: readonly { readonly value: number }[]) => ({ + total: table.reduce((n, { value }) => n + value, 0), +}); + +describe("older readers (S2)", () => { + it("get the composition mapped back with accepts[old].value, once per composition", async () => { + const { telemetry, counts } = await metered(); + const mapped: number[] = []; + const { host } = await loadedPlugins( + { + "@app/sum": (host) => { + host.serve(Sum, { + compose: total, + accepts: { + 1: { + value: (current) => { + mapped.push(current.total); + return current.total; + }, + }, + }, + }); + setter(host); + }, + }, + { telemetry }, + ); + const old = await host.registry(SumV1, context); + const current = await host.registry(Sum, context); + expect(old.value).toBe(1); + expect(old.value).toBe(1); + expect(current.value).toEqual({ total: 1 }); + await (await host.get(Setter, {}, context)).set(4, context); + expect(old.value).toBe(4); + expect(mapped).toEqual([1, 4]); + expect(counts()["chord.registry.adapts"]).toBe(2); + await host.close(context); + }); + + it("without a value mapping, and for a reader newer than the provider, fail `contract_mismatch`: broken, reported", async () => { + const Newer = local("app.newer", { version: 1, params: {}, members: { total: int() } }); + const Older = local("app.older", { version: 1, params: {}, members: { total: int() } }); + const { host, errors } = await loadedPlugins({ + "@app/sum": (host) => { + host.serve(Sum, { compose: total }); + host.serve(Newer, (_, self) => ({ total: self.derive([self.registry(SumV3)], ([v]) => v.total) })); + host.serve(Older, (_, self) => ({ total: self.derive([self.registry(SumV1)], ([v]) => v) })); + }, + }); + expect(await rejection(host.registry(SumV1, context))).toMatchObject({ code: "contract_mismatch" }); + expect(await rejection(host.registry(SumV3, context))).toMatchObject({ code: "contract_mismatch" }); + // Only a reload helps: no backoff, the mismatch itself the cause. + expect(failures(errors)).toEqual(["app.newer: broken", "app.older: broken"]); + expect(errors[0]).toMatchObject({ failure: { error: { code: "contract_mismatch" } } }); + await host.close(context); + }); + + it("maps an `undefined` composition (Sol, Astra)", async () => { + const Nothing = defineRegistry("app.nothing", { version: 2 }); + const NothingV1 = defineRegistry("app.nothing", { version: 1 }); + const { host } = await loadedPlugins({ + "@app/a": (h) => h.serve(Nothing, { compose: () => undefined, accepts: { 1: { value: () => "mapped" } } }), + }); + expect((await host.registry(NothingV1, context)).value).toBe("mapped"); + await host.close(context); + }); + + it("a failed mapping heals when the composition is the one mapped before (Astra)", async () => { + const Count = defineRegistry("app.count", { version: 2 }); + const CountV1 = defineRegistry("app.count", { version: 1 }); + const Counter = local("app.counter", { version: 1, params: {}, members: { set: method(int()) } }); + const { host } = await loadedPlugins({ + "@app/count": (h) => { + h.serve(Count, { + compose: (table) => table.reduce((n, { value }) => n + value, 0), + accepts: { + 1: { + value: (n) => { + if (n < 0) throw new Error("negative"); + return n; + }, + }, + }, + }); + h.serve(Counter, (_, self) => { + const value = self.writable(1); + self.contribute(Count, value); + return { + set: async (n: number) => { + value.set(n); + return null; + }, + }; + }); + }, + }); + const old = await host.registry(CountV1, context); + const calls: NativeWatchCall[] = []; + old.observe({ advance: (call) => calls.push(call), end: () => {} }, context); + const counter = await host.get(Counter, {}, context); + await counter.set(-1, context); + // Back to the composition mapped before (the same number): the mapping heals. + await counter.set(1, context); + expect(calls).toEqual([ + { kind: "failed", error: new Error("negative") }, + { kind: "value", value: 1 }, + ]); + await host.close(context); + }); +}); + +describe("readers (S3)", () => { + const Reader = local("app.reader", { version: 1, params: {}, members: { total: int() } }); + const serveReader = ( + host: Parameters[0]>[0], + R: Registry = Sum, + ) => host.serve(Reader, (_, self) => ({ total: self.derive([self.registry(R)], ([v]) => v.total) })); + + it("activate once the registry has a value; a reader of a registry nobody serves fails at once, `load` waits for nothing", async () => { + const started = Date.now(); + const { host, errors } = await loadedPlugins({ "@app/reader": (host) => serveReader(host) }); + expect(Date.now() - started).toBeLessThan(1_000); + expect(errors).toMatchObject([ + { + kind: "activation_failed", + object: "app.reader", + failure: { + kind: "transient", + error: { + code: "unavailable", + message: "app.reader: registry app.sum has no value", + cause: { code: "unavailable", reason: "undeclared" }, + }, + }, + }, + ]); + expect(await rejection(host.registry(Sum, context))).toMatchObject({ code: "unavailable", reason: "undeclared" }); + await host.close(context); + }); + + it("a reader failed on a first `compose` that threw is retried at once by the first good composition (R18)", async () => { + const { host, errors } = await loadedPlugins({ + "@app/sum": (host) => { + host.serve(Sum, { + compose: (table) => { + const sum = total(table); + if (sum.total === 0) throw new Error("empty"); + return sum; + }, + }); + serveReader(host); + }, + "@app/setter": (host) => setter(host, 0), + }); + expect(failures(errors)).toEqual(["app.reader: transient"]); + const setters = await host.get(Setter, {}, context); + await setters.set(3, context); + // The reader's backoff is a second at least: the composition retried it. + const reader = await host.get(Reader, {}, context, { readyTimeout: 500 }); + expect(reader.total.value).toBe(3); + await host.close(context); + }); + + it("once live, a later throwing `compose` changes nothing for the reader: `.value` keeps the last good one", async () => { + const { host } = await loadedPlugins({ + "@app/sum": (host) => { + host.serve(Sum, { + compose: (table) => { + const sum = total(table); + if (sum.total < 0) throw new Error("negative"); + return sum; + }, + }); + serveReader(host); + setter(host, 2); + }, + }); + const reader = await host.get(Reader, {}, context); + const setters = await host.get(Setter, {}, context); + expect(reader.total.value).toBe(2); + await setters.set(-1, context); + expect(reader.total.value).toBe(2); + expect(reader.state.value).toEqual({ kind: "live" }); + await host.close(context); + }); +}); + +describe("readers' memory", () => { + it("an ended reader is collected while the registry lives: its derive and its read leave the value", async () => { + const Reader = local("app.reader", { version: 1, params: { id: int() }, members: { total: int() } }); + const { host } = await loadedPlugins({ + "@app/sum": (host) => { + host.serve(Sum, { compose: total, accepts: { 1: { value: ({ total }) => total } } }); + host.serve(Reader, (_, self) => ({ + total: self.derive([self.registry(Sum), self.registry(SumV1)], ([v, old]) => v.total + old), + })); + setter(host); + }, + }); + const refs = await (async () => { + const reader = await host.get(Reader, { id: 1 }, context); + expect(reader.total.value).toBe(2); + reader.release(); + await new Promise((resolve) => setTimeout(resolve, 0)); + return objects.filter((ref) => ref.deref()?.label === "app.reader(1)"); + })(); + expect(refs.length).toBeGreaterThan(0); + await (await host.get(Setter, {}, context)).set(5, context); + await expectCollected(...refs); + await host.close(context); + }); +}); + +describe("values' memory", () => { + it("drops an older version's mapping once its last reader ended (Sol)", async () => { + const New = defineRegistry("app.version-gc", { version: 2 }); + const Old = defineRegistry("app.version-gc", { + version: 1, + }); + const Reader = local("app.old-reader", { version: 1, params: { id: int() }, members: { n: int() } }); + let mapped: WeakRef | undefined; + const { host } = await loadedPlugins({ + "@app/a": (h) => { + h.serve(New, { + compose: () => 1, + accepts: { + 1: { + value: (n) => { + const value = { n, payload: new Uint8Array(1_000_000) }; + mapped = new WeakRef(value); + return value; + }, + }, + }, + }); + h.serve(Reader, (_, self) => ({ n: self.derive([self.registry(Old)], ([v]) => v.n) })); + }, + }); + const reader = await host.get(Reader, { id: 1 }, context); + expect(reader.n.value).toBe(1); + reader.release(); + await flushed(); + await expectCollected(mapped!); + await host.close(context); + }); + + it("keeps no old composition once its readers stopped waiting and nobody follows it (Opus)", async () => { + const Big = defineRegistry("app.big", { version: 1 }); + let first = true; + const made: WeakRef[] = []; + const Waiting = local("app.waiting", { version: 1, params: {}, members: {} }); + const Counter = local("app.counter", { version: 1, params: {}, members: { set: method(int()) } }); + const { host } = await loadedPlugins({ + "@app/a": (h) => { + h.serve(Big, { + compose: (table) => { + if (first) throw new Error("first"); + const value = { payload: new Array(1000).fill(table.length) }; + made.push(new WeakRef(value)); + return value; + }, + }); + h.serve(Counter, (_, self) => { + const value = self.writable(0); + self.contribute(Big, value); + return { + set: async (n: number) => { + value.set(n); + return null; + }, + }; + }); + h.serve(Waiting, (_, self) => { + void self.registry(Big); + return {}; + }); + }, + }); + const counter = await host.get(Counter, {}, context); + first = false; + // The waiting reader's composition (pulled in the batch of this set) wakes it. + await counter.set(1, context); + await flushed(); + await counter.set(2, context); + void (await host.registry(Big, context)).value; + await settledHeap(); + // Only the current composition is alive. + expect(made.map((ref) => ref.deref() !== undefined)).toEqual([false, true]); + await host.close(context); + }); + + it("a value an app keeps after `host.close` keeps no closed host, and reads its last composition (Sol, Astra)", async () => { + const Kept = defineRegistry("app.kept", { version: 1 }); + const signal = new AbortController(); + const { weak, value } = await (async () => { + const { host } = await loadedPlugins({ "@app/a": (h) => h.serve(Kept, { compose: () => 7 }) }); + const value = await host.registry(Kept, context); + value.observe({ advance: () => {}, end: () => {} }, withAbortSignal(signal.signal, context)); + expect(getEventListeners(signal.signal, "abort")).toHaveLength(1); + await host.close(context); + // The observation ended with the host: its abort listener went with it. + expect(getEventListeners(signal.signal, "abort")).toHaveLength(0); + return { weak: new WeakRef(host), value }; + })(); + await expectCollected(weak); + expect(value.value).toBe(7); + }); +}); + +describe("observe", () => { + it("calls a synchronous observer inside the batch that composed; it counts as watching", async () => { + const Labels = defineRegistry("app.labels", { version: 1 }); + const Label = local("app.label", { version: 1, params: {}, members: { set: method(str()) } }); + const { host } = await loadedPlugins({ + "@app/labels": (host) => { + host.serve(Labels, { compose: (table) => table.map(({ value }) => value).join(",") }); + host.serve(Label, (_, self) => { + const label = self.writable("a"); + self.contribute(Labels, label); + return { + set: async (text: string) => { + label.set(text); + return null; + }, + }; + }); + }, + }); + const labels = await host.registry(Labels, context); + const calls: (NativeWatchCall | WatchEnd)[] = []; + const { context: cancellable, cancel } = withCancel(context); + const observation = labels.observe( + { advance: (call) => calls.push(call), end: (end) => calls.push(end) }, + cancellable, + ); + expect(observation.value).toBe("a"); + const label = await host.get(Label, {}, context); + await label.set("b", context); + // Already called: synchronously, in the set's batch. + expect(calls).toEqual([{ kind: "value", value: "b" }]); + cancel(); + await label.set("c", context); + expect(calls).toEqual([{ kind: "value", value: "b" }, { kind: "aborted" }]); + const again = labels.observe({ advance: (call) => calls.push(call), end: (end) => calls.push(end) }, context); + again.detach(); + await label.set("d", context); + expect(calls).toHaveLength(2); + labels.observe({ advance: () => {}, end: (end) => calls.push(end) }, context); + await host.close(context); + expect(calls.at(-1)).toEqual({ kind: "ended" }); + }); +}); diff --git a/packages/chord/test/next/state/native-source.test.ts b/packages/chord/test/next/state/native-source.test.ts new file mode 100644 index 000000000..cb31e8770 --- /dev/null +++ b/packages/chord/test/next/state/native-source.test.ts @@ -0,0 +1,152 @@ +import { describe, expect, expectTypeOf, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { int, local, method, str } from "../../../src/next/contracts/index.ts"; +import { ChordError } from "../../../src/next/data/index.ts"; +import type { Self } from "../../../src/next/host/index.ts"; +import { ChordObject } from "../../../src/next/host/object.ts"; +import { defineRegistry } from "../../../src/next/registries/index.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import type { NativeWatchCall, Watchable } from "../../../src/watch.ts"; +import { delivered, failLive, flushed, instancesOf, loaded, rejection } from "../support.ts"; + +const objects = instancesOf(ChordObject, "attempt"); + +const Count = defineRegistry("app.count", { version: 1 }); + +const Setter = local("app.setter", { version: 1, params: {}, members: { set: method(int()) } }); + +/** Serves `Count` (a sum that throws when negative) and `Setter`, which contributes a writable number. */ +function serveCount(h: Parameters[0]>[0]): void { + h.serve(Count, { + compose: (table) => { + const sum = table.reduce((n, { value }) => n + value, 0); + if (sum < 0) throw new Error(`negative: ${sum}`); + return sum; + }, + }); + h.serve(Setter, (_, self) => { + const value = self.writable(1); + self.contribute(Count, value); + return { + set: async (n: number) => { + value.set(n); + return null; + }, + }; + }); +} + +describe("native sources as derive sources", () => { + it("a derive reads JSON and native sources alike, marked when a native one changes", async () => { + const Mixed = local("app.mixed", { version: 1, params: {}, members: { label: str(), name: method(str()) } }); + const { host } = await loaded((h) => { + serveCount(h); + h.serve(Mixed, (_, self) => { + const name = self.writable("a"); + return { + label: self.derive([name, self.registry(Count)], ([n, count]) => `${n}:${count}`), + name: async (text: string) => { + name.set(text); + return null; + }, + }; + }); + }); + const mixed = await host.get(Mixed, {}, context); + await (await host.get(Setter, {}, context)).set(2, context); + expect(mixed.label.value).toBe("a:2"); + await mixed.name("b", context); + expect(mixed.label.value).toBe("b:2"); + await host.close(context); + }); + + it("`ref.state` is a derive source: a state member can show a dependency's reachability", async () => { + const Flaky = local("app.flaky", { version: 1, params: { id: int() }, members: {} }); + const Watcher = local("app.watcher", { version: 1, params: {}, members: { flaky: str() } }); + let flakySelf: Self | undefined; + const { host } = await loaded((h) => { + h.serve(Flaky, (_, self) => { + flakySelf = self; + return {}; + }); + h.serve(Watcher, (_, self) => { + const flaky = self.get(Flaky, { id: 1 }); + return { flaky: self.derive([flaky.state], ([state]) => state.kind) }; + }); + }); + const watcher = await host.get(Watcher, {}, context); + const kinds: string[] = []; + (await watcher.flaky.watch(context)).start( + async (call) => void (call.kind === "value" && kinds.push(call.value)), + ); + expect(watcher.flaky.value).toBe("live"); + failLive(objects, flakySelf!, new ChordError({ code: "unavailable", message: "down" })); + expect(watcher.flaky.value).toBe("failed"); + await delivered(); + expect(kinds).toEqual(["failed"]); + await host.close(context); + }); + + it("types a derive's sources: watchables of either kind, never a native's handle (Fable)", () => { + const declare = (self: Self): void => { + const handle = self.open( + async () => 1, + () => {}, + ); + // @ts-expect-error a native handle has a `value` but is no watchable + self.derive([handle], ([n]) => n); + expectTypeOf(self.derive([self.writable(1), constant("a")], ([n, s]) => s.length + n)).toEqualTypeOf< + Watchable + >(); + }; + expect(declare).toBeTypeOf("function"); + }); + + it("refuses a native source as a state member (Q14)", async () => { + const Bad = local("app.bad", { version: 1, params: {}, members: { n: int() } }); + const { host } = await loaded((h) => h.serve(Bad, (_, self) => ({ n: self.writable(1) as never }))); + expect(await rejection(host.get(Bad, {}, context))).toMatchObject({ + message: "app.bad: n: a state member is a JSON watchable", + }); + await host.close(context); + }); +}); + +describe("watching a native source that fails", () => { + it("a watch taken while it is failed opens at its last good value and gets the failure first", async () => { + const { host } = await loaded(serveCount); + const count = await host.registry(Count, context); + await (await host.get(Setter, {}, context)).set(-1, context); + const calls: NativeWatchCall[] = []; + const watch = await count.watch(context); + expect(watch.value).toBe(1); + watch.start(async (call) => void calls.push(call)); + await delivered(); + expect(calls).toEqual([{ kind: "failed", error: new Error("negative: -1") }]); + await host.close(context); + }); + + it("an `on` routine started while its source is failed waits for its next good value, never calls with the stale one", async () => { + const Follower = local("app.follower", { version: 1, params: { id: int() }, members: {} }); + const seen: number[] = []; + const { host } = await loaded((h) => { + serveCount(h); + h.serve(Follower, (_, self) => { + self.on(self.registry(Count), ({ value }) => void seen.push(value)); + return {}; + }); + }); + const setters = await host.get(Setter, {}, context); + await setters.set(-1, context); + // Read: the failure is composed before the follower starts. + void (await host.registry(Count, context)).value; + const follower = await host.get(Follower, { id: 1 }, context); + await flushed(); + expect(seen).toEqual([]); + await setters.set(4, context); + await delivered(); + expect(seen).toEqual([4]); + follower.release(); + await host.close(context); + }); +}); diff --git a/packages/chord/test/next/state/writable.test.ts b/packages/chord/test/next/state/writable.test.ts new file mode 100644 index 000000000..30b27309f --- /dev/null +++ b/packages/chord/test/next/state/writable.test.ts @@ -0,0 +1,210 @@ +import { getEventListeners } from "node:events"; +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context, withAbortSignal } from "../../../src/context/index.ts"; +import { int, local, method } from "../../../src/next/contracts/index.ts"; +import type { ChordHost } from "../../../src/next/host/host.ts"; +import type { Self } from "../../../src/next/host/index.ts"; +import { ChordObject } from "../../../src/next/host/object.ts"; +import { defineRegistry } from "../../../src/next/registries/index.ts"; +import type { Writable } from "../../../src/next/state/index.ts"; +import type { WritableSource } from "../../../src/next/state/internal.ts"; +import type { NativeWatchCall, WatchEnd } from "../../../src/watch.ts"; +import { delivered, expectCollected, instancesOf, loaded, metered, rejection, restartable } from "../support.ts"; + +const objects = instancesOf(ChordObject, "attempt"); + +const Counter = local("app.counter", { version: 1, params: {}, members: { doubled: int(), set: method(int()) } }); + +/** Serves `Counter` with a writable number, its double as a state member; the writable is handed out to the test. */ +async function counter(options: Parameters[1] = {}) { + let writable: Writable | undefined; + let self: Self | undefined; + const { host, errors } = await loaded((h) => { + h.serve(Counter, (_, made) => { + self = made; + const value = made.writable(1); + writable = value; + return { + doubled: made.derive([value], ([n]) => n * 2), + set: async (n: number) => { + value.set(n); + return null; + }, + }; + }); + }, options); + const reference = await host.get(Counter, {}, context); + return { host, errors, reference, writable: writable!, self: self! }; +} + +describe("writable", () => { + it("a new value (by identity) is a new frame; the same value notifies nobody and costs one comparison", async () => { + const { telemetry, counts } = await metered(); + const { host, reference, writable } = await counter({ telemetry }); + const calls: NativeWatchCall[] = []; + (await writable.watch(context)).start(async (call) => void calls.push(call)); + const doubled: number[] = []; + (await reference.doubled.watch(context)).start( + async (call) => void (call.kind === "value" && doubled.push(call.value)), + ); + const computes = counts()["chord.derive.computes"]; + writable.set(1); + await delivered(); + expect(calls).toEqual([]); + expect(counts()["chord.derive.computes"]).toBe(computes); + writable.set(2); + writable.set(3); + await delivered(); + expect(calls).toEqual([ + { kind: "value", value: 2 }, + { kind: "value", value: 3 }, + ]); + // A JSON watch folds what came while it waited: the derive's last value. + expect(doubled).toEqual([6]); + expect(writable.value).toBe(3); + await host.close(context); + }); + + it("inside a batch, sets mark it there: one frame with the last value, the derives over it computed once", async () => { + const { telemetry, counts } = await metered(); + const { host, reference, writable } = await counter({ telemetry }); + const calls: NativeWatchCall[] = []; + (await writable.watch(context)).start(async (call) => void calls.push(call)); + const doubled: number[] = []; + (await reference.doubled.watch(context)).start( + async (call) => void (call.kind === "value" && doubled.push(call.value)), + ); + const computes = counts()["chord.derive.computes"]!; + (host as unknown as ChordHost).batch.commit(() => { + writable.set(5); + writable.set(6); + }, context); + await delivered(); + expect(calls).toEqual([{ kind: "value", value: 6 }]); + expect(doubled).toEqual([12]); + expect(counts()["chord.derive.computes"]).toBe(computes + 1); + await host.close(context); + }); + + it("refuses a set inside a derive compute or a compose (`invalid/reentrant`)", async () => { + const Echo = defineRegistry("app.echo", { version: 1 }); + let inner: Writable | undefined; + const Bad = local("app.bad", { version: 1, params: {}, members: { bad: int() } }); + const Good = local("app.good", { version: 1, params: {}, members: {} }); + const { host, errors } = await loaded((h) => { + h.serve(Echo, { + compose: (table) => { + inner?.set(table.length + 100); + return table.length; + }, + }); + h.serve(Good, (_, self) => { + inner = self.writable(0); + self.contribute(Echo, inner); + return {}; + }); + h.serve(Bad, (_, self) => { + const own = self.writable(0); + return { + bad: self.derive([own], ([n]) => { + own.set(n + 1); + return n; + }), + }; + }); + }); + expect((await host.registry(Echo, context)).value).toBe(0); + expect(errors.find((error) => error.kind === "registry_failed")).toMatchObject({ + error: { code: "invalid", reason: "reentrant" }, + }); + expect(errors.find((error) => error.kind === "activation_failed")).toMatchObject({ + object: "app.bad", + failure: { kind: "broken", error: { code: "invalid", reason: "reentrant" } }, + }); + await host.close(context); + }); + + it("is the attempt's: `set` after its end throws `released`, its watches end `ended`, its value stays readable", async () => { + let writable: Writable | undefined; + const { host, restart } = await restartable(objects, (attempt, self) => { + if (attempt === 1) writable = self.writable(1); + }); + const watch = await writable!.watch(context); + await restart(); + expect(await watch.closed).toEqual({ kind: "ended" } satisfies WatchEnd); + expect(() => writable!.set(2)).toThrow("app.restartable(1): ended"); + expect(writable!.value).toBe(1); + await host.close(context); + }); + + it("is a contribution, a derive source and an `on` source", async () => { + const Values = defineRegistry("app.values", { version: 1 }); + const Source = local("app.source", { version: 1, params: {}, members: { set: method(int()), seen: int() } }); + const seen: number[] = []; + const { host } = await loaded((h) => { + h.serve(Values, { compose: (table) => table.map(({ value }) => value) }); + h.serve(Source, (_, self) => { + const value = self.writable(1); + self.contribute(Values, value); + self.on(value, ({ value: n }) => void seen.push(n)); + return { + seen: self.derive([value], ([n]) => n), + set: async (n: number) => { + value.set(n); + return null; + }, + }; + }); + }); + const source = await host.get(Source, {}, context); + const values = await host.registry(Values, context); + await source.set(2, context); + await delivered(); + expect(values.value).toEqual([2]); + expect(source.seen.value).toBe(2); + expect(seen).toEqual([1, 2]); + await host.close(context); + }); + + it("is collected with its attempt", async () => { + const refs = await (async () => { + const made: WeakRef[] = []; + const { host, restart } = await restartable(objects, (attempt, self) => { + if (attempt === 1) made.push(new WeakRef(self.writable({ big: new Array(1000).fill(0) }))); + }); + await restart(); + await host.close(context); + return made; + })(); + await expectCollected(...refs); + }); + + it("a set of the same value after its attempt ended throws `released` too (Sol, Astra)", async () => { + const { host, writable } = await counter(); + await host.close(context); + expect(() => writable.set(1)).toThrow("app.counter: ended"); + }); + + it("a writable an app keeps after `host.close` keeps no closed host; its observers' abort listeners go (Sol)", async () => { + const signal = new AbortController(); + const { weak, writable } = await (async () => { + const { host, writable } = await counter(); + (writable as WritableSource).observe( + { advance: () => {}, end: () => {} }, + withAbortSignal(signal.signal, context), + ); + expect(getEventListeners(signal.signal, "abort")).toHaveLength(1); + await host.close(context); + expect(getEventListeners(signal.signal, "abort")).toHaveLength(0); + return { weak: new WeakRef(host), writable }; + })(); + await expectCollected(weak); + expect(writable.value).toBe(1); + }); + + it("a late callback's set after the host closed throws `released`", async () => { + const { host, writable } = await counter(); + await host.close(context); + expect(await rejection(Promise.resolve().then(() => writable.set(9)))).toMatchObject({ code: "released" }); + }); +}); diff --git a/packages/chord/test/next/support.ts b/packages/chord/test/next/support.ts index 428dea36f..9796d534f 100644 --- a/packages/chord/test/next/support.ts +++ b/packages/chord/test/next/support.ts @@ -35,6 +35,7 @@ import { createHost, defineFacet, type FacetHost, + type FacetSelf, type Host, type HostError, type RefState, @@ -231,7 +232,7 @@ async function restartableOf void, + setup: (host: FacetHost, self: FacetSelf) => void, options: { readonly telemetry?: InMemoryTelemetryContext; readonly context?: Context; @@ -247,6 +248,21 @@ export async function loaded( return { host, errors }; } +/** + * A host with background errors collected, loaded with one facet per plugin, in the order given (`setups` by plugin + * name). + */ +export async function loadedPlugins( + setups: Readonly void>>, + options: { readonly telemetry?: InMemoryTelemetryContext; readonly readyTimeout?: number } = {}, +): Promise<{ readonly host: Host; readonly errors: HostError[] }> { + const errors: HostError[] = []; + const host = createHost({ ...options, onError: (error) => errors.push(error) }); + const facets = Object.entries(setups).map(([name, setup]) => ({ name, facet: defineFacet(setup) })); + await host.load(facets, BACKGROUND_CONTEXT); + return { host, errors }; +} + /** A store on a fresh SQLite `:memory:` database, as Chord's layers see it. */ export async function memoryStore(options?: { readonly outbox?: boolean }): Promise { const db = await openNodeSqliteDatabase(":memory:"); diff --git a/packages/durable/CHANGELOG.md b/packages/durable/CHANGELOG.md index 56763148d..92430f3e4 100644 --- a/packages/durable/CHANGELOG.md +++ b/packages/durable/CHANGELOG.md @@ -98,6 +98,8 @@ - `RunFailureReason` (`pi.live.runFailure.reason`) gains `nothing_to_answer` and `turn_limit`, also an `unanswered` submission's `reason`: exhaustive switches over it need cases ([#11043](https://github.com/earendil-works/pi/issues/11043)) - `pi.live.run` has a required `turns` field, the run's tool rounds so far; code that writes `run` itself sets it (`turns: 0`). `pi.live` is version 3: its `migrate` adds `turns: 0` to a stored run without it, version 1 or a version 2 an earlier unreleased build wrote, whose run in flight otherwise failed its next tool round with "Value contains a non-finite number" ([#11043](https://github.com/earendil-works/pi/issues/11043)) +- A Session's close ends synchronous observers (`observeDoc()`, `observeConversation()`, `observeTaskGraph()`, `observeCommits()`) `failed` with the error "session closed" instead of `ended`, so an adapter following a Session (Chord's `self.follow`) fails and follows the next one; `ended` stays for an observed owner that ended (a task document's task), and watches still end `ended`. Before: `end: (end) => { if (end.kind === "ended") done(); }`. After: `end: (end) => { if (end.kind === "ended" || end.kind === "failed") done(); }`. + ### Added - `StoreStorage` (`@earendil-works/pi-durable/storage/store`): Durable's `Storage` on Chord v2's store. `StoreStorage.open(store, owner, context)` keeps a Session's records as declared tables of the store under `owner` and its documents in the store's documents engine, takes its ids from the store's one allocator, and commits each batch as one write of the store's commit step, adopted in the batch of the host the store serves, so a Chord object and a Durable Session commit through one queue and one outbox. It passes the storage conformance suite; `close()` leaves the store open (the app owns it). @@ -115,6 +117,8 @@ - `HookApi.entry()` and `entries()`, and `TaskRuntime.entries()`: a hook or task reads the conversation's raw history, which compaction never changes, so a range bounded by entry IDs reads the same before and after a compaction and on a rerun ([#11141](https://github.com/earendil-works/pi/issues/11141)) - Run-end tasks: a `GenerationHooks.afterRun(run, api)` hook returns `{ tasks }`, built with the typed `runEndTask(task, input)`, after a run answered (a final answer or a `terminate` round; not when an `onYield` continuation is taken, not after a failure or an abort). The answer is committed and the run's inputs settle `done` first, so `Submission.wait()` resolves at the answer; the generation then creates the tasks as its children and waits for them, keeping `pi.live.run` (with `run.afterRun`) and the conversation busy, so idle waits include them, Stop aborts them, and a crash resumes them. Input queued meanwhile, steers included, is placed when they are terminal and starts the next run; a task admits a model-visible write or a fix-run follow-up in its own commit with `submit(tx, draft)`. A crash before the tasks exist asks the hooks again; a broken request (an uninstalled or built-in task, a non-JSON input, a throwing `initial()`) is reported and skipped. `run` names the run's entries: `first` (`pi.live.run.first`, carried into `runFailure.first` so a continue covers a failed run's history), `answer`, and `last` ([#11141](https://github.com/earendil-works/pi/issues/11141)) +- `registryReader(value, context)`: a `RegistryReader` over a Chord v2 registry value whose composition is a registry snapshot (`RegistryValue`): `snapshot()` reads the current composition, `subscribe` observes its compositions synchronously inside Chord's batch, so a Harness opened in a Chord object takes its extensions from a Chord registry. + ### Changed - Document, conversation view, and task graph watches use Chord's watch handles (`Watch`). diff --git a/packages/durable/docs/spec.md b/packages/durable/docs/spec.md index 4f3795e33..385f8e948 100644 --- a/packages/durable/docs/spec.md +++ b/packages/durable/docs/spec.md @@ -5431,9 +5431,11 @@ advances without a commit, when a registry change or a failed migration changes a task's `blocked` reason (section 9.4): that call runs on the Session line, outside any `adopt`, with a context of its own. A document observer follows its address as a document watch does, and `observeDoc()` gives `undefined` for an -absent one. `end` comes once: when the Session closes (`ended`) or fails -(`failed` with the error), or, for a task document, right after the retirement change of -the commit that makes its task terminal, inside that commit's `adopt` (`ended`). +absent one. `end` comes once: when the Session closes (`failed` with the error +"session closed": an adapter following the Session, such as Chord's `self.follow`, +fails and follows the next Session; watches end `ended`) or fails (`failed` with +the error), or, for a task document, right after the retirement change of the +commit that makes its task terminal, inside that commit's `adopt` (`ended`). `detach()` is idempotent, and nothing follows it, not even the rest of the commit underway; a detached observer is referenced by nothing of the Session. `context` governs only the attachment: cancelling it while attaching rejects diff --git a/packages/durable/src/harness/registry-reader.ts b/packages/durable/src/harness/registry-reader.ts new file mode 100644 index 000000000..97f8a44bd --- /dev/null +++ b/packages/durable/src/harness/registry-reader.ts @@ -0,0 +1,34 @@ +// A Harness's registry over a Chord registry (Chord v2 slice 8 §5): the composed value of p3's extensions registry, +// read as Durable's `RegistryReader`. + +import type { Context } from "@earendil-works/chord/context"; +import type { RegistryValue } from "@earendil-works/chord/next"; +import type { RegistryReader, RegistrySnapshot, ToolRegistration } from "./types.ts"; + +/** + * A `RegistryReader` over a Chord registry value whose composition is a registry snapshot: `snapshot()` reads the + * current composition (the last good one while `compose` fails), `subscribe` observes its compositions, its listener + * called synchronously inside the Chord batch that composed (so it only wakes the scheduler; a Chord call there + * throws `invalid/reentrant`). `context` is the observation's: aborting it detaches every listener. + */ +export function registryReader( + value: RegistryValue>, + context: Context, +): RegistryReader { + return { + snapshot: () => value.value, + subscribe: (listener) => { + const observation = value.observe( + { + advance: (call) => { + // A failed composition keeps the last snapshot: nothing to reconsider. + if (call.kind === "value") listener(); + }, + end: () => {}, + }, + context, + ); + return () => observation.detach(); + }, + }; +} diff --git a/packages/durable/src/harness/task-graph.ts b/packages/durable/src/harness/task-graph.ts index 2e0b78875..943eba33a 100644 --- a/packages/durable/src/harness/task-graph.ts +++ b/packages/durable/src/harness/task-graph.ts @@ -1,16 +1,10 @@ -import { - type Context, - type JsonValue, - Watch, - type WatchCall, - type WatchEnd, - type WatchHandle, -} from "@earendil-works/chord"; +import { type Context, type JsonValue, Watch, type WatchCall, type WatchHandle } from "@earendil-works/chord"; import { BACKGROUND_CONTEXT, withoutAbortSignal } from "@earendil-works/chord/context"; import { applyImmutable, type Op } from "@earendil-works/chord/delta"; import { + closeEndOf, + closeEnds, contained, - ENDED, guardObserver, type ObservedCommit, observation, @@ -110,11 +104,10 @@ export class TaskGraphView { blocking.subscribeBlocked(() => this.#refreshBlocked()); session.subscribeClose(() => { this.#closed = true; - const failure = this.#session.failure; - const end: WatchEnd = failure === undefined ? ENDED : { kind: "failed", error: failure.error }; + const ends = closeEnds(this.#session.failure); for (const observer of [...(this.#mount?.observers ?? [])]) { contained( - () => observer.end(end), + () => observer.end(closeEndOf(observer, ends)), (error) => session.report(error), ); } diff --git a/packages/durable/src/harness/view.ts b/packages/durable/src/harness/view.ts index d2999c47f..72cd9c890 100644 --- a/packages/durable/src/harness/view.ts +++ b/packages/durable/src/harness/view.ts @@ -10,8 +10,9 @@ import { withoutAbortSignal } from "@earendil-works/chord/context"; import { applyImmutable, type NonEmptyPath, type Op, type Path } from "@earendil-works/chord/delta"; import { idFromNumber } from "../ids.ts"; import { + closeEndOf, + closeEnds, contained, - ENDED, guardObserver, type ObservedCommit, observation, @@ -158,12 +159,11 @@ export class ConversationViews { }); session.subscribeClose(() => { this.#closed = true; - const failure = this.#session.failure; - const end: WatchEnd = failure === undefined ? ENDED : { kind: "failed", error: failure.error }; + const ends = closeEnds(this.#session.failure); for (const mount of this.#mounts.values()) for (const windowed of [...mount.observers]) { contained( - () => windowed.observer.end(end), + () => windowed.observer.end(closeEndOf(windowed.observer, ends)), (error) => this.report(error), ); } diff --git a/packages/durable/src/index.ts b/packages/durable/src/index.ts index 4b4671129..59068ad12 100644 --- a/packages/durable/src/index.ts +++ b/packages/durable/src/index.ts @@ -54,6 +54,7 @@ export { } from "./harness/live.ts"; export { ProviderDoc, type ProviderState } from "./harness/provider.ts"; export { createRegistry } from "./harness/registry.ts"; +export { registryReader } from "./harness/registry-reader.ts"; export type { BlockedReason } from "./harness/scheduler.ts"; export type { TaskGraph, TaskGraphNode, TaskGraphState, TaskGraphWatch } from "./harness/task-graph.ts"; export { DURABLE_TELEMETRY_SCHEMA } from "./harness/telemetry.ts"; diff --git a/packages/durable/src/session/session.ts b/packages/durable/src/session/session.ts index 5f5df8778..a6bb44556 100644 --- a/packages/durable/src/session/session.ts +++ b/packages/durable/src/session/session.ts @@ -612,8 +612,8 @@ export class SessionImpl implements Session { const observers = [...this.#commitListeners]; this.#commitListeners.clear(); // After the last publication: the commits admitted before the seal have settled, held changes are stored. - const end: WatchEnd = - this.#failure === undefined ? ENDED : { kind: "failed", error: this.#failure.error }; + // Commit observers are synchronous observers, all of them. + const end = closeEnds(this.#failure).observer; for (const observer of observers) observer.end(end); this.#internalListeners.clear(); this.#documents.clear(); @@ -1269,7 +1269,7 @@ export class SessionImpl implements Session { }); unsubscribeClose = this.subscribeClose(() => { detach(); - observer.end(this.#failure === undefined ? ENDED : { kind: "failed", error: this.#failure.error }); + observer.end(closeEndOf(observer, closeEnds(this.#failure))); }); return observer; } @@ -1419,6 +1419,28 @@ function toError(error: unknown): Error { /** The end of an observer whose source closed. */ export const ENDED: WatchEnd = { kind: "ended" }; +/** + * How a Session's close ends what follows it (decision "Slice 8 S1"), made once per close: an app's watch `ended`; a + * synchronous observer (an adapter following the Session, which may open it again) `failed` with "session closed", so + * it follows the next Session; `ended` stays for an observed owner that ended (a task document's task). A failed + * Session ends both `failed` with its error. + */ +export function closeEnds(failure: { readonly error: unknown } | undefined): { + readonly watch: WatchEnd; + readonly observer: WatchEnd; +} { + if (failure !== undefined) { + const failed: WatchEnd = { kind: "failed", error: failure.error }; + return { watch: failed, observer: failed }; + } + return { watch: ENDED, observer: { kind: "failed", error: new Error("session closed") } }; +} + +/** The end `ends` give `follower`: a watch's, or an observer's. */ +export function closeEndOf(follower: unknown, ends: ReturnType): WatchEnd { + return follower instanceof Watch ? ends.watch : ends.observer; +} + /** * The `Observation` of a guarded observer attached at `value`. Built outside the attaching closure, so a detached * observation keeps nothing but `value` and the spent guard alive. diff --git a/packages/durable/src/types.ts b/packages/durable/src/types.ts index 4f17a3711..c2aa21688 100644 --- a/packages/durable/src/types.ts +++ b/packages/durable/src/types.ts @@ -1101,8 +1101,9 @@ export type DocumentWatch = WatchHandle>; * may come before the attaching promise resolves; their ops apply from `Observation.value`. `advance` must only record * the call, synchronously and quickly, never await, call the Session, or throw. A throw is reported as * `listener_error`, detaches the observer, and ends it `listener_error` with the error. `end` is called once: `ended` - * when the Session closes or the observed scope's owner ends (a task document's task, after its retirement change, - * inside that commit's `adopt`), `failed` with the error that failed the Session; never after `detach()`. + * when the observed scope's owner ends (a task document's task, after its retirement change, inside that commit's + * `adopt`), `failed` with "session closed" when the Session closes (an adapter follows the next Session) or with the + * error that failed the Session; never after `detach()`. */ export type Observer = { advance(call: WatchCall, context: Context): void; @@ -1111,7 +1112,7 @@ export type Observer = { /** * A synchronous observer of stored commits (`Session.observeCommits()`): `advance` gets each `CommitPublication`, `end` - * the Session's close. + * the Session's close (`failed` with "session closed", or with the error that failed the Session). */ export type CommitObserver = { advance(publication: CommitPublication, context: Context): void; diff --git a/packages/durable/test/examples/37-harness-registry.ts b/packages/durable/test/examples/37-harness-registry.ts new file mode 100644 index 000000000..404a61a0d --- /dev/null +++ b/packages/durable/test/examples/37-harness-registry.ts @@ -0,0 +1,172 @@ +// A Durable harness in Chord objects, p3's pattern (Chord v2 slice 8): the object that opens Durable on the host's store +// takes its extensions from a Chord registry (`registryReader`), so any plugin extends the harness by contributing an +// extension; and it contributes its Harness to a second registry, through which another object follows its views +// (`self.follow`). When that Harness's Session closes, the follower fails and follows the next Harness, woken by its +// contribution (`self.retryOn`). +// Run from packages/durable: +// node --conditions=source --experimental-strip-types test/examples/37-harness-registry.ts + +import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context"; +import { + arr, + createHost, + defineFacet, + defineRegistry, + local, + method, + type Native, + openStore, + type RegistryTable, + sqliteBackend, + str, +} from "@earendil-works/chord/next"; +import { openNodeSqliteDatabase } from "@earendil-works/chord/sqlite/node"; +import { createModels } from "@earendil-works/pi-ai"; +import { + createRegistry, + defineExtension, + defineTask, + type Extension, + Harness, + type RegistrySnapshot, + ROOT_CONVERSATION_ID, + registryReader, +} from "../../src/index.ts"; +import { StoreStorage } from "../../src/storage/store/index.ts"; + +const context = BACKGROUND_CONTEXT; + +// 1. The core's registries: extensions in, Durable's registry snapshot out; and the Harness, for other objects. +const Extensions = defineRegistry("example.extensions", { version: 1 }); +const Harnesses = defineRegistry, Harness | undefined>("example.harnesses", { version: 1 }); + +/** Builds Durable's snapshot without throwing: an extension Durable refuses (a task name taken) is reported, left out. */ +function snapshotOf( + table: RegistryTable, + report: (conflict: { message: string; plugins: string[] }) => void, +) { + const registry = createRegistry(); + for (const { value, plugin } of table) { + try { + registry.install(value); + } catch (error) { + report({ message: (error as Error).message, plugins: [plugin] }); + } + } + return registry.snapshot(); +} + +const Core = local("example.harness", { + version: 1, + params: {}, + members: { greet: method(undefined, str()), close: method() }, +}); +const Transcript = local("example.transcript", { version: 1, params: {}, members: { kinds: arr(str()) } }); + +// 2. The core facet: the harness object opens Durable on the store with its extensions read from the registry. +const store = await openStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), context); +const core = defineFacet((host) => { + host.serve(Extensions, { compose: (table, { report }) => snapshotOf(table, report) }); + host.serve(Harnesses, { compose: (table) => table.at(-1)?.value.value }); + host.serve(Core, (_, self) => { + // A declared dependency: the object activates (and opens Durable) once the registry has a value. + const extensions = self.registry(Extensions); + const harness = self.open( + async (open) => { + const opened = await Harness.open( + await StoreStorage.open(store, Core.name, open), + { models: createModels(), registry: registryReader(extensions, open) }, + open, + ); + await opened.root(open); + return opened; + }, + (h, _reason, close) => h.close(close), + ); + self.contribute(Harnesses, harness); + // Following its own task graph, the object fails with its Session (S1), and its retry opens the next Harness. + self.follow((observer, ctx) => harness.value.observeTaskGraph(observer, ctx)); + return { + greet: async (ctx) => { + // The composed snapshot, read at use: is the task's extension installed? + if (extensions.value.task(Greet.definition.name) === undefined) return "no greeter installed"; + const root = await harness.value.conversation(ROOT_CONVERSATION_ID, ctx); + const id = await root!.commit( + (tx) => tx.createTask(Greet, undefined, { ownership: { kind: "conversation" } }), + ctx, + ); + harness.value.resume(); + return JSON.stringify((await harness.value.waitForTask(id, ctx)).state.outcome); + }, + close: async (ctx) => { + await harness.value.close(ctx); + return null; + }, + }; + }); + // 3. Another object follows the harness's conversation view through the registry. + host.serve(Transcript, (_, self) => { + const harness = self.registry(Harnesses); + self.retryOn(harness); + const view = self.follow(async (observer, ctx) => + harness.value?.observeConversation(ROOT_CONVERSATION_ID, observer, ctx), + ); + return { kinds: self.derive([view], ([v]) => v.entries.map((entry) => entry.kind)) }; + }); +}); + +// 4. Two plugins contribute extensions with the same task name, which Durable refuses: the first by plugin name +// (copycat) is installed, the other reported. +const Greet = defineTask({ + name: "example.greet", + version: 1, + initial: () => ({ phase: "run" }), + phases: { + run: async (_task, runtime, taskContext) => { + await runtime.commit(async (tx) => { + await tx.appendEntry(ROOT_CONVERSATION_ID, { kind: "example.greeted", data: {} }); + return { status: "terminal", outcome: { status: "completed", result: "hello" } }; + }, taskContext); + }, + }, + abort: async () => {}, +}); +const greeter = defineFacet((_, self) => + self.contribute(Extensions, defineExtension({ name: "greeter", tasks: [Greet] })), +); +const copycat = defineFacet((_, self) => + self.contribute(Extensions, defineExtension({ name: "copycat", tasks: [Greet] })), +); + +const host = createHost({ + store, + onError: (error) => console.log("onError:", error.kind, error.kind === "registry_conflict" ? error.message : ""), +}); +await host.load( + [ + { name: "@example/core", facet: core }, + { name: "@example/greeter", facet: greeter }, + { name: "@example/copycat", facet: copycat }, + ], + context, +); + +// 5. The task comes from a contributed extension; the transcript object sees its entry. (It failed once at start, before +// the harness object contributed its Harness: `activation_failed`, then the contribution woke it.) +const harness = host.hold(Core, {}, context); +const transcript = host.hold(Transcript, {}, context); +console.log("greet:", await harness.greet(context)); +await new Promise((resolve) => setTimeout(resolve, 10)); +console.log("transcript:", transcript.kinds.value); + +// 6. The Session closes: both objects fail; a retry of the harness object opens the next Harness (on the same store: +// the transcript is still there), whose contribution wakes the follower. +await harness.close(context).catch((error: Error) => console.log("close:", error.message)); +await new Promise((resolve) => setTimeout(resolve, 10)); +console.log("transcript after the close:", transcript.state.value.kind); +harness.retry(); +await new Promise((resolve) => setTimeout(resolve, 50)); +console.log("transcript after the retry:", transcript.state.value.kind, transcript.kinds.value); + +await host.close(context); +await store.close(context); diff --git a/packages/durable/test/harness-observers.test.ts b/packages/durable/test/harness-observers.test.ts index fdd6d7787..bbd3db12e 100644 --- a/packages/durable/test/harness-observers.test.ts +++ b/packages/durable/test/harness-observers.test.ts @@ -124,8 +124,11 @@ describe("observeConversation and observeTaskGraph", () => { await root.commit((tx) => tx.appendEntry(root.id, { kind: "note", data: {} }), context); expect(view.calls).toHaveLength(1); await watch.stop(); + const graphWatch = await harness.watchTaskGraph(context); await harness.close(context); - expect(graph.calls).toEqual([{ end: { kind: "ended" } }]); + // S1: a Session close fails an observer (it may follow the next Session); a watch ends `ended`. + expect(graph.calls).toEqual([{ end: { kind: "failed", error: new Error("session closed") } }]); + expect(await graphWatch.closed).toEqual({ kind: "ended" }); }); it("end with the error that failed the Harness", async () => { diff --git a/packages/durable/test/harness-registry-reader.test.ts b/packages/durable/test/harness-registry-reader.test.ts new file mode 100644 index 000000000..b2a82da35 --- /dev/null +++ b/packages/durable/test/harness-registry-reader.test.ts @@ -0,0 +1,198 @@ +import { BACKGROUND_CONTEXT as context } from "@earendil-works/chord/context"; +import { + arr, + createHost, + defineFacet, + defineRegistry, + type FacetHost, + local, + method, + type Native, + type RegistryTable, + str, +} from "@earendil-works/chord/next"; +import { createModels } from "@earendil-works/pi-ai/models"; +import { describe, expect, it } from "vitest"; +import { + createRegistry, + defineExtension, + defineTask, + type Extension, + Harness, + MemoryStorage, + type RegistrySnapshot, + ROOT_CONVERSATION_ID, + registryReader, +} from "../src/index.ts"; + +/** p3's pattern: the extensions registry composes Durable's snapshot, leaving out an extension Durable refuses. */ +const Extensions = defineRegistry("app.harness.extensions", { version: 1 }); + +function compose(table: RegistryTable, report: (conflict: { message: string; plugins: string[] }) => void) { + const registry = createRegistry(); + for (const { value, plugin } of table) { + try { + registry.install(value); + } catch (error) { + report({ message: (error as Error).message, plugins: [plugin] }); + } + } + return registry.snapshot(); +} + +const Done = defineTask({ + name: "app.done", + version: 1, + initial: () => ({ phase: "run" }), + phases: { + run: async (_task, runtime, taskContext) => { + await runtime.commit( + () => ({ status: "terminal", outcome: { status: "completed", result: null } }), + taskContext, + ); + }, + }, + abort: async () => {}, +}); + +/** A host-lifetime object contributing an extension it replaces with `install` (one with the task `app.done`). */ +const Plugin = local("app.plugin", { version: 1, params: {}, members: { install: method() } }); + +function serve(host: FacetHost): void { + host.serve(Extensions, { compose: (table, { report }) => compose(table, report) }); + host.serve(Plugin, (_, self) => { + const extension = self.writable(defineExtension({ name: "app.plugin" })); + self.contribute(Extensions, extension); + return { + install: async () => { + extension.set(defineExtension({ name: "app.plugin", tasks: [Done] })); + return null; + }, + }; + }); +} + +describe("registryReader", () => { + it("reads the composed snapshot; a listener is called synchronously, inside the batch that composed", async () => { + const host = createHost({ onError: () => {} }); + await host.load([{ name: "@app/core", facet: defineFacet(serve) }], context); + const value = await host.registry(Extensions, context); + const reader = registryReader(value, context); + expect(reader.snapshot().extension("app.plugin")?.tasks).toBeUndefined(); + const calls: string[] = []; + const unsubscribe = reader.subscribe(() => { + // Inside Chord's batch: a Chord call here is refused, so a listener only wakes its scheduler. + try { + value.observe({ advance: () => {}, end: () => {} }, context); + calls.push("outside"); + } catch (error) { + calls.push((error as { reason?: string }).reason ?? "other"); + } + }); + const plugin = await host.get(Plugin, {}, context); + await plugin.install(context); + expect(calls).toEqual(["reentrant"]); + expect(reader.snapshot().task("app.done")).toBe(Done); + unsubscribe(); + await host.close(context); + }); + + it("wakes a Harness's scheduler: a task blocked on its missing definition runs once a contribution brings it", async () => { + const host = createHost({ onError: () => {} }); + await host.load([{ name: "@app/core", facet: defineFacet(serve) }], context); + const harness = await Harness.open( + new MemoryStorage(), + { models: createModels(), registry: registryReader(await host.registry(Extensions, context), context) }, + context, + ); + harness.resume(); + const root = await harness.root(context); + const id = await root.commit( + (tx) => tx.createTask(Done, undefined, { ownership: { kind: "conversation" } }), + context, + ); + const done = harness.waitForTask(id, context); + await new Promise((resolve) => setTimeout(resolve, 10)); + await (await host.get(Plugin, {}, context)).install(context); + expect((await done).state.outcome).toEqual({ status: "completed", result: null }); + await harness.close(context); + await host.close(context); + }); +}); + +/** p3's pattern for Durable's views from other objects: the object that opens Durable contributes its Harness. */ +const Harnesses = defineRegistry, Harness | undefined>("app.harnesses", { version: 1 }); + +/** Opens a Harness (a new one per attempt), contributes it, and follows its task graph: its Session closing fails it. */ +const Durable = local("app.durable", { version: 1, params: {}, members: { close: method(), write: method(str()) } }); + +/** Follows the root conversation's entry kinds through the contributed Harness, from another object. */ +const Viewer = local("app.viewer", { version: 1, params: {}, members: { kinds: arr(str()) } }); + +function serveViews(host: FacetHost): void { + host.serve(Harnesses, { compose: (table) => table.at(-1)?.value.value }); + host.serve(Durable, (_, self) => { + const harness = self.open( + async (open) => { + const opened = await Harness.open( + new MemoryStorage(), + { models: createModels(), registry: createRegistry() }, + open, + ); + await opened.root(open); + return opened; + }, + (h, _reason, close) => h.close(close), + ); + self.contribute(Harnesses, harness); + self.follow((observer, ctx) => harness.value.observeTaskGraph(observer, ctx)); + return { + close: async (ctx) => { + await harness.value.close(ctx); + return null; + }, + write: async (kind, ctx) => { + const root = await harness.value.conversation(ROOT_CONVERSATION_ID, ctx); + await root!.submit({ type: "write", entry: { kind } }, ctx); + return null; + }, + }; + }); + host.serve(Viewer, (_, self) => { + const harness = self.registry(Harnesses); + self.retryOn(harness); + const view = self.follow(async (observer, ctx) => + harness.value === undefined + ? undefined + : harness.value.observeConversation(ROOT_CONVERSATION_ID, observer, ctx), + ); + return { kinds: self.derive([view], ([v]) => v.entries.map((entry) => entry.kind)) }; + }); +} + +describe("Durable's views from other objects (slice 8 §5, S1)", () => { + it("a follower of a contributed Harness fails when that Harness's Session closes, and follows the next one", async () => { + const host = createHost({ onError: () => {} }); + await host.load([{ name: "@app/core", facet: defineFacet(serveViews) }], context); + const durable = host.hold(Durable, {}, context); + const viewer = host.hold(Viewer, {}, context); + await new Promise((resolve) => setTimeout(resolve, 10)); + await durable.write("app.first", context); + await new Promise((resolve) => setTimeout(resolve, 10)); + expect(viewer.kinds.value).toContain("app.first"); + // The Session closes: both followers fail `transient` (S1); a retry opens the next Harness, whose contribution + // wakes the viewer (`retryOn`), which follows it. + // The call is cut off by its object's failure: the Session's close fails the object that opened it. + await expect(durable.close(context)).rejects.toThrow("session closed"); + await new Promise((resolve) => setTimeout(resolve, 10)); + expect(viewer.state.value).toMatchObject({ kind: "failed", cause: { kind: "transient" } }); + durable.retry(); + await new Promise((resolve) => setTimeout(resolve, 50)); + expect(viewer.state.value).toEqual({ kind: "live" }); + expect(viewer.kinds.value).not.toContain("app.first"); + await durable.write("app.second", context); + await new Promise((resolve) => setTimeout(resolve, 10)); + expect(viewer.kinds.value).toContain("app.second"); + await host.close(context); + }); +}); diff --git a/packages/durable/test/session-observers.test.ts b/packages/durable/test/session-observers.test.ts index 0dcbc324a..a22731cef 100644 --- a/packages/durable/test/session-observers.test.ts +++ b/packages/durable/test/session-observers.test.ts @@ -173,13 +173,31 @@ describe("observeDoc", () => { expect(calls).toEqual([]); }); - it("ends once when the Session closes", async () => { + it("ends once when the Session closes, `failed` (S1: an adapter follows the next Session), a watch `ended`", async () => { const { storage, session } = open(); await add(session); const { calls, observer } = recorder(storage); await session.observeDoc(SessionCounter, observer, context); + const watch = (await session.watchDoc(SessionCounter, context))!; await session.close(context); - expect(calls).toEqual([{ end: { kind: "ended" } }]); + expect(calls).toEqual([{ end: { kind: "failed", error: new Error("session closed") } }]); + expect(await watch.closed).toEqual({ kind: "ended" }); + }); + + it("ends each close's observers with an error of that close, not one shared by every Session", async () => { + const errors: unknown[] = []; + for (let i = 0; i < 2; i++) { + const { session } = open(); + await add(session); + await session.observeDoc( + SessionCounter, + { advance: () => {}, end: (end) => errors.push(end.kind === "failed" ? end.error : end) }, + context, + ); + await session.close(context); + } + expect(errors).toEqual([new Error("session closed"), new Error("session closed")]); + expect(errors[0]).not.toBe(errors[1]); }); it("ends with the error that failed the Session", async () => { @@ -368,14 +386,18 @@ describe("observeCommits", () => { return { calls, observer }; } - it("gets every stored commit, then `ended` once the Session closed, after its last publication", async () => { + it("gets every stored commit, then `failed` once the Session closed (S1), after its last publication", async () => { const { session } = open(); const { calls, observer } = commits(); session.observeCommits(observer); await add(session); await add(session); await session.close(context); - expect(calls).toEqual([{ seq: expect.any(Number) }, { seq: expect.any(Number) }, { end: { kind: "ended" } }]); + expect(calls).toEqual([ + { seq: expect.any(Number) }, + { seq: expect.any(Number) }, + { end: { kind: "failed", error: new Error("session closed") } }, + ]); }); it("ends `failed` with the error that failed the Session", async () => { diff --git a/scripts/check-entry-graphs.mjs b/scripts/check-entry-graphs.mjs index b3b033a73..388eb0012 100644 --- a/scripts/check-entry-graphs.mjs +++ b/scripts/check-entry-graphs.mjs @@ -67,8 +67,9 @@ const BUDGETS = { // The Harness's request identity reaches pi-ai's utils/identity.ts and its version constant. Partial commits // parse streamed tool-call arguments with pi-ai's utils/streaming-json.ts (and its json-parse.ts) to append what // argument strings grew by (harness/partial-changes.ts). Chord's watches fold pending ops with Delta's - // pending queue (delta/pending.ts). - maxFiles: 77, + // pending queue (delta/pending.ts). `registryReader` (harness/registry-reader.ts) reads a Chord v2 registry + // value through type-only imports. + maxFiles: 78, forbid: [ "packages/ai/src/index.ts", "packages/ai/src/utils/typebox-helpers.ts", From c466a35957e73876651d014d6c36d5ea47148344 Mon Sep 17 00:00:00 2001 From: Christian Klotz Date: Thu, 29 Oct 2026 00:13:41 +0200 Subject: [PATCH 005/175] fix(coding-agent): cap pi.exec output at 256 MiB (#11546) Closes #11360. ## Summary - Cap combined stdout/stderr with `ExecOptions.maxBuffer`, defaulting to the smaller of 256 MiB and the runtime string limit. Invalid explicit limits reject before spawning. - Terminate oversized commands and return `truncated: true`, `killed: true`, and a nonzero exit code. Close output pipes so writing descendants cannot keep the call pending, and discard incomplete UTF-8 suffixes rather than adding replacement characters. - Fix SIGKILL escalation for children that ignore SIGTERM, and simplify completion to `await` with one cleanup block and result. - Document the API and add regressions for byte limits, shell descendants, UTF-8 chunk boundaries, and 32-bit runtime limits. `ExecResult.truncated` is required. Existing extensions reading execution results continue working; typed mocks or wrappers constructing results must add the field. ## Verification - `npm run build` and the post-rebase `npm run build:offline` - `npm run check` - 39 focused execution and extension tests - `./test.sh` passes after building the env daemon. Earlier runs hit remote-cleanup and reftable-watcher timeouts; both passed in focused reruns and the final full run. --- packages/coding-agent/CHANGELOG.md | 3 + packages/coding-agent/docs/extensions.md | 6 +- packages/coding-agent/src/core/exec.ts | 190 +++++++++++------- .../test/exec-runtime-limit.test.ts | 25 +++ .../test/exec-utf8-chunks.test.ts | 53 +++++ packages/coding-agent/test/exec.test.ts | 108 +++++++++- .../git-merge-and-resolve-extension.test.ts | 4 +- .../input-transform-streaming-example.test.ts | 6 +- 8 files changed, 313 insertions(+), 82 deletions(-) create mode 100644 packages/coding-agent/test/exec-runtime-limit.test.ts create mode 100644 packages/coding-agent/test/exec-utf8-chunks.test.ts diff --git a/packages/coding-agent/CHANGELOG.md b/packages/coding-agent/CHANGELOG.md index 31571b8e9..6f09e904e 100644 --- a/packages/coding-agent/CHANGELOG.md +++ b/packages/coding-agent/CHANGELOG.md @@ -5,6 +5,7 @@ ### Added - Extension packages can now show their changelog after an update. Pi reads `pi.changelogPath` from the package's `package.json`, or `CHANGELOG.md` in the package root, and on the next fresh launch shows entries newer than the last seen version under "Extension Updates". The `collapseChangelog` setting condenses this to one line per package ([#5958](https://github.com/earendil-works/pi/issues/5958), [#8790](https://github.com/earendil-works/pi/pull/8790)) +- `pi.exec()` accepts a `maxBuffer` option that limits stdout and stderr combined, and its result has a `truncated` field that is true when output exceeded the limit ([#11360](https://github.com/earendil-works/pi/issues/11360)) - Device-code login (for example GitHub Copilot) now opens the verification page and copies the code when you press Enter. Providers can turn either action off with `openBrowserOnConfirm: false` or `copyCodeOnConfirm: false` ([#9282](https://github.com/earendil-works/pi/issues/9282), [#9301](https://github.com/earendil-works/pi/pull/9301)) ### Changed @@ -15,6 +16,8 @@ ### Fixed - Fixed a redundant system prompt update (tools, rules) when an extension set the same active tools in a different order, for example a reversed `pi.setActiveTools()` list. Tools that stay active now keep their order and newly activated tools are appended ([#11495](https://github.com/earendil-works/pi/issues/11495)) +- Fixed `pi.exec()` crashing pi with `RangeError: Invalid string length` when a command printed more output than fits in a string. The default output cap is the smaller of 256 MiB and the runtime string limit; past the cap, the command is terminated and the result has `truncated: true` and a nonzero exit code ([#11360](https://github.com/earendil-works/pi/issues/11360)) +- Fixed `pi.exec()` never returning when a command ignored SIGTERM after a timeout or abort. pi now sends SIGKILL 5 seconds later, as intended - Fixed `pi install` and `pi remove` adding ever deeper nested entries to the npm `package-lock.json` when the agent directory is reached through a symlink, such as `~/.pi` linked into a dotfiles repo. pi now passes the resolved real path to npm ([#11358](https://github.com/earendil-works/pi/issues/11358)) - MCP sign-in now rejects authorization endpoints and other OAuth metadata URLs that are not `http:` or `https:`. Before, a malicious MCP server could supply a `file:` or custom protocol URL, which pi passed to the platform URL handler. - Automatic `fd` and `rg` downloads no longer fetch x86-64 binaries on ARM32 and other architectures without a prebuilt release. pi now warns and recommends installing the tool with the system package manager ([#11209](https://github.com/earendil-works/pi/issues/11209)) diff --git a/packages/coding-agent/docs/extensions.md b/packages/coding-agent/docs/extensions.md index 2439d1b45..7666d5645 100644 --- a/packages/coding-agent/docs/extensions.md +++ b/packages/coding-agent/docs/extensions.md @@ -1868,13 +1868,15 @@ pi.on("session_start", () => { ### pi.exec(command, args, options?) -Execute a shell command. +Execute a command without a shell. ```typescript const result = await pi.exec("git", ["status"], { signal, timeout: 5000 }); -// result.stdout, result.stderr, result.code, result.killed +// result.stdout, result.stderr, result.code, result.killed, result.truncated ``` +Options: `signal` and `timeout` terminate the command, `cwd` sets the working directory, and `maxBuffer` limits stdout and stderr combined (default 256 MiB). When output exceeds `maxBuffer`, pi terminates the command and returns the output up to the limit with `truncated: true`, `killed: true`, and a nonzero `code`. `killed` is true whenever pi terminated the command: timeout, abort, or `maxBuffer`. A command that ignores SIGTERM gets SIGKILL after 5 seconds. + ### pi.getActiveTools() / pi.getAllTools() / pi.setActiveTools(names) Manage active tools. This works for both built-in tools and dynamically registered tools. `pi.getActiveTools()` returns the active tool names as `string[]`; `pi.getAllTools()` returns metadata for all configured tools. diff --git a/packages/coding-agent/src/core/exec.ts b/packages/coding-agent/src/core/exec.ts index cf4205f7f..b6e845f45 100644 --- a/packages/coding-agent/src/core/exec.ts +++ b/packages/coding-agent/src/core/exec.ts @@ -2,10 +2,14 @@ * Shared command execution utilities for extensions and custom tools. */ +import { constants as bufferConstants } from "node:buffer"; import { spawn } from "node:child_process"; import { StringDecoder } from "node:string_decoder"; import { waitForChildProcess } from "../utils/child-process.ts"; +const DEFAULT_EXEC_MAX_BUFFER = Math.min(256 * 1024 * 1024, bufferConstants.MAX_STRING_LENGTH); +const FORCE_KILL_DELAY_MS = 5000; + /** * Options for executing shell commands. */ @@ -16,6 +20,11 @@ export interface ExecOptions { timeout?: number; /** Working directory */ cwd?: string; + /** + * Combined stdout/stderr byte limit; exceeding it terminates the command with `truncated: true`. + * Nonnegative integer up to the runtime string limit. Default: 256 MiB, capped by that limit. + */ + maxBuffer?: number; } /** @@ -25,12 +34,15 @@ export interface ExecResult { stdout: string; stderr: string; code: number; + /** The command was terminated by pi: timeout, abort, or output over `maxBuffer`. */ killed: boolean; + /** Output exceeded `maxBuffer`; stdout and stderr are incomplete and `code` is nonzero. */ + truncated: boolean; } /** * Execute a shell command and return stdout/stderr/code. - * Supports timeout and abort signal. + * Supports timeout, abort signal, and an output cap (`maxBuffer`). */ export async function execCommand( command: string, @@ -38,79 +50,109 @@ export async function execCommand( cwd: string, options?: ExecOptions, ): Promise { - return new Promise((resolve) => { - const proc = spawn(command, args, { - cwd, - shell: false, - stdio: ["ignore", "pipe", "pipe"], - }); + const maxBuffer = options?.maxBuffer ?? DEFAULT_EXEC_MAX_BUFFER; + if (!Number.isInteger(maxBuffer) || maxBuffer < 0 || maxBuffer > bufferConstants.MAX_STRING_LENGTH) { + throw new RangeError(`maxBuffer must be an integer between 0 and ${bufferConstants.MAX_STRING_LENGTH} bytes`); + } - let stdout = ""; - let stderr = ""; - // Decode across chunks: a multi-byte UTF-8 character can be split between two chunks, - // and decoding each chunk separately would corrupt it (earendil-works/pi#11312). - const stdoutDecoder = new StringDecoder("utf8"); - const stderrDecoder = new StringDecoder("utf8"); - let killed = false; - let timeoutId: NodeJS.Timeout | undefined; - - const killProcess = () => { - if (!killed) { - killed = true; - proc.kill("SIGTERM"); - // Force kill after 5 seconds if SIGTERM doesn't work - setTimeout(() => { - if (!proc.killed) { - proc.kill("SIGKILL"); - } - }, 5000); - } - }; - - // Handle abort signal - if (options?.signal) { - if (options.signal.aborted) { - killProcess(); - } else { - options.signal.addEventListener("abort", killProcess, { once: true }); - } - } - - // Handle timeout - if (options?.timeout && options.timeout > 0) { - timeoutId = setTimeout(() => { - killProcess(); - }, options.timeout); - } - - proc.stdout?.on("data", (data) => { - stdout += stdoutDecoder.write(data); - }); - - proc.stderr?.on("data", (data) => { - stderr += stderrDecoder.write(data); - }); - - // Wait for process termination without hanging on inherited stdio handles - // held open by detached descendants. - waitForChildProcess(proc) - .then((code) => { - if (timeoutId) clearTimeout(timeoutId); - stdout += stdoutDecoder.end(); - stderr += stderrDecoder.end(); - if (options?.signal) { - options.signal.removeEventListener("abort", killProcess); - } - resolve({ stdout, stderr, code: code ?? 0, killed }); - }) - .catch((error: Error) => { - if (timeoutId) clearTimeout(timeoutId); - stdout += stdoutDecoder.end(); - stderr += stderrDecoder.end(); - if (options?.signal) { - options.signal.removeEventListener("abort", killProcess); - } - resolve({ stdout, stderr: stderr || error.message, code: 1, killed }); - }); + const proc = spawn(command, args, { + cwd, + shell: false, + stdio: ["ignore", "pipe", "pipe"], }); + + let stdout = ""; + let stderr = ""; + // Decode across chunks: a multi-byte UTF-8 character can be split between two chunks, + // and decoding each chunk separately would corrupt it (earendil-works/pi#11312). + const stdoutDecoder = new StringDecoder("utf8"); + const stderrDecoder = new StringDecoder("utf8"); + let outputBytes = 0; + let truncated = false; + let killed = false; + let timeoutId: NodeJS.Timeout | undefined; + let forceKillId: NodeJS.Timeout | undefined; + + const killProcess = () => { + if (killed) return; + killed = true; + proc.kill("SIGTERM"); + forceKillId = setTimeout(() => { + // `proc.killed` only says a signal was sent, so check whether the child actually exited. + if (proc.exitCode === null && proc.signalCode === null) { + proc.kill("SIGKILL"); + } + }, FORCE_KILL_DELAY_MS); + }; + + if (options?.signal) { + if (options.signal.aborted) { + killProcess(); + } else { + options.signal.addEventListener("abort", killProcess, { once: true }); + } + } + + if (options?.timeout && options.timeout > 0) { + timeoutId = setTimeout(killProcess, options.timeout); + } + + // Keep only bytes within the combined cap (#11360). Close both pipes on overflow so writing + // descendants cannot keep waitForChildProcess busy after the child exits. Pausing is not + // enough: Node resumes the child's stdio on exit. + const takeWithinCap = (data: Buffer): Buffer | undefined => { + if (truncated) return undefined; + const remaining = maxBuffer - outputBytes; + if (data.length <= remaining) { + outputBytes += data.length; + return data; + } + truncated = true; + proc.stdout?.destroy(); + proc.stderr?.destroy(); + killProcess(); + return remaining > 0 ? data.subarray(0, remaining) : undefined; + }; + + proc.stdout?.on("data", (data: Buffer) => { + const chunk = takeWithinCap(data); + if (chunk) stdout += stdoutDecoder.write(chunk); + }); + + proc.stderr?.on("data", (data: Buffer) => { + const chunk = takeWithinCap(data); + if (chunk) stderr += stderrDecoder.write(chunk); + }); + + let code: number | null; + try { + // Do not hang on inherited stdio handles held open by detached descendants. + code = await waitForChildProcess(proc); + } catch (error) { + if (!(error instanceof Error)) throw error; + // Child-process errors (e.g. ENOENT) are failed executions, not host failures. + code = 1; + if (!stderr) stderr = error.message; + } finally { + clearTimeout(timeoutId); + clearTimeout(forceKillId); + options?.signal?.removeEventListener("abort", killProcess); + } + + // On truncation, discard any incomplete UTF-8 suffix buffered by either decoder. + // Flushing it would append a replacement character that was not in the accepted output. + if (!truncated) { + stdout += stdoutDecoder.end(); + stderr += stderrDecoder.end(); + } + + const exitCode = code ?? 0; + return { + stdout, + stderr, + // A truncated result must never read as success. + code: truncated && exitCode === 0 ? 1 : exitCode, + killed, + truncated, + }; } diff --git a/packages/coding-agent/test/exec-runtime-limit.test.ts b/packages/coding-agent/test/exec-runtime-limit.test.ts new file mode 100644 index 000000000..879a45911 --- /dev/null +++ b/packages/coding-agent/test/exec-runtime-limit.test.ts @@ -0,0 +1,25 @@ +import * as buffer from "node:buffer"; +import { expect, it, vi } from "vitest"; +import { execCommand } from "../src/core/exec.ts"; + +vi.mock("node:buffer", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + // 32-bit Node's maximum is 16 bytes below 256 MiB. + constants: { ...actual.constants, MAX_STRING_LENGTH: 268_435_440 }, + }; +}); + +// Regression tests for https://github.com/earendil-works/pi/issues/11360 +it("executes with the default cap when the runtime string limit is below 256 MiB", async () => { + const result = await execCommand(process.execPath, ["-e", 'process.stdout.write("ok")'], process.cwd()); + + expect(result).toEqual({ stdout: "ok", stderr: "", code: 0, killed: false, truncated: false }); +}); + +it("still rejects an explicit cap above the runtime string limit", async () => { + await expect( + execCommand(process.execPath, ["-e", ""], process.cwd(), { maxBuffer: buffer.constants.MAX_STRING_LENGTH + 1 }), + ).rejects.toThrow(RangeError); +}); diff --git a/packages/coding-agent/test/exec-utf8-chunks.test.ts b/packages/coding-agent/test/exec-utf8-chunks.test.ts new file mode 100644 index 000000000..5dad0aae0 --- /dev/null +++ b/packages/coding-agent/test/exec-utf8-chunks.test.ts @@ -0,0 +1,53 @@ +import type * as ChildProcessModule from "node:child_process"; +import { ChildProcess, spawn } from "node:child_process"; +import { PassThrough } from "node:stream"; +import { setImmediate } from "node:timers/promises"; +import { expect, it, vi } from "vitest"; +import { execCommand } from "../src/core/exec.ts"; + +vi.mock("node:child_process", async (importOriginal) => ({ + ...(await importOriginal()), + spawn: vi.fn(), +})); + +// #11360: real streams control chunk boundaries without relying on OS pipe timing. +// Only process creation and termination are stubbed; decoding and process waiting are real. +it.each(["stdout", "stderr"] as const)( + "discards pending %s UTF-8 bytes when either stream exceeds the cap", + async (stream) => { + const child = new ChildProcess(); + const stdout = new PassThrough(); + const stderr = new PassThrough(); + child.stdout = stdout; + child.stderr = stderr; + vi.mocked(spawn).mockReturnValueOnce(child); + vi.spyOn(child, "kill").mockImplementation(() => { + queueMicrotask(() => { + child.emit("exit", null, "SIGTERM"); + child.emit("close", null, "SIGTERM"); + }); + return true; + }); + + const resultPromise = execCommand("test-command", [], process.cwd(), { maxBuffer: 3 }); + const output = stream === "stdout" ? stdout : stderr; + const otherOutput = stream === "stdout" ? stderr : stdout; + // The decoder first returns "a" and buffers the first byte of €. + output.write(Buffer.from([0x61, 0xe2])); + await setImmediate(); + // The other decoder also buffers the first byte of € before the shared cap is exceeded. + otherOutput.write(Buffer.from([0xe2])); + await setImmediate(); + output.write(Buffer.from([0x82, 0xac])); + + const result = await resultPromise; + expect(result).toMatchObject({ + stdout: stream === "stdout" ? "a" : "", + stderr: stream === "stderr" ? "a" : "", + truncated: true, + killed: true, + }); + expect(result.code).not.toBe(0); + expect(Buffer.byteLength(result.stdout) + Buffer.byteLength(result.stderr)).toBeLessThanOrEqual(3); + }, +); diff --git a/packages/coding-agent/test/exec.test.ts b/packages/coding-agent/test/exec.test.ts index 96232a8b3..df5a63817 100644 --- a/packages/coding-agent/test/exec.test.ts +++ b/packages/coding-agent/test/exec.test.ts @@ -1,3 +1,4 @@ +import { constants as bufferConstants } from "node:buffer"; import { describe, expect, it } from "vitest"; import { execCommand } from "../src/core/exec.ts"; @@ -9,6 +10,7 @@ describe("execCommand", () => { expect(result).toMatchObject({ code: 1, killed: false, + truncated: false, stdout: "", }); expect(result.stderr).toMatch(/ENOENT/); @@ -17,6 +19,110 @@ describe("execCommand", () => { it("keeps stderr empty when a command exits silently", async () => { const result = await execCommand(process.execPath, ["-e", "process.exit(1)"], process.cwd()); - expect(result).toEqual({ stdout: "", stderr: "", code: 1, killed: false }); + expect(result).toEqual({ stdout: "", stderr: "", code: 1, killed: false, truncated: false }); }); + + // Regression tests for https://github.com/earendil-works/pi/issues/11360 + it.each([ + -1, + 0.5, + Number.NaN, + Number.POSITIVE_INFINITY, + Number.NEGATIVE_INFINITY, + bufferConstants.MAX_STRING_LENGTH + 1, + ])("rejects invalid maxBuffer %s instead of attempting execution", async (maxBuffer) => { + // A missing command normally resolves to a failed result; invalid options must reject instead. + await expect( + execCommand("pi-command-that-does-not-exist-11360", [], process.cwd(), { maxBuffer }), + ).rejects.toThrow(RangeError); + }); + + it.each([0, bufferConstants.MAX_STRING_LENGTH])("accepts maxBuffer boundary %s", async (maxBuffer) => { + const result = await execCommand(process.execPath, ["-e", ""], process.cwd(), { maxBuffer }); + + expect(result).toEqual({ stdout: "", stderr: "", code: 0, killed: false, truncated: false }); + }); + + it("truncates all output when maxBuffer is zero", async () => { + const result = await execCommand(process.execPath, ["-e", 'process.stdout.write("x")'], process.cwd(), { + maxBuffer: 0, + }); + + expect(result).toMatchObject({ stdout: "", stderr: "", truncated: true, killed: true }); + expect(result.code).not.toBe(0); + }); + + it("caps combined output at maxBuffer, terminates the command, and reports truncation", async () => { + const script = + 'process.stderr.write("e".repeat(600)); process.stdout.write("o".repeat(600)); setInterval(() => process.stdout.write("x".repeat(1000)), 10);'; + const result = await execCommand(process.execPath, ["-e", script], process.cwd(), { maxBuffer: 1000 }); + + expect(result).toMatchObject({ truncated: true, killed: true }); + expect(result.code).not.toBe(0); + expect(Buffer.byteLength(result.stdout) + Buffer.byteLength(result.stderr)).toBe(1000); + }); + + // #11360: a byte cap must not turn a partial UTF-8 character into replacement output. + it.each([ + ["stdout", "€", 1, ""], + ["stderr", "€", 1, ""], + ["stdout", "a€🙂", 5, "a€"], + ["stderr", "a€🙂", 5, "a€"], + ] as const)( + "keeps complete UTF-8 characters from %s output %s capped at %i bytes", + async (stream, output, maxBuffer, expected) => { + const script = `process.${stream}.write(${JSON.stringify(output)})`; + const result = await execCommand(process.execPath, ["-e", script], process.cwd(), { maxBuffer }); + + expect(result).toMatchObject({ [stream]: expected, truncated: true, killed: true }); + expect(result.code).not.toBe(0); + expect(Buffer.byteLength(result.stdout) + Buffer.byteLength(result.stderr)).toBeLessThanOrEqual(maxBuffer); + }, + ); + + it("still flushes incomplete UTF-8 at normal EOF when output was not truncated", async () => { + const script = "process.stdout.write(Buffer.from([0xe2])); process.stderr.write(Buffer.from([0xf0]));"; + const result = await execCommand(process.execPath, ["-e", script], process.cwd()); + + expect(result).toEqual({ stdout: "�", stderr: "�", code: 0, killed: false, truncated: false }); + }); + + it("leaves output up to maxBuffer unchanged", async () => { + const result = await execCommand( + process.execPath, + ["-e", 'process.stdout.write("a".repeat(1000))'], + process.cwd(), + { + maxBuffer: 1000, + }, + ); + + expect(result).toEqual({ stdout: "a".repeat(1000), stderr: "", code: 0, killed: false, truncated: false }); + }); + + it.skipIf(process.platform === "win32")( + "returns when a shell's child keeps writing after the shell is terminated", + async () => { + // SIGTERM kills only `sh`; `yes` keeps the output pipes open and writing. + const result = await execCommand("sh", ["-c", "yes; echo done"], process.cwd(), { maxBuffer: 1000 }); + + expect(result).toMatchObject({ killed: true, truncated: true }); + expect(result.stdout).toBe("y\n".repeat(500)); + }, + 10_000, + ); + + it.skipIf(process.platform === "win32")( + "force-kills a command that ignores SIGTERM", + async () => { + // The handler is installed before the first write, so the kill triggered by maxBuffer + // cannot arrive before the child ignores SIGTERM. + const script = 'process.on("SIGTERM", () => {}); process.stdout.write("xx"); setInterval(() => {}, 1000);'; + const result = await execCommand(process.execPath, ["-e", script], process.cwd(), { maxBuffer: 1 }); + + expect(result).toMatchObject({ stdout: "x", killed: true, truncated: true }); + expect(result.code).not.toBe(0); + }, + 15_000, + ); }); diff --git a/packages/coding-agent/test/git-merge-and-resolve-extension.test.ts b/packages/coding-agent/test/git-merge-and-resolve-extension.test.ts index 7c8741c7e..7903a4078 100644 --- a/packages/coding-agent/test/git-merge-and-resolve-extension.test.ts +++ b/packages/coding-agent/test/git-merge-and-resolve-extension.test.ts @@ -7,8 +7,8 @@ import type { ExecResult, ExtensionAPI, ExtensionContext } from "../src/core/ext type AgentEndHandler = (event: { type: "agent_end" }, ctx: ExtensionContext) => Promise; -const ok: ExecResult = { stdout: "", stderr: "", code: 0, killed: false }; -const fail: ExecResult = { stdout: "", stderr: "error", code: 1, killed: false }; +const ok: ExecResult = { stdout: "", stderr: "", code: 0, killed: false, truncated: false }; +const fail: ExecResult = { stdout: "", stderr: "error", code: 1, killed: false, truncated: false }; /** Standard exec results for a clean repo tracking origin/main, not in a merge. */ function withUpstream(results: Map): Map { diff --git a/packages/coding-agent/test/input-transform-streaming-example.test.ts b/packages/coding-agent/test/input-transform-streaming-example.test.ts index faf95b6ba..0eb1ccf60 100644 --- a/packages/coding-agent/test/input-transform-streaming-example.test.ts +++ b/packages/coding-agent/test/input-transform-streaming-example.test.ts @@ -35,9 +35,9 @@ function setup(execResult: ExecResult) { describe("input-transform-streaming example", () => { const diffOutput = " src/index.ts | 5 ++---\n 1 file changed, 2 insertions(+), 3 deletions(-)"; - const gitSuccess: ExecResult = { stdout: diffOutput, stderr: "", code: 0, killed: false }; - const gitEmpty: ExecResult = { stdout: "", stderr: "", code: 0, killed: false }; - const gitFail: ExecResult = { stdout: "", stderr: "not a git repo", code: 128, killed: false }; + const gitSuccess: ExecResult = { stdout: diffOutput, stderr: "", code: 0, killed: false, truncated: false }; + const gitEmpty: ExecResult = { stdout: "", stderr: "", code: 0, killed: false, truncated: false }; + const gitFail: ExecResult = { stdout: "", stderr: "not a git repo", code: 128, killed: false, truncated: false }; it("skips exec during steering", async () => { const { emit, exec } = setup(gitSuccess); From 556860a386a03113eca146abd9cbb08e9391b7fb Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Wed, 28 Oct 2026 23:16:28 +0000 Subject: [PATCH 006/175] docs(chord): no client providers for now --- packages/chord/docs/next/decisions.md | 9 +++++++++ packages/chord/docs/next/design.md | 4 +++- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index f425d5bbf..59893c07b 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -1771,6 +1771,15 @@ validation and copies the v2 kernel added around Delta. setups a registry without a value cannot get one before a reload, so a reader then fails at once `transient` (`unavailable/undeclared`) and is retried when the registry gets its first value (R18), as `self.get` of a type not served here fails at once (refined 2026-10-27 after the second spec review: no `readyTimeout` wait at start); once live, a later failure of the registry changes nothing (last good value). +- **No client providers for now (owner, 2026-10-28: "yeah, no client providers for now"; reviews + `/tmp/barrier-{astra,fable,opus,sol}.md`):** clients stay unaddressable and serve only `local` types (design §3). + Client-side capabilities (a phone's camera, clipboard, a page's WebMCP tools) go through an owner's state, as the + Questions service: a tool asks an owner store (`request`, `wait`), clients watching its open requests carry them out + and answer with a method call (the answer is method arguments, checked against the owner's schema; bytes as `bytes()` + leaves, chunked if large); a client may advertise the tools it can run to such a store, tied to its presence. Why: + a client as a provider would feed untrusted results, state and ops into server and worker objects (consumers never + validate them), needs client addresses, routing down through relays and a principal for clients; the owner pattern + needs nothing new in Chord and survives link drops and several devices by the owner's rule. - **Design audit of slices 1-6a (owner, 2026-10-27; `/tmp/design-audit-{opus,sol}.md`):** the built code follows the design except for five bugs, fixed after 6b's commit with the auditors' reproductions as tests: on a failed activation references are released after the natives closed (§6, §13); a dependency going live while the dependent diff --git a/packages/chord/docs/next/design.md b/packages/chord/docs/next/design.md index b3ec994d1..dc38b5642 100644 --- a/packages/chord/docs/next/design.md +++ b/packages/chord/docs/next/design.md @@ -89,7 +89,9 @@ worker worker worker addressable by a typed address, started on demand | worker | yes, many | stamped by the parent | started on demand by the app | Clients serve only `local` types (they are not addressable): a host without a principal refuses to serve a remote type -(`load` throws; a remote object is owned by a principal of the host serving it; owner, 2026-10-24). A client's local +(`load` throws; a remote object is owned by a principal of the host serving it; owner, 2026-10-24). No client providers for now (owner, +2026-10-28): client-side capabilities (a camera, WebMCP tools) go through an owner's requests that clients answer +(section 20, the Questions service). A client's local calls have no principal; what a client calls on the server is stamped with the client's authenticated principal. Clients reach workers only through the server, which relays; workers reach each other through the server too. A detached process (e.g. a terminal host) would be another child kind of the server. Things that are not hosts: a code From e65b3b88c08412bb8ffbd21fb5dc5756064a6e5a Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 00:51:57 +0000 Subject: [PATCH 007/175] docs(chord): dependency model proposal for slice 7 and its explainer (Parked Until Resolved) --- packages/chord/docs/next/dependencies/book.md | 1674 +++++++++++ .../chord/docs/next/dependencies/index.html | 2647 +++++++++++++++++ .../chord/docs/next/dependencies/model.md | 419 +++ 3 files changed, 4740 insertions(+) create mode 100644 packages/chord/docs/next/dependencies/book.md create mode 100644 packages/chord/docs/next/dependencies/index.html create mode 100644 packages/chord/docs/next/dependencies/model.md diff --git a/packages/chord/docs/next/dependencies/book.md b/packages/chord/docs/next/dependencies/book.md new file mode 100644 index 000000000..19b550bdf --- /dev/null +++ b/packages/chord/docs/next/dependencies/book.md @@ -0,0 +1,1674 @@ +--- +title: Parked Until Resolved: one model for dependencies in Chord v2 +cover: Parked / Until Resolved +kicker: A design proposal · Chord v2 · slice 7 · 2026-10-29 +subtitle: What happens when what an object needs is not there yet, is somewhere else, has gone away or does not fit, told through the scenarios that decide it +stats: + - 5 → 1 | answers today to "the provider is not there" + - 0 | deadlines on waiting for a dependency + - 300+ | scenarios played through by four reviewers + - 6 | decisions left for the owner +note: | + A proposal, not a decision. It states the dependency model the redesign session drafted on the night of + 2026-10-28/29 from the owner's direction ("park until resolved"; "no difference between local and remote + resolution: a link going is like a plugin unloading") and five rounds of review by Astra, Fable, Opus and Sol, who + played more than 300 scenarios from p2's plugins and their own against it. The working document is + `/tmp/model/model.md`; the reviews are in `/tmp/dep-graph-*.md`, `/tmp/barrier-*.md` and `/tmp/model/review*-*.md`. + Code references are to pi `main` at `358f7d46a` (`packages/chord/src/next`). Notes marked *review* record a point + the reviewers found or still disagree on; *Owner* marks a decision that is his. +colophon: Parked Until Resolved · first edition · 2026-10-29 · Chord v2 design proposal for slice 7 +--- + +# Before we begin {.preface} + +It is a Tuesday morning and Mario opens a session on his phone. The server starts a worker for it. The worker +connects to the server, loads its plugins and opens the agent's harness. One of the plugins, code mode, needs the +harness before it can contribute its tools. Another, search, needs a feed that lives on the server. A third, sharing, +was removed from the worker's plugin set last night, but the notes view still asks for it. At the same moment the +browser tab holds the session's notes, which live on the worker that is still starting. + +Every one of these is the same question asked in different places: *what I need is not there yet; what now?* Chord +v2 answers it today in five different ways, depending on how the dependency was declared and where its provider +lives. A missing type on the same host breaks the object for good; the same type missing on another host makes it +retry every few seconds forever; a missing registry waits for a value; a reload that removes a type ends every +reference to it. And when two hosts start at once and need each other's objects in a chain that has no cycle at all, +both wait until a 60-second deadline breaks the knot. + +The owner's direction was short: an object should be **parked until its dependencies resolve**, and there should be +no difference between a dependency on this host and one on another: a link going away is like the provider's plugin +being unloaded, and a link coming back is like it being loaded. This book works out what that means, rule by rule, +and plays it through the scenarios that decide whether it holds. + +## Eight things to know + +:::findings +1. **A request outlives its provider.** A reference stands for a request (type, params, principal, version). It ends + only when its holder ends it, never because the provider is missing, went away or does not fit. +2. **One resolver for local and remote.** The address decides where a request goes; the provider found there is + checked; a missing provider here and a missing provider on another host give the same answer. +3. **Two kinds of not-yet.** Something under way (activating, reconnecting, installing, not published yet) shows as + `loading`; nothing under way (not served, no fitting version, a dependency failed) shows as `failed` with the new + cause `waiting`. +4. **Dependents park.** An object whose dependencies are not all live waits with no timer and no backoff, and is + woken by events: a dependency changing state, or a name becoming usable on a host. +5. **`broken` means the object's own fault.** Only its own code and cycles break an object; whatever it waits for is + `waiting`. +6. **Startup needs one short gate, not a barrier.** Requests wait only for the host's facets to be published, which + is synchronous; that ends the startup deadlock. Whether `load` then also waits for first outcomes is the owner's + call. +7. **One gap in cycle detection needs a choice.** Objects on two hosts that start at the same moment and need each + other are caught today only by a deadline. A probe can walk the waiting edges and find the ring without failing + anything (recommended); a narrowed deadline is simpler but polls, misses a ring, or never finds the cycle at all. +8. **Only callers and own work keep clocks.** Getters, parked calls and an object's own steps keep their deadlines; + waiting for a dependency has none. +::: + +## Words used throughout + +A **host** is one participant (the server, a session worker, a browser tab). A **provider** is what serves a type or +a registry on a host: a facet registered it with `host.serve`. A **request** is what a reference stands for: a type +or registry, its params, the principal it acts as and the consumer's contract version. Its **holder** made it and +owns it: an object's attempt (`self.get`, `self.registry`), a call (`ctx.get`), outside code (`host.get`, +`host.hold`) or a peer host (a `hold` arriving over a link). A request is **bound** when it points at an object (here) +or a hold (over a link), and **unbound** with a reason otherwise. An **attempt** is one try at activating an object: +its factory, its requests, its natives. A **dependency** is a request an attempt needs live before it may go live. An +attempt is **parked** while one of its dependencies is not live. A **publication** is the moment a host makes its +facets' declarations visible: the setups of `load`, the commit of a reload, the commit of an install. A **wake** is an +event that makes the resolver look at a request again. + +The running example is a small p3: a server **S** with the session catalog and the search feed; a worker **W** for +session `S1` with the harness, code mode, the notes store and the search indexer; and a browser tab **T** that shows +the notes. The plugins are `core`, `models`, `codemode`, `notes`, `search` and `sharing`. + +# Five Answers to One Question {#today lede="What Chord v2 does today when a dependency is missing, and why the answers disagree."} + +Chord v2 was built slice by slice, and each slice answered "the provider is not there" for the case in front of it. +Slice 1 made `self.get` of a type nobody serves throw, because a typo in a type name should fail loudly. Slice 3b made +remote holds survive link drops, so a missing provider on another host became a refused hold that the consumer retries. +Slice 8 made `self.registry` a dependency that waits for a value. Each answer was reasonable alone. Together they +give five outcomes for one situation. + +| Declared as | The provider is missing | Today | +| --- | --- | --- | +| `self.get(LocalType)` | no facet serves it | `self.get` throws, so the factory throws: the object is `broken` | +| `self.get(RemoteType)` | the worker does not serve it | the hold is refused `broken`; the dependent is `transient` and retried every 1 to 60 seconds, forever | +| `self.registry(R)` | no plugin serves `R` | `transient`, woken at `R`'s first value only | +| `self.registry(R)`, the reader newer than the provider | | `broken` | +| a reload removes the type | | every holder gets `gone`: ended for good | + +The differences are not about the situation. Whether the notes plugin is missing on this host or on the worker, the +notes view cannot work until someone installs it. Yet in one case the view is broken until a reload, in the other it +polls the worker every minute for the rest of the session, and the reload that finally brings the plugin back cannot +revive a reference that got `gone`. + +## The deadlock without a cycle + +The startup barrier has a worse problem. Sol reproduced it against the current code (`/tmp/dep-graph-sol-barrier.log`). + +:::sequence animate | The startup deadlock today. Nothing in the object graph is cyclic; the two barriers block each other. +actor s "server S" "load: waits for A" +actor w "worker W" "load: waits for BStore" +s -> s: setups: serve C; start eager A +s => w: A needs B: hold B +w -> w: setups: serve B; start eager BStore +w => s: BStore needs C: hold C +s -> s: Hold.start waits for S.pendingLoad +note s..w: S's load waits for A, A for B, B for BStore, BStore for C, C's hold for S's load +divider "60 s later" +s -> s: A's activation deadline fires, A fails +s --> w: C binds, BStore and B go live, A is retried +::: + +`C` has no dependencies and is registered the moment S's setups ran. Its hold waits anyway, because incoming holds +wait for `pendingLoad`, and `pendingLoad` is the whole of `load`: the setups and the barrier that waits for every +host-lifetime object, `A` included. Two jobs share one promise. Holds only need the first ("are the types +registered?"); the second ("has every plugin had its first try?") is for the app. Mixing them makes an acyclic chain +across two hosts wait until a timeout fires. + +## Timers standing in for events + +Both problems have the same root. Where Chord does not know when something will change, it polls or waits for a +deadline: the identity backoff retries a dependent whose remote provider refused it; the activation deadline breaks the +startup knot; the same deadline is the only answer to two objects on two hosts that start at the same moment and need +each other, a cycle the activation chains cannot see (design §6, "Known limitation"). + +A timer is the right tool when nothing else can tell Chord that the world changed: a disk that may be back, a worker +process that may start this time, a silent transport. It is the wrong tool when an event exists or can exist. A +missing plugin comes back through a reload, which is an event on the host that serves it. A dependency going live is +an event. A link opening is an event. The model below uses those events and keeps timers only where no event can +exist. + +# Requests That Wait {#requests lede="A reference is a request with a holder; it outlives every provider it meets."} + +The first change is a separation the code makes only halfway today: between what a reference **asks for** and what +currently **answers** it. When the notes view on W does `self.get(NoteStore, {})`, it asks for "the NoteStore of this +host, as the host, at contract version 3". Which object answers is a second question, and the answer can change: no +object yet, an object activating, a live one, a failed one, none again after a reload, a new one after the next. + +## A request and its holder + +A **request** is the asking part: type (or registry), params, principal and the consumer's contract version. Every +request has exactly one **holder**, which made it and is the only one that ends it. + +| Holder | Made by | Ends when | +| --- | --- | --- | +| an attempt | `self.get`, `self.registry`, a keyed pin `ref.notes.at(key)` | the attempt ends (its object ended, or a reload replaced it) | +| a call | `ctx.get` | the call settles | +| outside code | `host.get`, `host.hold`, `host.registry` | `release()`, or the getter gives up | +| a peer host | a `hold` arriving over a link | the peer releases it, or its link or origin closes | +| any of them | | revocation of its principal; its own host closes | + +That list is the whole of how a request ends. **A request never ends because its provider is missing, went away or +does not fit.** Today it does in two places: a `self.get` of a type nobody serves throws before the request exists, +and a reload that removes a type sends `gone`. Both go. + +## The address decides where + +A request goes to one place, and the address decides which. For a `local` type, the place is this +host. For a `remote` type, the consumer computes the address from the params with its own contract's +placement function (design §3): `Notes { session: "S1" }` is placed on `Session { id: "S1" }`. If that is this host's +address, the request goes here; otherwise over the route for that kind, or over the parent link. + +Today the code checks something else first: whether the type is registered here. A type placed on this host but not +registered yet is sent over the parent link, which is wrong twice. Sol's example: the server has address `Server`, +the catalog type is placed on `Server`, and no facet serves it yet. Today's `Placement.reference` sends the server's +own request over a parent link the server never had. And when the catalog plugin is installed later, nothing wakes +that request, because it was never registered as waiting here. + +:::flow wide | Placement by address. The destination follows from the address alone; only then is the provider found there checked. +row "Where" accent +box addr "the request's address" "from the consumer's own placement function" accent +box here "here" "a local type, or this host's address" +box away "a route or the parent" "every other address" blue +row "Then" muted +box check "check the provider here" "served? fitting version? authorize, own placement" +box hold "hold over the path" "the provider host checks it the same way" blue +arrow addr -> here +arrow addr -> away +arrow here -> check +arrow away -> hold +::: + +The rule, from the reviews (Sol B1, Astra F1): placement never looks at what is registered when the type names an +address. A request placed here whose provider is missing is **unbound here**, waiting for the name. A version check +compares the consumer with the provider actually selected, never with a same-name registration on the consumer's own +host (today `#assertRoute` does that before placement, so worker W1 serving `Notes` v1 rejects a `Notes` v2 request +placed on W2, which serves v2). + +One case keeps today's rule: a `remote` type declared **without placement** (allowed by `remote()`, used in tests and +for server types a tab reaches through its parent). It goes here if it is served here, otherwise over the parent. A +request of that kind made before the host has a parent link (a tab loads before it connects) waits here as not served +and is placed again when the host's first parent link connects (Opus, Sol); on a host that never connects, the root, +it stays here. Opus would rather require a placement on every remote type, which is simpler; it removes a supported +declaration, and the owner asked for no feature to go, so the draft keeps the fallback. + +## What the check finds + +Once placed, the provider found there is checked: served, a fitting version (the current one or an accepted older one, +with the fingerprint computed from the provider's own history), `authorize`, and the provider's own placement check. +The request is then **bound** (to an object here, or to a hold over a path), or **unbound** with a reason: + +| The check finds | The request | Shows | +| --- | --- | --- | +| a fitting provider | bound | its object's state | +| this host has not published its facets yet | unbound `unpublished` | `loading` | +| nobody serves the name at its address | unbound `not_served` | `failed { waiting }` | +| no fitting version, a fingerprint that differs at the same version, a placement the provider computes elsewhere | unbound `version` | `failed { waiting }` | +| the registry is served, but its first `compose`, or the reader's version mapping, threw | unbound `compose` | `failed { waiting }` | +| the provider host is installing, and the hold does not fit its current code | unbound `installing`, at the provider | `loading` | +| the path has no link (a drop, a route resolving) | waits on the path | `loading` | +| `authorize` refused, or the params are invalid | refused | `failed { broken }`, as today | + +The reasons that wait for a name (`unpublished`, `not_served`, `version`, `compose`) are kept in an index under that +name, on the host that decided them: the consumer's host for a local request, the provider's host for a hold that +arrived over a link. The provider already keeps such holds today as **refused** (`remote/hold.ts`), and `retry` re-runs +them. `installing` belongs to the provider host's own install; `no_link` lives on the path, where the next link holds +the request again, as today. Every index drops an entry when its request ends, so a released getter, a closed tab or a +revoked guest leaves nothing behind. A refused request (no authority, bad params) is never indexed: a later +publication cannot grant authority. + +## The publication gate + +There is one short gate. Before a host has run its facets' setups, it does not know what it serves, so it cannot tell +"not served" from "not registered yet". A worker connects to its parent before it loads (W6), so holds can arrive in +that window, and p2's web shell holds references before its own `load` resolved. + +So a request made on a host, or arriving on its links, before the host's **first publication** is not placed at all: +it is unbound `unpublished` and shows `loading`. The publication places it. Without this gate the request would be +placed by a half-empty registration table (Fable's trace: W1's own `Notes` sent over the parent, relayed back by the +server, forever), and a tab would see "not served" during every worker start (Opus's trace). + +The gate is not the barrier. It lasts only until the setups ran, and the setups are synchronous code at the start of +`load`: they register types and registries and contribute, and wait for nothing. That is the difference that ends +the startup deadlock (chapter VI). + +# Loading or Waiting {#states lede="What a reference shows while it is not live, and why a missing plugin is an answer rather than a spinner."} + +`ref.state` keeps its four kinds: `loading`, `live`, `failed`, `ended`. What changes is one new kind of failure cause. + +```ts +type RefState = + | { kind: "loading" } + | { kind: "live" } + | { kind: "failed"; cause: FailureCause; retryAt: number | undefined } + | { kind: "ended"; error: ChordError }; + +type FailureCause = + | { kind: "transient"; error: ChordError } // the object's own effect failed (an open, a follow): backoff + | { kind: "broken"; error: ChordError } // the object's own code, a cycle, or authority: retry() or a reload + | { kind: "waiting"; error: ChordError }; // not the object's fault: woken by an event, never by a timer +``` + +## The mapping + +Whether a request that is not live shows `loading` or `failed { waiting }` follows from what it waits for: + +| The request is | `ref.state` | Calls on it | +| --- | --- | --- | +| bound, its object activating, or parked with no failed blocker | `loading` | park, bounded | +| unbound `unpublished` or `installing`, or its path has no link | `loading` | park, bounded | +| unbound `not_served`, `version` or `compose` | `failed { waiting }`, no `retryAt` | reject at once with the cause | +| bound, its object failed | `failed` with the object's own cause and `retryAt`, as today | reject at once | +| bound, its object live | `live` | run | +| ended by its holder, revoked, or its host closed | `ended` | reject | + +The intuition is "is something under way?": an activation, a reconnect, an install, a load. When something is under +way, the reference waits and calls park; when nothing is, the reference says what is missing and calls fail fast. +The reviewers pointed out that "under way" is not something Chord can observe in general (a parent reconnect is the +app's, a `resolve` may hang), so the table is the rule and the intuition only explains it. + +## A link drop and an unload, side by side + +The owner's point was that a link going away is like the provider's plugin unloading. For the request they are the +same: it stays, it keeps its last values readable, it binds again when a provider is reachable, and its holder +notices nothing but the state. They differ in what is under way. + +:::steps | The notes view on T, once through a link drop and once through an unload on W ++++ Live +T's notes view holds `Notes { S1 }` on W through the server. `ref.state` is `live`; the view renders the notes. ++++ The link drops +W's link to the server closes (a crash, a network change). The request on T stays; the path has no link: unbound +`no_link`, `ref.state` `loading`. The last notes stay readable. A call made now parks, bounded by `readyTimeout`. +Calls already sent fail and are never resent: their outcome is unknown. ++++ The link comes back +The route resolves again, the request is held anew on the new link, `bound` arrives with a fresh snapshot. `live`. +Parked calls go out in order. ++++ The plugin is removed on W +A reload on W removes the `notes` plugin. The provider's object ends as today: its natives close, its callers are +rejected at once, its handlers drain. The request on T does not end: W sends `failed { waiting, not_served }` on +the bound hold, which unbinds it. `ref.state` is `failed { waiting }`; the last notes stay readable; new calls fail at once with "notes is +not served on Session S1". ++++ The plugin comes back +A later reload on W serves `Notes` again. The publication re-runs W's refused holds for that name: `loading`, then +`bound`. T's view is live again without having done anything. +::: + +One difference must stay: calls already sent. When a link drops, a sent call's outcome is unknown, so it fails and +is not resent. When a plugin unloads on the same host, its handlers are drained on the old code (design §21). Location +decides nothing else. + +:::review | Review: a third cause kind, or not +Opus and Fable recommend `waiting` as above. Astra would keep two kinds and mark event-waited failures as `transient` +with no `retryAt` (today's wire already carries a `transient` cause without `retryIn`, `remote/remote-object.ts`), so +no kind is added; the cost is that `transient` then means two things, a backoff under way or an event awaited. Sol would show `loading` with a diagnostic; that parks every call on a missing plugin for 60 +seconds and spins forever in the UI. The draft recommends `waiting`. **Owner** decision 3. +::: + +# Parked {#parked lede="An object whose dependencies are not all live waits, with no timer, and goes on the moment they are."} + +The owner's instinct was that an object is parked until its dependencies all cleanly resolve. This chapter makes that +exact: what a dependency is, what the parked object shows, what it keeps, and where parking ends. + +## What counts as a dependency + +A **dependency** is a request an attempt makes in its factory and needs live before the object may go live: + +- `self.get(Type, params)`, local or remote: one rule for both; +- `self.registry(R)`: a registry at the reader's version (chapter X); +- a keyed pin, `ref.notes.at(key)` on another object's keyed member: the key must be loaded. + +Two things are not dependencies. `self.follow(attach)` is the attempt's own step: its `attach` runs Durable's code +and may fail like an `open`, with the object's own `transient` failure and backoff (a follower that wants to recover +the moment a new harness appears declares `self.retryOn` on the registry it reads the harness through). And a +`ctx.get` in a method is not part of the object's activation at all: it is the call's request. + +## The dependency check + +An attempt runs its factory, hydrates its persisted singletons (they are the object's documents, kept across +attempts, as today), makes its requests, and is **parked** while any dependency is not live (and hydrated). What it +shows follows its dependencies: + +| Its dependencies | The parked object shows | +| --- | --- | +| any `loading`, none failed | `loading` | +| any `failed` | `failed { waiting, "waiting for X: " }`, no `retryAt`; the message keeps the nearest blocker and the root cause within the 500-character rule | +| one failed with `invalid/cycle` (a dependent off the ring) | `broken` with the same error: it would wait on a ring that never heals by itself | +| one `ended` (revoked, or its host closed) | `broken` with that error: the request cannot come back; `retry()` or a reload makes new requests | + +When every dependency is live, the attempt passes the **dependency check** and runs its own steps: its natives in +declaration order, its follow attaches, the member checks, its derives. Those are +bounded by its own `readyTimeout` deadline, as today. Then it goes live. + +The dependency check is the boundary between waiting and working (Astra's F3). Before it, the attempt holds no +natives; after it, a dependency that leaves `live` is treated as after live (R18): the attempt goes on, and its calls +through that reference meet the reference's state. So a parked attempt never holds a native, and an attempt that +opened a native is never parked. This is a deliberate rule, and Astra called it what it is: a weakening of "live +means every dependency is live". An object can finish opening after a dependency went away. Without that boundary a contributor unloading while the attempt opens its natives +would leave it half open and parked, or force a seal, a drain and a fresh attempt for a dependency the object may not +even use at once. + +:::sequence animate | Code mode on W parks on the harness and goes on the moment it is live. +actor cm "CodeMode" "host-lifetime, local" +actor h "Harness" "host-lifetime, local" +actor m "HarnessModels" "registry" +cm -> h: self.get(Harness): loading +note cm: parked: shows loading, no timer +h -> m: self.registry(HarnessModels): usable +h -> h: own steps: open Durable +h => cm: Harness live: a state change +cm -> cm: dependency check passes; open the sandbox +cm -> cm: live; contribute tools +::: + +## No clock while parked + +A parked attempt has no dependency deadline and no backoff. Its `readyTimeout` deadline is paused while it is +parked, as eager migrations already pause it (`attempt.ts`, `deadline.pause()`), and resumes with the budget left when +the attempt passes the check. Today a dependency that does not go live within 60 seconds fails its dependent +`transient`, which then retries with backoff from 1 to 60 seconds. Both go: the dependent is woken when the dependency +changes, and the time a dependency takes is bounded where it runs, by that dependency's own steps. + +## What a parked attempt keeps + +It keeps its documents (the object's), its declarations and its requests (R18's holding today), and its seal token. +It keeps no native. It ends with its object: when its last holder releases it, or, for a host-lifetime object, when +its plugin is removed or the host closes. A parked object +with params that nobody holds any more ends and releases its requests, so a whole parked chain goes when its top is +released. + +## `broken` is the object's own fault + +After the model, `broken` means exactly three things: the object's own code failed (a throwing factory, a derive that +throws at activation, a hydration or `migrate` that fails, a document fingerprint mismatch, an `open` that throws +`broken(...)`), the object is on a cycle, or it lost its authority (`forbidden`, a dependency that ended). Everything an object waits for, including a provider that is itself +broken, makes it `waiting`. The difference matters for recovery: a broken object needs `ref.retry()` or a reload of its +own plugin; a waiting one recovers by itself when what it waits for recovers. + +## Retry passes on + +A Retry button on a waiting object should do something. Opus's trace: the tab's `Panel` waits for the server's `Dash`, +which waits for `Catalog`, which is broken by a factory bug that a configuration fix would heal at the next attempt. +The user sees "waiting for Dash: waiting for Catalog: …" and presses Retry. Today `Panel` would re-check, find `Dash` +still failed and do nothing. + +So `ref.retry()` on a `failed { waiting }` object retries its failed blockers, hop by hop, with the existing `retry` +message over links: it retries the blockers that are `broken` or `transient` (the only ones a retry can change), +passes through the `waiting` ones, and leaves the rest alone. Each retry carries an operation id, so a diamond of +dependencies retries a shared leaf once. Retry of a `broken` or `transient` object starts a new attempt, as today. A +`retryOn` value call still retries a failed object as slice 8 decided; while the object is parked `waiting`, it does +nothing: its dependencies decide. + +## A reload of the parked object itself + +The parked attempt's factory never runs again while it is parked, with one exception: a reload of the object's own +plugin replaces the parked attempt with the new factory's attempt, as one handover: the new requests are made and +bound first, then the old ones released (Astra: a dependency held only by this object, unchanged by the reload, must +not end in between; today's `#activate` already binds before `#releasePrevious`). If the new code no longer declares +the missing dependency, the object can go live at once. + +## After live, nothing changes + +R18's second half stays: once an object is live, a dependency that leaves `live` (a link drop, a removal) leaves the +object live. Its calls that use the dependency get the dependency's state: they park while it is `loading` and reject +while it is `failed`. Reactivating every live consumer on every dependency loss would turn one worker disconnect into +closed natives, withdrawn contributions and aborted jobs on the server; the reviewers agreed it should not. + +# What Wakes Whom {#wakes lede="Two kinds of event replace every poll and every dependency deadline."} + +A model without timers lives or dies by its wakes: a wake that is lost leaves an object parked forever. So the wakes +are kept to two kinds of event, each with one owner, and every unbound request and every parked attempt is +subscribed to exactly the events that can change it. + +| Event | Who is told | Exists today? | +| --- | --- | --- | +| **a target changes state**: an object goes live, loading or failed, or gets a new cause; a hold receives `bound`, `loading` or `failed`, also after a re-hold on a new link | the requests bound to it, and the attempts that depend on those requests | partly: `dependencyLive` covers only "goes live" | +| **a name becomes usable on a host**: a publication (`load`'s setups, a reload's commit, an install's commit) serves a type, a registry or a version; a registry composes or maps a value at a version | the unbound requests indexed under that name on that host; a refused hold re-runs its start and sends `loading`, then `bound` or `failed` | partly: refused holds and `retry` exist; the re-run at publication is new; a registry wakes only at its first value | + +## Every change, not only "live" + +Draft 1 woke a parked attempt only when a dependency went live. Astra, Fable and Sol found the same hole: a dependency +that moves from `loading` to `failed`, or from one cause to another, leaves its dependent showing a stale state, and a +dependent that should now show `failed { waiting }` keeps parking calls. So a parked attempt re-evaluates on every +state change of any of its dependencies and updates what it shows. It leaves parking only when all of them are live. +Its factory never re-runs on a wake. + +Three details keep this correct and cheap: + +- **Coalesced, after the batch.** Wakes run after the batch that caused them (a publication, a commit, an inbound + `bound`), once per object, so a commit that changes three dependencies wakes their dependent once, and no factory + runs inside another owner's batch. +- **Remembered.** A change that arrives while the attempt runs its own steps or tears down is remembered and applied + after (today's R18 wake memory, generalized). +- **Indexed by name.** A publication touches only the unbound requests waiting for the names it serves. A reload of + `notes` never looks at requests waiting for `search`. + +## Across a link: no new message + +The wake across hosts needs no new message kind. Today a provider keeps a hold it could not serve as refused, and a +`retry` from the consumer re-runs its start. The model adds one thing: the provider re-runs its refused holds itself, +at its own publication, for the names that publication serves. + +:::sequence animate | The search indexer on W waits for the server's feed, which a reload on S adds. +actor w "worker W" "Indexer, parked" +actor s "server S" "search plugin missing" +w -> s: hold SearchFeed +s --> w: failed { waiting, not_served } +note w: Indexer shows failed { waiting }; no timer, no retries +divider "a reload on S adds the search plugin" +s -> s: publication serves SearchFeed; re-run refused holds for it +s --> w: loading +s --> w: bound (snapshot) +w => w: a target changed state: Indexer re-evaluates, passes the check, goes live +::: + +Two existing messages change: `failed` carries the new cause kind `waiting`, and `loading` and a `waiting` `failed` +carry the provider's attempt id for the cycle probe (chapter VIII). The pinned list of causes and its test change with +them, and `PROTOCOL_VERSION` stays 1 until p3 ships. + +## Registries + +A registry reader is woken at every transition of the registry's value at its version to usable: when the registry +is served, when it composes for the first time, when an older reader's mapping succeeds again after it failed. Today +`RegistryValueSource` wakes only at the very first good value (`#valued`), which a registry whose provider is removed +and added again would never pass a second time. + +# Starting Up {#startup lede="The barrier did two jobs with one promise; separated, the deadlock disappears."} + +`load` does three things today: it runs every facet's setup, it publishes them (types served, registries composed +once, contributions in their tables), and it waits until every host-lifetime object has had its first outcome. The +promise of the whole thing, `pendingLoad`, is also what incoming holds and `host.hold` wait for. That is the startup +deadlock of chapter I. + +## Publication first + +The model splits `load` at the publication: + +1. **Setups.** Every facet's setup runs. Setups only declare: `host.serve`, `self.contribute`, `host.references`. + They are synchronous and wait for nothing. +2. **Publication.** All declarations become visible at once, in one batch: types served, registries served and + composed once, contributions in their tables. Nothing activates inside that batch. +3. **Wakes.** After the batch, every request that waited for the publication (unbound `unpublished`) is placed, and + every unbound request for a name the publication serves is re-evaluated. +4. **First outcomes.** The host-lifetime objects activate. Whether `load` waits for them is the barrier question below. + +Requests wait for step 2 only. Since the setups wait for nothing, a host always reaches step 2, and a request from +another host can never wait for this host's activations. Chapter I's knot comes apart: + +:::sequence animate | The same startup as in chapter I, under the model. No deadline is needed. +actor s "server S" "" +actor w "worker W" "" +s -> s: setups, publication: C served +s => w: A needs B: hold B +w -> w: setups, publication: B served +w => s: BStore needs C: hold C +s --> w: bound C (S has published; C activates and goes live) +w -> w: BStore live, B live +w --> s: bound B +s -> s: A live; S's first pass done +::: + +## The barrier: (a) or (b) + +What is left of the barrier is the app's question "has every plugin had its first try?". R17 needs it: p3's base code +resumes the harness after `load`, so the first turn sees the tools of every plugin that loaded. Two shapes are on the +table. + +**(a) Chord keeps it.** `load` resolves when every host-lifetime object of this `load` has a **first outcome**: it is +live, failed by its own cause, or parked with no blocker that is a local object still without its first outcome. In +code, per object: + +```ts +const firstOutcome = (o: Obj): boolean => + o.live || o.ownFailure || (o.parked && o.blockers.every((b) => !b.local || firstOutcome(b))); +``` + +So `CodeMode → Harness`, both local, waits until the harness had its outcome: code mode is parked on a local object +that is still in its first attempt, so code mode is not done. A wait on another host, or on a dependency that failed, +ends the first pass: the barrier never waits for another host, a retry, a later contribution or a reload. The signal +is separate from the readiness dependents wait for (Sol: resolving that for a parked object would let its dependents +open natives too early), and it is counted once per object, never reopened by retries. An object of the cohort that +ends before its first outcome (its plugin removed, the host closing) settles its entry as ended, and `host.close` +settles a `load` that is still waiting (Astra). + +**(b) The app keeps it.** `load` resolves after the publication. Chord adds `host.startup`, a watchable of the +host-lifetime objects and their first outcomes with causes; base code decides when to resume (Fable). The tracker is +the same; only who awaits it changes. + +Three of the four reviewers recommend (a): the activation graph is Chord's, and every app that wants R17's behaviour +would otherwise build the same cohort tracker. Fable recommends (b): the policy belongs to the app, and p2 already has +three different ones. **Owner** decision 1. + +## What the barrier does not promise + +Neither shape can promise that every tool is there for the first turn, and the model does not pretend it can. Sol +reproduced it against today's code (`/tmp/barrier-sol-first-outcome.log`): object A's `open` fails on its first +attempt (A's first outcome); B goes live and contributes; the registry changes and `retryOn` wakes A for a second, +slow `open`; `load` resolves with only B's contribution in the table. Tools found in `onLive` (MCP) arrive later by +design. What a first turn really needs (a model, an environment, a reconciled working directory) is p3's explicit +admission check, as p2's `whenRegistered` and its harness prerequisites are today. + +:::note | What p2 does today +p2 does not wait for all host objects before the first turn. Its worker (`apps/p2/src/hosts/worker.ts`) loads with +`awaitHostObjects: false`, waits for the harness, resumes Durable, tells its parent `control.ready()` and only then +awaits `host.ready()`. It has to: its subagent tools and the search indexer need server types, and the server admits +the worker's link only after `control.ready()`. A barrier before that point would deadlock, exactly as in chapter I +(Astra and Sol, independently). +::: + +# Plugins Coming and Going {#reload lede="Removals, versions and rollouts across hosts, as events the requests wait for."} + +Slice 7 brings reload and rollout, and with them the cases where a provider disappears on purpose. Today's design +handles them with their own mechanisms: `gone` when a type is removed, a final `contract_mismatch` when a version is +dropped, `updating` with a deadline during a rollout. Under the model all of them are one thing: a request becomes +unbound with a reason, and the event that fixes the reason wakes it. + +## A type removed, and served again + +A reload that removes a type ends its objects as today: natives close with reason `end`, callers of running calls are +rejected at once, handlers drain. What changes is what happens to the requests that pointed at those objects. + +- **Over a link,** the provider sends `failed { waiting, not_served }` on the bound hold. A `failed` on a bound hold + unbinds it (Fable: one message, one rule; the consumer applies it as today's `#startOver` plus `failed`). The + provider detaches the hold's sources and drops its unsent pending ops and any queued `bound`. Two things it must not + do (Sol): free the ack slots of `bound` and `state` messages already sent, which stay counted until their `ack` or + the link's loss, or resend calls, which fail and free their slots once. +- **On the same host,** a `Binding` whose target ended by removal becomes unbound and accepts a second `bind`. Its + member sources re-bind by name (one frame from the last value, as on a restart today), keyed pins re-pin, parked + calls flush or reject, presence lists it again. This is the one genuinely new piece of machinery in the model + (Fable, Sol): today `bind` runs once per binding, and the ended object is forgotten. + +When a later commit serves the name again, the publication wakes every request indexed under it, on every host that +holds one, and they bind as if nothing had happened. + +## A version no longer accepted + +A reload can also keep the type and stop accepting an old version (it drops `accepts[1]`). Draft 1 ended the provider +object then; Sol and Astra caught it: the object, its documents, its jobs and its holders on the current version are +fine. Only the requests on version 1 become unbound `version`. The object's own plugin reload still restarts it if it +has natives (§21), as for any reload. + +## A type that moved + +A plugin can move a type from the worker to the server by changing its placement. A consumer's request carries the +address its own code computed, so the provider appearing on the server cannot wake it: the request is not waiting +there. The old host answers `not_served`; the consumer's own reload, which brings the new placement, makes a new +request. The model adds no redirect: moving state between hosts is a deliberate migration, not something Chord guesses. + +## Rollouts across hosts + +A rollout installs a revision on the server first, then on the workers, then on the tabs (design §21). Between those +installs, hosts run different code. The model needs only one rule for that: **each host knows its own install, and +nothing more.** + +- A provider host that is **installing** keeps holds that do not fit its current code as unbound `installing`: the + consumer sees `loading`, its calls park. When the install ends, by its commit or by its rejection, the host + re-evaluates them against the code it now runs (Astra, Sol: a rejected install must not leave them asleep). +- A provider host that is **queued** but not installing yet answers a mismatch with `failed { waiting, version }` + until its install commits and wakes the request. That is a visible blip of at most one install; Opus judged it + acceptable, and avoiding it would need revision fields in every answer. +- **Installs never wait for activations.** An install that waited for its objects to go live would deadlock a rollout + that runs in the other order: the worker's objects wait for the server's new types, the server's install waits for + the worker. +- A host whose **own** install was rejected fails its own requests at once with the reason (design §21). + +:::steps | The sharing plugin's contract goes from v1 to v2; the server installs first ++++ Before +S and W run revision R. W's `ShareStatus` holds the server's `Sharing` v1; T's tab holds W's `ShareBadge`. All `live`. ++++ S installs R+1 +R+1 makes `Sharing` v2, with no `accepts[1]`. While S stages it, S still runs R: W's v1 hold fits the running code +and stays `live`. A hold that arrived now from a host already on R+1 (a v2 hold) would not fit R: it would wait +`installing` (`loading`, calls park) until the install ends. ++++ S commits R+1 +The install ends. S re-evaluates: v1 is not served. W's hold becomes `failed { waiting, version: asked 1, served 2 }`. +`ShareStatus` stays live (R18 after live); its calls that use sharing reject at once with the cause. ++++ W installs R+1 +W's new code bundles `Sharing` v2. Its own reload's handover makes the new request at v2 and releases the old one. The +new hold binds; `ShareStatus` is fully back. T's `ShareBadge` never noticed anything but one stale value. +::: + +## Old calls + +Calls that were running when the reload came follow §21 unchanged: a pure swap lets them finish on their old handles; +a native restart drains them first; a removal rejects their callers at once and drains their handlers. Chord never +resends a call. + +# Finding Cycles {#cycles lede="Without the deadline, the one gap left in cycle detection needs an answer: a narrowed deadline, or a probe."} + +A provider that never appears keeps its dependents parked, and that is visible: they say what they wait for. A cycle is +different: A waits for B, B for A, every provider exists, and still neither can go live first. Today Chord has two detectors and a deadline. On one host, `self.get` follows the waiting edges of +activating objects. Across hosts, a hold carries the activation chain it was made in, and a provider refuses a hold +whose chain contains the object being activated. The gap is documented (design §6): when A on the server and B on a +worker start at the same moment, the holds carry `[A]` and `[B]`, neither finds the other, and both fail at the 60-second +deadline. The model removes the deadline, so the gap must close. + +## Two answers + +The owner decided this gap once already. On 2026-10-24, after the slice 4b reviews, he made simultaneous starts across +hosts "a documented limitation: they time out transient and retry; no probe message." That decision assumed the +activation deadline. The model removes the deadline for dependency waits, so the decision needs one of two +continuations, and this one is his (**Owner** decision 6): + +- **(A) The 4b decision, narrowed.** One deadline stays: an attempt parked on a dependency held over a link, whose + target attempt is known and still activating or parked, fails `transient` after its `readyTimeout` and retries with + backoff; its next attempt usually starts after the other side, and the activation chain in its hold finds the cycle. + Working it through showed the catch. A target that is *parked* is the common case of a chain across hosts waiting + for a missing plugin: a tab's view on a worker object that waits for an uninstalled type. Those dependents would time + out and poll again, as today, so "local = remote" would stop at the link. Narrowing the deadline to targets that are + `loading` avoids the polling but misses a real ring: A and B on two hosts wait on each other, and B also on a missing + X. B is `failed { waiting }`, so A is too; when X appears, both stay `failed { waiting }` on each other, with no + deadline to break it and no chain to see it. Worse (Fable, round 6): an object's activation chain is fixed + when it is created and its instance id survives retries, so a retried attempt sends the same hold with the same + chain; two simultaneous starts time out and retry in lockstep and are never found. The deadline would poll without + finding what it is for. +- **(B) The probe (recommended, pending review)** described in the rest of this chapter: no deadline at all, one new + message kind, and the most intricate rules in the model. Five review rounds found a counterexample in each of the + first four drafts; the version below answers all of them, with one choice left in rule 5. Slice 7 + would prove it with tests before the old deadline goes. + +The rest of the chapter describes (B): it is the answer that keeps one rule on both sides of a link, and its review +history shows what a complete answer takes. + +## Why a growing chain is not enough + +Draft 1 proposed letting chains grow: an activating object that receives a hold with a chain sends the grown chain on +to its own dependencies. Astra and Sol showed two flaws. Chains are cut at 32 entries today, so a ring of 33 objects +started at once is never found: the chain that reaches each object is always missing that object's own id. And a +chain records ancestry, not the current wait graph: once A stops waiting for B, B's chain still names A, and a later +wait of B on A looks like a cycle that is not there. + +## (B) The probe + +The model uses an edge-chasing probe (the idea of Chandy, Misra and Haas), for the rings that cross a link; on one host the `self.get` +walk stays, synchronous and exact, as today (Opus). +Five rounds of review shaped it; the rules below are the result. + +1. **Attempts, edges, tokens.** Every attempt has its own id, minted per attempt: a retry or a reload is a new + attempt (today the id is per instance and survives retries). A **wait edge** runs from a parked attempt to the + activating or parked attempt it waits for, through a bound request. Unbound waits and registry reads are never + edges: they are not deadlocks, they end when a provider appears. Every edge carries an **edge token**: a number from a counter the + attempt keeps, never reused, replaced by a fresh one on any interruption or change of that edge (Astra, Sol: two + requests to one object are two edges): a new target attempt (also when a local + target retries without a re-bind), any `loading` on a bound or pending hold (a relay re-holding upstream included), + an unbinding, a rebinding, a re-hold after a reconnect. An attempt is parked for one interval only (once past its + dependency check it never parks again), and attempt ids are never reused. A provider tells its consumers which attempt they wait for on every answer that is not `bound`: + `loading { hold, attempt }`, and a `waiting` `failed { hold, cause, attempt }`. +2. **A search per edge change.** Whenever a parked attempt gets a new edge, or a new token on one, it starts a + search: `probe { kind: "search", round, initiator, trail }`, with the next number of a counter the attempt keeps, + its own attempt id, and the trail of `(attempt, edge token)` hops crossed so far, starting with its + own. A ring closes at its last edge, + and that edge's source starts the search that finds it (Sol's simplification after Astra's counterexample: a ring + that forms only after a reconnect, with unchanged attempts, is otherwise missed). Searches are coalesced per batch. + On the same host a probe is a call on the target attempt, forwarded through a queue rather than recursion; over a + link it is a new message, `probe { hold, to, ... }`, consumer to provider, relayed by hold id with the origin. + **Probes are honoured on parent and route links and dropped on child links** (`accept`), whatever `trusted` says + (Opus): a tab learns attempt ids from `loading` and could otherwise break an object with a forged probe, while a + worker whose transport is untrusted must still take part. Clients serve no remote types, so no real cycle runs + through one. +3. **Forwarding.** An attempt that receives a search drops it if `to` is not its current attempt or if it is not + parked. If the initiator is itself, the search came back. If the attempt is already in the trail, it drops it: a ring + that does not contain the initiator is found by its own members' searches (Opus). Otherwise it forwards it along + every wait edge, appending itself and the edge's token to the trail, but only if its round number is higher than the one it stored for + that initiator; then it stores it. A late, older round is dropped even if this attempt never saw it: its initiator + has moved on (Astra, Opus). The seen map holds one number per initiator, up to 256 initiators, and goes with the + attempt. +4. **The check.** The initiator accepts a returned search while it is still parked, whatever round it carries (Astra: + a re-search must not invalidate an older search still under way); the check decides. It verifies its own hop, then + sends a + `probe { kind: "check", round, trail }` along the recorded trail: each hop passes it on only if it is still that + attempt, still parked, still holding the recorded edge token, pointing at the next hop's attempt. The check follows the + trail by position, ignores the seen map and dies on any mismatch. When it returns, the initiator verifies its own hop + once more. If the check comes + back, the initiator fails `broken invalid/cycle`, naming the path from the trail, and today's propagation breaks + the other activating objects on the ring and the dependents off it. +5. **No silent loss.** Astra found the last hole: a search can reach an attempt first by a shortcut and be dropped + there when it arrives again along the ring itself; if the shortcut breaks before the check, the check dies and + nothing is left to find the ring. Round 6 tried `dead` notices sent back to the initiator. Sol and Astra showed + that reliable notices need recovery state per search that the seen map's eviction may not drop, and that a long- + parked attempt would otherwise keep for every initiator that ever passed: a leak, unless every search gets a full + lifecycle with completion and cancellation. So rule 5 is a choice, part of **Owner** decision 6: + - **(B1) Re-search with backoff (recommended).** An attempt parked on a remote attempt starts a fresh search again + after a backoff, 1 second doubling to 60, reset by an edge change, whether it shows `loading` or `failed + { waiting }`; the timer stops when the attempt leaves parking. No `dead` messages and no recovery state; + eviction only costs work; a new search starts within one backoff period, so every stable ring that + crosses a link is eventually found (no wall-clock bound: delivery is only fair). The + timer runs as long as the attempt stays parked on a remote attempt: a view on a worker object that waits for a + missing plugin re-searches every minute until a reload (Fable). The timer sends probes only: it + never fails an object, never re-holds, never polls a provider. Its cost is one probe per parked remote dependent + per period, which dies at the leaves of the downstream wait graph. + - **(B2) Events only.** Every search is owned until it settles: branches report completion or abort along their + trails, a dropped probe answers `dead { initiator, round }` by a reverse route, edges remember the searches they + carry until those settle, and an initiator that ends or moves on cancels its old search. Complete without any + timer, bounded by the searches in flight, and the most intricate protocol in Chord; not yet specified to the + reviewers' satisfaction. + + Sol's correctness reference, enumerating every simple trail without dedupe, never suppresses a ring copy, but its + work is exponential on chains of diamonds. + +:::sequence animate | Two objects on two hosts start at the same moment and need each other. The probe finds it; nothing times out. +actor a "A on server S" "attempt a1" +actor b "B on worker W" "attempt b1" +a -> b: hold B (A activating) +b -> a: hold A (B activating) +b --> a: loading { attempt: b1 } +a --> b: loading { attempt: a1 } +a => b: search { to: b1, round: 1, initiator: a1, trail: [a1·t1] } +b => a: forward: search { to: a1, round: 1, trail: [a1·t1, b1·t1] } +a -> a: search 1 came back: check the trail +a => b: check { round: 1, trail }: b1 still waits on a1 with token t1 +b => a: check passes on +a -> a: check came back: broken invalid/cycle "A → B → A" +::: + +## Why a reported ring existed + +Draft 4 confirmed a cycle with a second, independent round. Sol showed a schedule in which two rounds pass through two +different stale paths, each of them half replaced by reloads, and together report a ring that never existed at any +moment (`/tmp/model/sol-probe4.log`). The check closes that: it walks the very path the search recorded and demands +that every edge still carry the token it had when the search crossed it. A token changes on every interruption and is +never reused, and an attempt is parked for one interval only, so an edge cannot vanish and come back unnoticed (Fable and Opus +checked the argument in round 5). Every edge of the trail therefore existed from the moment the +search crossed it until the moment the check crossed it; the search crossed all of them before the check began, so +when the check began, every edge of the ring existed at once. In Sol's schedule the check dies at `B`, whose edge now +points at another attempt. + +## Why every ring is found + +Under an eventually stable wait graph and fair delivery (endless edge churn can keep invalidating searches), and with +rule 5's (B1) or (B2) for a search suppressed behind a path that broke, a ring of parked attempts is closed by its last +edge. When that edge appears, its source starts a search; every other edge of the +ring exists at that moment, and every attempt on the ring waits, directly or not, on the source, so none of them can +go live before the search and its check come back. Rings of any length are found, through local and remote edges +alike. On a graph without cycles, searches die at the leaves; on a diamond, the seen map drops the second arrival. + +## What it does not promise + +- **A ring broken up just after.** A ring that existed when the check began may be broken up by a reload, a retry, an + unbinding or a revocation a moment later; it is still reported, because it did exist. The objects it breaks recover + as every broken object does, through `ref.retry()` or a reload of their plugin. +- **Linear work under every schedule.** With more than 256 initiators passing one attempt at once, an evicted entry lets + a search be forwarded again, so in an adversarial schedule the work can grow with the number of simple paths (Astra, Sol); never endlessly, since + a search never revisits an attempt already in its trail. + The seen map's memory stays bounded. Each edge change costs one search that crosses each downstream edge once, + and one check per returned search; with (B1), every attempt parked on a remote attempt adds one search per backoff + period, also while an acyclic chain only waits for a missing plugin. +- **Constant message size.** A probe carries its trail, so it grows with the length of the path it crossed; a search + that returns carries a trail as long as its ring. That is the price of the check; chains were cut at 32, trails are + not, because the trail is the proof. + +:::review | Review: the state of (B) +The check's argument (a reported ring existed at the moment the check began) was confirmed by all four reviewers in +rounds 5 and 6, given exact edge identity (the tokens). Completeness under an eventually stable graph holds with +(B1); with (B2) it holds once the search lifecycle is specified, which no draft has done yet. If the owner chooses (B), +slice 7 keeps today's deadline for activations waiting across links until the probe's tests pass: long rings, +diamonds, a ring closed after a reconnect with unchanged attempts, Sol's phantom schedule, Astra's broken shortcut, +Sol's eviction schedule, 300 concurrent initiators, edges added and removed, reloads, relays, forged probes. +::: + +# Callers {#callers lede="Getters, holds and calls get one rule: wait while something is under way, answer when nothing is."} + +Objects park; callers do not. A caller is code that wants an answer: a method that needs a reference now, a tool that +runs inside a turn, a tab that wants to render. The model keeps their deadlines and gives them one rule. + +## `host.hold` + +`host.hold` returns a reference at once and never rejects; only invalid params throw, also before `load`. That was +already the design; the model removes today's inconsistency: a missing local type throws after `load` but fails the +reference before it. Now a missing local type gives `loading` before the publication and `failed { waiting }` after +it, and the reference binds when a provider appears. A UI that wants to wait for something that may never come +holds it and shows its state. + +## `host.get`, `ctx.get`, `host.registry` + +The getters resolve when the reference is live. While it is `loading`, or `failed { transient }` (its object's own +retry is under way), they wait, bounded by their `readyTimeout`. On `failed { waiting }` or `failed { broken }` they +reject at once. On success, rejection or timeout their temporary wait goes; a rejected getter releases its request. +`host.registry` returns the `RegistryValue`, whose watches and observers are owned as they are today; it has no +`release()`. + +Opus's case decided the recommendation. A Durable tool runs `chord((ctx) => ctx.get(SharingView, { session }), ctx)` +while sharing is not installed. If getters waited through `waiting`, the tool, and with it the agent's turn, would +hang for 60 seconds and then fail; a call on the same reference would fail at once. With the rule above, the getter +and the call give the same answer at the same moment. The rule has a cost Opus named: a dependent whose only blocker is +an object recovering by itself (an SSH environment coming back, `transient` with its own backoff) is `waiting`, so a +getter on the dependent rejects at once. Code that wants to ride that out holds the reference instead of getting it. + +:::review | Owner decision 2: getters on `waiting` +Recommended: reject at once (Opus; Fable agreed in round 2). The alternative waits through `waiting` up to +`readyTimeout`, which keeps "local = remote" just as well but makes every getter on a missing plugin cost its full +deadline. +::: + +## Calls + +Calls on a `loading` reference park in order, each bounded by the reference's `readyTimeout` from the moment it was +made (never renewed by a reconnect or a reload). Calls on a `failed` reference reject at once with the cause. A call +already sent when its link drops, or when its object ends, fails and is never resent; its outcome is unknown, and +idempotency is the app's (a `requestId`, as p2's chat does). + +## The rule in one table + +| The reference | `host.hold` | getters | calls | +| --- | --- | --- | --- | +| `loading` | returns it | wait, bounded | park, bounded | +| `failed { transient }` | returns it | wait, bounded | reject | +| `failed { waiting }` | returns it | reject at once | reject | +| `failed { broken }` | returns it | reject at once | reject | +| `live` | returns it | resolve | run | + +# Registries {#registries lede="The same resolver on one host, with versions that can differ inside one revision."} + +A registry is host-local: its contributions are in-process values, often natives. Slice 8 made `self.registry(R)` a +dependency that waits for a value. The model keeps that and folds it into the one resolver. + +## A reader's request + +A reader asks for `R` at the version its own bundle was built against. The request is bound while `R` is served and +has a usable value at that version: composed, and for an older reader mapped back by the provider's +`accepts[old].value`. Otherwise it is unbound: `not_served` (no plugin serves `R` here), `version` (the provider does +not accept the reader's version, or the reader is newer), `compose` (`R` is served, but its first composition, or this +version's mapping, threw), or `loading` before the publication. The reader is woken at every transition of the value +at its version to usable. S2's `broken` for a newer reader and S3's backoff go; so does `#valued`, the latch that woke +readers only at the very first good value. + +A registry waits for a value, never for a particular contribution: an empty table composes. "Wait until model X is +there" is the app's admission check, not a registry rule; Durable's own "model unavailable" wait covers the harness. + +## Versions inside one host + +Draft 1 claimed that a registry's provider and its contributors always run the same version, because a host installs +a revision atomically. Astra corrected it: a revision is a set of plugin bundles, and a reload of the plugin that owns +the registry leaves every other plugin's contributors and readers on the version they were built against. So the +provider's adapters do real work inside one host: `accepts[old].contribution` maps an older contribution, `accepts[old]. +value` maps the composition back for an older reader. A contribution without an adapter is left out and reported; a +reader without one is `waiting` with `version`. + +## The last good value, and natives + +Slice 8 decided that a throwing `compose` keeps the last good composition, so an extension point stays available while +its owner fixes a bug. Astra found the case where that hands out something dead: the harness contributes its native +`H`; a reader is live; the harness ends; the next composition throws for an unrelated reason; `.value` still returns +the composition that contains the closed `H`. + +The model narrows the rule: `.value` keeps the last good composition only while every contribution it was built from +is still in the table. When one of them leaves (its contributor ended or was reloaded), the last good value is dropped +and `.value` throws until `compose` succeeds again. And `compose` represents a missing contributor as a value +(absent), never by throwing. + +:::review | Review: drop the last good value, or fence the reads +Sol would keep the last good value and instead require natives in a registry to be looked up at use through their +owner, without the guarantee. The draft takes Astra's narrower rule, because it keeps "a registry never hands out a +native whose contributor ended" without asking every contributor to wrap its natives. It narrows a decision of slice 8, +so it is the owner's: **Owner** decision 5. +::: + +# Keep-alive, Presence, Revocation {#housekeeping lede="What a waiting request costs, who counts it, and who may end it."} + +A parked object and an unbound request are resources: they hold memory, they may keep a worker process alive, and +they act for a principal. The model says how each of those works while nothing is bound. + +## Keep-alive and presence + +A worker stays alive while `host.activity` shows work: references held over links, calls, jobs, lingering objects. A +request that arrived over a link counts once in `host.activity.references` from its arrival to its end, bound or +unbound. Without that, a session whose tab waits for a plugin that is not installed would see its worker stopped every +idle period (five minutes in p2) and restarted by the next re-hold, forever (Opus). Presence is different: +`host.references` lists only bound references, so "who is looking at these notes" counts viewers, not waiting +requests. + +## Revocation + +Revocation ends the requests of a revoked principal, bound and unbound, at every hop. A guest whose grant is revoked +while their tab waits for a missing plugin does not get a reference later when the plugin comes back: the request is +gone, and nothing revives an ended request (no publication, no Retry, no new grant). A refused request (no authority) +is never indexed in the first place. + +## `gone` + +`gone` remains for one thing: revocation. A provider host that closes closes its links and sends nothing (today's +`#close`); its consumers see a link drop, `loading`, and hold again when the host comes back. A reload that removes a +type sends `failed { waiting }` instead of `gone` (chapter VII). + +## Parked objects and their jobs + +A parked object with params lives while it is referenced, like any object; when its last holder releases it, it ends +and releases its requests, so a parked chain goes from the top down. A host-lifetime object parks until its plugin is +removed or the host closes. Persisted job runs of a parked host-lifetime object are loaded with its documents, before +it parks; `run` and `finish` start only once it is live, and the run count keeps the object as today. + +# The Clock {#clock lede="Which timers stay, and why each of them has no event to replace it."} + +The owner expected the model to remove almost every timeout, "except maybe `readyTimeout`". That is what it does for +dependencies. Every remaining timer bounds something no event can end: a caller's patience, an object's own code, a +silent transport, a process that may start this time. + +| Timeout or backoff | After the model | Why | +| --- | --- | --- | +| `readyTimeout` of an attempt | only its own steps (natives, follow attaches, checks, derives); paused while parked | a slow dependency is bounded where it runs, by its own steps | +| `readyTimeout` of `host.get`, `ctx.get`, `host.registry` | kept | a caller must get an answer | +| `readyTimeout` of a parked call | kept | an action must not run long after it was asked for | +| identity backoff 1 to 60 s | only for the object's own `transient` failures (an `open`, a follow) | nothing tells Chord that a disk or a network came back | +| dependency backoff, polling of refused holds | **removed** | a dependency's recovery is an event | +| dependency deadline for waits across links | answer (A): kept only for an attempt waiting on a remote attempt that stays activating; answer (B): kept until the probe's tests pass, then **removed** | the simultaneous-cycle gap (chapter VIII) | +| probe re-search backoff, only with answer (B1) | new: 1 to 60 s while parked on a remote attempt | finds a ring whose search was suppressed; sends probes only, never fails or re-holds | +| route backoff per address | kept | a failed worker start has no event; it avoids fork storms | +| heartbeat 15 and 30 s | kept | the only detector of a silent transport | +| `closeTimeout` (natives, the drain, the routine handover) | kept | bounds other code's teardown | +| routine and job `finish` backoffs | kept | the object's own code throwing | +| `linger` | kept | a policy for warm objects, not a wait | +| rollout install timeout (designed, not built) | a watchdog on the install only | never a deadline for dependencies | + +So `readyTimeout` stays, but its meaning narrows to two things: how long a caller waits, and how long an object's own +activation steps may take. Waiting for someone else never counts against it. + +# A Morning, Step by Step {#morning lede="The preface's Tuesday morning, played through under the model from the first click to the afternoon's rollout."} + +The rules of the earlier chapters meet in one story. Mario opens session `S1` on his phone; the server S has the catalog +and the search feed; the worker W for `S1` has not started. W's plugin set has `core`, `models`, `codemode`, `notes` +and `search`, but not `sharing`, which was removed last night; the notes view still asks for it. Later in the day a +rollout brings sharing back, with a new contract. + +:::steps | Session S1 on Tuesday ++++ 1. The tab asks for the notes +T calls `host.hold(Notes, { session: "S1" })`. The address is `Session { id: "S1" }`, not T's own, so the request goes +over T's parent, the server. S has a route for `Session`: it resolves the address by starting W. T's reference shows +`loading`; the request waits on S's path until W's link exists. ++++ 2. W connects before it loads +W connects to S first (W6), then starts `load`. The hold for `Notes` arrives before W has published its facets: it is +unbound `unpublished` and waits, still `loading` for T. Nothing is placed by a half-empty table, so the hold is not +sent back to S. ++++ 3. W publishes +W's setups run, synchronously, and are published in one batch: `Notes`, `NoteStore`, `Harness`, `CodeMode`, `Indexer` +are served; `HarnessModels` and `HarnessExtensions` are served and composed once. After that batch, the waiting hold +for `Notes` is placed and binds; the host-lifetime objects start. ++++ 4. Code mode parks on the harness +`CodeMode` declared `self.get(Harness)`. The harness is still opening Durable, so `CodeMode` parks: `loading`, no timer. +The harness reads `HarnessModels`, which has a value (composed at the publication; the model runtime's contribution may arrive +after), finishes its own steps +and goes live. That state change wakes `CodeMode`, which passes its dependency check, opens its sandbox, goes live and +contributes its tools. ++++ 5. The indexer waits for the server +`Indexer` needs the server's `SearchFeed`. The hold goes over W's parent; S published long ago, so it binds at once, +`SearchFeed` is live, and `Indexer` goes live. Had S been starting at the same moment, the hold would have waited only +for S's publication, never for S's activations: the startup deadlock of chapter I cannot form. ++++ 6. The first pass +With barrier (a), W's `load` resolves once `Harness`, `CodeMode` and `Indexer` had their first outcomes. `CodeMode` +waited for the local harness, so its tools are in the table; base code resumes the harness, and the first turn sees +them. With (b), base code would read `host.startup` and decide the same. ++++ 7. The notes view asks for sharing +`Notes`'s factory declared `self.get(ShareStatus)`, a local type of the sharing plugin. Nobody serves it on W: the +request is unbound `not_served`, indexed under the name. `Notes` parks `failed { waiting, "waiting for ShareStatus: not +served on Session S1" }`. T's reference shows the same; a call from T rejects at once with that cause. No backoff, no +polling, no 60-second wait. ++++ 8. The phone changes networks +T's link to S drops. T's reference goes `loading`: its request stays, its last value stays readable, a call made now +parks. A call that was in flight fails and is not resent. T reconnects; the request is held again on the new link; W +answers as before: `failed { waiting }`. ++++ 9. Sharing comes back, with a new contract +In the afternoon a rollout brings the sharing plugin back at revision R+1, where `Sharing` on the server is v2. S +installs first and serves `Sharing` v2. W's install is queued; nothing on W asks for `Sharing` yet, because the sharing +plugin is not installed there, so nothing changes for W. (Had W still run a sharing facet built against v1 without an +adapter on S, its requests would show `failed { waiting, version }` until W installed: the ordinary window of a +server-first rollout.) ++++ 10. W installs R+1 +W's install commits: a publication. It serves `ShareStatus` again: the index wakes `Notes`'s request, which binds. +`ShareStatus` activates, holds the server's `Sharing` at v2 (W's new code bundles v2), goes live, and `Notes` passes its +dependency check, opens what it needs and goes live. T's reference becomes `live` without T having done anything. +::: + +What did not happen is the point. Nothing waited for a deadline; nothing retried on a timer; no reference ended +because a plugin was missing; no host waited for another host's activations; and every waiting object said what it +was waiting for. + +# Scenarios: Starting Up {#sc-startup lede="Twenty startups on p2's real hosts, under the model, with the barrier as (a) and as (b)."} + +These scenarios, from Astra, follow p2's real hosts: the server alone, workers started by a tab or by another worker, +web tabs, the TUI, restarts in the middle. Each row says what the model does, with the barrier in both shapes where +they differ, and what today's code does. The pattern across all of them: publication never waits for activation, so +dependency traffic between hosts always moves; what differs between (a) and (b) is only when base code is told that +the first pass is over, and neither shape promises every tool before the first turn. + +The Rule column cites sections of the working document, reproduced in appendix C. + +| # | Scenario | Under the model | Today | Rule | +| --- | --- | --- | --- | --- | +| 1 | The server starts alone. Catalog, ModelConfig, LoginService and ForwardStore initialize before any session worker exists. | Their local references are `loading` during initialization, then `live` or `failed` with an own cause. Under (a), `load` waits for these first outcomes; under (b), it returns after publication. Public admission still needs explicit security prerequisites; a completed first pass is not proof that every service works. | p2 starts its HTTP listener but gates requests and upgrades behind plugin startup (`apps/p2/src/hosts/server.ts`). Current next waits for eager objects' first live/failed outcomes (`host/host.ts:load`). | §§1–3, 6 | +| 2 | A tab's request reaches the server before its first facet publication. Catalog is declared by a later setup in that same load. | The request stays `loading` behind the publication gate. The complete declaration batch places it; Catalog activation then produces `bound` and the reference becomes `live`. Neither (a) nor (b) delays that hold until unrelated eager objects finish. | Current next's `Hold.start` waits for `pendingLoad`, which includes every eager object's first outcome (`remote/hold.ts`). It waits longer than declaration safety requires. | §§1, 4, 6 | +| 3 | Several server facets set up successfully, but the last setup throws. A tab is already waiting for Catalog. | No partial declaration batch may activate objects. Load fails and the host closes; local held references become `ended`, while the tab sees a link drop and remains `loading`. A replacement server's successful publication can satisfy the tab's retained request. | Current next closes the host on a throwing setup and activates no objects (`host/host.ts:load`). Setup failure rejects load directly. | §§1, 6, 10 | +| 4 | A session worker's Harness opens slowly. Its CodemodeRuntime startup object declares Harness and contributes tools only after its own sandbox opens. | Harness and CodemodeRuntime start `loading`; CodemodeRuntime waits without a dependency timer. Harness becoming `live` wakes CodemodeRuntime, which opens its sandbox and becomes `live`. Under (a), this local chain finishes before load returns and resume; under (b), the application must await the same startup outcomes or risk a first turn without those tools. | p2 resumes after Harness is ready, before `host.ready()` (`apps/p2/src/hosts/worker.ts`); CodemodeRuntime depends on Harness (`plugins/codemode/worker.ts`). Current next's load barrier waits for both first outcomes. | §§3, 6, 11 | +| 5 | A tab opens Chat for a stopped session. The server starts that session worker, whose Search Indexer needs server SearchFeed. | Chat remains `loading` while the route starts the worker. The worker connects and publishes before its application barrier; SearchFeed can bind immediately on the already-published server. Indexer becomes `live` when its prerequisites are ready, and Chat receives `bound` once its own prerequisites are ready. Under (a), a still-remote startup blocker can already count as a parked first outcome; neither option guarantees that every remote-backed feature precedes resume. | p2 connects IPC first but the parent admits it only after `control.ready()`; Harness resumes before the later `host.ready()` (`src/hosts/worker.ts`). Current next gates incoming holds on whole-host load (`remote/hold.ts`). | §§1, 3–4, 6 | +| 6 | A running worker calls ConversationBroker on the server. The broker sends to another session whose worker is stopped. | Only the second worker starts a new publication and first-pass sequence. Its PeerConversation reference is `loading` until that view binds; the first worker and broker remain `live`. Under (a), the new worker's scheduler waits for its first-pass report; under (b), its application must supply that admission rule. This call does not reopen the first worker's startup barrier. | p2 gets PeerConversation inside the broker's call and starts its worker through the session route (`plugins/conversations/server.ts`, `src/runtime/server.ts`). The spawned worker uses p2's Harness-before-host-ready ordering. | §§3, 6, 8 | +| 7 | Many tabs open the same stopped session while other tabs open several different sessions. | Requests to one worker share route resolution and provider identities where their principals match. Each tab's reference starts `loading` and becomes `live` on its own binding. Both barrier options remain per host; a slow worker cannot hold back every other worker's first turn. The application shows per-session startup status, not one global ready flag. | Current next shares resolution per route address (`host/route.ts`). p2's SessionHosts also serializes starts per session (`apps/p2/src/runtime/server.ts`). | §§1, 6 | +| 8 | During server startup, an eager Bootstrap object needs a worker's Agent. That worker's eager FeedOwner needs the server's independent Catalog. | Server and worker publication happen independently of their first-pass reports. Catalog binds while Bootstrap is still `loading`; that wakes FeedOwner, then Agent and Bootstrap can become `live`. Both (a) and (b) avoid a host-ready gate on this dependency traffic. The acyclic object graph needs no timeout to make progress. | Current next can deadlock its whole-host load gates until activation deadlines fail attempts: the Agent hold waits for worker load, while the Catalog hold waits for server load (`host/host.ts`, `remote/hold.ts`). This is an architectural example using p2-style server/worker dependencies, not a claim that p2 has a Bootstrap plugin. | §§1, 4, 6 | +| 9 | The worker has a healthy Harness, but its `p2.subagent-tools` startup object still waits for server Subagents. The first generation is ready to run locally. | `p2.subagent-tools` remains `loading` while its remote reference binds, or `failed` with `waiting` if that provider is unavailable. Under (a), the remote wait can settle its first outcome without its contribution; under (b), publication returns even earlier. Server `bound` later wakes `p2.subagent-tools` and publishes its tools for future phases. If the first turn requires those tools, application admission must wait explicitly. | p2 `p2.subagent-tools` declares both Harness and server Subagents (`plugins/subagents/worker.ts`). Its worker resumes Harness before admitting the upstream-dependent host objects (`src/hosts/worker.ts`). | §§2–4, 6 | +| 10 | In a p3 port, the core serves a Models registry whose empty composition is valid. The optional models contributor is absent when the worker starts. | The registry is usable, so this absence alone does not park Harness; Harness can become `live`. Neither (a) nor (b) waits for an uninstalled contributor. The generation's model selection or application admission must report model unavailable and wait for a contribution. Registry publication later wakes that domain-level wait; it is not a missing-provider activation failure. | p2 instead awaits the `HarnessConfig` default entry inside Harness's Durable opener (`src/runtime/worker.ts:serveHarness`). Without the models contribution, that opener does not finish normally. | §§3, 6, 9 | +| 11 | A session is configured for SSH. Harness's execution opener finds that the environment is invalid or cannot open. | Harness becomes `failed` with its own `transient` cause, or `broken` if the opener marks the error permanent. Dependents still before their readiness check become `failed` with `waiting`. Under (a), this is a first outcome, not a healthy Harness; under (b), load already returned but resume still requires a usable Harness. Effect retry or a configuration-triggered retry can later make it `live`; no local-execution fallback is implied. | p2 opens the session machine during Harness activation and fails rather than silently using the local machine (`src/runtime/worker.ts`). Worker startup failures are surfaced and can require explicit retry (`src/runtime/server.ts`). | §§2–3, 6, 11 | +| 12 | Prompt becomes live before its initial local instruction-file read finishes. A user immediately sends the first message. | Prompt's reference can be `live` while its file-loading work continues. Under (a), its InputGate contribution is present before a normal local first pass completes; the gate still controls when local input may proceed. Under (b), the application must ensure that contribution exists before admitting input. The model does not turn background file loading into a host-wide barrier. | p2 contributes InputGates and awaits the initial local load in its prompt sections; remote instructions deliberately load in the background (`plugins/prompt/worker.ts`). Object readiness is already distinct from instruction readiness. | §§3, 6, 9 | +| 13 | Search starts an archived session worker only to rebuild its historical index. Indexer has a long scan to perform. | Once SearchFeed and Harness are usable, Indexer becomes `live` even while its `scanned` state is false. Both (a) and (b) leave scan completion to the explicit IndexScan operation. The archived-worker application keeps Harness scheduling paused, waits for the scan result and then stops the worker. A startup report does not override that resume policy. | p2 passes `resume:false` for archived index-only starts, exposes `IndexScan.scanned`, and stops the archived worker after rebuilding (`plugins/search/server.ts`, `plugins/search/worker.ts`, `src/hosts/worker.ts`). | §§3, 6 | +| 14 | After a worker starts, a tab opens Workspace for a large directory. Its filesystem watcher is slow to initialize. | The on-demand Workspace reference is `loading` until the backend is usable, then `live`. Its background watch may still be starting. Neither barrier option waits for every future Workspace view or directory watch; a component that needs a complete initial scan must use explicit domain readiness. | p2 deliberately returns the Workspace backend before installing a potentially slow watcher (`plugins/workspace/worker.ts`). This work is not a universal worker-start prerequisite. | §§3, 6 | +| 15 | A worker restores durable work that can resume immediately. No new Chat.send call occurs, so there is no incoming-input gate to intercept it. | Harness opens with scheduling paused and can become `live` before all startup contributors. Under (a), base code resumes after the first-pass report and required application checks; under (b), base code must await equivalent outcomes itself. Input gates on new messages do not guard this replay. Each phase pins the contributions present when that phase begins. | p2 explicitly calls Harness.resume before `host.ready()` (`src/hosts/worker.ts`). Its Durable registry adapter snapshots the extension table per phase/version (`src/runtime/durable.ts`). | §§6, 9 | +| 16 | A previously loaded web tab starts while the server is offline. Its local UI facets can render an offline shell. | Publication of the tab's local facets does not require the server. Remote references stay `loading` on the absent path. Under (a), remote blockers can count as parked first outcomes; under (b), declaration publication returns directly. The shell renders, and later connection plus `bound` messages make the affected views `live`. | p2 starts reconnection before facet installation and renders React without awaiting a whole-host ready promise (`src/hosts/web/main.tsx`). It already supports an offline shell after the page has been loaded. | §§1–2, 6 | +| 17 | A TUI candidate loads before its launcher activates BridgeLink. A startup UI object declares a server reference, whose outgoing frames are buffered. | The reference remains `loading` on a remote dependency. Under (a), that remote wait is a parked first outcome, so candidate preparation can finish; under (b), publication alone finishes. Launcher activation flushes the frames, and `bound` wakes the UI object. Do not replace this dependency declaration with an own native opener that waits indefinitely for launcher activation. | p2's setupTui awaits facet load while BridgeLink buffers frames until `activate` (`src/hosts/tui-worker-runtime.ts`). Adding an eager remote-dependent host object under the old full-ready load rule could stall candidate preparation until a deadline. | §§1, 3, 6, 11 | +| 18 | A tab starts a worker, but the process crashes while Harness is still opening. The server launches a replacement. | The tab's Chat reference remains `loading` across the lost path and replacement startup. The replacement has its own publication and first-pass cohort; successful binding makes Chat `live`. Both (a) and (b) must keep transport/start failure distinct from a healthy startup report. A process manager may surface a startup error and require user retry rather than retrying forever. | p2 records worker startup failures and has an explicit retry path; it does not treat every failed boot as an automatically healing object (`src/runtime/server.ts:SessionHosts`). Current next's route layer independently retries failed resolution (`host/route.ts`). | §§1–2, 6, 11 | +| 19 | The server restarts while a worker is still starting. A detached terminal host already owns running PTYs. | Tabs see their server and worker references become `loading`; new hosts publish and bind them again. The application may stop old workers on parent loss, while the detached PTY owner survives. The old server's unfinished first-pass wait settles on close (its entries end). New startup reports must not wait for the detached PTYs to restart. | p2 workers stop on IPC disconnect, while the terminal driver detaches its socket without killing the detached PTYs (`src/hosts/worker.ts`, `plugins/terminals/detached.ts`). Current next settles ended objects through its existing load settlement path. | §§1, 6, 10 | +| 20 | A startup contributor in a v2 port reads a keyed configuration document from a local store. The store is live, but the first requested key still needs hydration. | The contributor remains `loading` until that pin is ready; store-live alone does not make the dependency usable. Under (a), this local key-loading work remains part of the first pass, then the contributor can become `live` and publish before resume. Under (b), publication returns earlier, so the application must await that readiness if the first turn needs the contribution. | Current next loads dependency pins before opening the consumer's natives and before its settled promise completes (`host/attempt.ts:pinKeys`). Its activation deadline currently also bounds this wait. This is a v2-port example, not an existing p2 keyed API. | §§3, 6, 11 | + +# Scenarios: References, Holds and Calls {#sc-refs lede="Getters, holds, calls, link drops, relays, release and revocation, played through."} + +Fable's scenarios use p2's names: `Catalog` and `Hosts` (the sessions plugin, on the server), `Sessions` (on the +server, called from worker tools), `SessionTransfer` (a worker type the server calls for a fork), `Notes { session }` +(a worker type a tab holds), `NoteStore` (local on the worker), `ForwardStore` (ports, on the worker) and `Forwards` +(radius, on the server), and `Harness` (host-lifetime, on the worker). S is the server, W1 the worker of session S1, T +a web tab. The pattern: a request waits where something is under way and answers where nothing is; nothing a holder +does is needed to recover; and an ended holder leaves nothing behind in any index. + +The Rule column cites sections of the working document, reproduced in appendix C. + +| # | Scenario | Under the model | Today | Rule | +| --- | --- | --- | --- | --- | +| 1 | `self.get(NoteStore)` in `Notes { S1 }`'s factory on W1; `NoteStore` is served and activating. | `Notes` makes its request, it binds to the `NoteStore` object, `Notes` parks `loading`. `NoteStore` goes live: a target state change wakes `Notes`, which opens its natives and goes live. No timer ran. | The same wait, bounded by `readyTimeout`; past it `Notes` fails `transient` and backs off (`attempt.ts` `waitFor`). | §3.1, §3.4, §4 | +| 2 | `self.get(Catalog)` from W1's `ForwardStore`; the parent link is up, S has published, `Catalog` is served there. | The request goes over the parent (placement `Server()`); S binds the hold; `bound` arrives; `ForwardStore` was `loading`, now goes on and live. | The same, after S's whole barrier (`Hold.start` awaits `pendingLoad`). | §1, §2 | +| 3 | The same, but `Catalog`'s plugin is not installed on S. | S indexes the hold `not_served` under "p2.catalog" and answers `failed { waiting, not_served }`; `ForwardStore`'s reference shows `failed { waiting }`, the object parks `failed { waiting, "waiting for p2.catalog: not served" }`. A reload on S serving `Catalog` re-runs the hold: `loading`, `bound`; the reference goes live, the dependent wakes. No backoff, no new hold. | The hold is refused `broken`; `ForwardStore` fails `transient` and re-holds every 1-60 s, refused each time (`remote/hold.ts` `#refuse`, `object.ts` backoff). | §1, §3.1, §4, §5 | +| 4 | `self.get(SessionInfo)` on W1 when the core facet is missing (same host, not served). | The request is unbound `not_served` on W1 itself; the dependent parks `failed { waiting }`; a reload serving the type binds it and wakes the dependent. | `#assertRoute` throws in the factory: the object is `broken` until `retry()` or a reload (`host.ts` `reference`). | §1, §3.1, §4 | +| 5 | A tab's `NotesView` declares `Notes { S1 }`; S routes `Session`, `resolve` forks W1, W1 connects and publishes. | The hold waits at T on its path (`no_link` while S resolves: `loading`), is relayed to W1, waits there `unpublished` until W1's setups, then binds; `bound` comes back through the relay. `NotesView` was `loading` the whole time and goes live. | The same chain of waits, each bounded by `readyTimeout`; a slow worker start past it fails the tab object `transient` and polls. | §1, §6.2 | +| 6 | `host.get(Notes, { session }, ctx)` from a tab's page code while W1 is starting. | The getter waits while the reference is `loading`, up to its `readyTimeout`, then rejects with the cause and releases its request. It resolves as soon as `bound` arrives. | The same. | §8 | +| 7 | `host.get(Catalog, {}, ctx)` on W1 when S does not serve `Catalog`. | The reference becomes `failed { waiting }` at the first answer; the getter rejects at once with "not served on server" (recommended; owner's choice to wait instead). | Rejects at once too (a broken failure), after the refused hold's `failed`. | §8 | +| 8 | `host.hold(Catalog, {}, ctx)` on W1 before W1's `load` ran. | The reference is returned at once, `loading` (`unpublished`); W1's publication places it over the parent; it follows the object for its whole life: `loading`, `live`, `failed { waiting }` if S drops the plugin, `live` again when it comes back. | Returned at once; placed once `load` resolved (`host.ts` `#hold` races `load`); a removal ends it for good with `gone`. | §1, §8, §5 | +| 9 | `host.hold(Missing, {}, ctx)` for a local type nobody serves here. | `loading` before the publication, `failed { waiting, not_served }` after it; a reload serving it binds the same reference. | Throws `unavailable` at once after `load`; before `load`, fails the reference broken once it resolved. | §8 | +| 10 | `await host.registry(HarnessModels, ctx)` by base code when the models plugin is missing. | Rejects at once with `unavailable/undeclared` (`waiting`); code that wants to wait declares the registry in an object or holds. A served registry whose first `compose` threw rejects the same way (`compose`). | Rejects at once (slice 8 build). | §8, §9 | +| 11 | `ctx.get(Sessions, {})` inside a worker tool through `host.root` (`plugins/sessions/worker.ts`) while the parent link is down. | The request waits on the path (`no_link`, `loading`); the getter waits up to `readyTimeout`; the link comes back, the path re-holds, `bound`, the getter resolves. The request ends when the tool call settles. | The same (`ctx.ts` `CallContext.get`, `path.ts` re-hold). | §1, §8 | +| 12 | A call `notes.add(args, ctx)` on a reference whose object is `loading` (W1 restarting). | The call parks in the reference's queue with its own `readyTimeout`; `bound` flushes the queue in order; past the deadline it rejects `unavailable/park_timeout`. | The same (`host/parked.ts`). | §8 | +| 13 | A call on a reference that is `failed { waiting }` (the plugin is not there). | Rejects at once with the cause, no parking: nothing is under way. | Rejects at once (the reference is failed). | §2, §8 | +| 14 | A call already sent to W1 when the IPC link drops. | The call fails `unavailable` and is never resent (its outcome is unknown); the reference goes `loading`; calls made after the drop park until the re-hold's `bound`. | The same (`consumer.ts` link close). | §2, §5 | +| 15 | The parent link drops while `ForwardStore` is parked `loading` on `Catalog` (S restarting). | The request stays on the path (`no_link`), `ForwardStore` stays `loading`; the reconnect holds it again, S answers `loading { attempt }` or `bound`; the dependent wakes on the state change. No deadline on the wait. | The dependent stays `loading` until `readyTimeout`, then `transient` with backoff (the audit's choice 6). | §1, §3.3, §4 | +| 16 | A route link to W1 drops while T holds `Notes { S1 }` through S's relay. | The relay sends `loading` downstream, rejects relayed calls in flight `unavailable`, re-resolves the address with the route backoff, holds again upstream with the kept request; `bound` is forwarded. The tab's dependents stay `loading`. | The same (`relay.ts`); the route backoff stays. | §1, §4, §11 | +| 17 | T's own link drops while its hold for `Notes { S1 }` is still unbound at W1 (`not_served`). | S sends `closed { origin }` upstream; W1 drops the indexed hold with the origin: no leak. T reconnects and holds anew. | The same for refused holds (`provider.ts` origin close). | §1, §10 | +| 18 | A tab closes while `NotesView` is parked on `Notes { S1 }`. | Its last holder releases: a parked object ends at once (it is not live, so no `linger`); its requests end, the relayed hold is released upstream, the index entries go. Nothing keeps the chain. | The same (`object.ts` `#idle`). | §1, §3.5, §10 | +| 19 | `host.revoke({ grant })` while a guest's objects on W1 are parked and its tab's holds wait unbound. | Every request of the principal ends, bound and unbound, at every hop: the parked objects end `forbidden/revoked`, the tab gets `gone` for each hold, the child link closes. Nothing revives them; a new login holds anew. | Bound objects end, holds waiting for `load` are ended (`host.ts` `revokeWaiting`); refused holds are released with the link. | §10 | +| 20 | A worker with nothing live: a tab holds `Notes { S1 }` but the notes plugin is missing on W1. | The unbound hold counts in `host.activity.references` from its arrival, so the supervisor does not stop W1 while the tab waits; it is not in `host.references` (presence lists bound references only). | A refused hold is not counted: the worker stops idle, restarts on the next hold, refuses again. | §10 | +| 21 | S holds `SessionTransfer` on W1 for a fork while W1's `Harness` is parked on the models registry (not served). | The hold binds to `SessionTransfer`, whose attempt parks on `Harness`: `loading`, then `failed { waiting, "waiting for p2.harness: waiting for registry …" }`, with the chain in the message; S's `ctx.get` rejects at once with it. A reload on W1 serving the models plugin wakes the harness, then `SessionTransfer`; a later `ctx.get` binds. | `SessionTransfer` fails `transient` after `readyTimeout`, polls; S's getter rejects at its own deadline. | §3.1, §3.4, §8 | +| 22 | `ref.retry()` from a tab's Retry button on a `failed { waiting }` reference whose blocker on W1 is `broken` (a throwing factory). | The retry passes over the link (`retry { hold }`) to the parked object, which passes it to its `broken` blocker: a new attempt; `waiting` blockers are passed through, `not_served` ones untouched. If the blocker goes live, the chain wakes. | `retry` reaches only the object the hold is on; a refused hold is started again. | §3.7 | +| 23 | Keyed pin: `Notes` pins `store.drafts.at(key)` on `NoteStore`; the key's load fails at the provider. | The pin is a dependency: `Notes` parks `failed { waiting on p2.note-store.drafts[key] }`; the provider's re-bind of that member (a retry) wakes it. | `dependency_failed` transient with backoff (`attempt.ts` `pinKeys`). | §3, §4 | + +# Scenarios: Providers, Versions, Registries {#sc-providers lede="Plugins missing, added, removed and broken; versions that do not fit; registries and the harness."} + +Opus's scenarios cover the provider side: plugins that are missing, arrive by a reload, are removed or break; versions +that fit through `accepts` or do not; registries with missing providers, skewed versions and last good values; +`self.follow` over a harness that restarts; and reloads of an object's own code. The pattern: what used to be `broken` +or polled becomes `waiting`, woken by the one event that can fix it, and an object's own code is still the only thing +that breaks it. + +The Rule column cites sections of the working document, reproduced in appendix C. + +| # | Scenario | Under the model | Today | Rule | +| --- | --- | --- | --- | --- | +| 1 | Worker W1 has no drafts plugin. Chat (chat plugin, on W1) declares `self.get(DraftStore)`. | Chat's request is unbound `not_served` and indexed under `DraftStore`. Chat is `failed { waiting, "waiting for DraftStore: not served" }`, with no timer. A tab's Panel holding Chat shows the same cause, and its calls reject at once. | `#assertRoute` throws in the factory, so Chat is `broken`. Only `ref.retry()` or a reload recovers it (`host.ts`). | §1, §2, §3.1 | +| 2 | A reload adds the drafts plugin to W1 while Chat waits (1). | The reload's commit publishes `DraftStore`. The index wakes Chat's request, which binds; DraftStore activates and goes live. Chat leaves parking, opens its natives and goes live; Panel goes `live`. | Chat stays `broken` unless the reload retries broken objects. | §4, §5 | +| 3 | Server S has no models plugin. On W1, the models plugin's host-lifetime ModelRuntime declares `self.get(ModelConfig)` (placed on S). | S keeps W1's hold as refused, `not_served`. ModelRuntime is `failed { waiting }`. W1's `load` counts it as a first outcome (parked on another host). The harness runs, and its generations report "model unavailable". | S refuses `broken` (`remote/hold.ts #refuse`). ModelRuntime is `transient` and polls: each backoff tick (1-60 s) makes a new hold across the link and reports again, for the host's life. | §1, §3, §6.3 | +| 4 | S gets the models plugin by a reload (3, continued). | S's commit re-runs its refused hold and sends `loading { hold, attempt }`, then `bound`. ModelRuntime wakes, opens its native and contributes to `HarnessModels`. The harness's next generation finds a model. | The next backoff tick re-holds; recovery waits up to 60 s. | §4 | +| 5 | A reload removes the ports plugin from S. Tabs hold `PortsView`; Radius (live) reads the `radius.forwards` registry that Ports contributed to. | PortsView's objects end, and their holds become unbound `not_served`: the provider sends `failed` on the bound hold, and the tabs show `failed { waiting }`. Radius gets a table without Ports, composes the empty value and keeps working. A later re-add re-runs the kept holds, which bind again. | The holds get `gone` and the references end for good (design §21). A re-add cannot revive them. | §5, §9 | +| 6 | A reload removes W1's file-index plugin. The live Files object (files plugin) holds `self.get(FileIndex)` locally. | Files' `Binding` becomes unbound `not_served`. Files stays live (R18). Its member sources keep their last values, and calls through that reference reject with the cause. When file-index is re-added, the binding binds again and the members re-bind by name (one frame). | The reference ends `released`; Files can never use FileIndex again without a restart. | §5 (locally), §3.10 | +| 7 | After a sessions plugin update, S's Catalog factory throws. W1's Chat waits on Catalog; a tab's Panel waits on Chat. The user presses Retry on Panel. | Catalog is `broken`; Chat and Panel are `failed { waiting, "waiting for Catalog: …" }`. Retry passes on hop by hop (T, then S's Catalog, through W1) and retries Catalog once. If the cause is gone (the config is fixed), Catalog goes live and Chat, then Panel, wake. | Chat polls by backoff with new holds. Retry on Panel retries only Panel, which is still blocked. | §3.7 | +| 8 | S's EnvManager (envs plugin) opens an SSH connection that fails because the remote host is offline. W1's environment objects wait on S's env views, which wait on EnvManager. | Only EnvManager backs off (`transient`, its own `retryAt`). Its dependents are `failed { waiting }` with no timers. When SSH comes back, EnvManager goes live and the chain wakes in order. A `host.get` on a dependent rejects at once (it is `waiting`); a hold just shows it. | Every level runs its own 1-60 s backoff and re-holds across the link at every tick. | §3.1, §3.3, §8 | +| 9 | A reload of the terminals plugin restarts S's TerminalService (natives: the terminal host process D). Terminals{session} objects are live and depend on it. | Native restart as in §21: new calls park while it is `loading`; running calls drain; natives close with reason `reload`; the factory runs; natives reopen. Its dependents stay live (R18); their calls park while it is `loading`. | The same; this is design §21, unchanged. | §5 (old calls), §3.10 | +| 10 | Worker W1 crashes while a tab views Chat{S1} through S. | S's relayed hold goes to its path, `no_link`, so the tab's reference is `loading` and its calls park, bounded. The route re-resolves (route backoff); the new W1 publishes, the held request binds, and a fresh `bound` reaches the tab. | The same, except a dependent that is activating fails at its 60 s deadline in between. | §1, §2 | +| 11 | A tab on Chat v1 meets W1 serving Chat v2 with `accepts[1]`. | The check accepts v1; the hold binds through the mappings (slice 7). The tab is `live`. | The same once slice 7 builds `accepts`. | §1, §5 | +| 12 | A tab on Chat v1 meets W1 serving Chat v3 without `accepts[1]`. | The hold is unbound `version`, so the tab shows `failed { waiting }` ("update the page") and its calls reject. The tab's own reload makes a new v3 request and releases the old one. If W1 rolled back to v2, the old hold would bind again in place. | Refused `broken`, or `gone` at the reload that dropped v1: ended for good. | §1, §5 | +| 13 | A tab reloaded to v2 holds Chat while W1 is installing v2. | W1 keeps the hold `installing`: the tab is `loading` and its calls park, bounded. W1's install commit re-evaluates the hold, which binds v2. If W1's install is rejected, the hold is re-evaluated at that end and becomes `failed { waiting, version }`. | Refused `broken` (`contract_mismatch`); design §21's `updating` is not built. | §5 | +| 14 | A plugin changed Chat's contract without bumping the version: a tab on v1 (fingerprint A), W1 on v1 (fingerprint B). | Unbound `version` (a fingerprint that differs at the same version), so `failed { waiting }`. It is not `broken`: the consumer did nothing wrong, and its next reload or a fixed provider heals it. | Refused `broken`, `contract_mismatch`. | §1 | +| 15 | A reload of chat makes Chat v4 accept v3 but drop v2. Tabs hold Chat at v2 and at v3. | Only the v2 holds unbind (`version`). The object, its v3 holds, documents, jobs and natives stay. | Design §21: the version's holds end with `gone`. | §5 | +| 16 | S has no web-shell plugin. Files, artifacts and terminals contribute routes to `Http`; proxy-marker reads `Http`. | The contributions wait in the table. Proxy-marker's reader is unbound `not_served` at once after the publication: `failed { waiting }`. A reload that adds web-shell serves and composes `Http`, the index wakes the reader, and it goes live. | Slice 8 S3: the reader fails `transient` (`unavailable/undeclared`) at once and is woken at the first value only. | §9 | +| 17 | The codemode plugin is reloaded alone with a `HarnessExtensions` token v3; the core harness serves v2. | Codemode's v3 contribution is left out and reported (`contribution_refused`). A reader bundling v3 is unbound `version`: `failed { waiting }`. A core reload to v3 makes it usable, which wakes the reader. | The contribution is refused the same way; the reader is `broken` (S2). | §9 | +| 18 | Core is reloaded alone, serving `HarnessExtensions` v3 with `accepts[2]` (contribution and value). | v2 contributions are adapted once per value; v2 readers get the composition mapped back once per composition. A reader at v1 (no mapping) is `failed { waiting, version }`. | The same, except the v1 reader is `broken` (S2). | §9 | +| 19 | The harness's `HarnessExtensions` compose throws at the first composition (an owner bug). | The registry is served but has no usable value, so the Harness's request is unbound `compose`: `failed { waiting }`. The barrier counts its first outcome. A reload that fixes compose makes the value usable, and the Harness wakes and activates. | S3: the Harness fails `transient` at once, woken at the first good value. | §1, §9 | +| 20 | Codemode contributes its sandbox (a native) to `HarnessExtensions`. Later another contributor's entry makes compose throw, and then codemode restarts. | While compose throws, `.value` keeps the last good composition. When codemode's contribution leaves, that last good value is dropped (it holds the closed sandbox), and `.value` throws until compose succeeds again. No reader is handed a closed native. | Slice 8 keeps the last good value regardless, so a reader could use the closed sandbox. | §9 | +| 21 | Transcript (on W1) follows the harness through `Harnesses.value.observeConversation(...)` and declares `retryOn(Harnesses)`. The harness's Session fails and the harness object restarts. | `self.follow` is Transcript's own step: S1 ends the observer `failed`, so Transcript fails `transient` (its own backoff). When the new harness contributes, `retryOn` wakes it, and it attaches to the new harness and gets one fresh frame. | The same (slice 8 S1 and `retryOn`); unchanged by the model. | §3 (follow), §3.8 | +| 22 | A reload of chat swaps the live Chat in place (no natives). Its new factory adds a `self.get(Sharing)`, and the sharing plugin is absent. | Owner item 4, recommended: Chat parks. Its holders see `failed { waiting, "waiting for Sharing: not served" }` and its calls reject until sharing appears. The alternative keeps the old code until the dependency resolves. | Not built (slice 7); design §21 swaps the object and opens its gate. | §3.9, §13.4 | +| 23 | Chat is parked on DraftStore (1). A chat reload ships code that no longer needs DraftStore. | One handover: the new attempt's requests bind first, then the old ones are released. Chat activates on the new code and goes live; the old `not_served` index entry goes. | Not built; today a broken Chat is retried by the reload (slice 7 plan). | §3.9 | +| 24 | A reload removes the models plugin from W1 while ModelRuntime is parked on S's ModelConfig (3). | ModelRuntime (host-lifetime) ends with its plugin. Its requests end, and W1's refused hold at S is released (a `release`). Nothing stays waiting. | The type's objects end (§21), and its backoff polling stops with it. | §3.5, §1 | + +# Scenarios: Rollouts and Cycles {#sc-rollout lede="Delayed facets across hosts, rollout windows, chains over three hosts, and the probe at work."} + +Sol's scenarios cover the windows in which hosts run different code: a server facet ahead of its worker facet and the +reverse, tabs older or newer than the server, installs that are queued, staged or rejected, a type moved between host +kinds; then chains across three hosts, diamonds and rings, the probe's own corner cases and a forged probe. Row 20 is a +case round 3 found against the previous draft; the rule that fixes it is in chapter VIII. The pattern: a version that +does not fit is a request waiting for a publication, never an error that ends it; an install answers only for its own +host; and a cycle is found by the attempt whose edge closed it. + +The Rule column cites sections of the working document, reproduced in appendix C. + +| # | Scenario | Under the model | Today | Rule | +| --- | --- | --- | --- | --- | +| 1 | The models plugin's server facet upgrades first. Browser and TUI model pickers still use its older Models contract, which the new provider accepts. | Their references remain usable through the old-version adapters. If an owning native restarts, affected references show `loading` until its fresh binding is ready. A caller that later installs the new contract releases its old request and makes a new one. | `host/host.ts:#assertMatch` accepts only exact versions and fingerprints. Accepted-version serving across reload is not built in next; p2 uses its older cluster reload runtime. | §1, §5 | +| 2 | A worker's new search facet makes SearchIndexer depend on SearchFeed v2. The server still serves v1; its search facet installs later. | SearchFeed's request is `failed { waiting }` for the version mismatch. SearchIndexer is also `failed { waiting }`, without a dependency retry timer. The server's later publication rechecks the request; SearchFeed becomes `live` and wakes the indexer. | A remote mismatch is refused as broken; its dependent fails transient and retries with backoff. See `host/attempt.ts` and `remote/hold.ts`; p2's real indexer declares this server dependency in `plugins/search/worker.ts`. | §2–5 | +| 3 | A new web workspace facet arrives before the selected worker's matching Workspace contract. The user opens the workspace panel during this window. | The panel's held Workspace reference becomes `failed { waiting }`; an awaited getter rejects under the recommended getter rule. The old worker object and compatible readers are unaffected. Installing the worker facet wakes the held request and supplies a fresh binding. | Exact-match refusal reaches the client; a dependent object can enter R18 backoff. Next has no install-aware wake protocol yet; p2's workspace UI and worker facets live in `plugins/workspace/`. | §2, §5, §8 | +| 4 | A tab sleeps while the worker upgrades Workspace and accepts its old version. The tab reconnects without updating its code. | The retained request shows `loading` while its path is down. Re-holding asks for the tab's original version, and `bound` supplies the mapped snapshot, returning it to `live`. No old method index is interpreted as a new one. | Path re-holding already exists in `remote/path.ts`, but unequal versions still fail in `host/host.ts`. Reload-time adapters are slice 7 work. | §1, §5 | +| 5 | The server publishes a new Models contract without accepting the old TUI version. The TUI launcher downloads its replacement bundle later. | The old TUI request stays held but shows `failed { waiting }`. The old runtime releases it when the launcher swaps workers. The replacement runtime makes a new-version request and can become `live`; the old request is not silently relabeled. | p2 polls for TUI bundles and swaps sandbox workers in `hosts/tui-client.ts`. Next does not yet implement reload-time version recovery. | §1, §5, §8 | +| 6 | The server and tabs already use revision R5 when a route starts a worker from an older plugin set. The worker connects before loading its current facets. | Business requests stay `loading` until publication or show `failed { waiting }` against published incompatible code. The loader installs the desired code set without awaiting activation. Publication then rechecks waiting requests and the actual objects activate; a route must not wait for the worker's activation barrier before admitting its link. | p2 sends selected worker facets through WorkerInit and uses a separate control-ready sequence. Next has the publication/activation admission deadlock described in `host/host.ts:load` and `remote/hold.ts:start`. | §1, §5–6 | +| 7 | A tab installs new workspace code while the worker's matching install is queued but has not started. | The old worker answers `failed { waiting }`, not `loading` merely because another host knows an upgrade is planned. A held reference survives; a getter rejects. The worker's own install and its completion re-evaluate the request, so it can later become `live`. | Next has no revision/install status handling; it returns an exact-match failure. The model deliberately accepts this rollout-window blip instead of adding revision fields to mismatch messages. | §2, §5, §8 | +| 8 | The worker's search facet stages a new contract, so an incompatible request is held as installing. Staging fails and the old search code remains published. | The request was `loading` during the install. Rejection is an install-end event even without publication, so the request is checked against the old provider and becomes `failed { waiting }` for the remaining mismatch. Its getter rejects rather than waiting forever for a commit that will not occur. | Next has no installer. p2's `hosts/server.ts:performReload` retains published code and bundles when preparation fails, through a different reload protocol. | §4–5, §8 | +| 9 | A worker's own envs facet install is rejected while it needs a capability from the new code set. A provider elsewhere becoming ready cannot repair that rejected local install. | Requests blocked by the worker's own rejected install fail promptly with its reason. Published old code is not replaced by half-installed new code. A later successful local install or explicit retry is needed; waiting for a remote activation is not an install strategy. | Next has no own-install state. Design §21 specifies the prompt own-rejection rule; p2 reports reload refusal while retaining its previous selection. | §5 | +| 10 | The worker ports tool facet is added before the server ports facet. The tool is registered successfully, but the first invocation looks up server Ports with `ctx.get`. | The tool owner remains `live`: this is a call-time lookup, not a declared factory dependency. The missing Ports request is `failed { waiting }`, so the invocation reports unavailable and its call-scoped request ends. After the server facet publishes, a later invocation can acquire a live Ports object. | This is how `plugins/ports/worker.ts` declares the tool: its callback does `ctx.get(Ports)`. Next's remote lookup currently receives a broken refusal; no provider-publication recovery is built. | §1–3, §5, §8 | +| 11 | The server removes Ports first, while old worker tool code and an old browser Ports sheet remain loaded. The worker and browser facets update afterward. | The browser's held request becomes `failed { waiting }` without ending permanently. Ordinary live worker owners stay `live`; their next Ports lookup fails while the provider is absent. Updating those consumers removes obsolete surfaces, or a later provider publication can restore a still-held request. | p2 normally refuses removal when outside-batch consumers still hold a removed type (`hosts/server.ts:refuseHeldRemoval`). Next's target design originally ended removed-type references; the retained-request behavior is new. | §3.10, §5 | +| 12 | Notes(session=7) moves from a server facet to a worker facet. An old tab still computes the server address; a new tab computes worker 7. | The old request stays at the old address and shows `failed { waiting }`; publishing Notes on the worker cannot wake it there. The new tab routes to worker 7 and can become `live`. Updating the old consumer computes a new address and creates a new request; the model adds no redirect. | Placement is already computed by the caller and checked by the provider in `host/placement.ts`. Next has no relocation reload implementation. | §1, §5 | +| 13 | Worker 1 still serves Notes v1 for its own session. Its new code requests Notes v2 placed on worker 2, which already serves v2. | Resolution chooses worker 2 before checking its version. The request can become `live` without waiting for worker 1's unrelated registration to upgrade. Worker 2's `bound` wakes any dependent waiting on that request. | `host/host.ts:#assertRoute` checks a same-name local registration before `Placement.reference`, so worker 1's v1 can wrongly reject this v2 request. | §1 | +| 14 | The models plugin's server facet updates now, but the worker facet updates hours later. Each host has its own native model registry and compatible local contributors. | Each host composes its own table and its local consumers can stay `live`. A worker does not wait for the server's registry revision: native registries are host-local. Only an actual remote object contract used between those facets creates a cross-host version wait. | p2 also has independent server ModelConfig and WorkerModels resources (`plugins/models/server.ts`, `worker.ts`). Next's registry adapters already handle local reader/contributor versions; atomic registry replacement is not built. | §5, §9 | +| 15 | A tab Panel depends on a worker SessionView, which depends on a server Catalog view. Catalog's new facet arrives last. | Catalog's missing request is `failed { waiting }`; SessionView and Panel show waiting with the nearest blocker and root cause. The server publication binds Catalog and makes it `live`. Its state change wakes SessionView; that provider's `bound` then wakes Panel, with no dependent backoff loops. | R18 retains dependency references but fails dependents transient and uses identity backoff. `host/object.ts:dependencyLive` already skips that backoff on a live event; registration-based recovery is missing. | §1–4 | +| 16 | A server Summary waits for views on two workers. Both views reach a shared local CatalogStore on the server, whose own required provider is absent: a diamond, not a cycle. | CatalogStore is `failed { waiting }`, and waiting propagates through the views to Summary. Probes reach the common store, but its unavailable request has no target attempt and no edge back to Summary. It must not report `invalid/cycle`; adding the missing provider later wakes the chain. | Current local waiting-edge traversal and trusted activation chains distinguish cycles from shared dependencies. Dependency failures still use R18's transient/backoff path (`host/object.ts`, `host/attempt.ts`). | §3–4, §7 | +| 17 | Thirty-three object attempts on trusted server/worker links form a ring and start together. Each reference resolves to the next attempt in that ring. | All participants stay pre-activation, showing `loading` or waiting according to their blockers. With answer (A), the attempts waiting across links time out after `readyTimeout`, retry with backoff, and the activation chains of the later attempts find the ring. With answer (B): the attempt whose edge closed the ring last starts a search; it returns, a check along its recorded path returns, and the initiator fails `broken invalid/cycle`; propagation breaks the others and the dependents off the ring. The probe carries its trail of 33 hops; no cap cuts it. | Concurrent cross-host cycles can escape fixed activation chains and rely on readyTimeout/backoff. `host/object.ts:assertNoCycle` and design §6 document that limitation; next has no probe. | §3.1, §7 | +| 18 | A ring is being confirmed when one worker reloads its facet and removes its outgoing dependency, repairing the ring. Some probe returns are already in flight. | With answer (B): New attempts have new ids, and obsolete target ids are discarded. If the check passed before the repair, the ring is reported (it existed when the check began); if the repair came first, the check dies at the replaced attempt and nothing is reported. Unchanged attempts can therefore become `broken invalid/cycle` and need `ref.retry()` or their own plugin reload; provider readiness alone does not retry broken objects. | Next has no probe or reload implementation. Broken cycle failures already require retry, and `host/object.ts:retry` starts another attempt. | §3.6–9, §7 | +| 19 | A malicious tab learns a worker attempt id and sends a forged probe naming that same attempt as its initiator through the server relay. | With answer (B): The server drops the probe because it arrives on a child link, before any forwarding. The worker object and legitimate references keep their states. A child link never carries cycle evidence. | There is no probe message today. `remote/provider.ts` already strips activation chains from untrusted holds rather than trusting a child to declare ancestry. | §7.2 | +| 20 | A→B→C wait while C's return path to A is disconnected. Sixty-five unrelated initiating rounds pass through C before reconnect makes C→A a known edge. | With answer (B): Found in round 3 against draft 3 (C could have evicted every round from the ring). Under the model the re-hold gives C a new edge token, so C starts its own search; it crosses A and B and returns to C, the check along its path confirms, and C fails `broken invalid/cycle`. Foreign rounds and evictions play no part. | Current re-hold carries an activation chain (`remote/remote-object.ts:attach`), which can detect a small newly reachable ring; otherwise the dependency deadline remains its fallback. No probe-round cache exists today. | §7.2–3 | + +# Open Questions {.epilogue lede="What the owner decides, and what slice 7's spec must still pin down."} + +The model replaces five answers to "the provider is not there" with one: the request stays, the dependent parks, an +event wakes it. It ends the startup deadlock with a gate that lasts only for the synchronous setups, finds cycles with a probe +instead of a deadline (or, if the owner prefers, one narrowed deadline), and keeps timers only where no event can exist. Four reviewers played more than 300 +scenarios against five drafts; the catalog in the chapters before this one is a selection. + +## Decisions for the owner + +1. **The barrier** (chapter VI): (a) Chord's `load` waits for the first outcomes of this host's host-lifetime objects, + with the local-blocker rule; or (b) `load` resolves after the publication and the app reads `host.startup`. + Recommended (a); Astra, Opus and Sol for (a), Fable for (b). +2. **Getters on `waiting`** (chapter IX): reject at once (recommended) or wait up to `readyTimeout`. +3. **`waiting` as a cause kind** (chapter III): a new kind (recommended; Opus, Fable), `transient` without `retryAt` + (Astra), or `loading` with a diagnostic (Sol). +4. **A pure swap whose new factory needs a dependency that is not there:** the object parks (its gate closes, its + holders see `loading` or `failed { waiting }`, calls park or reject; recommended), or it stays on the old code until + the dependency resolves (old and new code would then overlap without bound, against §21's "a host is at R or R−1"). +5. **The last good value** (chapter X): kept only while every contribution it was built from is still in the table + (recommended; Astra), or kept as slice 8 decided, with natives in a registry looked up at use through their owner + (Sol). +6. **Cycles across hosts** (chapter VIII): (B) the probe, with (B1) a re-search backoff that sends probes only + (recommended) or (B2) a full search lifecycle with no timer at all; or (A) the slice 4b decision narrowed to one + deadline, which polls chains across hosts again, misses a ring of waiting attempts, and retries in lockstep without + finding the cycle it is for. + +## For slice 7's spec + +- The test list: the startup deadlock resolving without `readyTimeout`; an early hold placed here, not through the + parent; remove-then-serve with a live dependent's calls recovering; a 1,000-hold publication wave under the window; + the probe on long rings, diamonds, edge changes, reloads, relays, reconnects and forged probes; GC of a parked + three-level chain after its top is released; every index entry gone after release, revocation and link loss. +- The local re-binding of a `Binding` (member sources, keyed pins, parked calls, presence) and the unbinding of a live + hold under flow control, both with their race tests. +- The wire: `waiting` in the `failed` cause schema, the attempt id on `loading` and `failed`, the `probe` message, the + pinned list; `PROTOCOL_VERSION` stays 1 until p3 ships. +- The design changes listed in appendix A, folded into `design.md` and the decision log in the same commit as the + owner's decisions. + +# What Changes {.appendix} + +| Where | Today | Under the model | +| --- | --- | --- | +| R18, activation | a dependency that does not go live within `readyTimeout` fails the dependent `transient`; backoff 1 to 60 s | the dependent parks with no timer; `failed { waiting }` when a dependency failed; woken on every state change of a dependency; the deadline paused while parked | +| R18, the boundary | "live" means every dependency was live at the end of activation | the dependency check is the boundary: after it, a dependency leaving `live` is treated as after live | +| `self.get` of a missing `local` type | throws in the factory: `broken` | unbound `not_served`: `failed { waiting }`, woken by the publication that serves it | +| a remote hold the provider cannot serve | refused `broken`; the dependent polls by backoff | refused and kept at the provider, re-run at its publication; `failed { waiting }` | +| design §3, placement | "served here" decides "here" | the address decides; a remote type without placement keeps today's rule | +| the start barrier, `pendingLoad` | holds wait for all of `load` | holds wait only for the publication; the barrier is (a) or (b) | +| S2, a registry reader newer than its provider | `broken` | `failed { waiting, version }` | +| S3, a registry without a value | `transient`, woken at the first value only | `failed { waiting }`, woken at every transition to usable | +| slice 8, the last good value | kept whenever `compose` throws | kept only while every contribution it was built from is still in the table | +| design §6, cycles | a walk on one host, chains across hosts, a documented gap closed by the deadline | the gap of simultaneous starts across hosts: (A) one narrowed deadline, or (B) a probe: a search per edge change, a check along the recorded path, edge tokens, `dead` notices | +| design §7, `host.hold` | throws for a missing local type after `load` | always returns a reference | +| design §7, getters | wait through every failure up to `readyTimeout`, reject on `broken` | wait through `loading` and `transient`; reject at once on `waiting` and `broken` | +| design §21, removal | `gone` to every holder | `failed { waiting, not_served }`; the request binds again when the name is served | +| design §21, a dropped version | `contract_mismatch`, final | only that version's requests unbind; the object stays | +| design §21, rollout | `updating` with revision fields and a deadline | each host knows its own install; `installing` re-evaluated at the install's end; installs never await activations | +| `FailureCause` | `transient`, `broken` | adds `waiting` | +| the wire | | `failed` cause `waiting`; the attempt id on `loading` and `failed`; with (B), one new message, `probe` (`search`, `check`, `dead`) | +| unchanged | | an object's own code reload (§21); R18 after live; `retryOn` outside parking; `self.follow`'s R4; the call window and flow control; revocation's semantics; no client providers | + +# Glossary {.appendix} + +**Attempt:** one try at activating an object: its factory, requests, natives; its own id since the model. **Bound:** a +request pointing at an object here or a hold over a path. **Dependency:** a request an attempt needs live before it +may go live (`self.get`, `self.registry`, a keyed pin). **Dependency check:** the moment all of an attempt's +dependencies are live; after it the attempt runs its own steps. **Edge generation:** a number of a wait edge, incremented by its +source whenever the request binds anew. **First outcome:** an object is live, failed by its own cause, or parked with no local +blocker still in its first attempt. **Holder:** whoever made a request and alone ends it. **Parked:** an attempt +waiting for a dependency, with no timer. **Probe:** the message that walks wait edges to find cycles: a `search`, then a `check` along the recorded path. **Publication:** +a host making its facets' declarations visible: `load`'s setups, a reload's commit, an install's commit. **Refused:** a +request without authority or with invalid params; never indexed. **Request:** what a reference stands for: type or +registry, params, principal, version. **Round:** the number of one search, from a counter per attempt. **Seen set:** the newest round number an +attempt has forwarded, per initiator (up to 256). **Unbound:** a request with no provider at the moment, with a reason (`unpublished`, `not_served`, +`version`, `compose`, `installing`, no link). **Waiting:** the cause of a failure that is not the object's own, woken by +an event. **Wait edge:** a parked attempt waiting for another attempt. **Wake:** an event that makes the resolver look +at a request again. + +# The Working Document {.appendix} + +The proposal as the reviewers checked it, `/tmp/model/model.md`, draft 4. Its section numbers are the ones the +scenario catalogs cite. + +Status: proposal for the owner, not decided. Built in slice 7 with reload and rollout. Sources: the owner's direction +("park until resolved"; "no difference between local and remote resolution: a link going is like a plugin +unloading"), the reviews (`/tmp/dep-mechanism-*.md`, `/tmp/dep-graph-*.md`, `/tmp/barrier-*.md`, +`/tmp/model/review{1,2}-*.md`), the code at pi `358f7d46a`. **Owner** marks an open decision. + +Draft 7 (round 6): (A) also retries in lockstep without finding the cycle (Fable); the probe's recovery after a +suppressed search becomes a choice between (B1) a re-search backoff for probes only (recommended) and (B2) a full +search lifecycle without timers, since `dead` notices alone need recovery state that eviction may not drop (Astra, Sol). +Causes are compared by value on wakes (Opus). +Round 7: a returned search is accepted while its initiator is parked, whatever its round (Astra: re-searches must not +invalidate searches under way); detection is eventual, not within one backoff period; the local walk stays (Opus). + +Draft 6 (round 5): cycles across hosts become an owner choice between (A) the owner's slice 4b decision (simultaneous +starts across hosts time out, no probe message), narrowed to one deadline, and (B) the probe, completed with edge +tokens, no revisits, and `dead` notices back to the initiator (Astra: a check that dies silently can lose the only +search that survived dedupe). (A) turned out to either poll chains across hosts again or miss a ring of `waiting` +attempts, so (B) is recommended, pending its review. + +Draft 5 (round 4, Sol's phantom cycle): the second round is replaced by a check along the recorded trail of +`(attempt, edge generation)` hops; a reported ring existed at one moment; probes grow with their path. + +Draft 4 after round 4 (probe only): round numbers per attempt, forwarded only when newer than the stored one; probes +honoured on parent and route links, dropped on child links; completeness stated under an eventually stable graph; the +repair caveat broadened to any concurrent edge removal; a new edge generation resets confirmation; a bounded display +trail. + +Draft 4 changes (round 3): the probe starts a fresh round on every new or changed wait edge (a ring closes at its last +edge, whose source finds it), keeps one seen entry per initiator, accepts only its current round, and states its +cost honestly; every host drops probes from untrusted links; local forwarding is iterative; no-placement remote +requests are re-placed when a host's first parent link connects; a cohort object that ends before its first outcome +settles the barrier; the last-good refinement is marked for the owner. + +Draft 3 changes (round 2): remote types without placement keep today's fallback (here if served, else the parent); +hydration stays before parking; no copied `retryAt`; a dependency `ended` breaks its dependent; a reload's handover +binds new requests before releasing old ones; retry passes only to `broken` and `transient` blockers, once per +operation; `installing` re-evaluated at every install end (commit or rejection); a registry's last good value dropped +when one of its contributors leaves; the probe with round ids, bounded seen sets, edge-generation triggers, attempt +ids on every non-live answer and relays dropping probes from untrusted links; one cycle mechanism, local and remote. + +## 0. The problem + +The same situation, "what I need is not there", has five outcomes today, depending only on how it is declared: + +| Declared as | Provider missing | Today | +| --- | --- | --- | +| `self.get(LocalType)` | no facet serves it | `self.get` throws, the factory throws: `broken` | +| `self.get(RemoteType)` | the worker does not serve it | the hold is refused `broken`; the dependent `transient`, polled 1-60 s forever | +| `self.registry(R)` | no provider | `transient`, woken at R's first value only | +| `self.registry(R)`, reader newer | | `broken` | +| a reload removes the type | | holders get `gone`: ended for good | + +And the startup barrier deadlocks an acyclic graph across two hosts (Sol, `/tmp/dep-graph-sol-barrier.log`): S's eager +A needs W's B, B needs S's C; W's hold for C waits for S's whole `load`, which waits for A. Only `readyTimeout` ends it. + +## 1. Requests and one resolver + +- **A request** is what a reference stands for: type (or registry), params, principal, the consumer's contract version. + Its holder makes it and owns it: an attempt (`self.get`, `self.registry`, keyed pins), a call (`ctx.get`), outside + code (`host.get`, `host.hold`, `host.registry`), a link's peer (`hold`). **It ends only when its holder ends it** (the + attempt ends, the call settles, `release()`, a getter gives up, the peer's link or origin closes, revocation, the + holding host closes), never because its provider is missing, went away or does not fit. +- **Placement by address** (round 1): a `local` type goes here; a `remote` type with a placement goes to the address + its placement (computed from the consumer's own contract) gives: here if that is this host's address, whether served + here or not, else over the route for that kind or the parent (design §3). A `remote` type **without placement** keeps + today's rule (round 2: Fable, Opus, Astra; no feature removed): here if served here, otherwise over the parent. A + request of that kind made before the host has a parent link waits here `not_served` and is **re-placed when the + host's first parent link connects** (round 3: Opus N5, Sol 3: a tab loads before it connects); on a host that never + connects (the root), it stays here. Opus would rather require a placement on every `remote()`; that removes a + supported declaration, so the draft keeps the fallback. A hold arriving for an address that is not this host's is relayed or + refused (`invalid/placement`) as today; a relay never captures a hold because it serves the same name. +- **Then the selected provider is checked** (served, version or accepted version and fingerprint, `authorize`, the + provider's own placement check). Version checks never look at a same-name registration elsewhere. +- **The publication gate:** a request made on this host, or arriving on its links, before the host's first + publication (§6) is not placed yet; it shows `loading` and is placed by the publication. The gate lasts only for the + synchronous setups, never for activation, so it cannot deadlock. +- **Outcomes of the check:** + + | The check finds | Request | + | --- | --- | + | a fitting provider | bound (to the object here, or to the hold over a path) | + | not published yet | unbound `unpublished`: `loading` | + | nobody serves the name at its address | unbound `not_served`: `failed { waiting }` | + | no fitting version, a fingerprint that differs at the same version, a placement the provider computes elsewhere (code skew) | unbound `version`: `failed { waiting }` | + | the registry is served, but its first `compose` (or the reader's version mapping) threw | unbound `compose`: `failed { waiting }` | + | the provider host is installing a revision and the hold does not fit its current code | unbound `installing` at the provider: `loading` | + | the path has no link (a drop, a route resolving) | on the path: `loading` | + | `authorize` refused, invalid params | `failed { broken, forbidden \| invalid }`, as today, never indexed | + + The unbound reasons that wait for a name (`unpublished`, `not_served`, `version`, `compose`) are indexed under it on + the host that decided them: the consumer's host for a local request; the provider's host for a hold that arrived + over a link (it already keeps such holds as refused: `remote/hold.ts`). `installing` is the provider host's own, + re-evaluated at its install's end; `no_link` lives on the path, where the next link holds it again. Indexes drop an + entry when its request ends; GC-tested. + +## 2. States + +`ref.state` keeps its four kinds; `FailureCause` gains one kind, `waiting`: + +```ts +type RefState = + | { kind: "loading" } + | { kind: "live" } + | { kind: "failed"; cause: FailureCause; retryAt: number | undefined } + | { kind: "ended"; error: ChordError }; + +type FailureCause = + | { kind: "transient"; error: ChordError } // the object's own effect failed (an open, a follow): backoff + | { kind: "broken"; error: ChordError } // the object's own code, a cycle, or authority: retry() or a reload + | { kind: "waiting"; error: ChordError }; // not the object's fault: no provider, no fitting version, a + // dependency failed; retryAt undefined, woken by an event +``` + +| The request is | `ref.state` | Calls | +| --- | --- | --- | +| bound, its object activating or parked with no failed blocker | `loading` | park, bounded | +| unbound `unpublished` or `installing`, or its path has no link | `loading` | park, bounded | +| unbound `not_served`, `version` or `compose` | `failed { waiting }` | reject at once | +| bound, its object failed | `failed` with the object's cause, as today | reject at once | +| bound, its object live | `live` | run | +| ended by its holder, revoked, or its host closed | `ended` | reject | + +**Owner:** `waiting` as a new cause kind (Opus, Fable; recommended: the cause says "not your fault, an event will wake +you" and a UI can say what is missing) or no new kind: `transient` with no `retryAt` (Astra; today's wire already +carries a `transient` cause without `retryIn`, `remote-object.ts`), or `loading` with a diagnostic (Sol; parks every +call on a missing plugin for its full `readyTimeout`). + +## 3. Dependents: parked until their dependencies resolve + +A **dependency** is what an attempt declares and needs before it goes live: `self.get` (local or remote), +`self.registry`, keyed pins (`at(key)` on another object's keyed member). `self.follow` is the attempt's own step (its +attach runs Durable's code), not a dependency; a follower declares `retryOn` on the registry it reads the harness +through, or recovers by its own backoff. + +1. **Order, as today up to the check:** the factory runs; persisted singletons hydrate (the object's documents, kept + across attempts); the requests are made. The attempt is **parked** while any dependency is not `live` (hydrated): + + | Its dependencies | The dependent shows | + | --- | --- | + | any `loading`, none failed | `loading` | + | any `failed` | `failed { waiting, "waiting for X: " }`, `retryAt` undefined; the message keeps the nearest blocker and the root cause within the 500-character rule | + | one failed with `invalid/cycle` (a dependent off the ring) | `broken` with the same error: it would wait on a ring that never heals by itself | + | one `ended` (revoked, or its host closed) | `broken` with that error: the request cannot come back; `retry()` or a reload makes new requests (a guard: a dependency ends only with its holder's principal or host, so the dependent ends too; Fable) | + +2. **The dependency check is the activation boundary:** once every dependency is `live`, the attempt runs its own + steps (natives in order, follow attaches, member checks, derives), bounded by its own `readyTimeout` deadline, and + goes live. A dependency that leaves `live` after the check is treated as after live (R18): the attempt goes on. This + is a deliberate rule (Astra: a weakening of "live means every dependency is live"): an object can finish opening + after a dependency went away, and its calls through that reference meet its state. So a parked attempt holds no + natives, and an attempt that opened natives is never parked. +3. **No timer while parked:** no dependency deadline, no backoff. The attempt's deadline is paused while parked (as + eager migrations pause it) and resumes with the budget left. +4. **Wakes, one rule:** the dependent re-evaluates on every state change of any of its dependencies (live, loading, + failed, a new cause compared by value, ended; Opus: otherwise a ring of waiting attempts passes its causes round and + round), updating what it shows; it leaves parking when all are live. The factory never re-runs + while parked. Wakes are coalesced per object and run after the batch that caused them; a change during the + attempt's own steps or its teardown is remembered. +5. **Kept while parked:** its documents (the object's), its declarations and requests, its seal token; no natives. + Ended with the object (its last holder; a host-lifetime object when its plugin is removed or the host closes). +6. **`broken` only for the object's own code** (a throwing factory, a derive throwing at activation, hydration, + `migrate`, a document fingerprint mismatch, an `open` throwing `broken(...)`), **cycles** (§7) and **authority** + (`forbidden`, an ended dependency). +7. **Retry passes on:** `ref.retry()` on a `failed { waiting }` object retries its `broken` and `transient` blockers, + hop by hop (the existing `retry` message over links), once per retry operation (an id dedupes diamonds); `waiting` + blockers pass it on; nothing else is touched. Retry of a `broken` or `transient` object starts a new attempt, as + today. +8. **`retryOn`** wakes failed objects as decided in slice 8 (a value call retries a `transient` or `broken` object); + while the object is parked `waiting`, a value call does nothing: its dependencies decide. +9. **Own-code reload of a parked object:** its plugin's reload replaces the parked attempt with the new factory's, + as one handover: the new requests are made and bound first, then the old ones released (round 2, Astra N4: an + unchanged dependency held only by this object must not end in between; today's `#activate` binds before + `#releasePrevious`). +10. **After live, R18 unchanged:** a dependency leaving `live` leaves its dependent live. + +## 4. Wakes: two events + +| Event | Re-evaluates | +| --- | --- | +| **a target changes state**: a dependency's object goes live, loading, fails, gets a new cause or ends; a hold's `bound`, `loading`, `failed` arrive, also after a re-hold on a new link | the requests bound to it, and their dependents (§3.4) | +| **a name becomes usable, an install ends, or the first parent link connects, on a host**: a publication (`load`'s setups, a reload's commit, an install's commit) serves a type, a registry, a version; a registry composes or maps a value at a version; an install ends without a commit (rejected) | the unbound requests indexed under those names; the provider's `installing` holds; at the first parent link, the +no-placement remote requests waiting here; a refused hold re-runs its start and sends `loading`, then `bound` or `failed` | + +No new message kind for wakes: `loading`, `bound`, `failed` and `retry` exist. The wire's `failed` carries the new cause +kind `waiting`; `loading` and a `waiting` `failed` carry the provider's attempt id for the probe (§7). + +## 5. Removal, versions, rollout + +- **A reload that removes a type:** its objects end as today (natives close, reason `end`; callers rejected at once; + handlers drain). Their requests are not ended: each becomes unbound `not_served` and binds again if a later commit + serves the name. + - **Locally:** a `Binding` whose target ended by removal becomes unbound and accepts a second `bind`: member sources + re-bind by name (one frame from the last value, as on a restart), keyed pins re-pin, parked calls flush or + reject, presence re-lists. The one new piece of machinery. + - **Over a link:** a `failed` on a bound hold unbinds it (Fable: one message, one rule; the consumer applies it as + `#startOver` plus `failed`); the provider detaches the hold's sources and drops its unsent pending ops and queued + `bound`; the ack slots of `bound`/`state` messages already sent stay counted until their `ack` or the link's loss; + calls in flight on it are rejected and free their slots once. +- **A reload that stops accepting a version** unbinds only the requests on that version (`version`); the object stays + (its own plugin's reload still restarts it if it has natives, §21). +- **A type moved to another host kind:** the request carries the address its own consumer code computed; the old host + answers `not_served`; the consumer's own reload makes a new request. No redirect. +- **A consumer whose code moves to a new version** releases its old request and makes a new one (today). +- **Rollout, without revision fields:** each host knows only its own install. A provider host that installs keeps + holds that do not fit its current code `installing` (`loading`) and re-evaluates them when the install ends, by its + commit or by its rejection. A provider host that is queued but not installing yet answers a mismatch `failed + { waiting, version }` until its install commits and wakes it: a visible blip of at most one install, accepted (Opus). + An install never waits for activations (a reverse-order rollout would deadlock); a host whose own install was + rejected fails its own requests at once with the reason (design §21). The loader keeps its own duties (the desired + code set, digests, serialized installs). +- **Old calls across a reload** follow §21 unchanged: a pure swap lets them finish on old handles; a native restart + drains them first; a removal rejects their callers at once and drains the handlers. Calls are never resent. + +## 6. Startup: publication, then first outcomes + +1. **Publication:** `load` runs every facet's setup, then publishes all at once in one batch (types served, + registries served and composed once, contributions in their tables), then wakes (§4) after that batch. Nothing + activates inside it. +2. **No gate beyond it:** holds and `host.hold` wait only for the publication, never for the barrier. This ends the + startup deadlock: W's hold for C binds as soon as S's setups published; B goes live, then A. +3. **The barrier** (**Owner:** (a) or (b); recommended (a), three reviewers of four): + - **(a):** `load` resolves when every host-lifetime object of this `load` has a first outcome: + `firstOutcome(o) = o.live || o.ownFailure || (o.parked && o.blockers.every((b) => !b.local || firstOutcome(b)))`, + where `local` means executing on this host (an object here, a keyed pin still loading here), not the contract's + kind. So `CodeMode → Harness` waits for the harness's outcome; a wait on another host or on a failed dependency + ends the first pass. One `firstOutcome` signal per object, separate from the readiness dependents wait for (Sol), + counted once, never reopened by retries. An object of the cohort that ends before its first outcome (its plugin + removed, the host closing) settles its entry as ended; `host.close` settles the waiting `load` (Astra R3). + - **What (a) promises:** every host-lifetime object on this host had its first try, and the contributions of those + that went live are in their tables. **Not:** every tool that will arrive (a parked or slow contributor, tools found + in `onLive`, MCP; Sol's probe `/tmp/barrier-sol-first-outcome.log`). What a first turn needs is p3's explicit + admission check. + - **(b):** `load` resolves after the publication; Chord adds `host.startup`, a watchable of the host-lifetime + objects' first outcomes with their causes; the app decides (Fable). The same tracker, awaited by the app. + - p2 today resumes its harness without waiting for all host objects (`apps/p2/src/hosts/worker.ts`), because its + worker objects that need server types are reachable only after `control.ready()`. + +## 7. Cycles + +On one host, cycles are found exactly as today: at `self.get`, following the waiting edges of activating and parked +attempts. Across hosts, a hold carries the activation chain it was made in (today), which finds every cycle whose +objects start one after another. What remains is the documented gap (design §6): objects on two hosts that start at +the same moment and need each other. The owner decided on 2026-10-24 (slice 4b review decisions (2)) that this is a +documented limitation that times out and retries, with no probe message. Without dependency deadlines that timeout no +longer exists, so the gap needs one of two answers (**Owner**): + +- **(A) The 4b decision, narrowed.** One deadline stays: an attempt parked on a dependency held over a link whose target + attempt is known and still activating or parked fails `transient` after its `readyTimeout` and retries with backoff; + its next attempt usually starts after the other side and its chain finds the cycle. It has a cost that decides + against it: the target being *parked* is the common case of a chain across hosts waiting for a missing plugin (a + tab's view on a worker object that waits for an uninstalled type), and those dependents would time out and poll + again, as today, so "local = remote" would not hold across hosts. Narrowing it to targets that are `loading` (not + `failed { waiting }`) avoids that but misses a real ring: A and B on two hosts wait on each other, B also on a + missing X; B is `failed { waiting }`, so A is too; when X appears, both stay `failed { waiting }` on each other, with + no deadline and no chain to find it. And even the timeout does not find what it is meant to (round 6, Fable): an + object's chain is fixed when it is created and its instance id survives retries, so a retried attempt sends the same + hold with the same chain; two simultaneous starts time out and retry in lockstep and are never found. +- **(B) The probe (recommended)**, below: no deadline at all, one new message kind, the most intricate rule set in the + model. Five + review rounds found a counterexample in each of the first four drafts; draft 6's version answers all of them, but it + has not been reviewed again, and slice 7 would have to prove it with tests before the deadline could go. + +### (B) The probe + +Replaces the activation chain in holds. On one host the `self.get` walk stays (it is synchronous and exact: a ring +with no remote edge is found there, where (B1)'s timer does not run; Opus, round 7); the probe serves the rings that +cross a link. It replaces draft 1's bounded chain too +draft 1's bounded chain (a ring of 33 is never found with chains cut at 32; chains keep stale ancestry). Draft 5 replaces the +second round by a check along the recorded path (round 4, Sol: two rounds through different stale paths can report a +ring that never existed, `/tmp/model/sol-probe4.log`). + +1. **Attempts, edges, tokens.** Every attempt has its own id (uuidv7 per attempt: a retry or reload is a new + attempt). A **wait edge** runs from a parked attempt to the parked or activating attempt it waits for, through a + bound request (unbound waits and registry reads are never edges; they are not deadlocks). Each edge carries an **edge + token** (round 5, Astra, Sol): a number from a counter per attempt, never reused, replaced by a fresh one on any + interruption or change of that edge, so two requests to one object are two edges: a new target attempt (also a + local target's retry, without a re-bind), any `loading` on a bound or pending hold (a relay's upstream re-hold + included), an unbinding, a rebinding, a re-hold after a reconnect. An attempt is parked for one interval only (one + that passed its dependency check never parks again) and ids are never reused. A provider tells its consumers the attempt they wait for on + every non-live answer: `loading { hold, attempt }` and a `waiting` `failed { hold, cause, attempt }`. +2. **A search per edge change.** Whenever a parked attempt gets a new edge or a new token on one, it starts a + search: `probe { kind: "search", round, initiator, trail }` along each of its wait edges, `round` the next number of + a counter the attempt keeps, `trail` the list of `(attempt, edge token)` hops it crossed, starting with + the initiator's own. A ring closes at its last + edge, and that edge's source starts the search that finds it. Coalesced per batch. Locally a call on the target + attempt, forwarded through a queue (no recursion); over a link the message `probe { hold, to, ... }`, consumer to + provider, relayed by hold id with the origin. **Probes are honoured on parent and route links and dropped on child + links** (`accept`), whatever `trusted` says. +3. **Forwarding.** An attempt receiving a search drops it if `to` is not its current attempt or it is not parked. If + the initiator is itself, the search returned (4). If the attempt is already in the trail, it drops it (a ring that does not contain the initiator is + found by its own members' searches; Opus). Otherwise it forwards it along every wait edge, appending `(itself, the edge's + token)` to the trail, only if the round number is higher than the one it stored for that + initiator (older or equal rounds are dropped: the initiator superseded them), and stores it. The seen map holds one + number per initiator, up to 256 initiators, and goes with the attempt. +4. **The check.** A returned search is accepted while its initiating attempt is still parked, whatever round it + carries (round 7, Astra: a re-search must not invalidate an older search still under way); the check decides. The + initiator then sends `probe { kind: "check", round, trail }` along the recorded trail: the initiator first verifies its own hop; each hop passes it on only if it is still + that attempt, still parked, still has the recorded edge token, pointing at the next hop's attempt; otherwise the + check dies (also on a `to` mismatch). The check follows the trail by position and bypasses the seen + map. The initiator verifies its own hop again when the check returns. If the check returns, the initiator fails `broken invalid/cycle`, the path named for display from the trail, + and today's propagation breaks the other activating objects on the ring and the dependents off it. 5. **No silent loss.** Dedupe can drop the copy of a search that travelled the ring in favour of a copy whose path + later breaks (Astra, round 5); then the check dies and nothing is left to find the ring. Round 6 tried `dead` + notices back to the initiator; Sol and Astra showed that reliable notices need recovery state per search that + eviction may not drop and that a long-parked attempt would otherwise keep for every initiator that ever passed (a + leak), i.e. a full search lifecycle with cancellation. Two ways (**Owner**, part of decision 6): + - **(B1) Re-search with backoff (recommended).** An attempt parked on a remote attempt starts a fresh search again + after a backoff (1 s doubling to 60 s, reset by an edge change), whether it shows `loading` or `failed { waiting }`; + the timer stops when the attempt leaves parking. No `dead` messages, no recovery state, eviction + only costs work; a new search starts within one backoff period, so every stable ring that crosses a link is + eventually found (no wall-clock bound: delivery is only fair; a ring on one host is the walk's). The timer runs as long as the attempt stays parked on a remote attempt: a view on a worker object that + waits for a missing plugin re-searches every 60 s until a reload (Fable: cheap, but for as long as it waits). The timer sends probes only: it never + fails an object, re-holds or polls a provider. Cost: one probe per parked remote dependent per period, which + crosses the downstream wait graph and dies at its leaves (an unbound wait is not an edge). + - **(B2) Events only.** Each search is owned until it settles: branches report completion or abort back along their + trails, a dropped probe answers `dead { initiator, round }` with a reverse route, an edge records the searches it + carries until they settle, and an initiator that ends or starts a new round cancels its old one along the same + routes. Complete without a timer, bounded by the searches in flight, and the most intricate protocol in Chord; not + yet specified to the reviewers' satisfaction. + Rejected: enumerating every simple trail without dedupe (Sol's correctness reference): no ring copy is ever + suppressed, but work is exponential on chains of diamonds. +- **Why a reported ring existed:** every edge of the trail kept its token from the moment the search crossed it + to the moment the check crossed it; the search crossed all of them before the check started, so at the moment the + check started every edge of the ring existed at once. A token changes on every interruption and is never reused, so an edge cannot + vanish and come back unnoticed. Sol's counterexample fails at the check: `C1` is no longer `B`'s target. +- **Why every ring is found** (under an eventually stable wait graph and fair delivery; endless edge churn can keep + invalidating searches; with rule 5's (B1) or (B2) for a search suppressed behind a path that broke): a ring is closed by its last edge; its source starts a search when the edge appears; every + other edge exists then and every attempt on the ring waits transitively on the source, so none goes live before the + search and the check return. Rings of any length, local and remote mixed. On a DAG searches die at the leaves; on a + diamond the second arrival is dropped. +- **What it does not promise:** (1) a ring that existed when the check started may be broken up by a concurrent reload, + retry, unbinding or revocation just after; it is still reported (it did exist). Its broken objects recover as every + broken object does: `ref.retry()` or a reload of their plugin. (2) Linear work under every schedule: with more than + 256 initiators passing one attempt at once, an evicted entry lets a search be forwarded again, so work can grow with + the number of simple paths (never endlessly: a search never revisits an attempt in its trail). (3) Constant message size: a probe carries its trail, so its size grows with the length of the + path it crossed (a returning search's trail is as long as its ring); no cap cuts it, since the trail is the proof. +- **Cost:** one search per edge change, crossing each downstream edge once, one check per returned search; with (B1), + one more search per backoff period for every attempt parked on a remote attempt, also while an acyclic chain waits + for a missing plugin, plus the trail bytes. The bounded memory is the seen map's; probes in flight are bounded by + the searches started. +- **Transition:** today's dependency deadline for activations waiting across links stays until the probe's tests pass + in slice 7 (long rings, diamonds, a ring closed after a reconnect with unchanged attempt ids, Sol's phantom schedule, + 300 concurrent initiators, edges added and removed, reloads, relays, forged probes); then it goes in the same slice. + +## 8. Getters, holds, calls: one rule + +- **`host.hold`** returns a reference at once; only invalid params throw (also before `load`). A missing local type + gives `failed { waiting }` after the publication, `loading` before it. +- **`host.get`, `ctx.get` and `host.registry`:** resolve when live; wait while `loading`, or `failed { transient }` + (its own retry is under way), up to their `readyTimeout`; reject at once on `failed { waiting }` or `{ broken }`. On + success, rejection or timeout their temporary wait goes; `host.registry` returns the `RegistryValue` with its + ordinary watch ownership (no `release()`). **Owner:** recommended (nobody waits 60 s for a plugin that is not + there; code that wants to wait holds instead), or wait through `waiting` too. +- **Calls** on a `loading` reference park, each bounded by the reference's `readyTimeout` from when it was made; on a + `failed` one they reject at once with the cause; sent calls are never resent. + +## 9. Registries + +- The same resolver on one host: a reader's request for R at its version is bound while R is served and has a usable + value at that version (composed; mapped by `accepts[old].value` for an older reader), otherwise unbound + `not_served`, `version` or `compose`, or `loading` before the publication. Woken at every transition to usable + (`#valued`'s first-value latch goes). S2's `broken` and S3's backoff go. +- **Version skew within one host:** a reload of the registry's plugin alone leaves other plugins' contributors and + readers on their bundled versions; `accepts[old].contribution` and `.value` keep them working, unsupported + contributions are left out and reported, unsupported readers are `waiting` (`version`). +- A registry waits for a value, never for a particular contribution (an empty table composes). After live, a later + `compose` failure changes nothing for the reader: `.value` keeps the last good composition, **but only while every + contribution it was built from is still in the table** (round 2, Astra N2): when one of them leaves (its + contributor ended or was reloaded), the last good value is dropped and `.value` throws until `compose` succeeds, so + it never hands out a native whose contributor ended. `compose` represents a missing contributor as a value. +- Registries stay host-local (contributions are in-process values). + +## 10. Keep-alive, presence, revocation, jobs + +- A request over a link counts once in `host.activity.references` from its arrival to its end, bound or unbound; + presence (`host.references`) lists only bound references. +- Revocation ends the requests of a revoked principal, bound and unbound, at every hop; nothing revives them. +- `gone` stays only for revocation (a provider host closing closes its links and sends nothing; its consumers see a + drop, `loading`, and hold again on the next link). +- Persisted job runs of a parked host-lifetime object are loaded with its documents; `run` and `finish` start only + once it is live; the run count keeps the object as today. + +## 11. Timeouts after the model + +| Timeout or backoff | After | Why | +| --- | --- | --- | +| `readyTimeout` of an attempt | only its own steps; paused while parked | a slow dependency is bounded where it runs | +| `readyTimeout` of `host.get`, `ctx.get`, `host.registry` | kept | a caller gets an answer | +| `readyTimeout` of a parked call | kept | an action must not run long after it was asked for | +| identity backoff 1-60 s | only for the object's own `transient` failures | nothing else says the disk or network is back | +| dependency backoff, polling of refused holds | removed | recovery is an event | +| dependency deadline across links | (A): only for a parked attempt whose remote target attempt stays activating or parked; (B): until the probe's tests pass, then removed | the simultaneous-cycle gap | +| probe re-search backoff (B1) | new, only with (B1): 1 s doubling to 60 s while parked on a remote attempt | finds a ring whose search was suppressed; sends probes only, never fails or re-holds | +| route backoff, heartbeat, `closeTimeout`, routine and `finish` backoffs, `linger` | kept | not dependency waits | +| rollout install timeout (not built) | a watchdog on the install only | never a deadline for dependencies | + +## 12. What changes in the design + +R18 (dependency failures `waiting`, no backoff, the deadline paused while parked, the dependency check as boundary, +wakes on every change, an ended dependency breaks), S2 and S3 (`waiting`, woken at every usable value), slice 8's +last-good rule (dropped when a contribution leaves), §3's placement table (by address; the publication gate), §6 (the +simultaneous-cycle gap: (A) one narrowed deadline or (B) the probe), §7 (`host.hold`, getters, calls), §21 (no `gone` on +removal; a dropped version unbinds only its requests; installs never await activations; rollout without revision +fields; the reload handover), the start barrier (R17, W6), `FailureCause` (+ `waiting`), `loading`/`failed` with the +attempt id, one message kind (`probe`, tagged `search` or `check`). Unchanged: what an object's own code reload does (§21), R18 after live, +`retryOn` outside parking, `self.follow`'s R4, the call window and flow control, revocation, no client providers. + +## 13. Open for the owner + +1. The barrier: (a) or (b) (§6.3). +2. Getters on `waiting`: reject at once (recommended) or wait up to `readyTimeout` (§8). +3. `waiting` as a new cause kind (recommended) or not (§2). +4. A pure swap whose new factory declares a dependency that is not there: the object parks (its gate closes, holders + see `loading` or `failed { waiting }`, calls park or reject; recommended) or stays on the old code until it + resolves (old and new code would overlap without bound, against §21's "a host is at R or R−1"). +5. **The last good value** (§9): kept only while every contribution it was built from is still in the table (draft; + Astra), or kept as slice 8 decided, with natives in a registry looked up at use through their owner (Sol). +6. **Cycles across hosts** (§7): (B) the probe, with (B1) a re-search backoff for probes only (recommended) or (B2) a + full search lifecycle without any timer; or (A) the slice 4b decision narrowed to one deadline, which polls chains + across hosts again, misses a ring of `waiting` attempts, and retries in lockstep without finding the cycle it is for. + diff --git a/packages/chord/docs/next/dependencies/index.html b/packages/chord/docs/next/dependencies/index.html new file mode 100644 index 000000000..3bc8daa6f --- /dev/null +++ b/packages/chord/docs/next/dependencies/index.html @@ -0,0 +1,2647 @@ + + +Parked Until Resolved: one model for dependencies in Chord v2 + + + + +
+
+
A design proposal · Chord v2 · slice 7 · 2026-10-29
+

Parked
Until Resolved

+

What happens when what an object needs is not there yet, is somewhere else, has gone away or does not fit, told through the scenarios that decide it

+
5 → 1answers today to "the provider is not there"
0deadlines on waiting for a dependency
300+scenarios played through by four reviewers
6decisions left for the owner
+

A proposal, not a decision. It states the dependency model the redesign session drafted on the night of +2026-10-28/29 from the owner's direction ("park until resolved"; "no difference between local and remote +resolution: a link going is like a plugin unloading") and five rounds of review by Astra, Fable, Opus and Sol, who +played more than 300 scenarios from p2's plugins and their own against it. The working document is +/tmp/model/model.md; the reviews are in /tmp/dep-graph-*.md, /tmp/barrier-*.md and /tmp/model/review*-*.md. +Code references are to pi main at 358f7d46a (packages/chord/src/next). Notes marked review record a point +the reviewers found or still disagree on; Owner marks a decision that is his.

+
+
+ + +
+

Before we begin

+

It is a Tuesday morning and Mario opens a session on his phone. The server starts a worker for it. The worker +connects to the server, loads its plugins and opens the agent's harness. One of the plugins, code mode, needs the +harness before it can contribute its tools. Another, search, needs a feed that lives on the server. A third, sharing, +was removed from the worker's plugin set last night, but the notes view still asks for it. At the same moment the +browser tab holds the session's notes, which live on the worker that is still starting.

+

Every one of these is the same question asked in different places: what I need is not there yet; what now? Chord +v2 answers it today in five different ways, depending on how the dependency was declared and where its provider +lives. A missing type on the same host breaks the object for good; the same type missing on another host makes it +retry every few seconds forever; a missing registry waits for a value; a reload that removes a type ends every +reference to it. And when two hosts start at once and need each other's objects in a chain that has no cycle at all, +both wait until a 60-second deadline breaks the knot.

+

The owner's direction was short: an object should be parked until its dependencies resolve, and there should be +no difference between a dependency on this host and one on another: a link going away is like the provider's plugin +being unloaded, and a link coming back is like it being loaded. This book works out what that means, rule by rule, +and plays it through the scenarios that decide whether it holds.

+

Eight things to know

+
    +
  1. A request outlives its provider. A reference stands for a request (type, params, principal, version). It ends +only when its holder ends it, never because the provider is missing, went away or does not fit.
    2. +
  2. One resolver for local and remote. The address decides where a request goes; the provider found there is +checked; a missing provider here and a missing provider on another host give the same answer.
    3. +
  3. Two kinds of not-yet. Something under way (activating, reconnecting, installing, not published yet) shows as +loading; nothing under way (not served, no fitting version, a dependency failed) shows as failed with the new +cause waiting.
    4. +
  4. Dependents park. An object whose dependencies are not all live waits with no timer and no backoff, and is +woken by events: a dependency changing state, or a name becoming usable on a host.
    5. +
  5. broken means the object's own fault. Only its own code and cycles break an object; whatever it waits for is +waiting.
    6. +
  6. Startup needs one short gate, not a barrier. Requests wait only for the host's facets to be published, which +is synchronous; that ends the startup deadlock. Whether load then also waits for first outcomes is the owner's +call.
    7. +
  7. One gap in cycle detection needs a choice. Objects on two hosts that start at the same moment and need each +other are caught today only by a deadline. A probe can walk the waiting edges and find the ring without failing +anything (recommended); a narrowed deadline is simpler but polls, misses a ring, or never finds the cycle at all.
    8. +
  8. Only callers and own work keep clocks. Getters, parked calls and an object's own steps keep their deadlines; +waiting for a dependency has none.
    9. +
+

Words used throughout

+

A host is one participant (the server, a session worker, a browser tab). A provider is what serves a type or +a registry on a host: a facet registered it with host.serve. A request is what a reference stands for: a type +or registry, its params, the principal it acts as and the consumer's contract version. Its holder made it and +owns it: an object's attempt (self.get, self.registry), a call (ctx.get), outside code (host.get, +host.hold) or a peer host (a hold arriving over a link). A request is bound when it points at an object (here) +or a hold (over a link), and unbound with a reason otherwise. An attempt is one try at activating an object: +its factory, its requests, its natives. A dependency is a request an attempt needs live before it may go live. An +attempt is parked while one of its dependencies is not live. A publication is the moment a host makes its +facets' declarations visible: the setups of load, the commit of a reload, the commit of an install. A wake is an +event that makes the resolver look at a request again.

+

The running example is a small p3: a server S with the session catalog and the search feed; a worker W for +session S1 with the harness, code mode, the notes store and the search indexer; and a browser tab T that shows +the notes. The plugins are core, models, codemode, notes, search and sharing.

+
+ +
I

Five Answers to One Question

What Chord v2 does today when a dependency is missing, and why the answers disagree.

+

Chord v2 was built slice by slice, and each slice answered "the provider is not there" for the case in front of it. +Slice 1 made self.get of a type nobody serves throw, because a typo in a type name should fail loudly. Slice 3b made +remote holds survive link drops, so a missing provider on another host became a refused hold that the consumer retries. +Slice 8 made self.registry a dependency that waits for a value. Each answer was reasonable alone. Together they +give five outcomes for one situation.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Declared asThe provider is missingToday
self.get(LocalType)no facet serves itself.get throws, so the factory throws: the object is broken
self.get(RemoteType)the worker does not serve itthe hold is refused broken; the dependent is transient and retried every 1 to 60 seconds, forever
self.registry(R)no plugin serves Rtransient, woken at R's first value only
self.registry(R), the reader newer than the providerbroken
a reload removes the typeevery holder gets gone: ended for good
+

The differences are not about the situation. Whether the notes plugin is missing on this host or on the worker, the +notes view cannot work until someone installs it. Yet in one case the view is broken until a reload, in the other it +polls the worker every minute for the rest of the session, and the reload that finally brings the plugin back cannot +revive a reference that got gone.

+

The deadlock without a cycle

+

The startup barrier has a worse problem. Sol reproduced it against the current code (/tmp/dep-graph-sol-barrier.log).

+
server Sload: waits for Aworker Wload: waits for BStoresetups: serve C; start eager AA needs B: hold Bsetups: serve B; start eager BStoreBStore needs C: hold CHold.start waits for S.pendingLoadS's load waits for A, A for B, B for BStore, BStore for C, C's hold for S's load60 S LATERA's activation deadline fires, A failsC binds, BStore and B go live, A is retried
Figure 1. The startup deadlock today. Nothing in the object graph is cyclic; the two barriers block each other.
+

C has no dependencies and is registered the moment S's setups ran. Its hold waits anyway, because incoming holds +wait for pendingLoad, and pendingLoad is the whole of load: the setups and the barrier that waits for every +host-lifetime object, A included. Two jobs share one promise. Holds only need the first ("are the types +registered?"); the second ("has every plugin had its first try?") is for the app. Mixing them makes an acyclic chain +across two hosts wait until a timeout fires.

+

Timers standing in for events

+

Both problems have the same root. Where Chord does not know when something will change, it polls or waits for a +deadline: the identity backoff retries a dependent whose remote provider refused it; the activation deadline breaks the +startup knot; the same deadline is the only answer to two objects on two hosts that start at the same moment and need +each other, a cycle the activation chains cannot see (design §6, "Known limitation").

+

A timer is the right tool when nothing else can tell Chord that the world changed: a disk that may be back, a worker +process that may start this time, a silent transport. It is the wrong tool when an event exists or can exist. A +missing plugin comes back through a reload, which is an event on the host that serves it. A dependency going live is +an event. A link opening is an event. The model below uses those events and keeps timers only where no event can +exist.

+
+ +
II

Requests That Wait

A reference is a request with a holder; it outlives every provider it meets.

+

The first change is a separation the code makes only halfway today: between what a reference asks for and what +currently answers it. When the notes view on W does self.get(NoteStore, {}), it asks for "the NoteStore of this +host, as the host, at contract version 3". Which object answers is a second question, and the answer can change: no +object yet, an object activating, a live one, a failed one, none again after a reload, a new one after the next.

+

A request and its holder

+

A request is the asking part: type (or registry), params, principal and the consumer's contract version. Every +request has exactly one holder, which made it and is the only one that ends it.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
HolderMade byEnds when
an attemptself.get, self.registry, a keyed pin ref.notes.at(key)the attempt ends (its object ended, or a reload replaced it)
a callctx.getthe call settles
outside codehost.get, host.hold, host.registryrelease(), or the getter gives up
a peer hosta hold arriving over a linkthe peer releases it, or its link or origin closes
any of themrevocation of its principal; its own host closes
+

That list is the whole of how a request ends. A request never ends because its provider is missing, went away or +does not fit. Today it does in two places: a self.get of a type nobody serves throws before the request exists, +and a reload that removes a type sends gone. Both go.

+

The address decides where

+

A request goes to one place, and the address decides which. For a local type, the place is this +host. For a remote type, the consumer computes the address from the params with its own contract's +placement function (design §3): Notes { session: "S1" } is placed on Session { id: "S1" }. If that is this host's +address, the request goes here; otherwise over the route for that kind, or over the parent link.

+

Today the code checks something else first: whether the type is registered here. A type placed on this host but not +registered yet is sent over the parent link, which is wrong twice. Sol's example: the server has address Server, +the catalog type is placed on Server, and no facet serves it yet. Today's Placement.reference sends the server's +own request over a parent link the server never had. And when the catalog plugin is installed later, nothing wakes +that request, because it was never registered as waiting here.

+
WHERETHENthe request's addressfrom the consumer's own placementfunctionherea local type, or this host's addressa route or the parentevery other addresscheck the provider hereserved? fitting version? authorize,own placementhold over the paththe provider host checks it the sameway
Figure 2. Placement by address. The destination follows from the address alone; only then is the provider found there checked.
+

The rule, from the reviews (Sol B1, Astra F1): placement never looks at what is registered when the type names an +address. A request placed here whose provider is missing is unbound here, waiting for the name. A version check +compares the consumer with the provider actually selected, never with a same-name registration on the consumer's own +host (today #assertRoute does that before placement, so worker W1 serving Notes v1 rejects a Notes v2 request +placed on W2, which serves v2).

+

One case keeps today's rule: a remote type declared without placement (allowed by remote(), used in tests and +for server types a tab reaches through its parent). It goes here if it is served here, otherwise over the parent. A +request of that kind made before the host has a parent link (a tab loads before it connects) waits here as not served +and is placed again when the host's first parent link connects (Opus, Sol); on a host that never connects, the root, +it stays here. Opus would rather require a placement on every remote type, which is simpler; it removes a supported +declaration, and the owner asked for no feature to go, so the draft keeps the fallback.

+

What the check finds

+

Once placed, the provider found there is checked: served, a fitting version (the current one or an accepted older one, +with the fingerprint computed from the provider's own history), authorize, and the provider's own placement check. +The request is then bound (to an object here, or to a hold over a path), or unbound with a reason:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
The check findsThe requestShows
a fitting providerboundits object's state
this host has not published its facets yetunbound unpublishedloading
nobody serves the name at its addressunbound not_servedfailed { waiting }
no fitting version, a fingerprint that differs at the same version, a placement the provider computes elsewhereunbound versionfailed { waiting }
the registry is served, but its first compose, or the reader's version mapping, threwunbound composefailed { waiting }
the provider host is installing, and the hold does not fit its current codeunbound installing, at the providerloading
the path has no link (a drop, a route resolving)waits on the pathloading
authorize refused, or the params are invalidrefusedfailed { broken }, as today
+

The reasons that wait for a name (unpublished, not_served, version, compose) are kept in an index under that +name, on the host that decided them: the consumer's host for a local request, the provider's host for a hold that +arrived over a link. The provider already keeps such holds today as refused (remote/hold.ts), and retry re-runs +them. installing belongs to the provider host's own install; no_link lives on the path, where the next link holds +the request again, as today. Every index drops an entry when its request ends, so a released getter, a closed tab or a +revoked guest leaves nothing behind. A refused request (no authority, bad params) is never indexed: a later +publication cannot grant authority.

+

The publication gate

+

There is one short gate. Before a host has run its facets' setups, it does not know what it serves, so it cannot tell +"not served" from "not registered yet". A worker connects to its parent before it loads (W6), so holds can arrive in +that window, and p2's web shell holds references before its own load resolved.

+

So a request made on a host, or arriving on its links, before the host's first publication is not placed at all: +it is unbound unpublished and shows loading. The publication places it. Without this gate the request would be +placed by a half-empty registration table (Fable's trace: W1's own Notes sent over the parent, relayed back by the +server, forever), and a tab would see "not served" during every worker start (Opus's trace).

+

The gate is not the barrier. It lasts only until the setups ran, and the setups are synchronous code at the start of +load: they register types and registries and contribute, and wait for nothing. That is the difference that ends +the startup deadlock (chapter VI).

+
+ +
III

Loading or Waiting

What a reference shows while it is not live, and why a missing plugin is an answer rather than a spinner.

+

ref.state keeps its four kinds: loading, live, failed, ended. What changes is one new kind of failure cause.

+
type RefState =
+  | { kind: "loading" }
+  | { kind: "live" }
+  | { kind: "failed"; cause: FailureCause; retryAt: number | undefined }
+  | { kind: "ended"; error: ChordError };
+
+type FailureCause =
+  | { kind: "transient"; error: ChordError }   // the object's own effect failed (an open, a follow): backoff
+  | { kind: "broken"; error: ChordError }      // the object's own code, a cycle, or authority: retry() or a reload
+  | { kind: "waiting"; error: ChordError };    // not the object's fault: woken by an event, never by a timer
+

The mapping

+

Whether a request that is not live shows loading or failed { waiting } follows from what it waits for:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
The request isref.stateCalls on it
bound, its object activating, or parked with no failed blockerloadingpark, bounded
unbound unpublished or installing, or its path has no linkloadingpark, bounded
unbound not_served, version or composefailed { waiting }, no retryAtreject at once with the cause
bound, its object failedfailed with the object's own cause and retryAt, as todayreject at once
bound, its object liveliverun
ended by its holder, revoked, or its host closedendedreject
+

The intuition is "is something under way?": an activation, a reconnect, an install, a load. When something is under +way, the reference waits and calls park; when nothing is, the reference says what is missing and calls fail fast. +The reviewers pointed out that "under way" is not something Chord can observe in general (a parent reconnect is the +app's, a resolve may hang), so the table is the rule and the intuition only explains it.

+ +

The owner's point was that a link going away is like the provider's plugin unloading. For the request they are the +same: it stays, it keeps its last values readable, it binds again when a provider is reachable, and its holder +notices nothing but the state. They differ in what is under way.

+
The notes view on T, once through a link drop and once through an unload on W

Live

T's notes view holds Notes { S1 } on W through the server. ref.state is live; the view renders the notes.

+

The link drops

W's link to the server closes (a crash, a network change). The request on T stays; the path has no link: unbound +no_link, ref.state loading. The last notes stay readable. A call made now parks, bounded by readyTimeout. +Calls already sent fail and are never resent: their outcome is unknown.

+

The link comes back

The route resolves again, the request is held anew on the new link, bound arrives with a fresh snapshot. live. +Parked calls go out in order.

+

The plugin is removed on W

A reload on W removes the notes plugin. The provider's object ends as today: its natives close, its callers are +rejected at once, its handlers drain. The request on T does not end: W sends failed { waiting, not_served } on +the bound hold, which unbinds it. ref.state is failed { waiting }; the last notes stay readable; new calls fail at once with "notes is +not served on Session S1".

+

The plugin comes back

A later reload on W serves Notes again. The publication re-runs W's refused holds for that name: loading, then +bound. T's view is live again without having done anything.

+
+

One difference must stay: calls already sent. When a link drops, a sent call's outcome is unknown, so it fails and +is not resent. When a plugin unloads on the same host, its handlers are drained on the old code (design §21). Location +decides nothing else.

+
Review: a third cause kind, or not

Opus and Fable recommend waiting as above. Astra would keep two kinds and mark event-waited failures as transient +with no retryAt (today's wire already carries a transient cause without retryIn, remote/remote-object.ts), so +no kind is added; the cost is that transient then means two things, a backoff under way or an event awaited. Sol would show loading with a diagnostic; that parks every call on a missing plugin for 60 +seconds and spins forever in the UI. The draft recommends waiting. Owner decision 3.

+
+
+ +
IV

Parked

An object whose dependencies are not all live waits, with no timer, and goes on the moment they are.

+

The owner's instinct was that an object is parked until its dependencies all cleanly resolve. This chapter makes that +exact: what a dependency is, what the parked object shows, what it keeps, and where parking ends.

+

What counts as a dependency

+

A dependency is a request an attempt makes in its factory and needs live before the object may go live:

+
    +
  • self.get(Type, params), local or remote: one rule for both;
    • +
  • self.registry(R): a registry at the reader's version (chapter X);
    • +
  • a keyed pin, ref.notes.at(key) on another object's keyed member: the key must be loaded.
    • +
+

Two things are not dependencies. self.follow(attach) is the attempt's own step: its attach runs Durable's code +and may fail like an open, with the object's own transient failure and backoff (a follower that wants to recover +the moment a new harness appears declares self.retryOn on the registry it reads the harness through). And a +ctx.get in a method is not part of the object's activation at all: it is the call's request.

+

The dependency check

+

An attempt runs its factory, hydrates its persisted singletons (they are the object's documents, kept across +attempts, as today), makes its requests, and is parked while any dependency is not live (and hydrated). What it +shows follows its dependencies:

+ + + + + + + + + + + + + + + + + + + + + + + +
Its dependenciesThe parked object shows
any loading, none failedloading
any failedfailed { waiting, "waiting for X: <X's cause>" }, no retryAt; the message keeps the nearest blocker and the root cause within the 500-character rule
one failed with invalid/cycle (a dependent off the ring)broken with the same error: it would wait on a ring that never heals by itself
one ended (revoked, or its host closed)broken with that error: the request cannot come back; retry() or a reload makes new requests
+

When every dependency is live, the attempt passes the dependency check and runs its own steps: its natives in +declaration order, its follow attaches, the member checks, its derives. Those are +bounded by its own readyTimeout deadline, as today. Then it goes live.

+

The dependency check is the boundary between waiting and working (Astra's F3). Before it, the attempt holds no +natives; after it, a dependency that leaves live is treated as after live (R18): the attempt goes on, and its calls +through that reference meet the reference's state. So a parked attempt never holds a native, and an attempt that +opened a native is never parked. This is a deliberate rule, and Astra called it what it is: a weakening of "live +means every dependency is live". An object can finish opening after a dependency went away. Without that boundary a contributor unloading while the attempt opens its natives +would leave it half open and parked, or force a seal, a drain and a fresh attempt for a dependency the object may not +even use at once.

+
CodeModehost-lifetime, localHarnesshost-lifetime, localHarnessModelsregistryself.get(Harness): loadingparked: shows loading, no timerself.registry(HarnessModels): usableown steps: open DurableHarness live: a state changedependency check passes; open the sandboxlive; contribute tools
Figure 3. Code mode on W parks on the harness and goes on the moment it is live.
+

No clock while parked

+

A parked attempt has no dependency deadline and no backoff. Its readyTimeout deadline is paused while it is +parked, as eager migrations already pause it (attempt.ts, deadline.pause()), and resumes with the budget left when +the attempt passes the check. Today a dependency that does not go live within 60 seconds fails its dependent +transient, which then retries with backoff from 1 to 60 seconds. Both go: the dependent is woken when the dependency +changes, and the time a dependency takes is bounded where it runs, by that dependency's own steps.

+

What a parked attempt keeps

+

It keeps its documents (the object's), its declarations and its requests (R18's holding today), and its seal token. +It keeps no native. It ends with its object: when its last holder releases it, or, for a host-lifetime object, when +its plugin is removed or the host closes. A parked object +with params that nobody holds any more ends and releases its requests, so a whole parked chain goes when its top is +released.

+

broken is the object's own fault

+

After the model, broken means exactly three things: the object's own code failed (a throwing factory, a derive that +throws at activation, a hydration or migrate that fails, a document fingerprint mismatch, an open that throws +broken(...)), the object is on a cycle, or it lost its authority (forbidden, a dependency that ended). Everything an object waits for, including a provider that is itself +broken, makes it waiting. The difference matters for recovery: a broken object needs ref.retry() or a reload of its +own plugin; a waiting one recovers by itself when what it waits for recovers.

+

Retry passes on

+

A Retry button on a waiting object should do something. Opus's trace: the tab's Panel waits for the server's Dash, +which waits for Catalog, which is broken by a factory bug that a configuration fix would heal at the next attempt. +The user sees "waiting for Dash: waiting for Catalog: …" and presses Retry. Today Panel would re-check, find Dash +still failed and do nothing.

+

So ref.retry() on a failed { waiting } object retries its failed blockers, hop by hop, with the existing retry +message over links: it retries the blockers that are broken or transient (the only ones a retry can change), +passes through the waiting ones, and leaves the rest alone. Each retry carries an operation id, so a diamond of +dependencies retries a shared leaf once. Retry of a broken or transient object starts a new attempt, as today. A +retryOn value call still retries a failed object as slice 8 decided; while the object is parked waiting, it does +nothing: its dependencies decide.

+

A reload of the parked object itself

+

The parked attempt's factory never runs again while it is parked, with one exception: a reload of the object's own +plugin replaces the parked attempt with the new factory's attempt, as one handover: the new requests are made and +bound first, then the old ones released (Astra: a dependency held only by this object, unchanged by the reload, must +not end in between; today's #activate already binds before #releasePrevious). If the new code no longer declares +the missing dependency, the object can go live at once.

+

After live, nothing changes

+

R18's second half stays: once an object is live, a dependency that leaves live (a link drop, a removal) leaves the +object live. Its calls that use the dependency get the dependency's state: they park while it is loading and reject +while it is failed. Reactivating every live consumer on every dependency loss would turn one worker disconnect into +closed natives, withdrawn contributions and aborted jobs on the server; the reviewers agreed it should not.

+
+ +
V

What Wakes Whom

Two kinds of event replace every poll and every dependency deadline.

+

A model without timers lives or dies by its wakes: a wake that is lost leaves an object parked forever. So the wakes +are kept to two kinds of event, each with one owner, and every unbound request and every parked attempt is +subscribed to exactly the events that can change it.

+ + + + + + + + + + + + + + + + + + +
EventWho is toldExists today?
a target changes state: an object goes live, loading or failed, or gets a new cause; a hold receives bound, loading or failed, also after a re-hold on a new linkthe requests bound to it, and the attempts that depend on those requestspartly: dependencyLive covers only "goes live"
a name becomes usable on a host: a publication (load's setups, a reload's commit, an install's commit) serves a type, a registry or a version; a registry composes or maps a value at a versionthe unbound requests indexed under that name on that host; a refused hold re-runs its start and sends loading, then bound or failedpartly: refused holds and retry exist; the re-run at publication is new; a registry wakes only at its first value
+

Every change, not only "live"

+

Draft 1 woke a parked attempt only when a dependency went live. Astra, Fable and Sol found the same hole: a dependency +that moves from loading to failed, or from one cause to another, leaves its dependent showing a stale state, and a +dependent that should now show failed { waiting } keeps parking calls. So a parked attempt re-evaluates on every +state change of any of its dependencies and updates what it shows. It leaves parking only when all of them are live. +Its factory never re-runs on a wake.

+

Three details keep this correct and cheap:

+
    +
  • Coalesced, after the batch. Wakes run after the batch that caused them (a publication, a commit, an inbound +bound), once per object, so a commit that changes three dependencies wakes their dependent once, and no factory +runs inside another owner's batch.
    • +
  • Remembered. A change that arrives while the attempt runs its own steps or tears down is remembered and applied +after (today's R18 wake memory, generalized).
    • +
  • Indexed by name. A publication touches only the unbound requests waiting for the names it serves. A reload of +notes never looks at requests waiting for search.
    • +
+ +

The wake across hosts needs no new message kind. Today a provider keeps a hold it could not serve as refused, and a +retry from the consumer re-runs its start. The model adds one thing: the provider re-runs its refused holds itself, +at its own publication, for the names that publication serves.

+
worker WIndexer, parkedserver Ssearch plugin missinghold SearchFeedfailed { waiting, not_served }Indexer shows failed { waiting }; no timer, no retriesA RELOAD ON S ADDS THE SEARCH PLUGINpublication serves SearchFeed; re-run refused holds for itloadingbound (snapshot)a target changed state: Indexer re-evaluates, passes the check, goes live
Figure 4. The search indexer on W waits for the server's feed, which a reload on S adds.
+

Two existing messages change: failed carries the new cause kind waiting, and loading and a waiting failed +carry the provider's attempt id for the cycle probe (chapter VIII). The pinned list of causes and its test change with +them, and PROTOCOL_VERSION stays 1 until p3 ships.

+

Registries

+

A registry reader is woken at every transition of the registry's value at its version to usable: when the registry +is served, when it composes for the first time, when an older reader's mapping succeeds again after it failed. Today +RegistryValueSource wakes only at the very first good value (#valued), which a registry whose provider is removed +and added again would never pass a second time.

+
+ +
VI

Starting Up

The barrier did two jobs with one promise; separated, the deadlock disappears.

+

load does three things today: it runs every facet's setup, it publishes them (types served, registries composed +once, contributions in their tables), and it waits until every host-lifetime object has had its first outcome. The +promise of the whole thing, pendingLoad, is also what incoming holds and host.hold wait for. That is the startup +deadlock of chapter I.

+

Publication first

+

The model splits load at the publication:

+
    +
  1. Setups. Every facet's setup runs. Setups only declare: host.serve, self.contribute, host.references. +They are synchronous and wait for nothing.
    2. +
  2. Publication. All declarations become visible at once, in one batch: types served, registries served and +composed once, contributions in their tables. Nothing activates inside that batch.
    3. +
  3. Wakes. After the batch, every request that waited for the publication (unbound unpublished) is placed, and +every unbound request for a name the publication serves is re-evaluated.
    4. +
  4. First outcomes. The host-lifetime objects activate. Whether load waits for them is the barrier question below.
    5. +
+

Requests wait for step 2 only. Since the setups wait for nothing, a host always reaches step 2, and a request from +another host can never wait for this host's activations. Chapter I's knot comes apart:

+
server Sworker Wsetups, publication: C servedA needs B: hold Bsetups, publication: B servedBStore needs C: hold Cbound C (S has published; C activates and goes live)BStore live, B livebound BA live; S's first pass done
Figure 5. The same startup as in chapter I, under the model. No deadline is needed.
+

The barrier: (a) or (b)

+

What is left of the barrier is the app's question "has every plugin had its first try?". R17 needs it: p3's base code +resumes the harness after load, so the first turn sees the tools of every plugin that loaded. Two shapes are on the +table.

+

(a) Chord keeps it. load resolves when every host-lifetime object of this load has a first outcome: it is +live, failed by its own cause, or parked with no blocker that is a local object still without its first outcome. In +code, per object:

+
const firstOutcome = (o: Obj): boolean =>
+  o.live || o.ownFailure || (o.parked && o.blockers.every((b) => !b.local || firstOutcome(b)));
+

So CodeMode → Harness, both local, waits until the harness had its outcome: code mode is parked on a local object +that is still in its first attempt, so code mode is not done. A wait on another host, or on a dependency that failed, +ends the first pass: the barrier never waits for another host, a retry, a later contribution or a reload. The signal +is separate from the readiness dependents wait for (Sol: resolving that for a parked object would let its dependents +open natives too early), and it is counted once per object, never reopened by retries. An object of the cohort that +ends before its first outcome (its plugin removed, the host closing) settles its entry as ended, and host.close +settles a load that is still waiting (Astra).

+

(b) The app keeps it. load resolves after the publication. Chord adds host.startup, a watchable of the +host-lifetime objects and their first outcomes with causes; base code decides when to resume (Fable). The tracker is +the same; only who awaits it changes.

+

Three of the four reviewers recommend (a): the activation graph is Chord's, and every app that wants R17's behaviour +would otherwise build the same cohort tracker. Fable recommends (b): the policy belongs to the app, and p2 already has +three different ones. Owner decision 1.

+

What the barrier does not promise

+

Neither shape can promise that every tool is there for the first turn, and the model does not pretend it can. Sol +reproduced it against today's code (/tmp/barrier-sol-first-outcome.log): object A's open fails on its first +attempt (A's first outcome); B goes live and contributes; the registry changes and retryOn wakes A for a second, +slow open; load resolves with only B's contribution in the table. Tools found in onLive (MCP) arrive later by +design. What a first turn really needs (a model, an environment, a reconciled working directory) is p3's explicit +admission check, as p2's whenRegistered and its harness prerequisites are today.

+
What p2 does today

p2 does not wait for all host objects before the first turn. Its worker (apps/p2/src/hosts/worker.ts) loads with +awaitHostObjects: false, waits for the harness, resumes Durable, tells its parent control.ready() and only then +awaits host.ready(). It has to: its subagent tools and the search indexer need server types, and the server admits +the worker's link only after control.ready(). A barrier before that point would deadlock, exactly as in chapter I +(Astra and Sol, independently).

+
+
+ +
VII

Plugins Coming and Going

Removals, versions and rollouts across hosts, as events the requests wait for.

+

Slice 7 brings reload and rollout, and with them the cases where a provider disappears on purpose. Today's design +handles them with their own mechanisms: gone when a type is removed, a final contract_mismatch when a version is +dropped, updating with a deadline during a rollout. Under the model all of them are one thing: a request becomes +unbound with a reason, and the event that fixes the reason wakes it.

+

A type removed, and served again

+

A reload that removes a type ends its objects as today: natives close with reason end, callers of running calls are +rejected at once, handlers drain. What changes is what happens to the requests that pointed at those objects.

+
    +
  • Over a link, the provider sends failed { waiting, not_served } on the bound hold. A failed on a bound hold +unbinds it (Fable: one message, one rule; the consumer applies it as today's #startOver plus failed). The +provider detaches the hold's sources and drops its unsent pending ops and any queued bound. Two things it must not +do (Sol): free the ack slots of bound and state messages already sent, which stay counted until their ack or +the link's loss, or resend calls, which fail and free their slots once.
    • +
  • On the same host, a Binding whose target ended by removal becomes unbound and accepts a second bind. Its +member sources re-bind by name (one frame from the last value, as on a restart today), keyed pins re-pin, parked +calls flush or reject, presence lists it again. This is the one genuinely new piece of machinery in the model +(Fable, Sol): today bind runs once per binding, and the ended object is forgotten.
    • +
+

When a later commit serves the name again, the publication wakes every request indexed under it, on every host that +holds one, and they bind as if nothing had happened.

+

A version no longer accepted

+

A reload can also keep the type and stop accepting an old version (it drops accepts[1]). Draft 1 ended the provider +object then; Sol and Astra caught it: the object, its documents, its jobs and its holders on the current version are +fine. Only the requests on version 1 become unbound version. The object's own plugin reload still restarts it if it +has natives (§21), as for any reload.

+

A type that moved

+

A plugin can move a type from the worker to the server by changing its placement. A consumer's request carries the +address its own code computed, so the provider appearing on the server cannot wake it: the request is not waiting +there. The old host answers not_served; the consumer's own reload, which brings the new placement, makes a new +request. The model adds no redirect: moving state between hosts is a deliberate migration, not something Chord guesses.

+

Rollouts across hosts

+

A rollout installs a revision on the server first, then on the workers, then on the tabs (design §21). Between those +installs, hosts run different code. The model needs only one rule for that: each host knows its own install, and +nothing more.

+
    +
  • A provider host that is installing keeps holds that do not fit its current code as unbound installing: the +consumer sees loading, its calls park. When the install ends, by its commit or by its rejection, the host +re-evaluates them against the code it now runs (Astra, Sol: a rejected install must not leave them asleep).
    • +
  • A provider host that is queued but not installing yet answers a mismatch with failed { waiting, version } +until its install commits and wakes the request. That is a visible blip of at most one install; Opus judged it +acceptable, and avoiding it would need revision fields in every answer.
    • +
  • Installs never wait for activations. An install that waited for its objects to go live would deadlock a rollout +that runs in the other order: the worker's objects wait for the server's new types, the server's install waits for +the worker.
    • +
  • A host whose own install was rejected fails its own requests at once with the reason (design §21).
    • +
+
The sharing plugin's contract goes from v1 to v2; the server installs first

Before

S and W run revision R. W's ShareStatus holds the server's Sharing v1; T's tab holds W's ShareBadge. All live.

+

S installs R+1

R+1 makes Sharing v2, with no accepts[1]. While S stages it, S still runs R: W's v1 hold fits the running code +and stays live. A hold that arrived now from a host already on R+1 (a v2 hold) would not fit R: it would wait +installing (loading, calls park) until the install ends.

+

S commits R+1

The install ends. S re-evaluates: v1 is not served. W's hold becomes failed { waiting, version: asked 1, served 2 }. +ShareStatus stays live (R18 after live); its calls that use sharing reject at once with the cause.

+

W installs R+1

W's new code bundles Sharing v2. Its own reload's handover makes the new request at v2 and releases the old one. The +new hold binds; ShareStatus is fully back. T's ShareBadge never noticed anything but one stale value.

+
+

Old calls

+

Calls that were running when the reload came follow §21 unchanged: a pure swap lets them finish on their old handles; +a native restart drains them first; a removal rejects their callers at once and drains their handlers. Chord never +resends a call.

+
+ +
VIII

Finding Cycles

Without the deadline, the one gap left in cycle detection needs an answer: a narrowed deadline, or a probe.

+

A provider that never appears keeps its dependents parked, and that is visible: they say what they wait for. A cycle is +different: A waits for B, B for A, every provider exists, and still neither can go live first. Today Chord has two detectors and a deadline. On one host, self.get follows the waiting edges of +activating objects. Across hosts, a hold carries the activation chain it was made in, and a provider refuses a hold +whose chain contains the object being activated. The gap is documented (design §6): when A on the server and B on a +worker start at the same moment, the holds carry [A] and [B], neither finds the other, and both fail at the 60-second +deadline. The model removes the deadline, so the gap must close.

+

Two answers

+

The owner decided this gap once already. On 2026-10-24, after the slice 4b reviews, he made simultaneous starts across +hosts "a documented limitation: they time out transient and retry; no probe message." That decision assumed the +activation deadline. The model removes the deadline for dependency waits, so the decision needs one of two +continuations, and this one is his (Owner decision 6):

+
    +
  • (A) The 4b decision, narrowed. One deadline stays: an attempt parked on a dependency held over a link, whose +target attempt is known and still activating or parked, fails transient after its readyTimeout and retries with +backoff; its next attempt usually starts after the other side, and the activation chain in its hold finds the cycle. +Working it through showed the catch. A target that is parked is the common case of a chain across hosts waiting +for a missing plugin: a tab's view on a worker object that waits for an uninstalled type. Those dependents would time +out and poll again, as today, so "local = remote" would stop at the link. Narrowing the deadline to targets that are +loading avoids the polling but misses a real ring: A and B on two hosts wait on each other, and B also on a missing +X. B is failed { waiting }, so A is too; when X appears, both stay failed { waiting } on each other, with no +deadline to break it and no chain to see it. Worse (Fable, round 6): an object's activation chain is fixed +when it is created and its instance id survives retries, so a retried attempt sends the same hold with the same +chain; two simultaneous starts time out and retry in lockstep and are never found. The deadline would poll without +finding what it is for.
    • +
  • (B) The probe (recommended, pending review) described in the rest of this chapter: no deadline at all, one new +message kind, and the most intricate rules in the model. Five review rounds found a counterexample in each of the +first four drafts; the version below answers all of them, with one choice left in rule 5. Slice 7 +would prove it with tests before the old deadline goes.
    • +
+

The rest of the chapter describes (B): it is the answer that keeps one rule on both sides of a link, and its review +history shows what a complete answer takes.

+

Why a growing chain is not enough

+

Draft 1 proposed letting chains grow: an activating object that receives a hold with a chain sends the grown chain on +to its own dependencies. Astra and Sol showed two flaws. Chains are cut at 32 entries today, so a ring of 33 objects +started at once is never found: the chain that reaches each object is always missing that object's own id. And a +chain records ancestry, not the current wait graph: once A stops waiting for B, B's chain still names A, and a later +wait of B on A looks like a cycle that is not there.

+

(B) The probe

+

The model uses an edge-chasing probe (the idea of Chandy, Misra and Haas), for the rings that cross a link; on one host the self.get +walk stays, synchronous and exact, as today (Opus). +Five rounds of review shaped it; the rules below are the result.

+
    +

  1. Attempts, edges, tokens. Every attempt has its own id, minted per attempt: a retry or a reload is a new +attempt (today the id is per instance and survives retries). A wait edge runs from a parked attempt to the +activating or parked attempt it waits for, through a bound request. Unbound waits and registry reads are never +edges: they are not deadlocks, they end when a provider appears. Every edge carries an edge token: a number from a counter the +attempt keeps, never reused, replaced by a fresh one on any interruption or change of that edge (Astra, Sol: two +requests to one object are two edges): a new target attempt (also when a local +target retries without a re-bind), any loading on a bound or pending hold (a relay re-holding upstream included), +an unbinding, a rebinding, a re-hold after a reconnect. An attempt is parked for one interval only (once past its +dependency check it never parks again), and attempt ids are never reused. A provider tells its consumers which attempt they wait for on every answer that is not bound: +loading { hold, attempt }, and a waiting failed { hold, cause, attempt }.

    +
    2. +

  2. A search per edge change. Whenever a parked attempt gets a new edge, or a new token on one, it starts a +search: probe { kind: "search", round, initiator, trail }, with the next number of a counter the attempt keeps, +its own attempt id, and the trail of (attempt, edge token) hops crossed so far, starting with its +own. A ring closes at its last edge, +and that edge's source starts the search that finds it (Sol's simplification after Astra's counterexample: a ring +that forms only after a reconnect, with unchanged attempts, is otherwise missed). Searches are coalesced per batch. +On the same host a probe is a call on the target attempt, forwarded through a queue rather than recursion; over a +link it is a new message, probe { hold, to, ... }, consumer to provider, relayed by hold id with the origin. +Probes are honoured on parent and route links and dropped on child links (accept), whatever trusted says +(Opus): a tab learns attempt ids from loading and could otherwise break an object with a forged probe, while a +worker whose transport is untrusted must still take part. Clients serve no remote types, so no real cycle runs +through one.

    +
    3. +

  3. Forwarding. An attempt that receives a search drops it if to is not its current attempt or if it is not +parked. If the initiator is itself, the search came back. If the attempt is already in the trail, it drops it: a ring +that does not contain the initiator is found by its own members' searches (Opus). Otherwise it forwards it along +every wait edge, appending itself and the edge's token to the trail, but only if its round number is higher than the one it stored for +that initiator; then it stores it. A late, older round is dropped even if this attempt never saw it: its initiator +has moved on (Astra, Opus). The seen map holds one number per initiator, up to 256 initiators, and goes with the +attempt.

    +
    4. +

  4. The check. The initiator accepts a returned search while it is still parked, whatever round it carries (Astra: +a re-search must not invalidate an older search still under way); the check decides. It verifies its own hop, then +sends a +probe { kind: "check", round, trail } along the recorded trail: each hop passes it on only if it is still that +attempt, still parked, still holding the recorded edge token, pointing at the next hop's attempt. The check follows the +trail by position, ignores the seen map and dies on any mismatch. When it returns, the initiator verifies its own hop +once more. If the check comes +back, the initiator fails broken invalid/cycle, naming the path from the trail, and today's propagation breaks +the other activating objects on the ring and the dependents off it.

    +
    5. +

  5. No silent loss. Astra found the last hole: a search can reach an attempt first by a shortcut and be dropped +there when it arrives again along the ring itself; if the shortcut breaks before the check, the check dies and +nothing is left to find the ring. Round 6 tried dead notices sent back to the initiator. Sol and Astra showed +that reliable notices need recovery state per search that the seen map's eviction may not drop, and that a long- +parked attempt would otherwise keep for every initiator that ever passed: a leak, unless every search gets a full +lifecycle with completion and cancellation. So rule 5 is a choice, part of Owner decision 6:

    +
      +
    • (B1) Re-search with backoff (recommended). An attempt parked on a remote attempt starts a fresh search again +after a backoff, 1 second doubling to 60, reset by an edge change, whether it shows loading or failed { waiting }; the timer stops when the attempt leaves parking. No dead messages and no recovery state; +eviction only costs work; a new search starts within one backoff period, so every stable ring that +crosses a link is eventually found (no wall-clock bound: delivery is only fair). The +timer runs as long as the attempt stays parked on a remote attempt: a view on a worker object that waits for a +missing plugin re-searches every minute until a reload (Fable). The timer sends probes only: it +never fails an object, never re-holds, never polls a provider. Its cost is one probe per parked remote dependent +per period, which dies at the leaves of the downstream wait graph.
      • +
    • (B2) Events only. Every search is owned until it settles: branches report completion or abort along their +trails, a dropped probe answers dead { initiator, round } by a reverse route, edges remember the searches they +carry until those settle, and an initiator that ends or moves on cancels its old search. Complete without any +timer, bounded by the searches in flight, and the most intricate protocol in Chord; not yet specified to the +reviewers' satisfaction.
      • +
    +

    Sol's correctness reference, enumerating every simple trail without dedupe, never suppresses a ring copy, but its +work is exponential on chains of diamonds.

    +
    6. +
+
A on server Sattempt a1B on worker Wattempt b1hold B (A activating)hold A (B activating)loading { attempt: b1 }loading { attempt: a1 }search { to: b1, round: 1, initiator: a1, trail: [a1·t1] }forward: search { to: a1, round: 1, trail: [a1·t1, b1·t1] }search 1 came back: check the trailcheck { round: 1, trail }: b1 still waits on a1 with token t1check passes oncheck came back: broken invalid/cycle "A → B → A"
Figure 6. Two objects on two hosts start at the same moment and need each other. The probe finds it; nothing times out.
+

Why a reported ring existed

+

Draft 4 confirmed a cycle with a second, independent round. Sol showed a schedule in which two rounds pass through two +different stale paths, each of them half replaced by reloads, and together report a ring that never existed at any +moment (/tmp/model/sol-probe4.log). The check closes that: it walks the very path the search recorded and demands +that every edge still carry the token it had when the search crossed it. A token changes on every interruption and is +never reused, and an attempt is parked for one interval only, so an edge cannot vanish and come back unnoticed (Fable and Opus +checked the argument in round 5). Every edge of the trail therefore existed from the moment the +search crossed it until the moment the check crossed it; the search crossed all of them before the check began, so +when the check began, every edge of the ring existed at once. In Sol's schedule the check dies at B, whose edge now +points at another attempt.

+

Why every ring is found

+

Under an eventually stable wait graph and fair delivery (endless edge churn can keep invalidating searches), and with +rule 5's (B1) or (B2) for a search suppressed behind a path that broke, a ring of parked attempts is closed by its last +edge. When that edge appears, its source starts a search; every other edge of the +ring exists at that moment, and every attempt on the ring waits, directly or not, on the source, so none of them can +go live before the search and its check come back. Rings of any length are found, through local and remote edges +alike. On a graph without cycles, searches die at the leaves; on a diamond, the seen map drops the second arrival.

+

What it does not promise

+
    +
  • A ring broken up just after. A ring that existed when the check began may be broken up by a reload, a retry, an +unbinding or a revocation a moment later; it is still reported, because it did exist. The objects it breaks recover +as every broken object does, through ref.retry() or a reload of their plugin.
    • +
  • Linear work under every schedule. With more than 256 initiators passing one attempt at once, an evicted entry lets +a search be forwarded again, so in an adversarial schedule the work can grow with the number of simple paths (Astra, Sol); never endlessly, since +a search never revisits an attempt already in its trail. +The seen map's memory stays bounded. Each edge change costs one search that crosses each downstream edge once, +and one check per returned search; with (B1), every attempt parked on a remote attempt adds one search per backoff +period, also while an acyclic chain only waits for a missing plugin.
    • +
  • Constant message size. A probe carries its trail, so it grows with the length of the path it crossed; a search +that returns carries a trail as long as its ring. That is the price of the check; chains were cut at 32, trails are +not, because the trail is the proof.
    • +
+
Review: the state of (B)

The check's argument (a reported ring existed at the moment the check began) was confirmed by all four reviewers in +rounds 5 and 6, given exact edge identity (the tokens). Completeness under an eventually stable graph holds with +(B1); with (B2) it holds once the search lifecycle is specified, which no draft has done yet. If the owner chooses (B), +slice 7 keeps today's deadline for activations waiting across links until the probe's tests pass: long rings, +diamonds, a ring closed after a reconnect with unchanged attempts, Sol's phantom schedule, Astra's broken shortcut, +Sol's eviction schedule, 300 concurrent initiators, edges added and removed, reloads, relays, forged probes.

+
+
+ +
IX

Callers

Getters, holds and calls get one rule: wait while something is under way, answer when nothing is.

+

Objects park; callers do not. A caller is code that wants an answer: a method that needs a reference now, a tool that +runs inside a turn, a tab that wants to render. The model keeps their deadlines and gives them one rule.

+

host.hold

+

host.hold returns a reference at once and never rejects; only invalid params throw, also before load. That was +already the design; the model removes today's inconsistency: a missing local type throws after load but fails the +reference before it. Now a missing local type gives loading before the publication and failed { waiting } after +it, and the reference binds when a provider appears. A UI that wants to wait for something that may never come +holds it and shows its state.

+

host.get, ctx.get, host.registry

+

The getters resolve when the reference is live. While it is loading, or failed { transient } (its object's own +retry is under way), they wait, bounded by their readyTimeout. On failed { waiting } or failed { broken } they +reject at once. On success, rejection or timeout their temporary wait goes; a rejected getter releases its request. +host.registry returns the RegistryValue, whose watches and observers are owned as they are today; it has no +release().

+

Opus's case decided the recommendation. A Durable tool runs chord((ctx) => ctx.get(SharingView, { session }), ctx) +while sharing is not installed. If getters waited through waiting, the tool, and with it the agent's turn, would +hang for 60 seconds and then fail; a call on the same reference would fail at once. With the rule above, the getter +and the call give the same answer at the same moment. The rule has a cost Opus named: a dependent whose only blocker is +an object recovering by itself (an SSH environment coming back, transient with its own backoff) is waiting, so a +getter on the dependent rejects at once. Code that wants to ride that out holds the reference instead of getting it.

+
Owner decision 2: getters on waiting

Recommended: reject at once (Opus; Fable agreed in round 2). The alternative waits through waiting up to +readyTimeout, which keeps "local = remote" just as well but makes every getter on a missing plugin cost its full +deadline.

+
+

Calls

+

Calls on a loading reference park in order, each bounded by the reference's readyTimeout from the moment it was +made (never renewed by a reconnect or a reload). Calls on a failed reference reject at once with the cause. A call +already sent when its link drops, or when its object ends, fails and is never resent; its outcome is unknown, and +idempotency is the app's (a requestId, as p2's chat does).

+

The rule in one table

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
The referencehost.holdgetterscalls
loadingreturns itwait, boundedpark, bounded
failed { transient }returns itwait, boundedreject
failed { waiting }returns itreject at oncereject
failed { broken }returns itreject at oncereject
livereturns itresolverun
+
+ +
X

Registries

The same resolver on one host, with versions that can differ inside one revision.

+

A registry is host-local: its contributions are in-process values, often natives. Slice 8 made self.registry(R) a +dependency that waits for a value. The model keeps that and folds it into the one resolver.

+

A reader's request

+

A reader asks for R at the version its own bundle was built against. The request is bound while R is served and +has a usable value at that version: composed, and for an older reader mapped back by the provider's +accepts[old].value. Otherwise it is unbound: not_served (no plugin serves R here), version (the provider does +not accept the reader's version, or the reader is newer), compose (R is served, but its first composition, or this +version's mapping, threw), or loading before the publication. The reader is woken at every transition of the value +at its version to usable. S2's broken for a newer reader and S3's backoff go; so does #valued, the latch that woke +readers only at the very first good value.

+

A registry waits for a value, never for a particular contribution: an empty table composes. "Wait until model X is +there" is the app's admission check, not a registry rule; Durable's own "model unavailable" wait covers the harness.

+

Versions inside one host

+

Draft 1 claimed that a registry's provider and its contributors always run the same version, because a host installs +a revision atomically. Astra corrected it: a revision is a set of plugin bundles, and a reload of the plugin that owns +the registry leaves every other plugin's contributors and readers on the version they were built against. So the +provider's adapters do real work inside one host: accepts[old].contribution maps an older contribution, accepts[old]. value maps the composition back for an older reader. A contribution without an adapter is left out and reported; a +reader without one is waiting with version.

+

The last good value, and natives

+

Slice 8 decided that a throwing compose keeps the last good composition, so an extension point stays available while +its owner fixes a bug. Astra found the case where that hands out something dead: the harness contributes its native +H; a reader is live; the harness ends; the next composition throws for an unrelated reason; .value still returns +the composition that contains the closed H.

+

The model narrows the rule: .value keeps the last good composition only while every contribution it was built from +is still in the table. When one of them leaves (its contributor ended or was reloaded), the last good value is dropped +and .value throws until compose succeeds again. And compose represents a missing contributor as a value +(absent), never by throwing.

+
Review: drop the last good value, or fence the reads

Sol would keep the last good value and instead require natives in a registry to be looked up at use through their +owner, without the guarantee. The draft takes Astra's narrower rule, because it keeps "a registry never hands out a +native whose contributor ended" without asking every contributor to wrap its natives. It narrows a decision of slice 8, +so it is the owner's: Owner decision 5.

+
+
+ +
XI

Keep-alive, Presence, Revocation

What a waiting request costs, who counts it, and who may end it.

+

A parked object and an unbound request are resources: they hold memory, they may keep a worker process alive, and +they act for a principal. The model says how each of those works while nothing is bound.

+

Keep-alive and presence

+

A worker stays alive while host.activity shows work: references held over links, calls, jobs, lingering objects. A +request that arrived over a link counts once in host.activity.references from its arrival to its end, bound or +unbound. Without that, a session whose tab waits for a plugin that is not installed would see its worker stopped every +idle period (five minutes in p2) and restarted by the next re-hold, forever (Opus). Presence is different: +host.references lists only bound references, so "who is looking at these notes" counts viewers, not waiting +requests.

+

Revocation

+

Revocation ends the requests of a revoked principal, bound and unbound, at every hop. A guest whose grant is revoked +while their tab waits for a missing plugin does not get a reference later when the plugin comes back: the request is +gone, and nothing revives an ended request (no publication, no Retry, no new grant). A refused request (no authority) +is never indexed in the first place.

+

gone

+

gone remains for one thing: revocation. A provider host that closes closes its links and sends nothing (today's +#close); its consumers see a link drop, loading, and hold again when the host comes back. A reload that removes a +type sends failed { waiting } instead of gone (chapter VII).

+

Parked objects and their jobs

+

A parked object with params lives while it is referenced, like any object; when its last holder releases it, it ends +and releases its requests, so a parked chain goes from the top down. A host-lifetime object parks until its plugin is +removed or the host closes. Persisted job runs of a parked host-lifetime object are loaded with its documents, before +it parks; run and finish start only once it is live, and the run count keeps the object as today.

+
+ +
XII

The Clock

Which timers stay, and why each of them has no event to replace it.

+

The owner expected the model to remove almost every timeout, "except maybe readyTimeout". That is what it does for +dependencies. Every remaining timer bounds something no event can end: a caller's patience, an object's own code, a +silent transport, a process that may start this time.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Timeout or backoffAfter the modelWhy
readyTimeout of an attemptonly its own steps (natives, follow attaches, checks, derives); paused while parkeda slow dependency is bounded where it runs, by its own steps
readyTimeout of host.get, ctx.get, host.registrykepta caller must get an answer
readyTimeout of a parked callkeptan action must not run long after it was asked for
identity backoff 1 to 60 sonly for the object's own transient failures (an open, a follow)nothing tells Chord that a disk or a network came back
dependency backoff, polling of refused holdsremoveda dependency's recovery is an event
dependency deadline for waits across linksanswer (A): kept only for an attempt waiting on a remote attempt that stays activating; answer (B): kept until the probe's tests pass, then removedthe simultaneous-cycle gap (chapter VIII)
probe re-search backoff, only with answer (B1)new: 1 to 60 s while parked on a remote attemptfinds a ring whose search was suppressed; sends probes only, never fails or re-holds
route backoff per addresskepta failed worker start has no event; it avoids fork storms
heartbeat 15 and 30 skeptthe only detector of a silent transport
closeTimeout (natives, the drain, the routine handover)keptbounds other code's teardown
routine and job finish backoffskeptthe object's own code throwing
lingerkepta policy for warm objects, not a wait
rollout install timeout (designed, not built)a watchdog on the install onlynever a deadline for dependencies
+

So readyTimeout stays, but its meaning narrows to two things: how long a caller waits, and how long an object's own +activation steps may take. Waiting for someone else never counts against it.

+
+ +
XIII

A Morning, Step by Step

The preface's Tuesday morning, played through under the model from the first click to the afternoon's rollout.

+

The rules of the earlier chapters meet in one story. Mario opens session S1 on his phone; the server S has the catalog +and the search feed; the worker W for S1 has not started. W's plugin set has core, models, codemode, notes +and search, but not sharing, which was removed last night; the notes view still asks for it. Later in the day a +rollout brings sharing back, with a new contract.

+
Session S1 on Tuesday

1. The tab asks for the notes

T calls host.hold(Notes, { session: "S1" }). The address is Session { id: "S1" }, not T's own, so the request goes +over T's parent, the server. S has a route for Session: it resolves the address by starting W. T's reference shows +loading; the request waits on S's path until W's link exists.

+

2. W connects before it loads

W connects to S first (W6), then starts load. The hold for Notes arrives before W has published its facets: it is +unbound unpublished and waits, still loading for T. Nothing is placed by a half-empty table, so the hold is not +sent back to S.

+

3. W publishes

W's setups run, synchronously, and are published in one batch: Notes, NoteStore, Harness, CodeMode, Indexer +are served; HarnessModels and HarnessExtensions are served and composed once. After that batch, the waiting hold +for Notes is placed and binds; the host-lifetime objects start.

+

4. Code mode parks on the harness

CodeMode declared self.get(Harness). The harness is still opening Durable, so CodeMode parks: loading, no timer. +The harness reads HarnessModels, which has a value (composed at the publication; the model runtime's contribution may arrive +after), finishes its own steps +and goes live. That state change wakes CodeMode, which passes its dependency check, opens its sandbox, goes live and +contributes its tools.

+

5. The indexer waits for the server

Indexer needs the server's SearchFeed. The hold goes over W's parent; S published long ago, so it binds at once, +SearchFeed is live, and Indexer goes live. Had S been starting at the same moment, the hold would have waited only +for S's publication, never for S's activations: the startup deadlock of chapter I cannot form.

+

6. The first pass

With barrier (a), W's load resolves once Harness, CodeMode and Indexer had their first outcomes. CodeMode +waited for the local harness, so its tools are in the table; base code resumes the harness, and the first turn sees +them. With (b), base code would read host.startup and decide the same.

+

7. The notes view asks for sharing

Notes's factory declared self.get(ShareStatus), a local type of the sharing plugin. Nobody serves it on W: the +request is unbound not_served, indexed under the name. Notes parks failed { waiting, "waiting for ShareStatus: not served on Session S1" }. T's reference shows the same; a call from T rejects at once with that cause. No backoff, no +polling, no 60-second wait.

+

8. The phone changes networks

T's link to S drops. T's reference goes loading: its request stays, its last value stays readable, a call made now +parks. A call that was in flight fails and is not resent. T reconnects; the request is held again on the new link; W +answers as before: failed { waiting }.

+

9. Sharing comes back, with a new contract

In the afternoon a rollout brings the sharing plugin back at revision R+1, where Sharing on the server is v2. S +installs first and serves Sharing v2. W's install is queued; nothing on W asks for Sharing yet, because the sharing +plugin is not installed there, so nothing changes for W. (Had W still run a sharing facet built against v1 without an +adapter on S, its requests would show failed { waiting, version } until W installed: the ordinary window of a +server-first rollout.)

+

10. W installs R+1

W's install commits: a publication. It serves ShareStatus again: the index wakes Notes's request, which binds. +ShareStatus activates, holds the server's Sharing at v2 (W's new code bundles v2), goes live, and Notes passes its +dependency check, opens what it needs and goes live. T's reference becomes live without T having done anything.

+
+

What did not happen is the point. Nothing waited for a deadline; nothing retried on a timer; no reference ended +because a plugin was missing; no host waited for another host's activations; and every waiting object said what it +was waiting for.

+
+ +
XIV

Scenarios: Starting Up

Twenty startups on p2's real hosts, under the model, with the barrier as (a) and as (b).

+

These scenarios, from Astra, follow p2's real hosts: the server alone, workers started by a tab or by another worker, +web tabs, the TUI, restarts in the middle. Each row says what the model does, with the barrier in both shapes where +they differ, and what today's code does. The pattern across all of them: publication never waits for activation, so +dependency traffic between hosts always moves; what differs between (a) and (b) is only when base code is told that +the first pass is over, and neither shape promises every tool before the first turn.

+

The Rule column cites sections of the working document, reproduced in appendix C.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
#ScenarioUnder the modelTodayRule
1The server starts alone. Catalog, ModelConfig, LoginService and ForwardStore initialize before any session worker exists.Their local references are loading during initialization, then live or failed with an own cause. Under (a), load waits for these first outcomes; under (b), it returns after publication. Public admission still needs explicit security prerequisites; a completed first pass is not proof that every service works.p2 starts its HTTP listener but gates requests and upgrades behind plugin startup (apps/p2/src/hosts/server.ts). Current next waits for eager objects' first live/failed outcomes (host/host.ts:load).§§1–3, 6
2A tab's request reaches the server before its first facet publication. Catalog is declared by a later setup in that same load.The request stays loading behind the publication gate. The complete declaration batch places it; Catalog activation then produces bound and the reference becomes live. Neither (a) nor (b) delays that hold until unrelated eager objects finish.Current next's Hold.start waits for pendingLoad, which includes every eager object's first outcome (remote/hold.ts). It waits longer than declaration safety requires.§§1, 4, 6
3Several server facets set up successfully, but the last setup throws. A tab is already waiting for Catalog.No partial declaration batch may activate objects. Load fails and the host closes; local held references become ended, while the tab sees a link drop and remains loading. A replacement server's successful publication can satisfy the tab's retained request.Current next closes the host on a throwing setup and activates no objects (host/host.ts:load). Setup failure rejects load directly.§§1, 6, 10
4A session worker's Harness opens slowly. Its CodemodeRuntime startup object declares Harness and contributes tools only after its own sandbox opens.Harness and CodemodeRuntime start loading; CodemodeRuntime waits without a dependency timer. Harness becoming live wakes CodemodeRuntime, which opens its sandbox and becomes live. Under (a), this local chain finishes before load returns and resume; under (b), the application must await the same startup outcomes or risk a first turn without those tools.p2 resumes after Harness is ready, before host.ready() (apps/p2/src/hosts/worker.ts); CodemodeRuntime depends on Harness (plugins/codemode/worker.ts). Current next's load barrier waits for both first outcomes.§§3, 6, 11
5A tab opens Chat for a stopped session. The server starts that session worker, whose Search Indexer needs server SearchFeed.Chat remains loading while the route starts the worker. The worker connects and publishes before its application barrier; SearchFeed can bind immediately on the already-published server. Indexer becomes live when its prerequisites are ready, and Chat receives bound once its own prerequisites are ready. Under (a), a still-remote startup blocker can already count as a parked first outcome; neither option guarantees that every remote-backed feature precedes resume.p2 connects IPC first but the parent admits it only after control.ready(); Harness resumes before the later host.ready() (src/hosts/worker.ts). Current next gates incoming holds on whole-host load (remote/hold.ts).§§1, 3–4, 6
6A running worker calls ConversationBroker on the server. The broker sends to another session whose worker is stopped.Only the second worker starts a new publication and first-pass sequence. Its PeerConversation reference is loading until that view binds; the first worker and broker remain live. Under (a), the new worker's scheduler waits for its first-pass report; under (b), its application must supply that admission rule. This call does not reopen the first worker's startup barrier.p2 gets PeerConversation inside the broker's call and starts its worker through the session route (plugins/conversations/server.ts, src/runtime/server.ts). The spawned worker uses p2's Harness-before-host-ready ordering.§§3, 6, 8
7Many tabs open the same stopped session while other tabs open several different sessions.Requests to one worker share route resolution and provider identities where their principals match. Each tab's reference starts loading and becomes live on its own binding. Both barrier options remain per host; a slow worker cannot hold back every other worker's first turn. The application shows per-session startup status, not one global ready flag.Current next shares resolution per route address (host/route.ts). p2's SessionHosts also serializes starts per session (apps/p2/src/runtime/server.ts).§§1, 6
8During server startup, an eager Bootstrap object needs a worker's Agent. That worker's eager FeedOwner needs the server's independent Catalog.Server and worker publication happen independently of their first-pass reports. Catalog binds while Bootstrap is still loading; that wakes FeedOwner, then Agent and Bootstrap can become live. Both (a) and (b) avoid a host-ready gate on this dependency traffic. The acyclic object graph needs no timeout to make progress.Current next can deadlock its whole-host load gates until activation deadlines fail attempts: the Agent hold waits for worker load, while the Catalog hold waits for server load (host/host.ts, remote/hold.ts). This is an architectural example using p2-style server/worker dependencies, not a claim that p2 has a Bootstrap plugin.§§1, 4, 6
9The worker has a healthy Harness, but its p2.subagent-tools startup object still waits for server Subagents. The first generation is ready to run locally.p2.subagent-tools remains loading while its remote reference binds, or failed with waiting if that provider is unavailable. Under (a), the remote wait can settle its first outcome without its contribution; under (b), publication returns even earlier. Server bound later wakes p2.subagent-tools and publishes its tools for future phases. If the first turn requires those tools, application admission must wait explicitly.p2 p2.subagent-tools declares both Harness and server Subagents (plugins/subagents/worker.ts). Its worker resumes Harness before admitting the upstream-dependent host objects (src/hosts/worker.ts).§§2–4, 6
10In a p3 port, the core serves a Models registry whose empty composition is valid. The optional models contributor is absent when the worker starts.The registry is usable, so this absence alone does not park Harness; Harness can become live. Neither (a) nor (b) waits for an uninstalled contributor. The generation's model selection or application admission must report model unavailable and wait for a contribution. Registry publication later wakes that domain-level wait; it is not a missing-provider activation failure.p2 instead awaits the HarnessConfig default entry inside Harness's Durable opener (src/runtime/worker.ts:serveHarness). Without the models contribution, that opener does not finish normally.§§3, 6, 9
11A session is configured for SSH. Harness's execution opener finds that the environment is invalid or cannot open.Harness becomes failed with its own transient cause, or broken if the opener marks the error permanent. Dependents still before their readiness check become failed with waiting. Under (a), this is a first outcome, not a healthy Harness; under (b), load already returned but resume still requires a usable Harness. Effect retry or a configuration-triggered retry can later make it live; no local-execution fallback is implied.p2 opens the session machine during Harness activation and fails rather than silently using the local machine (src/runtime/worker.ts). Worker startup failures are surfaced and can require explicit retry (src/runtime/server.ts).§§2–3, 6, 11
12Prompt becomes live before its initial local instruction-file read finishes. A user immediately sends the first message.Prompt's reference can be live while its file-loading work continues. Under (a), its InputGate contribution is present before a normal local first pass completes; the gate still controls when local input may proceed. Under (b), the application must ensure that contribution exists before admitting input. The model does not turn background file loading into a host-wide barrier.p2 contributes InputGates and awaits the initial local load in its prompt sections; remote instructions deliberately load in the background (plugins/prompt/worker.ts). Object readiness is already distinct from instruction readiness.§§3, 6, 9
13Search starts an archived session worker only to rebuild its historical index. Indexer has a long scan to perform.Once SearchFeed and Harness are usable, Indexer becomes live even while its scanned state is false. Both (a) and (b) leave scan completion to the explicit IndexScan operation. The archived-worker application keeps Harness scheduling paused, waits for the scan result and then stops the worker. A startup report does not override that resume policy.p2 passes resume:false for archived index-only starts, exposes IndexScan.scanned, and stops the archived worker after rebuilding (plugins/search/server.ts, plugins/search/worker.ts, src/hosts/worker.ts).§§3, 6
14After a worker starts, a tab opens Workspace for a large directory. Its filesystem watcher is slow to initialize.The on-demand Workspace reference is loading until the backend is usable, then live. Its background watch may still be starting. Neither barrier option waits for every future Workspace view or directory watch; a component that needs a complete initial scan must use explicit domain readiness.p2 deliberately returns the Workspace backend before installing a potentially slow watcher (plugins/workspace/worker.ts). This work is not a universal worker-start prerequisite.§§3, 6
15A worker restores durable work that can resume immediately. No new Chat.send call occurs, so there is no incoming-input gate to intercept it.Harness opens with scheduling paused and can become live before all startup contributors. Under (a), base code resumes after the first-pass report and required application checks; under (b), base code must await equivalent outcomes itself. Input gates on new messages do not guard this replay. Each phase pins the contributions present when that phase begins.p2 explicitly calls Harness.resume before host.ready() (src/hosts/worker.ts). Its Durable registry adapter snapshots the extension table per phase/version (src/runtime/durable.ts).§§6, 9
16A previously loaded web tab starts while the server is offline. Its local UI facets can render an offline shell.Publication of the tab's local facets does not require the server. Remote references stay loading on the absent path. Under (a), remote blockers can count as parked first outcomes; under (b), declaration publication returns directly. The shell renders, and later connection plus bound messages make the affected views live.p2 starts reconnection before facet installation and renders React without awaiting a whole-host ready promise (src/hosts/web/main.tsx). It already supports an offline shell after the page has been loaded.§§1–2, 6
17A TUI candidate loads before its launcher activates BridgeLink. A startup UI object declares a server reference, whose outgoing frames are buffered.The reference remains loading on a remote dependency. Under (a), that remote wait is a parked first outcome, so candidate preparation can finish; under (b), publication alone finishes. Launcher activation flushes the frames, and bound wakes the UI object. Do not replace this dependency declaration with an own native opener that waits indefinitely for launcher activation.p2's setupTui awaits facet load while BridgeLink buffers frames until activate (src/hosts/tui-worker-runtime.ts). Adding an eager remote-dependent host object under the old full-ready load rule could stall candidate preparation until a deadline.§§1, 3, 6, 11
18A tab starts a worker, but the process crashes while Harness is still opening. The server launches a replacement.The tab's Chat reference remains loading across the lost path and replacement startup. The replacement has its own publication and first-pass cohort; successful binding makes Chat live. Both (a) and (b) must keep transport/start failure distinct from a healthy startup report. A process manager may surface a startup error and require user retry rather than retrying forever.p2 records worker startup failures and has an explicit retry path; it does not treat every failed boot as an automatically healing object (src/runtime/server.ts:SessionHosts). Current next's route layer independently retries failed resolution (host/route.ts).§§1–2, 6, 11
19The server restarts while a worker is still starting. A detached terminal host already owns running PTYs.Tabs see their server and worker references become loading; new hosts publish and bind them again. The application may stop old workers on parent loss, while the detached PTY owner survives. The old server's unfinished first-pass wait settles on close (its entries end). New startup reports must not wait for the detached PTYs to restart.p2 workers stop on IPC disconnect, while the terminal driver detaches its socket without killing the detached PTYs (src/hosts/worker.ts, plugins/terminals/detached.ts). Current next settles ended objects through its existing load settlement path.§§1, 6, 10
20A startup contributor in a v2 port reads a keyed configuration document from a local store. The store is live, but the first requested key still needs hydration.The contributor remains loading until that pin is ready; store-live alone does not make the dependency usable. Under (a), this local key-loading work remains part of the first pass, then the contributor can become live and publish before resume. Under (b), publication returns earlier, so the application must await that readiness if the first turn needs the contribution.Current next loads dependency pins before opening the consumer's natives and before its settled promise completes (host/attempt.ts:pinKeys). Its activation deadline currently also bounds this wait. This is a v2-port example, not an existing p2 keyed API.§§3, 6, 11
+
+ +
XV

Scenarios: References, Holds and Calls

Getters, holds, calls, link drops, relays, release and revocation, played through.

+

Fable's scenarios use p2's names: Catalog and Hosts (the sessions plugin, on the server), Sessions (on the +server, called from worker tools), SessionTransfer (a worker type the server calls for a fork), Notes { session } +(a worker type a tab holds), NoteStore (local on the worker), ForwardStore (ports, on the worker) and Forwards +(radius, on the server), and Harness (host-lifetime, on the worker). S is the server, W1 the worker of session S1, T +a web tab. The pattern: a request waits where something is under way and answers where nothing is; nothing a holder +does is needed to recover; and an ended holder leaves nothing behind in any index.

+

The Rule column cites sections of the working document, reproduced in appendix C.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
#ScenarioUnder the modelTodayRule
1self.get(NoteStore) in Notes { S1 }'s factory on W1; NoteStore is served and activating.Notes makes its request, it binds to the NoteStore object, Notes parks loading. NoteStore goes live: a target state change wakes Notes, which opens its natives and goes live. No timer ran.The same wait, bounded by readyTimeout; past it Notes fails transient and backs off (attempt.ts waitFor).§3.1, §3.4, §4
2self.get(Catalog) from W1's ForwardStore; the parent link is up, S has published, Catalog is served there.The request goes over the parent (placement Server()); S binds the hold; bound arrives; ForwardStore was loading, now goes on and live.The same, after S's whole barrier (Hold.start awaits pendingLoad).§1, §2
3The same, but Catalog's plugin is not installed on S.S indexes the hold not_served under "p2.catalog" and answers failed { waiting, not_served }; ForwardStore's reference shows failed { waiting }, the object parks failed { waiting, "waiting for p2.catalog: not served" }. A reload on S serving Catalog re-runs the hold: loading, bound; the reference goes live, the dependent wakes. No backoff, no new hold.The hold is refused broken; ForwardStore fails transient and re-holds every 1-60 s, refused each time (remote/hold.ts #refuse, object.ts backoff).§1, §3.1, §4, §5
4self.get(SessionInfo) on W1 when the core facet is missing (same host, not served).The request is unbound not_served on W1 itself; the dependent parks failed { waiting }; a reload serving the type binds it and wakes the dependent.#assertRoute throws in the factory: the object is broken until retry() or a reload (host.ts reference).§1, §3.1, §4
5A tab's NotesView declares Notes { S1 }; S routes Session, resolve forks W1, W1 connects and publishes.The hold waits at T on its path (no_link while S resolves: loading), is relayed to W1, waits there unpublished until W1's setups, then binds; bound comes back through the relay. NotesView was loading the whole time and goes live.The same chain of waits, each bounded by readyTimeout; a slow worker start past it fails the tab object transient and polls.§1, §6.2
6host.get(Notes, { session }, ctx) from a tab's page code while W1 is starting.The getter waits while the reference is loading, up to its readyTimeout, then rejects with the cause and releases its request. It resolves as soon as bound arrives.The same.§8
7host.get(Catalog, {}, ctx) on W1 when S does not serve Catalog.The reference becomes failed { waiting } at the first answer; the getter rejects at once with "not served on server" (recommended; owner's choice to wait instead).Rejects at once too (a broken failure), after the refused hold's failed.§8
8host.hold(Catalog, {}, ctx) on W1 before W1's load ran.The reference is returned at once, loading (unpublished); W1's publication places it over the parent; it follows the object for its whole life: loading, live, failed { waiting } if S drops the plugin, live again when it comes back.Returned at once; placed once load resolved (host.ts #hold races load); a removal ends it for good with gone.§1, §8, §5
9host.hold(Missing, {}, ctx) for a local type nobody serves here.loading before the publication, failed { waiting, not_served } after it; a reload serving it binds the same reference.Throws unavailable at once after load; before load, fails the reference broken once it resolved.§8
10await host.registry(HarnessModels, ctx) by base code when the models plugin is missing.Rejects at once with unavailable/undeclared (waiting); code that wants to wait declares the registry in an object or holds. A served registry whose first compose threw rejects the same way (compose).Rejects at once (slice 8 build).§8, §9
11ctx.get(Sessions, {}) inside a worker tool through host.root (plugins/sessions/worker.ts) while the parent link is down.The request waits on the path (no_link, loading); the getter waits up to readyTimeout; the link comes back, the path re-holds, bound, the getter resolves. The request ends when the tool call settles.The same (ctx.ts CallContext.get, path.ts re-hold).§1, §8
12A call notes.add(args, ctx) on a reference whose object is loading (W1 restarting).The call parks in the reference's queue with its own readyTimeout; bound flushes the queue in order; past the deadline it rejects unavailable/park_timeout.The same (host/parked.ts).§8
13A call on a reference that is failed { waiting } (the plugin is not there).Rejects at once with the cause, no parking: nothing is under way.Rejects at once (the reference is failed).§2, §8
14A call already sent to W1 when the IPC link drops.The call fails unavailable and is never resent (its outcome is unknown); the reference goes loading; calls made after the drop park until the re-hold's bound.The same (consumer.ts link close).§2, §5
15The parent link drops while ForwardStore is parked loading on Catalog (S restarting).The request stays on the path (no_link), ForwardStore stays loading; the reconnect holds it again, S answers loading { attempt } or bound; the dependent wakes on the state change. No deadline on the wait.The dependent stays loading until readyTimeout, then transient with backoff (the audit's choice 6).§1, §3.3, §4
16A route link to W1 drops while T holds Notes { S1 } through S's relay.The relay sends loading downstream, rejects relayed calls in flight unavailable, re-resolves the address with the route backoff, holds again upstream with the kept request; bound is forwarded. The tab's dependents stay loading.The same (relay.ts); the route backoff stays.§1, §4, §11
17T's own link drops while its hold for Notes { S1 } is still unbound at W1 (not_served).S sends closed { origin } upstream; W1 drops the indexed hold with the origin: no leak. T reconnects and holds anew.The same for refused holds (provider.ts origin close).§1, §10
18A tab closes while NotesView is parked on Notes { S1 }.Its last holder releases: a parked object ends at once (it is not live, so no linger); its requests end, the relayed hold is released upstream, the index entries go. Nothing keeps the chain.The same (object.ts #idle).§1, §3.5, §10
19host.revoke({ grant }) while a guest's objects on W1 are parked and its tab's holds wait unbound.Every request of the principal ends, bound and unbound, at every hop: the parked objects end forbidden/revoked, the tab gets gone for each hold, the child link closes. Nothing revives them; a new login holds anew.Bound objects end, holds waiting for load are ended (host.ts revokeWaiting); refused holds are released with the link.§10
20A worker with nothing live: a tab holds Notes { S1 } but the notes plugin is missing on W1.The unbound hold counts in host.activity.references from its arrival, so the supervisor does not stop W1 while the tab waits; it is not in host.references (presence lists bound references only).A refused hold is not counted: the worker stops idle, restarts on the next hold, refuses again.§10
21S holds SessionTransfer on W1 for a fork while W1's Harness is parked on the models registry (not served).The hold binds to SessionTransfer, whose attempt parks on Harness: loading, then failed { waiting, "waiting for p2.harness: waiting for registry …" }, with the chain in the message; S's ctx.get rejects at once with it. A reload on W1 serving the models plugin wakes the harness, then SessionTransfer; a later ctx.get binds.SessionTransfer fails transient after readyTimeout, polls; S's getter rejects at its own deadline.§3.1, §3.4, §8
22ref.retry() from a tab's Retry button on a failed { waiting } reference whose blocker on W1 is broken (a throwing factory).The retry passes over the link (retry { hold }) to the parked object, which passes it to its broken blocker: a new attempt; waiting blockers are passed through, not_served ones untouched. If the blocker goes live, the chain wakes.retry reaches only the object the hold is on; a refused hold is started again.§3.7
23Keyed pin: Notes pins store.drafts.at(key) on NoteStore; the key's load fails at the provider.The pin is a dependency: Notes parks failed { waiting on p2.note-store.drafts[key] }; the provider's re-bind of that member (a retry) wakes it.dependency_failed transient with backoff (attempt.ts pinKeys).§3, §4
+
+ +
XVI

Scenarios: Providers, Versions, Registries

Plugins missing, added, removed and broken; versions that do not fit; registries and the harness.

+

Opus's scenarios cover the provider side: plugins that are missing, arrive by a reload, are removed or break; versions +that fit through accepts or do not; registries with missing providers, skewed versions and last good values; +self.follow over a harness that restarts; and reloads of an object's own code. The pattern: what used to be broken +or polled becomes waiting, woken by the one event that can fix it, and an object's own code is still the only thing +that breaks it.

+

The Rule column cites sections of the working document, reproduced in appendix C.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
#ScenarioUnder the modelTodayRule
1Worker W1 has no drafts plugin. Chat (chat plugin, on W1) declares self.get(DraftStore).Chat's request is unbound not_served and indexed under DraftStore. Chat is failed { waiting, "waiting for DraftStore: not served" }, with no timer. A tab's Panel holding Chat shows the same cause, and its calls reject at once.#assertRoute throws in the factory, so Chat is broken. Only ref.retry() or a reload recovers it (host.ts).§1, §2, §3.1
2A reload adds the drafts plugin to W1 while Chat waits (1).The reload's commit publishes DraftStore. The index wakes Chat's request, which binds; DraftStore activates and goes live. Chat leaves parking, opens its natives and goes live; Panel goes live.Chat stays broken unless the reload retries broken objects.§4, §5
3Server S has no models plugin. On W1, the models plugin's host-lifetime ModelRuntime declares self.get(ModelConfig) (placed on S).S keeps W1's hold as refused, not_served. ModelRuntime is failed { waiting }. W1's load counts it as a first outcome (parked on another host). The harness runs, and its generations report "model unavailable".S refuses broken (remote/hold.ts #refuse). ModelRuntime is transient and polls: each backoff tick (1-60 s) makes a new hold across the link and reports again, for the host's life.§1, §3, §6.3
4S gets the models plugin by a reload (3, continued).S's commit re-runs its refused hold and sends loading { hold, attempt }, then bound. ModelRuntime wakes, opens its native and contributes to HarnessModels. The harness's next generation finds a model.The next backoff tick re-holds; recovery waits up to 60 s.§4
5A reload removes the ports plugin from S. Tabs hold PortsView; Radius (live) reads the radius.forwards registry that Ports contributed to.PortsView's objects end, and their holds become unbound not_served: the provider sends failed on the bound hold, and the tabs show failed { waiting }. Radius gets a table without Ports, composes the empty value and keeps working. A later re-add re-runs the kept holds, which bind again.The holds get gone and the references end for good (design §21). A re-add cannot revive them.§5, §9
6A reload removes W1's file-index plugin. The live Files object (files plugin) holds self.get(FileIndex) locally.Files' Binding becomes unbound not_served. Files stays live (R18). Its member sources keep their last values, and calls through that reference reject with the cause. When file-index is re-added, the binding binds again and the members re-bind by name (one frame).The reference ends released; Files can never use FileIndex again without a restart.§5 (locally), §3.10
7After a sessions plugin update, S's Catalog factory throws. W1's Chat waits on Catalog; a tab's Panel waits on Chat. The user presses Retry on Panel.Catalog is broken; Chat and Panel are failed { waiting, "waiting for Catalog: …" }. Retry passes on hop by hop (T, then S's Catalog, through W1) and retries Catalog once. If the cause is gone (the config is fixed), Catalog goes live and Chat, then Panel, wake.Chat polls by backoff with new holds. Retry on Panel retries only Panel, which is still blocked.§3.7
8S's EnvManager (envs plugin) opens an SSH connection that fails because the remote host is offline. W1's environment objects wait on S's env views, which wait on EnvManager.Only EnvManager backs off (transient, its own retryAt). Its dependents are failed { waiting } with no timers. When SSH comes back, EnvManager goes live and the chain wakes in order. A host.get on a dependent rejects at once (it is waiting); a hold just shows it.Every level runs its own 1-60 s backoff and re-holds across the link at every tick.§3.1, §3.3, §8
9A reload of the terminals plugin restarts S's TerminalService (natives: the terminal host process D). Terminals{session} objects are live and depend on it.Native restart as in §21: new calls park while it is loading; running calls drain; natives close with reason reload; the factory runs; natives reopen. Its dependents stay live (R18); their calls park while it is loading.The same; this is design §21, unchanged.§5 (old calls), §3.10
10Worker W1 crashes while a tab views Chat{S1} through S.S's relayed hold goes to its path, no_link, so the tab's reference is loading and its calls park, bounded. The route re-resolves (route backoff); the new W1 publishes, the held request binds, and a fresh bound reaches the tab.The same, except a dependent that is activating fails at its 60 s deadline in between.§1, §2
11A tab on Chat v1 meets W1 serving Chat v2 with accepts[1].The check accepts v1; the hold binds through the mappings (slice 7). The tab is live.The same once slice 7 builds accepts.§1, §5
12A tab on Chat v1 meets W1 serving Chat v3 without accepts[1].The hold is unbound version, so the tab shows failed { waiting } ("update the page") and its calls reject. The tab's own reload makes a new v3 request and releases the old one. If W1 rolled back to v2, the old hold would bind again in place.Refused broken, or gone at the reload that dropped v1: ended for good.§1, §5
13A tab reloaded to v2 holds Chat while W1 is installing v2.W1 keeps the hold installing: the tab is loading and its calls park, bounded. W1's install commit re-evaluates the hold, which binds v2. If W1's install is rejected, the hold is re-evaluated at that end and becomes failed { waiting, version }.Refused broken (contract_mismatch); design §21's updating is not built.§5
14A plugin changed Chat's contract without bumping the version: a tab on v1 (fingerprint A), W1 on v1 (fingerprint B).Unbound version (a fingerprint that differs at the same version), so failed { waiting }. It is not broken: the consumer did nothing wrong, and its next reload or a fixed provider heals it.Refused broken, contract_mismatch.§1
15A reload of chat makes Chat v4 accept v3 but drop v2. Tabs hold Chat at v2 and at v3.Only the v2 holds unbind (version). The object, its v3 holds, documents, jobs and natives stay.Design §21: the version's holds end with gone.§5
16S has no web-shell plugin. Files, artifacts and terminals contribute routes to Http; proxy-marker reads Http.The contributions wait in the table. Proxy-marker's reader is unbound not_served at once after the publication: failed { waiting }. A reload that adds web-shell serves and composes Http, the index wakes the reader, and it goes live.Slice 8 S3: the reader fails transient (unavailable/undeclared) at once and is woken at the first value only.§9
17The codemode plugin is reloaded alone with a HarnessExtensions token v3; the core harness serves v2.Codemode's v3 contribution is left out and reported (contribution_refused). A reader bundling v3 is unbound version: failed { waiting }. A core reload to v3 makes it usable, which wakes the reader.The contribution is refused the same way; the reader is broken (S2).§9
18Core is reloaded alone, serving HarnessExtensions v3 with accepts[2] (contribution and value).v2 contributions are adapted once per value; v2 readers get the composition mapped back once per composition. A reader at v1 (no mapping) is failed { waiting, version }.The same, except the v1 reader is broken (S2).§9
19The harness's HarnessExtensions compose throws at the first composition (an owner bug).The registry is served but has no usable value, so the Harness's request is unbound compose: failed { waiting }. The barrier counts its first outcome. A reload that fixes compose makes the value usable, and the Harness wakes and activates.S3: the Harness fails transient at once, woken at the first good value.§1, §9
20Codemode contributes its sandbox (a native) to HarnessExtensions. Later another contributor's entry makes compose throw, and then codemode restarts.While compose throws, .value keeps the last good composition. When codemode's contribution leaves, that last good value is dropped (it holds the closed sandbox), and .value throws until compose succeeds again. No reader is handed a closed native.Slice 8 keeps the last good value regardless, so a reader could use the closed sandbox.§9
21Transcript (on W1) follows the harness through Harnesses.value.observeConversation(...) and declares retryOn(Harnesses). The harness's Session fails and the harness object restarts.self.follow is Transcript's own step: S1 ends the observer failed, so Transcript fails transient (its own backoff). When the new harness contributes, retryOn wakes it, and it attaches to the new harness and gets one fresh frame.The same (slice 8 S1 and retryOn); unchanged by the model.§3 (follow), §3.8
22A reload of chat swaps the live Chat in place (no natives). Its new factory adds a self.get(Sharing), and the sharing plugin is absent.Owner item 4, recommended: Chat parks. Its holders see failed { waiting, "waiting for Sharing: not served" } and its calls reject until sharing appears. The alternative keeps the old code until the dependency resolves.Not built (slice 7); design §21 swaps the object and opens its gate.§3.9, §13.4
23Chat is parked on DraftStore (1). A chat reload ships code that no longer needs DraftStore.One handover: the new attempt's requests bind first, then the old ones are released. Chat activates on the new code and goes live; the old not_served index entry goes.Not built; today a broken Chat is retried by the reload (slice 7 plan).§3.9
24A reload removes the models plugin from W1 while ModelRuntime is parked on S's ModelConfig (3).ModelRuntime (host-lifetime) ends with its plugin. Its requests end, and W1's refused hold at S is released (a release). Nothing stays waiting.The type's objects end (§21), and its backoff polling stops with it.§3.5, §1
+
+ +
XVII

Scenarios: Rollouts and Cycles

Delayed facets across hosts, rollout windows, chains over three hosts, and the probe at work.

+

Sol's scenarios cover the windows in which hosts run different code: a server facet ahead of its worker facet and the +reverse, tabs older or newer than the server, installs that are queued, staged or rejected, a type moved between host +kinds; then chains across three hosts, diamonds and rings, the probe's own corner cases and a forged probe. Row 20 is a +case round 3 found against the previous draft; the rule that fixes it is in chapter VIII. The pattern: a version that +does not fit is a request waiting for a publication, never an error that ends it; an install answers only for its own +host; and a cycle is found by the attempt whose edge closed it.

+

The Rule column cites sections of the working document, reproduced in appendix C.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
#ScenarioUnder the modelTodayRule
1The models plugin's server facet upgrades first. Browser and TUI model pickers still use its older Models contract, which the new provider accepts.Their references remain usable through the old-version adapters. If an owning native restarts, affected references show loading until its fresh binding is ready. A caller that later installs the new contract releases its old request and makes a new one.host/host.ts:#assertMatch accepts only exact versions and fingerprints. Accepted-version serving across reload is not built in next; p2 uses its older cluster reload runtime.§1, §5
2A worker's new search facet makes SearchIndexer depend on SearchFeed v2. The server still serves v1; its search facet installs later.SearchFeed's request is failed { waiting } for the version mismatch. SearchIndexer is also failed { waiting }, without a dependency retry timer. The server's later publication rechecks the request; SearchFeed becomes live and wakes the indexer.A remote mismatch is refused as broken; its dependent fails transient and retries with backoff. See host/attempt.ts and remote/hold.ts; p2's real indexer declares this server dependency in plugins/search/worker.ts.§2–5
3A new web workspace facet arrives before the selected worker's matching Workspace contract. The user opens the workspace panel during this window.The panel's held Workspace reference becomes failed { waiting }; an awaited getter rejects under the recommended getter rule. The old worker object and compatible readers are unaffected. Installing the worker facet wakes the held request and supplies a fresh binding.Exact-match refusal reaches the client; a dependent object can enter R18 backoff. Next has no install-aware wake protocol yet; p2's workspace UI and worker facets live in plugins/workspace/.§2, §5, §8
4A tab sleeps while the worker upgrades Workspace and accepts its old version. The tab reconnects without updating its code.The retained request shows loading while its path is down. Re-holding asks for the tab's original version, and bound supplies the mapped snapshot, returning it to live. No old method index is interpreted as a new one.Path re-holding already exists in remote/path.ts, but unequal versions still fail in host/host.ts. Reload-time adapters are slice 7 work.§1, §5
5The server publishes a new Models contract without accepting the old TUI version. The TUI launcher downloads its replacement bundle later.The old TUI request stays held but shows failed { waiting }. The old runtime releases it when the launcher swaps workers. The replacement runtime makes a new-version request and can become live; the old request is not silently relabeled.p2 polls for TUI bundles and swaps sandbox workers in hosts/tui-client.ts. Next does not yet implement reload-time version recovery.§1, §5, §8
6The server and tabs already use revision R5 when a route starts a worker from an older plugin set. The worker connects before loading its current facets.Business requests stay loading until publication or show failed { waiting } against published incompatible code. The loader installs the desired code set without awaiting activation. Publication then rechecks waiting requests and the actual objects activate; a route must not wait for the worker's activation barrier before admitting its link.p2 sends selected worker facets through WorkerInit and uses a separate control-ready sequence. Next has the publication/activation admission deadlock described in host/host.ts:load and remote/hold.ts:start.§1, §5–6
7A tab installs new workspace code while the worker's matching install is queued but has not started.The old worker answers failed { waiting }, not loading merely because another host knows an upgrade is planned. A held reference survives; a getter rejects. The worker's own install and its completion re-evaluate the request, so it can later become live.Next has no revision/install status handling; it returns an exact-match failure. The model deliberately accepts this rollout-window blip instead of adding revision fields to mismatch messages.§2, §5, §8
8The worker's search facet stages a new contract, so an incompatible request is held as installing. Staging fails and the old search code remains published.The request was loading during the install. Rejection is an install-end event even without publication, so the request is checked against the old provider and becomes failed { waiting } for the remaining mismatch. Its getter rejects rather than waiting forever for a commit that will not occur.Next has no installer. p2's hosts/server.ts:performReload retains published code and bundles when preparation fails, through a different reload protocol.§4–5, §8
9A worker's own envs facet install is rejected while it needs a capability from the new code set. A provider elsewhere becoming ready cannot repair that rejected local install.Requests blocked by the worker's own rejected install fail promptly with its reason. Published old code is not replaced by half-installed new code. A later successful local install or explicit retry is needed; waiting for a remote activation is not an install strategy.Next has no own-install state. Design §21 specifies the prompt own-rejection rule; p2 reports reload refusal while retaining its previous selection.§5
10The worker ports tool facet is added before the server ports facet. The tool is registered successfully, but the first invocation looks up server Ports with ctx.get.The tool owner remains live: this is a call-time lookup, not a declared factory dependency. The missing Ports request is failed { waiting }, so the invocation reports unavailable and its call-scoped request ends. After the server facet publishes, a later invocation can acquire a live Ports object.This is how plugins/ports/worker.ts declares the tool: its callback does ctx.get(Ports). Next's remote lookup currently receives a broken refusal; no provider-publication recovery is built.§1–3, §5, §8
11The server removes Ports first, while old worker tool code and an old browser Ports sheet remain loaded. The worker and browser facets update afterward.The browser's held request becomes failed { waiting } without ending permanently. Ordinary live worker owners stay live; their next Ports lookup fails while the provider is absent. Updating those consumers removes obsolete surfaces, or a later provider publication can restore a still-held request.p2 normally refuses removal when outside-batch consumers still hold a removed type (hosts/server.ts:refuseHeldRemoval). Next's target design originally ended removed-type references; the retained-request behavior is new.§3.10, §5
12Notes(session=7) moves from a server facet to a worker facet. An old tab still computes the server address; a new tab computes worker 7.The old request stays at the old address and shows failed { waiting }; publishing Notes on the worker cannot wake it there. The new tab routes to worker 7 and can become live. Updating the old consumer computes a new address and creates a new request; the model adds no redirect.Placement is already computed by the caller and checked by the provider in host/placement.ts. Next has no relocation reload implementation.§1, §5
13Worker 1 still serves Notes v1 for its own session. Its new code requests Notes v2 placed on worker 2, which already serves v2.Resolution chooses worker 2 before checking its version. The request can become live without waiting for worker 1's unrelated registration to upgrade. Worker 2's bound wakes any dependent waiting on that request.host/host.ts:#assertRoute checks a same-name local registration before Placement.reference, so worker 1's v1 can wrongly reject this v2 request.§1
14The models plugin's server facet updates now, but the worker facet updates hours later. Each host has its own native model registry and compatible local contributors.Each host composes its own table and its local consumers can stay live. A worker does not wait for the server's registry revision: native registries are host-local. Only an actual remote object contract used between those facets creates a cross-host version wait.p2 also has independent server ModelConfig and WorkerModels resources (plugins/models/server.ts, worker.ts). Next's registry adapters already handle local reader/contributor versions; atomic registry replacement is not built.§5, §9
15A tab Panel depends on a worker SessionView, which depends on a server Catalog view. Catalog's new facet arrives last.Catalog's missing request is failed { waiting }; SessionView and Panel show waiting with the nearest blocker and root cause. The server publication binds Catalog and makes it live. Its state change wakes SessionView; that provider's bound then wakes Panel, with no dependent backoff loops.R18 retains dependency references but fails dependents transient and uses identity backoff. host/object.ts:dependencyLive already skips that backoff on a live event; registration-based recovery is missing.§1–4
16A server Summary waits for views on two workers. Both views reach a shared local CatalogStore on the server, whose own required provider is absent: a diamond, not a cycle.CatalogStore is failed { waiting }, and waiting propagates through the views to Summary. Probes reach the common store, but its unavailable request has no target attempt and no edge back to Summary. It must not report invalid/cycle; adding the missing provider later wakes the chain.Current local waiting-edge traversal and trusted activation chains distinguish cycles from shared dependencies. Dependency failures still use R18's transient/backoff path (host/object.ts, host/attempt.ts).§3–4, §7
17Thirty-three object attempts on trusted server/worker links form a ring and start together. Each reference resolves to the next attempt in that ring.All participants stay pre-activation, showing loading or waiting according to their blockers. With answer (A), the attempts waiting across links time out after readyTimeout, retry with backoff, and the activation chains of the later attempts find the ring. With answer (B): the attempt whose edge closed the ring last starts a search; it returns, a check along its recorded path returns, and the initiator fails broken invalid/cycle; propagation breaks the others and the dependents off the ring. The probe carries its trail of 33 hops; no cap cuts it.Concurrent cross-host cycles can escape fixed activation chains and rely on readyTimeout/backoff. host/object.ts:assertNoCycle and design §6 document that limitation; next has no probe.§3.1, §7
18A ring is being confirmed when one worker reloads its facet and removes its outgoing dependency, repairing the ring. Some probe returns are already in flight.With answer (B): New attempts have new ids, and obsolete target ids are discarded. If the check passed before the repair, the ring is reported (it existed when the check began); if the repair came first, the check dies at the replaced attempt and nothing is reported. Unchanged attempts can therefore become broken invalid/cycle and need ref.retry() or their own plugin reload; provider readiness alone does not retry broken objects.Next has no probe or reload implementation. Broken cycle failures already require retry, and host/object.ts:retry starts another attempt.§3.6–9, §7
19A malicious tab learns a worker attempt id and sends a forged probe naming that same attempt as its initiator through the server relay.With answer (B): The server drops the probe because it arrives on a child link, before any forwarding. The worker object and legitimate references keep their states. A child link never carries cycle evidence.There is no probe message today. remote/provider.ts already strips activation chains from untrusted holds rather than trusting a child to declare ancestry.§7.2
20A→B→C wait while C's return path to A is disconnected. Sixty-five unrelated initiating rounds pass through C before reconnect makes C→A a known edge.With answer (B): Found in round 3 against draft 3 (C could have evicted every round from the ring). Under the model the re-hold gives C a new edge token, so C starts its own search; it crosses A and B and returns to C, the check along its path confirms, and C fails broken invalid/cycle. Foreign rounds and evictions play no part.Current re-hold carries an activation chain (remote/remote-object.ts:attach), which can detect a small newly reachable ring; otherwise the dependency deadline remains its fallback. No probe-round cache exists today.§7.2–3
+
+ +
∴

Open Questions

What the owner decides, and what slice 7's spec must still pin down.

+

The model replaces five answers to "the provider is not there" with one: the request stays, the dependent parks, an +event wakes it. It ends the startup deadlock with a gate that lasts only for the synchronous setups, finds cycles with a probe +instead of a deadline (or, if the owner prefers, one narrowed deadline), and keeps timers only where no event can exist. Four reviewers played more than 300 +scenarios against five drafts; the catalog in the chapters before this one is a selection.

+

Decisions for the owner

+
    +
  1. The barrier (chapter VI): (a) Chord's load waits for the first outcomes of this host's host-lifetime objects, +with the local-blocker rule; or (b) load resolves after the publication and the app reads host.startup. +Recommended (a); Astra, Opus and Sol for (a), Fable for (b).
    2. +
  2. Getters on waiting (chapter IX): reject at once (recommended) or wait up to readyTimeout.
    3. +
  3. waiting as a cause kind (chapter III): a new kind (recommended; Opus, Fable), transient without retryAt +(Astra), or loading with a diagnostic (Sol).
    4. +
  4. A pure swap whose new factory needs a dependency that is not there: the object parks (its gate closes, its +holders see loading or failed { waiting }, calls park or reject; recommended), or it stays on the old code until +the dependency resolves (old and new code would then overlap without bound, against §21's "a host is at R or R−1").
    5. +
  5. The last good value (chapter X): kept only while every contribution it was built from is still in the table +(recommended; Astra), or kept as slice 8 decided, with natives in a registry looked up at use through their owner +(Sol).
    6. +
  6. Cycles across hosts (chapter VIII): (B) the probe, with (B1) a re-search backoff that sends probes only +(recommended) or (B2) a full search lifecycle with no timer at all; or (A) the slice 4b decision narrowed to one +deadline, which polls chains across hosts again, misses a ring of waiting attempts, and retries in lockstep without +finding the cycle it is for.
    7. +
+

For slice 7's spec

+
    +
  • The test list: the startup deadlock resolving without readyTimeout; an early hold placed here, not through the +parent; remove-then-serve with a live dependent's calls recovering; a 1,000-hold publication wave under the window; +the probe on long rings, diamonds, edge changes, reloads, relays, reconnects and forged probes; GC of a parked +three-level chain after its top is released; every index entry gone after release, revocation and link loss.
    • +
  • The local re-binding of a Binding (member sources, keyed pins, parked calls, presence) and the unbinding of a live +hold under flow control, both with their race tests.
    • +
  • The wire: waiting in the failed cause schema, the attempt id on loading and failed, the probe message, the +pinned list; PROTOCOL_VERSION stays 1 until p3 ships.
    • +
  • The design changes listed in appendix A, folded into design.md and the decision log in the same commit as the +owner's decisions.
    • +
+
+ +
A

What Changes

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
WhereTodayUnder the model
R18, activationa dependency that does not go live within readyTimeout fails the dependent transient; backoff 1 to 60 sthe dependent parks with no timer; failed { waiting } when a dependency failed; woken on every state change of a dependency; the deadline paused while parked
R18, the boundary"live" means every dependency was live at the end of activationthe dependency check is the boundary: after it, a dependency leaving live is treated as after live
self.get of a missing local typethrows in the factory: brokenunbound not_served: failed { waiting }, woken by the publication that serves it
a remote hold the provider cannot serverefused broken; the dependent polls by backoffrefused and kept at the provider, re-run at its publication; failed { waiting }
design §3, placement"served here" decides "here"the address decides; a remote type without placement keeps today's rule
the start barrier, pendingLoadholds wait for all of loadholds wait only for the publication; the barrier is (a) or (b)
S2, a registry reader newer than its providerbrokenfailed { waiting, version }
S3, a registry without a valuetransient, woken at the first value onlyfailed { waiting }, woken at every transition to usable
slice 8, the last good valuekept whenever compose throwskept only while every contribution it was built from is still in the table
design §6, cyclesa walk on one host, chains across hosts, a documented gap closed by the deadlinethe gap of simultaneous starts across hosts: (A) one narrowed deadline, or (B) a probe: a search per edge change, a check along the recorded path, edge tokens, dead notices
design §7, host.holdthrows for a missing local type after loadalways returns a reference
design §7, getterswait through every failure up to readyTimeout, reject on brokenwait through loading and transient; reject at once on waiting and broken
design §21, removalgone to every holderfailed { waiting, not_served }; the request binds again when the name is served
design §21, a dropped versioncontract_mismatch, finalonly that version's requests unbind; the object stays
design §21, rolloutupdating with revision fields and a deadlineeach host knows its own install; installing re-evaluated at the install's end; installs never await activations
FailureCausetransient, brokenadds waiting
the wirefailed cause waiting; the attempt id on loading and failed; with (B), one new message, probe (search, check, dead)
unchangedan object's own code reload (§21); R18 after live; retryOn outside parking; self.follow's R4; the call window and flow control; revocation's semantics; no client providers
+
+ +
B

Glossary

+

Attempt: one try at activating an object: its factory, requests, natives; its own id since the model. Bound: a +request pointing at an object here or a hold over a path. Dependency: a request an attempt needs live before it +may go live (self.get, self.registry, a keyed pin). Dependency check: the moment all of an attempt's +dependencies are live; after it the attempt runs its own steps. Edge generation: a number of a wait edge, incremented by its +source whenever the request binds anew. First outcome: an object is live, failed by its own cause, or parked with no local +blocker still in its first attempt. Holder: whoever made a request and alone ends it. Parked: an attempt +waiting for a dependency, with no timer. Probe: the message that walks wait edges to find cycles: a search, then a check along the recorded path. Publication: +a host making its facets' declarations visible: load's setups, a reload's commit, an install's commit. Refused: a +request without authority or with invalid params; never indexed. Request: what a reference stands for: type or +registry, params, principal, version. Round: the number of one search, from a counter per attempt. Seen set: the newest round number an +attempt has forwarded, per initiator (up to 256). Unbound: a request with no provider at the moment, with a reason (unpublished, not_served, +version, compose, installing, no link). Waiting: the cause of a failure that is not the object's own, woken by +an event. Wait edge: a parked attempt waiting for another attempt. Wake: an event that makes the resolver look +at a request again.

+
+ +
C

The Working Document

+

The proposal as the reviewers checked it, /tmp/model/model.md, draft 4. Its section numbers are the ones the +scenario catalogs cite.

+

Status: proposal for the owner, not decided. Built in slice 7 with reload and rollout. Sources: the owner's direction +("park until resolved"; "no difference between local and remote resolution: a link going is like a plugin +unloading"), the reviews (/tmp/dep-mechanism-*.md, /tmp/dep-graph-*.md, /tmp/barrier-*.md, +/tmp/model/review{1,2}-*.md), the code at pi 358f7d46a. Owner marks an open decision.

+

Draft 7 (round 6): (A) also retries in lockstep without finding the cycle (Fable); the probe's recovery after a +suppressed search becomes a choice between (B1) a re-search backoff for probes only (recommended) and (B2) a full +search lifecycle without timers, since dead notices alone need recovery state that eviction may not drop (Astra, Sol). +Causes are compared by value on wakes (Opus). +Round 7: a returned search is accepted while its initiator is parked, whatever its round (Astra: re-searches must not +invalidate searches under way); detection is eventual, not within one backoff period; the local walk stays (Opus).

+

Draft 6 (round 5): cycles across hosts become an owner choice between (A) the owner's slice 4b decision (simultaneous +starts across hosts time out, no probe message), narrowed to one deadline, and (B) the probe, completed with edge +tokens, no revisits, and dead notices back to the initiator (Astra: a check that dies silently can lose the only +search that survived dedupe). (A) turned out to either poll chains across hosts again or miss a ring of waiting +attempts, so (B) is recommended, pending its review.

+

Draft 5 (round 4, Sol's phantom cycle): the second round is replaced by a check along the recorded trail of +(attempt, edge generation) hops; a reported ring existed at one moment; probes grow with their path.

+

Draft 4 after round 4 (probe only): round numbers per attempt, forwarded only when newer than the stored one; probes +honoured on parent and route links, dropped on child links; completeness stated under an eventually stable graph; the +repair caveat broadened to any concurrent edge removal; a new edge generation resets confirmation; a bounded display +trail.

+

Draft 4 changes (round 3): the probe starts a fresh round on every new or changed wait edge (a ring closes at its last +edge, whose source finds it), keeps one seen entry per initiator, accepts only its current round, and states its +cost honestly; every host drops probes from untrusted links; local forwarding is iterative; no-placement remote +requests are re-placed when a host's first parent link connects; a cohort object that ends before its first outcome +settles the barrier; the last-good refinement is marked for the owner.

+

Draft 3 changes (round 2): remote types without placement keep today's fallback (here if served, else the parent); +hydration stays before parking; no copied retryAt; a dependency ended breaks its dependent; a reload's handover +binds new requests before releasing old ones; retry passes only to broken and transient blockers, once per +operation; installing re-evaluated at every install end (commit or rejection); a registry's last good value dropped +when one of its contributors leaves; the probe with round ids, bounded seen sets, edge-generation triggers, attempt +ids on every non-live answer and relays dropping probes from untrusted links; one cycle mechanism, local and remote.

+

0. The problem

+

The same situation, "what I need is not there", has five outcomes today, depending only on how it is declared:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Declared asProvider missingToday
self.get(LocalType)no facet serves itself.get throws, the factory throws: broken
self.get(RemoteType)the worker does not serve itthe hold is refused broken; the dependent transient, polled 1-60 s forever
self.registry(R)no providertransient, woken at R's first value only
self.registry(R), reader newerbroken
a reload removes the typeholders get gone: ended for good
+

And the startup barrier deadlocks an acyclic graph across two hosts (Sol, /tmp/dep-graph-sol-barrier.log): S's eager +A needs W's B, B needs S's C; W's hold for C waits for S's whole load, which waits for A. Only readyTimeout ends it.

+

1. Requests and one resolver

+
    +

  • A request is what a reference stands for: type (or registry), params, principal, the consumer's contract version. +Its holder makes it and owns it: an attempt (self.get, self.registry, keyed pins), a call (ctx.get), outside +code (host.get, host.hold, host.registry), a link's peer (hold). It ends only when its holder ends it (the +attempt ends, the call settles, release(), a getter gives up, the peer's link or origin closes, revocation, the +holding host closes), never because its provider is missing, went away or does not fit.

    +
    • +

  • Placement by address (round 1): a local type goes here; a remote type with a placement goes to the address +its placement (computed from the consumer's own contract) gives: here if that is this host's address, whether served +here or not, else over the route for that kind or the parent (design §3). A remote type without placement keeps +today's rule (round 2: Fable, Opus, Astra; no feature removed): here if served here, otherwise over the parent. A +request of that kind made before the host has a parent link waits here not_served and is re-placed when the +host's first parent link connects (round 3: Opus N5, Sol 3: a tab loads before it connects); on a host that never +connects (the root), it stays here. Opus would rather require a placement on every remote(); that removes a +supported declaration, so the draft keeps the fallback. A hold arriving for an address that is not this host's is relayed or +refused (invalid/placement) as today; a relay never captures a hold because it serves the same name.

    +
    • +

  • Then the selected provider is checked (served, version or accepted version and fingerprint, authorize, the +provider's own placement check). Version checks never look at a same-name registration elsewhere.

    +
    • +

  • The publication gate: a request made on this host, or arriving on its links, before the host's first +publication (§6) is not placed yet; it shows loading and is placed by the publication. The gate lasts only for the +synchronous setups, never for activation, so it cannot deadlock.

    +
    • +

  • Outcomes of the check:

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    The check findsRequest
    a fitting providerbound (to the object here, or to the hold over a path)
    not published yetunbound unpublished: loading
    nobody serves the name at its addressunbound not_served: failed { waiting }
    no fitting version, a fingerprint that differs at the same version, a placement the provider computes elsewhere (code skew)unbound version: failed { waiting }
    the registry is served, but its first compose (or the reader's version mapping) threwunbound compose: failed { waiting }
    the provider host is installing a revision and the hold does not fit its current codeunbound installing at the provider: loading
    the path has no link (a drop, a route resolving)on the path: loading
    authorize refused, invalid paramsfailed { broken, forbidden | invalid }, as today, never indexed
    +

    The unbound reasons that wait for a name (unpublished, not_served, version, compose) are indexed under it on +the host that decided them: the consumer's host for a local request; the provider's host for a hold that arrived +over a link (it already keeps such holds as refused: remote/hold.ts). installing is the provider host's own, +re-evaluated at its install's end; no_link lives on the path, where the next link holds it again. Indexes drop an +entry when its request ends; GC-tested.

    +
    • +
+

2. States

+

ref.state keeps its four kinds; FailureCause gains one kind, waiting:

+
type RefState =
+  | { kind: "loading" }
+  | { kind: "live" }
+  | { kind: "failed"; cause: FailureCause; retryAt: number | undefined }
+  | { kind: "ended"; error: ChordError };
+
+type FailureCause =
+  | { kind: "transient"; error: ChordError }   // the object's own effect failed (an open, a follow): backoff
+  | { kind: "broken"; error: ChordError }      // the object's own code, a cycle, or authority: retry() or a reload
+  | { kind: "waiting"; error: ChordError };    // not the object's fault: no provider, no fitting version, a
+                                               //   dependency failed; retryAt undefined, woken by an event
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
The request isref.stateCalls
bound, its object activating or parked with no failed blockerloadingpark, bounded
unbound unpublished or installing, or its path has no linkloadingpark, bounded
unbound not_served, version or composefailed { waiting }reject at once
bound, its object failedfailed with the object's cause, as todayreject at once
bound, its object liveliverun
ended by its holder, revoked, or its host closedendedreject
+

Owner: waiting as a new cause kind (Opus, Fable; recommended: the cause says "not your fault, an event will wake +you" and a UI can say what is missing) or no new kind: transient with no retryAt (Astra; today's wire already +carries a transient cause without retryIn, remote-object.ts), or loading with a diagnostic (Sol; parks every +call on a missing plugin for its full readyTimeout).

+

3. Dependents: parked until their dependencies resolve

+

A dependency is what an attempt declares and needs before it goes live: self.get (local or remote), +self.registry, keyed pins (at(key) on another object's keyed member). self.follow is the attempt's own step (its +attach runs Durable's code), not a dependency; a follower declares retryOn on the registry it reads the harness +through, or recovers by its own backoff.

+
    +

  1. Order, as today up to the check: the factory runs; persisted singletons hydrate (the object's documents, kept +across attempts); the requests are made. The attempt is parked while any dependency is not live (hydrated):

    + + + + + + + + + + + + + + + + + + + + + + + +
    Its dependenciesThe dependent shows
    any loading, none failedloading
    any failedfailed { waiting, "waiting for X: <X's cause>" }, retryAt undefined; the message keeps the nearest blocker and the root cause within the 500-character rule
    one failed with invalid/cycle (a dependent off the ring)broken with the same error: it would wait on a ring that never heals by itself
    one ended (revoked, or its host closed)broken with that error: the request cannot come back; retry() or a reload makes new requests (a guard: a dependency ends only with its holder's principal or host, so the dependent ends too; Fable)
    +
    2. +

  2. The dependency check is the activation boundary: once every dependency is live, the attempt runs its own +steps (natives in order, follow attaches, member checks, derives), bounded by its own readyTimeout deadline, and +goes live. A dependency that leaves live after the check is treated as after live (R18): the attempt goes on. This +is a deliberate rule (Astra: a weakening of "live means every dependency is live"): an object can finish opening +after a dependency went away, and its calls through that reference meet its state. So a parked attempt holds no +natives, and an attempt that opened natives is never parked.

    +
    3. +

  3. No timer while parked: no dependency deadline, no backoff. The attempt's deadline is paused while parked (as +eager migrations pause it) and resumes with the budget left.

    +
    4. +

  4. Wakes, one rule: the dependent re-evaluates on every state change of any of its dependencies (live, loading, +failed, a new cause compared by value, ended; Opus: otherwise a ring of waiting attempts passes its causes round and +round), updating what it shows; it leaves parking when all are live. The factory never re-runs +while parked. Wakes are coalesced per object and run after the batch that caused them; a change during the +attempt's own steps or its teardown is remembered.

    +
    5. +

  5. Kept while parked: its documents (the object's), its declarations and requests, its seal token; no natives. +Ended with the object (its last holder; a host-lifetime object when its plugin is removed or the host closes).

    +
    6. +

  6. broken only for the object's own code (a throwing factory, a derive throwing at activation, hydration, +migrate, a document fingerprint mismatch, an open throwing broken(...)), cycles (§7) and authority +(forbidden, an ended dependency).

    +
    7. +

  7. Retry passes on: ref.retry() on a failed { waiting } object retries its broken and transient blockers, +hop by hop (the existing retry message over links), once per retry operation (an id dedupes diamonds); waiting +blockers pass it on; nothing else is touched. Retry of a broken or transient object starts a new attempt, as +today.

    +
    8. +

  8. retryOn wakes failed objects as decided in slice 8 (a value call retries a transient or broken object); +while the object is parked waiting, a value call does nothing: its dependencies decide.

    +
    9. +

  9. Own-code reload of a parked object: its plugin's reload replaces the parked attempt with the new factory's, +as one handover: the new requests are made and bound first, then the old ones released (round 2, Astra N4: an +unchanged dependency held only by this object must not end in between; today's #activate binds before +#releasePrevious).

    +
    10. +

  10. After live, R18 unchanged: a dependency leaving live leaves its dependent live.

    +
    11. +
+

4. Wakes: two events

+ + + + + + + + + + + + + + + + + + + +
EventRe-evaluates
a target changes state: a dependency's object goes live, loading, fails, gets a new cause or ends; a hold's bound, loading, failed arrive, also after a re-hold on a new linkthe requests bound to it, and their dependents (§3.4)
a name becomes usable, an install ends, or the first parent link connects, on a host: a publication (load's setups, a reload's commit, an install's commit) serves a type, a registry, a version; a registry composes or maps a value at a version; an install ends without a commit (rejected)the unbound requests indexed under those names; the provider's installing holds; at the first parent link, the
no-placement remote requests waiting here; a refused hold re-runs its start and sends loading, then bound or failed
+

No new message kind for wakes: loading, bound, failed and retry exist. The wire's failed carries the new cause +kind waiting; loading and a waiting failed carry the provider's attempt id for the probe (§7).

+

5. Removal, versions, rollout

+
    +
  • A reload that removes a type: its objects end as today (natives close, reason end; callers rejected at once; +handlers drain). Their requests are not ended: each becomes unbound not_served and binds again if a later commit +serves the name.
      +
    • Locally: a Binding whose target ended by removal becomes unbound and accepts a second bind: member sources +re-bind by name (one frame from the last value, as on a restart), keyed pins re-pin, parked calls flush or +reject, presence re-lists. The one new piece of machinery.
      • +
    • Over a link: a failed on a bound hold unbinds it (Fable: one message, one rule; the consumer applies it as +#startOver plus failed); the provider detaches the hold's sources and drops its unsent pending ops and queued +bound; the ack slots of bound/state messages already sent stay counted until their ack or the link's loss; +calls in flight on it are rejected and free their slots once.
      • +
    +
    • +
  • A reload that stops accepting a version unbinds only the requests on that version (version); the object stays +(its own plugin's reload still restarts it if it has natives, §21).
    • +
  • A type moved to another host kind: the request carries the address its own consumer code computed; the old host +answers not_served; the consumer's own reload makes a new request. No redirect.
    • +
  • A consumer whose code moves to a new version releases its old request and makes a new one (today).
    • +
  • Rollout, without revision fields: each host knows only its own install. A provider host that installs keeps +holds that do not fit its current code installing (loading) and re-evaluates them when the install ends, by its +commit or by its rejection. A provider host that is queued but not installing yet answers a mismatch failed { waiting, version } until its install commits and wakes it: a visible blip of at most one install, accepted (Opus). +An install never waits for activations (a reverse-order rollout would deadlock); a host whose own install was +rejected fails its own requests at once with the reason (design §21). The loader keeps its own duties (the desired +code set, digests, serialized installs).
    • +
  • Old calls across a reload follow §21 unchanged: a pure swap lets them finish on old handles; a native restart +drains them first; a removal rejects their callers at once and drains the handlers. Calls are never resent.
    • +
+

6. Startup: publication, then first outcomes

+
    +
  1. Publication: load runs every facet's setup, then publishes all at once in one batch (types served, +registries served and composed once, contributions in their tables), then wakes (§4) after that batch. Nothing +activates inside it.
    2. +
  2. No gate beyond it: holds and host.hold wait only for the publication, never for the barrier. This ends the +startup deadlock: W's hold for C binds as soon as S's setups published; B goes live, then A.
    3. +
  3. The barrier (Owner: (a) or (b); recommended (a), three reviewers of four):
      +
    • (a): load resolves when every host-lifetime object of this load has a first outcome: +firstOutcome(o) = o.live || o.ownFailure || (o.parked && o.blockers.every((b) => !b.local || firstOutcome(b))), +where local means executing on this host (an object here, a keyed pin still loading here), not the contract's +kind. So CodeMode → Harness waits for the harness's outcome; a wait on another host or on a failed dependency +ends the first pass. One firstOutcome signal per object, separate from the readiness dependents wait for (Sol), +counted once, never reopened by retries. An object of the cohort that ends before its first outcome (its plugin +removed, the host closing) settles its entry as ended; host.close settles the waiting load (Astra R3).
      • +
    • What (a) promises: every host-lifetime object on this host had its first try, and the contributions of those +that went live are in their tables. Not: every tool that will arrive (a parked or slow contributor, tools found +in onLive, MCP; Sol's probe /tmp/barrier-sol-first-outcome.log). What a first turn needs is p3's explicit +admission check.
      • +
    • (b): load resolves after the publication; Chord adds host.startup, a watchable of the host-lifetime +objects' first outcomes with their causes; the app decides (Fable). The same tracker, awaited by the app.
      • +
    • p2 today resumes its harness without waiting for all host objects (apps/p2/src/hosts/worker.ts), because its +worker objects that need server types are reachable only after control.ready().
      • +
    +
    4. +
+

7. Cycles

+

On one host, cycles are found exactly as today: at self.get, following the waiting edges of activating and parked +attempts. Across hosts, a hold carries the activation chain it was made in (today), which finds every cycle whose +objects start one after another. What remains is the documented gap (design §6): objects on two hosts that start at +the same moment and need each other. The owner decided on 2026-10-24 (slice 4b review decisions (2)) that this is a +documented limitation that times out and retries, with no probe message. Without dependency deadlines that timeout no +longer exists, so the gap needs one of two answers (Owner):

+
    +
  • (A) The 4b decision, narrowed. One deadline stays: an attempt parked on a dependency held over a link whose target +attempt is known and still activating or parked fails transient after its readyTimeout and retries with backoff; +its next attempt usually starts after the other side and its chain finds the cycle. It has a cost that decides +against it: the target being parked is the common case of a chain across hosts waiting for a missing plugin (a +tab's view on a worker object that waits for an uninstalled type), and those dependents would time out and poll +again, as today, so "local = remote" would not hold across hosts. Narrowing it to targets that are loading (not +failed { waiting }) avoids that but misses a real ring: A and B on two hosts wait on each other, B also on a +missing X; B is failed { waiting }, so A is too; when X appears, both stay failed { waiting } on each other, with +no deadline and no chain to find it. And even the timeout does not find what it is meant to (round 6, Fable): an +object's chain is fixed when it is created and its instance id survives retries, so a retried attempt sends the same +hold with the same chain; two simultaneous starts time out and retry in lockstep and are never found.
    • +
  • (B) The probe (recommended), below: no deadline at all, one new message kind, the most intricate rule set in the +model. Five +review rounds found a counterexample in each of the first four drafts; draft 6's version answers all of them, but it +has not been reviewed again, and slice 7 would have to prove it with tests before the deadline could go.
    • +
+

(B) The probe

+

Replaces the activation chain in holds. On one host the self.get walk stays (it is synchronous and exact: a ring +with no remote edge is found there, where (B1)'s timer does not run; Opus, round 7); the probe serves the rings that +cross a link. It replaces draft 1's bounded chain too +draft 1's bounded chain (a ring of 33 is never found with chains cut at 32; chains keep stale ancestry). Draft 5 replaces the +second round by a check along the recorded path (round 4, Sol: two rounds through different stale paths can report a +ring that never existed, /tmp/model/sol-probe4.log).

+
    +
  1. Attempts, edges, tokens. Every attempt has its own id (uuidv7 per attempt: a retry or reload is a new +attempt). A wait edge runs from a parked attempt to the parked or activating attempt it waits for, through a +bound request (unbound waits and registry reads are never edges; they are not deadlocks). Each edge carries an edge +token (round 5, Astra, Sol): a number from a counter per attempt, never reused, replaced by a fresh one on any +interruption or change of that edge, so two requests to one object are two edges: a new target attempt (also a +local target's retry, without a re-bind), any loading on a bound or pending hold (a relay's upstream re-hold +included), an unbinding, a rebinding, a re-hold after a reconnect. An attempt is parked for one interval only (one +that passed its dependency check never parks again) and ids are never reused. A provider tells its consumers the attempt they wait for on +every non-live answer: loading { hold, attempt } and a waiting failed { hold, cause, attempt }.
    2. +
  2. A search per edge change. Whenever a parked attempt gets a new edge or a new token on one, it starts a +search: probe { kind: "search", round, initiator, trail } along each of its wait edges, round the next number of +a counter the attempt keeps, trail the list of (attempt, edge token) hops it crossed, starting with +the initiator's own. A ring closes at its last +edge, and that edge's source starts the search that finds it. Coalesced per batch. Locally a call on the target +attempt, forwarded through a queue (no recursion); over a link the message probe { hold, to, ... }, consumer to +provider, relayed by hold id with the origin. Probes are honoured on parent and route links and dropped on child +links (accept), whatever trusted says.
    3. +
  3. Forwarding. An attempt receiving a search drops it if to is not its current attempt or it is not parked. If +the initiator is itself, the search returned (4). If the attempt is already in the trail, it drops it (a ring that does not contain the initiator is +found by its own members' searches; Opus). Otherwise it forwards it along every wait edge, appending (itself, the edge's token) to the trail, only if the round number is higher than the one it stored for that +initiator (older or equal rounds are dropped: the initiator superseded them), and stores it. The seen map holds one +number per initiator, up to 256 initiators, and goes with the attempt.
    4. +
  4. The check. A returned search is accepted while its initiating attempt is still parked, whatever round it +carries (round 7, Astra: a re-search must not invalidate an older search still under way); the check decides. The +initiator then sends probe { kind: "check", round, trail } along the recorded trail: the initiator first verifies its own hop; each hop passes it on only if it is still +that attempt, still parked, still has the recorded edge token, pointing at the next hop's attempt; otherwise the +check dies (also on a to mismatch). The check follows the trail by position and bypasses the seen +map. The initiator verifies its own hop again when the check returns. If the check returns, the initiator fails broken invalid/cycle, the path named for display from the trail, +and today's propagation breaks the other activating objects on the ring and the dependents off it. 5. No silent loss. Dedupe can drop the copy of a search that travelled the ring in favour of a copy whose path +later breaks (Astra, round 5); then the check dies and nothing is left to find the ring. Round 6 tried dead +notices back to the initiator; Sol and Astra showed that reliable notices need recovery state per search that +eviction may not drop and that a long-parked attempt would otherwise keep for every initiator that ever passed (a +leak), i.e. a full search lifecycle with cancellation. Two ways (Owner, part of decision 6):
      +
    • (B1) Re-search with backoff (recommended). An attempt parked on a remote attempt starts a fresh search again +after a backoff (1 s doubling to 60 s, reset by an edge change), whether it shows loading or failed { waiting }; +the timer stops when the attempt leaves parking. No dead messages, no recovery state, eviction +only costs work; a new search starts within one backoff period, so every stable ring that crosses a link is +eventually found (no wall-clock bound: delivery is only fair; a ring on one host is the walk's). The timer runs as long as the attempt stays parked on a remote attempt: a view on a worker object that +waits for a missing plugin re-searches every 60 s until a reload (Fable: cheap, but for as long as it waits). The timer sends probes only: it never +fails an object, re-holds or polls a provider. Cost: one probe per parked remote dependent per period, which +crosses the downstream wait graph and dies at its leaves (an unbound wait is not an edge).
      • +
    • (B2) Events only. Each search is owned until it settles: branches report completion or abort back along their +trails, a dropped probe answers dead { initiator, round } with a reverse route, an edge records the searches it +carries until they settle, and an initiator that ends or starts a new round cancels its old one along the same +routes. Complete without a timer, bounded by the searches in flight, and the most intricate protocol in Chord; not +yet specified to the reviewers' satisfaction. +Rejected: enumerating every simple trail without dedupe (Sol's correctness reference): no ring copy is ever +suppressed, but work is exponential on chains of diamonds.
      • +
    +
    5. +
+
    +
  • Why a reported ring existed: every edge of the trail kept its token from the moment the search crossed it +to the moment the check crossed it; the search crossed all of them before the check started, so at the moment the +check started every edge of the ring existed at once. A token changes on every interruption and is never reused, so an edge cannot +vanish and come back unnoticed. Sol's counterexample fails at the check: C1 is no longer B's target.
    • +
  • Why every ring is found (under an eventually stable wait graph and fair delivery; endless edge churn can keep +invalidating searches; with rule 5's (B1) or (B2) for a search suppressed behind a path that broke): a ring is closed by its last edge; its source starts a search when the edge appears; every +other edge exists then and every attempt on the ring waits transitively on the source, so none goes live before the +search and the check return. Rings of any length, local and remote mixed. On a DAG searches die at the leaves; on a +diamond the second arrival is dropped.
    • +
  • What it does not promise: (1) a ring that existed when the check started may be broken up by a concurrent reload, +retry, unbinding or revocation just after; it is still reported (it did exist). Its broken objects recover as every +broken object does: ref.retry() or a reload of their plugin. (2) Linear work under every schedule: with more than +256 initiators passing one attempt at once, an evicted entry lets a search be forwarded again, so work can grow with +the number of simple paths (never endlessly: a search never revisits an attempt in its trail). (3) Constant message size: a probe carries its trail, so its size grows with the length of the +path it crossed (a returning search's trail is as long as its ring); no cap cuts it, since the trail is the proof.
    • +
  • Cost: one search per edge change, crossing each downstream edge once, one check per returned search; with (B1), +one more search per backoff period for every attempt parked on a remote attempt, also while an acyclic chain waits +for a missing plugin, plus the trail bytes. The bounded memory is the seen map's; probes in flight are bounded by +the searches started.
    • +
  • Transition: today's dependency deadline for activations waiting across links stays until the probe's tests pass +in slice 7 (long rings, diamonds, a ring closed after a reconnect with unchanged attempt ids, Sol's phantom schedule, +300 concurrent initiators, edges added and removed, reloads, relays, forged probes); then it goes in the same slice.
    • +
+

8. Getters, holds, calls: one rule

+
    +
  • host.hold returns a reference at once; only invalid params throw (also before load). A missing local type +gives failed { waiting } after the publication, loading before it.
    • +
  • host.get, ctx.get and host.registry: resolve when live; wait while loading, or failed { transient } +(its own retry is under way), up to their readyTimeout; reject at once on failed { waiting } or { broken }. On +success, rejection or timeout their temporary wait goes; host.registry returns the RegistryValue with its +ordinary watch ownership (no release()). Owner: recommended (nobody waits 60 s for a plugin that is not +there; code that wants to wait holds instead), or wait through waiting too.
    • +
  • Calls on a loading reference park, each bounded by the reference's readyTimeout from when it was made; on a +failed one they reject at once with the cause; sent calls are never resent.
    • +
+

9. Registries

+
    +
  • The same resolver on one host: a reader's request for R at its version is bound while R is served and has a usable +value at that version (composed; mapped by accepts[old].value for an older reader), otherwise unbound +not_served, version or compose, or loading before the publication. Woken at every transition to usable +(#valued's first-value latch goes). S2's broken and S3's backoff go.
    • +
  • Version skew within one host: a reload of the registry's plugin alone leaves other plugins' contributors and +readers on their bundled versions; accepts[old].contribution and .value keep them working, unsupported +contributions are left out and reported, unsupported readers are waiting (version).
    • +
  • A registry waits for a value, never for a particular contribution (an empty table composes). After live, a later +compose failure changes nothing for the reader: .value keeps the last good composition, but only while every +contribution it was built from is still in the table (round 2, Astra N2): when one of them leaves (its +contributor ended or was reloaded), the last good value is dropped and .value throws until compose succeeds, so +it never hands out a native whose contributor ended. compose represents a missing contributor as a value.
    • +
  • Registries stay host-local (contributions are in-process values).
    • +
+

10. Keep-alive, presence, revocation, jobs

+
    +
  • A request over a link counts once in host.activity.references from its arrival to its end, bound or unbound; +presence (host.references) lists only bound references.
    • +
  • Revocation ends the requests of a revoked principal, bound and unbound, at every hop; nothing revives them.
    • +
  • gone stays only for revocation (a provider host closing closes its links and sends nothing; its consumers see a +drop, loading, and hold again on the next link).
    • +
  • Persisted job runs of a parked host-lifetime object are loaded with its documents; run and finish start only +once it is live; the run count keeps the object as today.
    • +
+

11. Timeouts after the model

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Timeout or backoffAfterWhy
readyTimeout of an attemptonly its own steps; paused while parkeda slow dependency is bounded where it runs
readyTimeout of host.get, ctx.get, host.registrykepta caller gets an answer
readyTimeout of a parked callkeptan action must not run long after it was asked for
identity backoff 1-60 sonly for the object's own transient failuresnothing else says the disk or network is back
dependency backoff, polling of refused holdsremovedrecovery is an event
dependency deadline across links(A): only for a parked attempt whose remote target attempt stays activating or parked; (B): until the probe's tests pass, then removedthe simultaneous-cycle gap
probe re-search backoff (B1)new, only with (B1): 1 s doubling to 60 s while parked on a remote attemptfinds a ring whose search was suppressed; sends probes only, never fails or re-holds
route backoff, heartbeat, closeTimeout, routine and finish backoffs, lingerkeptnot dependency waits
rollout install timeout (not built)a watchdog on the install onlynever a deadline for dependencies
+

12. What changes in the design

+

R18 (dependency failures waiting, no backoff, the deadline paused while parked, the dependency check as boundary, +wakes on every change, an ended dependency breaks), S2 and S3 (waiting, woken at every usable value), slice 8's +last-good rule (dropped when a contribution leaves), §3's placement table (by address; the publication gate), §6 (the +simultaneous-cycle gap: (A) one narrowed deadline or (B) the probe), §7 (host.hold, getters, calls), §21 (no gone on +removal; a dropped version unbinds only its requests; installs never await activations; rollout without revision +fields; the reload handover), the start barrier (R17, W6), FailureCause (+ waiting), loading/failed with the +attempt id, one message kind (probe, tagged search or check). Unchanged: what an object's own code reload does (§21), R18 after live, +retryOn outside parking, self.follow's R4, the call window and flow control, revocation, no client providers.

+

13. Open for the owner

+
    +
  1. The barrier: (a) or (b) (§6.3).
    2. +
  2. Getters on waiting: reject at once (recommended) or wait up to readyTimeout (§8).
    3. +
  3. waiting as a new cause kind (recommended) or not (§2).
    4. +
  4. A pure swap whose new factory declares a dependency that is not there: the object parks (its gate closes, holders +see loading or failed { waiting }, calls park or reject; recommended) or stays on the old code until it +resolves (old and new code would overlap without bound, against §21's "a host is at R or R−1").
    5. +
  5. The last good value (§9): kept only while every contribution it was built from is still in the table (draft; +Astra), or kept as slice 8 decided, with natives in a registry looked up at use through their owner (Sol).
    6. +
  6. Cycles across hosts (§7): (B) the probe, with (B1) a re-search backoff for probes only (recommended) or (B2) a +full search lifecycle without any timer; or (A) the slice 4b decision narrowed to one deadline, which polls chains +across hosts again, misses a ring of waiting attempts, and retries in lockstep without finding the cycle it is for.
    7. +
+
+ +

Parked Until Resolved · first edition · 2026-10-29 · Chord v2 design proposal for slice 7

+
+ + diff --git a/packages/chord/docs/next/dependencies/model.md b/packages/chord/docs/next/dependencies/model.md new file mode 100644 index 000000000..f29f430d0 --- /dev/null +++ b/packages/chord/docs/next/dependencies/model.md @@ -0,0 +1,419 @@ +# Dependency resolution in Chord v2: one model (draft 7, redesign session, 2026-10-29) + +Status: proposal for the owner, not decided. Built in slice 7 with reload and rollout. Sources: the owner's direction +("park until resolved"; "no difference between local and remote resolution: a link going is like a plugin +unloading"), the reviews (`/tmp/dep-mechanism-*.md`, `/tmp/dep-graph-*.md`, `/tmp/barrier-*.md`, +`/tmp/model/review{1,2}-*.md`), the code at pi `358f7d46a`. **Owner** marks an open decision. + +Draft 7 (round 6): (A) also retries in lockstep without finding the cycle (Fable); the probe's recovery after a +suppressed search becomes a choice between (B1) a re-search backoff for probes only (recommended) and (B2) a full +search lifecycle without timers, since `dead` notices alone need recovery state that eviction may not drop (Astra, Sol). +Causes are compared by value on wakes (Opus). +Round 7: a returned search is accepted while its initiator is parked, whatever its round (Astra: re-searches must not +invalidate searches under way); detection is eventual, not within one backoff period; the local walk stays (Opus). + +Draft 6 (round 5): cycles across hosts become an owner choice between (A) the owner's slice 4b decision (simultaneous +starts across hosts time out, no probe message), narrowed to one deadline, and (B) the probe, completed with edge +tokens, no revisits, and `dead` notices back to the initiator (Astra: a check that dies silently can lose the only +search that survived dedupe). (A) turned out to either poll chains across hosts again or miss a ring of `waiting` +attempts, so (B) is recommended, pending its review. + +Draft 5 (round 4, Sol's phantom cycle): the second round is replaced by a check along the recorded trail of +`(attempt, edge generation)` hops; a reported ring existed at one moment; probes grow with their path. + +Draft 4 after round 4 (probe only): round numbers per attempt, forwarded only when newer than the stored one; probes +honoured on parent and route links, dropped on child links; completeness stated under an eventually stable graph; the +repair caveat broadened to any concurrent edge removal; a new edge generation resets confirmation; a bounded display +trail. + +Draft 4 changes (round 3): the probe starts a fresh round on every new or changed wait edge (a ring closes at its last +edge, whose source finds it), keeps one seen entry per initiator, accepts only its current round, and states its +cost honestly; every host drops probes from untrusted links; local forwarding is iterative; no-placement remote +requests are re-placed when a host's first parent link connects; a cohort object that ends before its first outcome +settles the barrier; the last-good refinement is marked for the owner. + +Draft 3 changes (round 2): remote types without placement keep today's fallback (here if served, else the parent); +hydration stays before parking; no copied `retryAt`; a dependency `ended` breaks its dependent; a reload's handover +binds new requests before releasing old ones; retry passes only to `broken` and `transient` blockers, once per +operation; `installing` re-evaluated at every install end (commit or rejection); a registry's last good value dropped +when one of its contributors leaves; the probe with round ids, bounded seen sets, edge-generation triggers, attempt +ids on every non-live answer and relays dropping probes from untrusted links; one cycle mechanism, local and remote. + +## 0. The problem + +The same situation, "what I need is not there", has five outcomes today, depending only on how it is declared: + +| Declared as | Provider missing | Today | +| --- | --- | --- | +| `self.get(LocalType)` | no facet serves it | `self.get` throws, the factory throws: `broken` | +| `self.get(RemoteType)` | the worker does not serve it | the hold is refused `broken`; the dependent `transient`, polled 1-60 s forever | +| `self.registry(R)` | no provider | `transient`, woken at R's first value only | +| `self.registry(R)`, reader newer | | `broken` | +| a reload removes the type | | holders get `gone`: ended for good | + +And the startup barrier deadlocks an acyclic graph across two hosts (Sol, `/tmp/dep-graph-sol-barrier.log`): S's eager +A needs W's B, B needs S's C; W's hold for C waits for S's whole `load`, which waits for A. Only `readyTimeout` ends it. + +## 1. Requests and one resolver + +- **A request** is what a reference stands for: type (or registry), params, principal, the consumer's contract version. + Its holder makes it and owns it: an attempt (`self.get`, `self.registry`, keyed pins), a call (`ctx.get`), outside + code (`host.get`, `host.hold`, `host.registry`), a link's peer (`hold`). **It ends only when its holder ends it** (the + attempt ends, the call settles, `release()`, a getter gives up, the peer's link or origin closes, revocation, the + holding host closes), never because its provider is missing, went away or does not fit. +- **Placement by address** (round 1): a `local` type goes here; a `remote` type with a placement goes to the address + its placement (computed from the consumer's own contract) gives: here if that is this host's address, whether served + here or not, else over the route for that kind or the parent (design §3). A `remote` type **without placement** keeps + today's rule (round 2: Fable, Opus, Astra; no feature removed): here if served here, otherwise over the parent. A + request of that kind made before the host has a parent link waits here `not_served` and is **re-placed when the + host's first parent link connects** (round 3: Opus N5, Sol 3: a tab loads before it connects); on a host that never + connects (the root), it stays here. Opus would rather require a placement on every `remote()`; that removes a + supported declaration, so the draft keeps the fallback. A hold arriving for an address that is not this host's is relayed or + refused (`invalid/placement`) as today; a relay never captures a hold because it serves the same name. +- **Then the selected provider is checked** (served, version or accepted version and fingerprint, `authorize`, the + provider's own placement check). Version checks never look at a same-name registration elsewhere. +- **The publication gate:** a request made on this host, or arriving on its links, before the host's first + publication (§6) is not placed yet; it shows `loading` and is placed by the publication. The gate lasts only for the + synchronous setups, never for activation, so it cannot deadlock. +- **Outcomes of the check:** + + | The check finds | Request | + | --- | --- | + | a fitting provider | bound (to the object here, or to the hold over a path) | + | not published yet | unbound `unpublished`: `loading` | + | nobody serves the name at its address | unbound `not_served`: `failed { waiting }` | + | no fitting version, a fingerprint that differs at the same version, a placement the provider computes elsewhere (code skew) | unbound `version`: `failed { waiting }` | + | the registry is served, but its first `compose` (or the reader's version mapping) threw | unbound `compose`: `failed { waiting }` | + | the provider host is installing a revision and the hold does not fit its current code | unbound `installing` at the provider: `loading` | + | the path has no link (a drop, a route resolving) | on the path: `loading` | + | `authorize` refused, invalid params | `failed { broken, forbidden \| invalid }`, as today, never indexed | + + The unbound reasons that wait for a name (`unpublished`, `not_served`, `version`, `compose`) are indexed under it on + the host that decided them: the consumer's host for a local request; the provider's host for a hold that arrived + over a link (it already keeps such holds as refused: `remote/hold.ts`). `installing` is the provider host's own, + re-evaluated at its install's end; `no_link` lives on the path, where the next link holds it again. Indexes drop an + entry when its request ends; GC-tested. + +## 2. States + +`ref.state` keeps its four kinds; `FailureCause` gains one kind, `waiting`: + +```ts +type RefState = + | { kind: "loading" } + | { kind: "live" } + | { kind: "failed"; cause: FailureCause; retryAt: number | undefined } + | { kind: "ended"; error: ChordError }; + +type FailureCause = + | { kind: "transient"; error: ChordError } // the object's own effect failed (an open, a follow): backoff + | { kind: "broken"; error: ChordError } // the object's own code, a cycle, or authority: retry() or a reload + | { kind: "waiting"; error: ChordError }; // not the object's fault: no provider, no fitting version, a + // dependency failed; retryAt undefined, woken by an event +``` + +| The request is | `ref.state` | Calls | +| --- | --- | --- | +| bound, its object activating or parked with no failed blocker | `loading` | park, bounded | +| unbound `unpublished` or `installing`, or its path has no link | `loading` | park, bounded | +| unbound `not_served`, `version` or `compose` | `failed { waiting }` | reject at once | +| bound, its object failed | `failed` with the object's cause, as today | reject at once | +| bound, its object live | `live` | run | +| ended by its holder, revoked, or its host closed | `ended` | reject | + +**Owner:** `waiting` as a new cause kind (Opus, Fable; recommended: the cause says "not your fault, an event will wake +you" and a UI can say what is missing) or no new kind: `transient` with no `retryAt` (Astra; today's wire already +carries a `transient` cause without `retryIn`, `remote-object.ts`), or `loading` with a diagnostic (Sol; parks every +call on a missing plugin for its full `readyTimeout`). + +## 3. Dependents: parked until their dependencies resolve + +A **dependency** is what an attempt declares and needs before it goes live: `self.get` (local or remote), +`self.registry`, keyed pins (`at(key)` on another object's keyed member). `self.follow` is the attempt's own step (its +attach runs Durable's code), not a dependency; a follower declares `retryOn` on the registry it reads the harness +through, or recovers by its own backoff. + +1. **Order, as today up to the check:** the factory runs; persisted singletons hydrate (the object's documents, kept + across attempts); the requests are made. The attempt is **parked** while any dependency is not `live` (hydrated): + + | Its dependencies | The dependent shows | + | --- | --- | + | any `loading`, none failed | `loading` | + | any `failed` | `failed { waiting, "waiting for X: " }`, `retryAt` undefined; the message keeps the nearest blocker and the root cause within the 500-character rule | + | one failed with `invalid/cycle` (a dependent off the ring) | `broken` with the same error: it would wait on a ring that never heals by itself | + | one `ended` (revoked, or its host closed) | `broken` with that error: the request cannot come back; `retry()` or a reload makes new requests (a guard: a dependency ends only with its holder's principal or host, so the dependent ends too; Fable) | + +2. **The dependency check is the activation boundary:** once every dependency is `live`, the attempt runs its own + steps (natives in order, follow attaches, member checks, derives), bounded by its own `readyTimeout` deadline, and + goes live. A dependency that leaves `live` after the check is treated as after live (R18): the attempt goes on. This + is a deliberate rule (Astra: a weakening of "live means every dependency is live"): an object can finish opening + after a dependency went away, and its calls through that reference meet its state. So a parked attempt holds no + natives, and an attempt that opened natives is never parked. +3. **No timer while parked:** no dependency deadline, no backoff. The attempt's deadline is paused while parked (as + eager migrations pause it) and resumes with the budget left. +4. **Wakes, one rule:** the dependent re-evaluates on every state change of any of its dependencies (live, loading, + failed, a new cause compared by value, ended; Opus: otherwise a ring of waiting attempts passes its causes round and + round), updating what it shows; it leaves parking when all are live. The factory never re-runs + while parked. Wakes are coalesced per object and run after the batch that caused them; a change during the + attempt's own steps or its teardown is remembered. +5. **Kept while parked:** its documents (the object's), its declarations and requests, its seal token; no natives. + Ended with the object (its last holder; a host-lifetime object when its plugin is removed or the host closes). +6. **`broken` only for the object's own code** (a throwing factory, a derive throwing at activation, hydration, + `migrate`, a document fingerprint mismatch, an `open` throwing `broken(...)`), **cycles** (§7) and **authority** + (`forbidden`, an ended dependency). +7. **Retry passes on:** `ref.retry()` on a `failed { waiting }` object retries its `broken` and `transient` blockers, + hop by hop (the existing `retry` message over links), once per retry operation (an id dedupes diamonds); `waiting` + blockers pass it on; nothing else is touched. Retry of a `broken` or `transient` object starts a new attempt, as + today. +8. **`retryOn`** wakes failed objects as decided in slice 8 (a value call retries a `transient` or `broken` object); + while the object is parked `waiting`, a value call does nothing: its dependencies decide. +9. **Own-code reload of a parked object:** its plugin's reload replaces the parked attempt with the new factory's, + as one handover: the new requests are made and bound first, then the old ones released (round 2, Astra N4: an + unchanged dependency held only by this object must not end in between; today's `#activate` binds before + `#releasePrevious`). +10. **After live, R18 unchanged:** a dependency leaving `live` leaves its dependent live. + +## 4. Wakes: two events + +| Event | Re-evaluates | +| --- | --- | +| **a target changes state**: a dependency's object goes live, loading, fails, gets a new cause or ends; a hold's `bound`, `loading`, `failed` arrive, also after a re-hold on a new link | the requests bound to it, and their dependents (§3.4) | +| **a name becomes usable, an install ends, or the first parent link connects, on a host**: a publication (`load`'s setups, a reload's commit, an install's commit) serves a type, a registry, a version; a registry composes or maps a value at a version; an install ends without a commit (rejected) | the unbound requests indexed under those names; the provider's `installing` holds; at the first parent link, the +no-placement remote requests waiting here; a refused hold re-runs its start and sends `loading`, then `bound` or `failed` | + +No new message kind for wakes: `loading`, `bound`, `failed` and `retry` exist. The wire's `failed` carries the new cause +kind `waiting`; `loading` and a `waiting` `failed` carry the provider's attempt id for the probe (§7). + +## 5. Removal, versions, rollout + +- **A reload that removes a type:** its objects end as today (natives close, reason `end`; callers rejected at once; + handlers drain). Their requests are not ended: each becomes unbound `not_served` and binds again if a later commit + serves the name. + - **Locally:** a `Binding` whose target ended by removal becomes unbound and accepts a second `bind`: member sources + re-bind by name (one frame from the last value, as on a restart), keyed pins re-pin, parked calls flush or + reject, presence re-lists. The one new piece of machinery. + - **Over a link:** a `failed` on a bound hold unbinds it (Fable: one message, one rule; the consumer applies it as + `#startOver` plus `failed`); the provider detaches the hold's sources and drops its unsent pending ops and queued + `bound`; the ack slots of `bound`/`state` messages already sent stay counted until their `ack` or the link's loss; + calls in flight on it are rejected and free their slots once. +- **A reload that stops accepting a version** unbinds only the requests on that version (`version`); the object stays + (its own plugin's reload still restarts it if it has natives, §21). +- **A type moved to another host kind:** the request carries the address its own consumer code computed; the old host + answers `not_served`; the consumer's own reload makes a new request. No redirect. +- **A consumer whose code moves to a new version** releases its old request and makes a new one (today). +- **Rollout, without revision fields:** each host knows only its own install. A provider host that installs keeps + holds that do not fit its current code `installing` (`loading`) and re-evaluates them when the install ends, by its + commit or by its rejection. A provider host that is queued but not installing yet answers a mismatch `failed + { waiting, version }` until its install commits and wakes it: a visible blip of at most one install, accepted (Opus). + An install never waits for activations (a reverse-order rollout would deadlock); a host whose own install was + rejected fails its own requests at once with the reason (design §21). The loader keeps its own duties (the desired + code set, digests, serialized installs). +- **Old calls across a reload** follow §21 unchanged: a pure swap lets them finish on old handles; a native restart + drains them first; a removal rejects their callers at once and drains the handlers. Calls are never resent. + +## 6. Startup: publication, then first outcomes + +1. **Publication:** `load` runs every facet's setup, then publishes all at once in one batch (types served, + registries served and composed once, contributions in their tables), then wakes (§4) after that batch. Nothing + activates inside it. +2. **No gate beyond it:** holds and `host.hold` wait only for the publication, never for the barrier. This ends the + startup deadlock: W's hold for C binds as soon as S's setups published; B goes live, then A. +3. **The barrier** (**Owner:** (a) or (b); recommended (a), three reviewers of four): + - **(a):** `load` resolves when every host-lifetime object of this `load` has a first outcome: + `firstOutcome(o) = o.live || o.ownFailure || (o.parked && o.blockers.every((b) => !b.local || firstOutcome(b)))`, + where `local` means executing on this host (an object here, a keyed pin still loading here), not the contract's + kind. So `CodeMode → Harness` waits for the harness's outcome; a wait on another host or on a failed dependency + ends the first pass. One `firstOutcome` signal per object, separate from the readiness dependents wait for (Sol), + counted once, never reopened by retries. An object of the cohort that ends before its first outcome (its plugin + removed, the host closing) settles its entry as ended; `host.close` settles the waiting `load` (Astra R3). + - **What (a) promises:** every host-lifetime object on this host had its first try, and the contributions of those + that went live are in their tables. **Not:** every tool that will arrive (a parked or slow contributor, tools found + in `onLive`, MCP; Sol's probe `/tmp/barrier-sol-first-outcome.log`). What a first turn needs is p3's explicit + admission check. + - **(b):** `load` resolves after the publication; Chord adds `host.startup`, a watchable of the host-lifetime + objects' first outcomes with their causes; the app decides (Fable). The same tracker, awaited by the app. + - p2 today resumes its harness without waiting for all host objects (`apps/p2/src/hosts/worker.ts`), because its + worker objects that need server types are reachable only after `control.ready()`. + +## 7. Cycles + +On one host, cycles are found exactly as today: at `self.get`, following the waiting edges of activating and parked +attempts. Across hosts, a hold carries the activation chain it was made in (today), which finds every cycle whose +objects start one after another. What remains is the documented gap (design §6): objects on two hosts that start at +the same moment and need each other. The owner decided on 2026-10-24 (slice 4b review decisions (2)) that this is a +documented limitation that times out and retries, with no probe message. Without dependency deadlines that timeout no +longer exists, so the gap needs one of two answers (**Owner**): + +- **(A) The 4b decision, narrowed.** One deadline stays: an attempt parked on a dependency held over a link whose target + attempt is known and still activating or parked fails `transient` after its `readyTimeout` and retries with backoff; + its next attempt usually starts after the other side and its chain finds the cycle. It has a cost that decides + against it: the target being *parked* is the common case of a chain across hosts waiting for a missing plugin (a + tab's view on a worker object that waits for an uninstalled type), and those dependents would time out and poll + again, as today, so "local = remote" would not hold across hosts. Narrowing it to targets that are `loading` (not + `failed { waiting }`) avoids that but misses a real ring: A and B on two hosts wait on each other, B also on a + missing X; B is `failed { waiting }`, so A is too; when X appears, both stay `failed { waiting }` on each other, with + no deadline and no chain to find it. And even the timeout does not find what it is meant to (round 6, Fable): an + object's chain is fixed when it is created and its instance id survives retries, so a retried attempt sends the same + hold with the same chain; two simultaneous starts time out and retry in lockstep and are never found. +- **(B) The probe (recommended)**, below: no deadline at all, one new message kind, the most intricate rule set in the + model. Five + review rounds found a counterexample in each of the first four drafts; draft 6's version answers all of them, but it + has not been reviewed again, and slice 7 would have to prove it with tests before the deadline could go. + +### (B) The probe + +Replaces the activation chain in holds. On one host the `self.get` walk stays (it is synchronous and exact: a ring +with no remote edge is found there, where (B1)'s timer does not run; Opus, round 7); the probe serves the rings that +cross a link. It replaces draft 1's bounded chain too +draft 1's bounded chain (a ring of 33 is never found with chains cut at 32; chains keep stale ancestry). Draft 5 replaces the +second round by a check along the recorded path (round 4, Sol: two rounds through different stale paths can report a +ring that never existed, `/tmp/model/sol-probe4.log`). + +1. **Attempts, edges, tokens.** Every attempt has its own id (uuidv7 per attempt: a retry or reload is a new + attempt). A **wait edge** runs from a parked attempt to the parked or activating attempt it waits for, through a + bound request (unbound waits and registry reads are never edges; they are not deadlocks). Each edge carries an **edge + token** (round 5, Astra, Sol): a number from a counter per attempt, never reused, replaced by a fresh one on any + interruption or change of that edge, so two requests to one object are two edges: a new target attempt (also a + local target's retry, without a re-bind), any `loading` on a bound or pending hold (a relay's upstream re-hold + included), an unbinding, a rebinding, a re-hold after a reconnect. An attempt is parked for one interval only (one + that passed its dependency check never parks again) and ids are never reused. A provider tells its consumers the attempt they wait for on + every non-live answer: `loading { hold, attempt }` and a `waiting` `failed { hold, cause, attempt }`. +2. **A search per edge change.** Whenever a parked attempt gets a new edge or a new token on one, it starts a + search: `probe { kind: "search", round, initiator, trail }` along each of its wait edges, `round` the next number of + a counter the attempt keeps, `trail` the list of `(attempt, edge token)` hops it crossed, starting with + the initiator's own. A ring closes at its last + edge, and that edge's source starts the search that finds it. Coalesced per batch. Locally a call on the target + attempt, forwarded through a queue (no recursion); over a link the message `probe { hold, to, ... }`, consumer to + provider, relayed by hold id with the origin. **Probes are honoured on parent and route links and dropped on child + links** (`accept`), whatever `trusted` says. +3. **Forwarding.** An attempt receiving a search drops it if `to` is not its current attempt or it is not parked. If + the initiator is itself, the search returned (4). If the attempt is already in the trail, it drops it (a ring that does not contain the initiator is + found by its own members' searches; Opus). Otherwise it forwards it along every wait edge, appending `(itself, the edge's + token)` to the trail, only if the round number is higher than the one it stored for that + initiator (older or equal rounds are dropped: the initiator superseded them), and stores it. The seen map holds one + number per initiator, up to 256 initiators, and goes with the attempt. +4. **The check.** A returned search is accepted while its initiating attempt is still parked, whatever round it + carries (round 7, Astra: a re-search must not invalidate an older search still under way); the check decides. The + initiator then sends `probe { kind: "check", round, trail }` along the recorded trail: the initiator first verifies its own hop; each hop passes it on only if it is still + that attempt, still parked, still has the recorded edge token, pointing at the next hop's attempt; otherwise the + check dies (also on a `to` mismatch). The check follows the trail by position and bypasses the seen + map. The initiator verifies its own hop again when the check returns. If the check returns, the initiator fails `broken invalid/cycle`, the path named for display from the trail, + and today's propagation breaks the other activating objects on the ring and the dependents off it. 5. **No silent loss.** Dedupe can drop the copy of a search that travelled the ring in favour of a copy whose path + later breaks (Astra, round 5); then the check dies and nothing is left to find the ring. Round 6 tried `dead` + notices back to the initiator; Sol and Astra showed that reliable notices need recovery state per search that + eviction may not drop and that a long-parked attempt would otherwise keep for every initiator that ever passed (a + leak), i.e. a full search lifecycle with cancellation. Two ways (**Owner**, part of decision 6): + - **(B1) Re-search with backoff (recommended).** An attempt parked on a remote attempt starts a fresh search again + after a backoff (1 s doubling to 60 s, reset by an edge change), whether it shows `loading` or `failed { waiting }`; + the timer stops when the attempt leaves parking. No `dead` messages, no recovery state, eviction + only costs work; a new search starts within one backoff period, so every stable ring that crosses a link is + eventually found (no wall-clock bound: delivery is only fair; a ring on one host is the walk's). The timer runs as long as the attempt stays parked on a remote attempt: a view on a worker object that + waits for a missing plugin re-searches every 60 s until a reload (Fable: cheap, but for as long as it waits). The timer sends probes only: it never + fails an object, re-holds or polls a provider. Cost: one probe per parked remote dependent per period, which + crosses the downstream wait graph and dies at its leaves (an unbound wait is not an edge). + - **(B2) Events only.** Each search is owned until it settles: branches report completion or abort back along their + trails, a dropped probe answers `dead { initiator, round }` with a reverse route, an edge records the searches it + carries until they settle, and an initiator that ends or starts a new round cancels its old one along the same + routes. Complete without a timer, bounded by the searches in flight, and the most intricate protocol in Chord; not + yet specified to the reviewers' satisfaction. + Rejected: enumerating every simple trail without dedupe (Sol's correctness reference): no ring copy is ever + suppressed, but work is exponential on chains of diamonds. +- **Why a reported ring existed:** every edge of the trail kept its token from the moment the search crossed it + to the moment the check crossed it; the search crossed all of them before the check started, so at the moment the + check started every edge of the ring existed at once. A token changes on every interruption and is never reused, so an edge cannot + vanish and come back unnoticed. Sol's counterexample fails at the check: `C1` is no longer `B`'s target. +- **Why every ring is found** (under an eventually stable wait graph and fair delivery; endless edge churn can keep + invalidating searches; with rule 5's (B1) or (B2) for a search suppressed behind a path that broke): a ring is closed by its last edge; its source starts a search when the edge appears; every + other edge exists then and every attempt on the ring waits transitively on the source, so none goes live before the + search and the check return. Rings of any length, local and remote mixed. On a DAG searches die at the leaves; on a + diamond the second arrival is dropped. +- **What it does not promise:** (1) a ring that existed when the check started may be broken up by a concurrent reload, + retry, unbinding or revocation just after; it is still reported (it did exist). Its broken objects recover as every + broken object does: `ref.retry()` or a reload of their plugin. (2) Linear work under every schedule: with more than + 256 initiators passing one attempt at once, an evicted entry lets a search be forwarded again, so work can grow with + the number of simple paths (never endlessly: a search never revisits an attempt in its trail). (3) Constant message size: a probe carries its trail, so its size grows with the length of the + path it crossed (a returning search's trail is as long as its ring); no cap cuts it, since the trail is the proof. +- **Cost:** one search per edge change, crossing each downstream edge once, one check per returned search; with (B1), + one more search per backoff period for every attempt parked on a remote attempt, also while an acyclic chain waits + for a missing plugin, plus the trail bytes. The bounded memory is the seen map's; probes in flight are bounded by + the searches started. +- **Transition:** today's dependency deadline for activations waiting across links stays until the probe's tests pass + in slice 7 (long rings, diamonds, a ring closed after a reconnect with unchanged attempt ids, Sol's phantom schedule, + 300 concurrent initiators, edges added and removed, reloads, relays, forged probes); then it goes in the same slice. + +## 8. Getters, holds, calls: one rule + +- **`host.hold`** returns a reference at once; only invalid params throw (also before `load`). A missing local type + gives `failed { waiting }` after the publication, `loading` before it. +- **`host.get`, `ctx.get` and `host.registry`:** resolve when live; wait while `loading`, or `failed { transient }` + (its own retry is under way), up to their `readyTimeout`; reject at once on `failed { waiting }` or `{ broken }`. On + success, rejection or timeout their temporary wait goes; `host.registry` returns the `RegistryValue` with its + ordinary watch ownership (no `release()`). **Owner:** recommended (nobody waits 60 s for a plugin that is not + there; code that wants to wait holds instead), or wait through `waiting` too. +- **Calls** on a `loading` reference park, each bounded by the reference's `readyTimeout` from when it was made; on a + `failed` one they reject at once with the cause; sent calls are never resent. + +## 9. Registries + +- The same resolver on one host: a reader's request for R at its version is bound while R is served and has a usable + value at that version (composed; mapped by `accepts[old].value` for an older reader), otherwise unbound + `not_served`, `version` or `compose`, or `loading` before the publication. Woken at every transition to usable + (`#valued`'s first-value latch goes). S2's `broken` and S3's backoff go. +- **Version skew within one host:** a reload of the registry's plugin alone leaves other plugins' contributors and + readers on their bundled versions; `accepts[old].contribution` and `.value` keep them working, unsupported + contributions are left out and reported, unsupported readers are `waiting` (`version`). +- A registry waits for a value, never for a particular contribution (an empty table composes). After live, a later + `compose` failure changes nothing for the reader: `.value` keeps the last good composition, **but only while every + contribution it was built from is still in the table** (round 2, Astra N2): when one of them leaves (its + contributor ended or was reloaded), the last good value is dropped and `.value` throws until `compose` succeeds, so + it never hands out a native whose contributor ended. `compose` represents a missing contributor as a value. +- Registries stay host-local (contributions are in-process values). + +## 10. Keep-alive, presence, revocation, jobs + +- A request over a link counts once in `host.activity.references` from its arrival to its end, bound or unbound; + presence (`host.references`) lists only bound references. +- Revocation ends the requests of a revoked principal, bound and unbound, at every hop; nothing revives them. +- `gone` stays only for revocation (a provider host closing closes its links and sends nothing; its consumers see a + drop, `loading`, and hold again on the next link). +- Persisted job runs of a parked host-lifetime object are loaded with its documents; `run` and `finish` start only + once it is live; the run count keeps the object as today. + +## 11. Timeouts after the model + +| Timeout or backoff | After | Why | +| --- | --- | --- | +| `readyTimeout` of an attempt | only its own steps; paused while parked | a slow dependency is bounded where it runs | +| `readyTimeout` of `host.get`, `ctx.get`, `host.registry` | kept | a caller gets an answer | +| `readyTimeout` of a parked call | kept | an action must not run long after it was asked for | +| identity backoff 1-60 s | only for the object's own `transient` failures | nothing else says the disk or network is back | +| dependency backoff, polling of refused holds | removed | recovery is an event | +| dependency deadline across links | (A): only for a parked attempt whose remote target attempt stays activating or parked; (B): until the probe's tests pass, then removed | the simultaneous-cycle gap | +| probe re-search backoff (B1) | new, only with (B1): 1 s doubling to 60 s while parked on a remote attempt | finds a ring whose search was suppressed; sends probes only, never fails or re-holds | +| route backoff, heartbeat, `closeTimeout`, routine and `finish` backoffs, `linger` | kept | not dependency waits | +| rollout install timeout (not built) | a watchdog on the install only | never a deadline for dependencies | + +## 12. What changes in the design + +R18 (dependency failures `waiting`, no backoff, the deadline paused while parked, the dependency check as boundary, +wakes on every change, an ended dependency breaks), S2 and S3 (`waiting`, woken at every usable value), slice 8's +last-good rule (dropped when a contribution leaves), §3's placement table (by address; the publication gate), §6 (the +simultaneous-cycle gap: (A) one narrowed deadline or (B) the probe), §7 (`host.hold`, getters, calls), §21 (no `gone` on +removal; a dropped version unbinds only its requests; installs never await activations; rollout without revision +fields; the reload handover), the start barrier (R17, W6), `FailureCause` (+ `waiting`), `loading`/`failed` with the +attempt id, one message kind (`probe`, tagged `search` or `check`). Unchanged: what an object's own code reload does (§21), R18 after live, +`retryOn` outside parking, `self.follow`'s R4, the call window and flow control, revocation, no client providers. + +## 13. Open for the owner + +1. The barrier: (a) or (b) (§6.3). +2. Getters on `waiting`: reject at once (recommended) or wait up to `readyTimeout` (§8). +3. `waiting` as a new cause kind (recommended) or not (§2). +4. A pure swap whose new factory declares a dependency that is not there: the object parks (its gate closes, holders + see `loading` or `failed { waiting }`, calls park or reject; recommended) or stays on the old code until it + resolves (old and new code would overlap without bound, against §21's "a host is at R or R−1"). +5. **The last good value** (§9): kept only while every contribution it was built from is still in the table (draft; + Astra), or kept as slice 8 decided, with natives in a registry looked up at use through their owner (Sol). +6. **Cycles across hosts** (§7): (B) the probe, with (B1) a re-search backoff for probes only (recommended) or (B2) a + full search lifecycle without any timer; or (A) the slice 4b decision narrowed to one deadline, which polls chains + across hosts again, misses a ring of `waiting` attempts, and retries in lockstep without finding the cycle it is for. From 67c63c0f7ff41a55176683b0f55bbd8fb0f4935e Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 06:49:17 +0000 Subject: [PATCH 008/175] docs(chord): dependency model: the dependency check applies R18 after live, not a rule of its own --- packages/chord/docs/next/dependencies/book.md | 11 ++++++----- packages/chord/docs/next/dependencies/index.html | 11 ++++++----- packages/chord/docs/next/dependencies/model.md | 6 +++--- 3 files changed, 15 insertions(+), 13 deletions(-) diff --git a/packages/chord/docs/next/dependencies/book.md b/packages/chord/docs/next/dependencies/book.md index 19b550bdf..4d9a45678 100644 --- a/packages/chord/docs/next/dependencies/book.md +++ b/packages/chord/docs/next/dependencies/book.md @@ -352,8 +352,9 @@ bounded by its own `readyTimeout` deadline, as today. Then it goes live. The dependency check is the boundary between waiting and working (Astra's F3). Before it, the attempt holds no natives; after it, a dependency that leaves `live` is treated as after live (R18): the attempt goes on, and its calls through that reference meet the reference's state. So a parked attempt never holds a native, and an attempt that -opened a native is never parked. This is a deliberate rule, and Astra called it what it is: a weakening of "live -means every dependency is live". An object can finish opening after a dependency went away. Without that boundary a contributor unloading while the attempt opens its natives +opened a native is never parked. It is not a rule of its own: it is R18's rule for live objects (a dependency that +goes away leaves the object live; same-host calls in flight finish, remote ones fail, new calls meet the reference's +state), applied from the dependency check on. Without that boundary a contributor unloading while the attempt opens its natives would leave it half open and parked, or force a seal, a drain and a fresh attempt for a dependency the object may not even use at once. @@ -1399,9 +1400,9 @@ through, or recovers by its own backoff. 2. **The dependency check is the activation boundary:** once every dependency is `live`, the attempt runs its own steps (natives in order, follow attaches, member checks, derives), bounded by its own `readyTimeout` deadline, and - goes live. A dependency that leaves `live` after the check is treated as after live (R18): the attempt goes on. This - is a deliberate rule (Astra: a weakening of "live means every dependency is live"): an object can finish opening - after a dependency went away, and its calls through that reference meet its state. So a parked attempt holds no + goes live. A dependency that leaves `live` after the check is treated as after live (R18): the attempt goes on. This is + R18's rule for live objects applied from the check on, not a rule of its own (owner, 2026-10-29: a dependency can + go away at any point of an object's life; the object stays live, its calls through that reference meet its state). So a parked attempt holds no natives, and an attempt that opened natives is never parked. 3. **No timer while parked:** no dependency deadline, no backoff. The attempt's deadline is paused while parked (as eager migrations pause it) and resumes with the budget left. diff --git a/packages/chord/docs/next/dependencies/index.html b/packages/chord/docs/next/dependencies/index.html index 3bc8daa6f..3cbb8f533 100644 --- a/packages/chord/docs/next/dependencies/index.html +++ b/packages/chord/docs/next/dependencies/index.html @@ -538,8 +538,9 @@ bounded by its own readyTimeout deadline, as today. Then it goes li

The dependency check is the boundary between waiting and working (Astra's F3). Before it, the attempt holds no natives; after it, a dependency that leaves live is treated as after live (R18): the attempt goes on, and its calls through that reference meet the reference's state. So a parked attempt never holds a native, and an attempt that -opened a native is never parked. This is a deliberate rule, and Astra called it what it is: a weakening of "live -means every dependency is live". An object can finish opening after a dependency went away. Without that boundary a contributor unloading while the attempt opens its natives +opened a native is never parked. It is not a rule of its own: it is R18's rule for live objects (a dependency that +goes away leaves the object live; same-host calls in flight finish, remote ones fail, new calls meet the reference's +state), applied from the dependency check on. Without that boundary a contributor unloading while the attempt opens its natives would leave it half open and parked, or force a seal, a drain and a fresh attempt for a dependency the object may not even use at once.

CodeModehost-lifetime, localHarnesshost-lifetime, localHarnessModelsregistryself.get(Harness): loadingparked: shows loading, no timerself.registry(HarnessModels): usableown steps: open DurableHarness live: a state changedependency check passes; open the sandboxlive; contribute tools
Figure 3. Code mode on W parks on the harness and goes on the moment it is live.
@@ -2256,9 +2257,9 @@ across attempts); the requests are made. The attempt is parked

  • The dependency check is the activation boundary: once every dependency is live, the attempt runs its own steps (natives in order, follow attaches, member checks, derives), bounded by its own readyTimeout deadline, and -goes live. A dependency that leaves live after the check is treated as after live (R18): the attempt goes on. This -is a deliberate rule (Astra: a weakening of "live means every dependency is live"): an object can finish opening -after a dependency went away, and its calls through that reference meet its state. So a parked attempt holds no +goes live. A dependency that leaves live after the check is treated as after live (R18): the attempt goes on. This is +R18's rule for live objects applied from the check on, not a rule of its own (owner, 2026-10-29: a dependency can +go away at any point of an object's life; the object stays live, its calls through that reference meet its state). So a parked attempt holds no natives, and an attempt that opened natives is never parked.

  • No timer while parked: no dependency deadline, no backoff. The attempt's deadline is paused while parked (as diff --git a/packages/chord/docs/next/dependencies/model.md b/packages/chord/docs/next/dependencies/model.md index f29f430d0..172f2a688 100644 --- a/packages/chord/docs/next/dependencies/model.md +++ b/packages/chord/docs/next/dependencies/model.md @@ -145,9 +145,9 @@ through, or recovers by its own backoff. 2. **The dependency check is the activation boundary:** once every dependency is `live`, the attempt runs its own steps (natives in order, follow attaches, member checks, derives), bounded by its own `readyTimeout` deadline, and - goes live. A dependency that leaves `live` after the check is treated as after live (R18): the attempt goes on. This - is a deliberate rule (Astra: a weakening of "live means every dependency is live"): an object can finish opening - after a dependency went away, and its calls through that reference meet its state. So a parked attempt holds no + goes live. A dependency that leaves `live` after the check is treated as after live (R18): the attempt goes on. This is + R18's rule for live objects applied from the check on, not a rule of its own (owner, 2026-10-29: a dependency can + go away at any point of an object's life; the object stays live, its calls through that reference meet its state). So a parked attempt holds no natives, and an attempt that opened natives is never parked. 3. **No timer while parked:** no dependency deadline, no backoff. The attempt's deadline is paused while parked (as eager migrations pause it) and resumes with the budget left. From 1b760044ec6686f669f09500dfeaf8f04d3af49d Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 07:06:37 +0000 Subject: [PATCH 009/175] docs(chord): note: every object may contribute (for later) --- packages/chord/docs/next/decisions.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 59893c07b..17b8e8aa4 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2337,6 +2337,10 @@ Everything else from earlier lists is decided (see the entries above and `packag - **Future goal: deploy Chord apps to Cloudflare Durable Objects and similar platforms.** Hot reload there is not needed; composition through plugins is (plugin set fixed at deploy). Research in `research/chord-on-durable-objects.md`. - Deferred until a concrete case: jobs as objects and where a flow lives (when login is ported); project plugins. +- **Every object may contribute** (owner, 2026-10-29: "note it down as something to consider for the future; can be + changed later without a need for migrations"): lift R21 so objects with params and `remote` objects contribute too; + the entry lives while the object is live, its identity is the object's (type, params, principal key); `contribute` + on every `self`. Not decided; R21 stands. - p3 domain points (decided per domain when p3 is designed): `SessionFiles {session}` (proposed), session delete, connection pools, config files with a single writer, file formats as versioned interfaces, the terminal host process as a revision artifact. From 13dee755297b9b2a3d1366b1296984fb45978636 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 07:17:22 +0000 Subject: [PATCH 010/175] docs(chord): every remote type declares its placement --- packages/chord/docs/next/decisions.md | 6 ++++++ packages/chord/docs/next/design.md | 6 +++--- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 17b8e8aa4..d4ba29fe0 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -1771,6 +1771,12 @@ validation and copies the v2 kernel added around Delta. setups a registry without a value cannot get one before a reload, so a reader then fails at once `transient` (`unavailable/undeclared`) and is retried when the registry gets its first value (R18), as `self.get` of a type not served here fails at once (refined 2026-10-27 after the second spec review: no `readyTimeout` wait at start); once live, a later failure of the registry changes nothing (last good value). +- **Every remote type declares its placement (owner, 2026-10-29: "yes, let's remove it"):** `placement` is required in + `remote()` (a type error without it, and a declaration error at runtime); a remote type without placement, a leftover + of slice 3a (remote objects over one link, before host kinds in 4a), is gone, and with it the fallback "here if served + here, otherwise the parent" and its re-placement at the first parent link. Every remote request is placed by its + address alone (dependency model, `packages/chord/docs/next/dependencies/`). Tests and examples get `placement: () => + Server()` or their own host kind. - **No client providers for now (owner, 2026-10-28: "yeah, no client providers for now"; reviews `/tmp/barrier-{astra,fable,opus,sol}.md`):** clients stay unaddressable and serve only `local` types (design §3). Client-side capabilities (a phone's camera, clipboard, a page's WebMCP tools) go through an owner's state, as the diff --git a/packages/chord/docs/next/design.md b/packages/chord/docs/next/design.md index dc38b5642..2772ee9e8 100644 --- a/packages/chord/docs/next/design.md +++ b/packages/chord/docs/next/design.md @@ -117,8 +117,8 @@ export const Catalog = remote("catalog", { placement: () => Server(), … }); - `hostKind(name, params?)` declares a kind of addressable host; its params follow the params policy of contracts (section 5). `Session({ id: "S1" })` is an **address**: `{ kind, params, key }`, the key built like an identity key (`session|S1`); two addresses are equal when their keys are. No host-name strings appear in app code. -- **`placement`** (owner, 2026-10-21) is a typed function of the params returning an address; `local` contracts have - none. It is not part of the contract fingerprint; a changed rule surfaces as a refusal (below). +- **`placement`** (owner, 2026-10-21) is a typed function of the params returning an address, required on every + `remote` contract (owner, 2026-10-29); `local` contracts have none. It is not part of the contract fingerprint; a changed rule surfaces as a refusal (below). - **`params`** (owner, 2026-10-21; was `input`) select the instance among objects of its type and are part of its identity (section 5). - **The caller computes the address** and sends it in the `hold`; a host that relays routes by that address alone and @@ -129,7 +129,7 @@ export const Catalog = remote("catalog", { placement: () => Server(), … }); | Case | Goes | | --- | --- | - | the type is served here and its placement (computed here from the params) is this host's address, or the host has no address, or the type no placement | here | + | the type is served here and its placement (computed here from the params) is this host's address, or the host has no address | here | | a reference made here (`host.get`/`hold`, `self.get`, `ctx.get`) to a served type placed on another host | on, as if not served here (next rows): worker S1 reaches `Notes { session: "S2" }` through the parent | | an arriving hold for a served type whose params place it on another host | broken `invalid/placement`; the address the hold names only routes it | | the hold names an address of a kind this host has a route for | over that route's link (the address's params checked against the kind, then `authorize`) | From 0ddc953102a06424e9d731d60ace171dfcd19f06 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 07:17:49 +0000 Subject: [PATCH 011/175] docs(chord): dependency model: placement required on remote types --- packages/chord/docs/next/dependencies/book.md | 24 ++++++-------- .../chord/docs/next/dependencies/index.html | 31 +++++++------------ .../chord/docs/next/dependencies/model.md | 12 +++---- 3 files changed, 24 insertions(+), 43 deletions(-) diff --git a/packages/chord/docs/next/dependencies/book.md b/packages/chord/docs/next/dependencies/book.md index 4d9a45678..15665a434 100644 --- a/packages/chord/docs/next/dependencies/book.md +++ b/packages/chord/docs/next/dependencies/book.md @@ -195,12 +195,10 @@ compares the consumer with the provider actually selected, never with a same-nam host (today `#assertRoute` does that before placement, so worker W1 serving `Notes` v1 rejects a `Notes` v2 request placed on W2, which serves v2). -One case keeps today's rule: a `remote` type declared **without placement** (allowed by `remote()`, used in tests and -for server types a tab reaches through its parent). It goes here if it is served here, otherwise over the parent. A -request of that kind made before the host has a parent link (a tab loads before it connects) waits here as not served -and is placed again when the host's first parent link connects (Opus, Sol); on a host that never connects, the root, -it stays here. Opus would rather require a placement on every remote type, which is simpler; it removes a supported -declaration, and the owner asked for no feature to go, so the draft keeps the fallback. +Every `remote` type declares its placement (owner, 2026-10-29). Until then `remote()` allowed leaving it out, a +leftover of slice 3a, which built remote objects over one link before host kinds existed; such a type went "here if +served, otherwise the parent", which needed its own fallback and a re-placement when a tab that loads before it +connects gets its first parent link (Opus, Sol). Opus and Fable proposed making it required; the owner agreed. ## What the check finds @@ -1218,7 +1216,7 @@ scenarios against five drafts; the catalog in the chapters before this one is a | R18, the boundary | "live" means every dependency was live at the end of activation | the dependency check is the boundary: after it, a dependency leaving `live` is treated as after live | | `self.get` of a missing `local` type | throws in the factory: `broken` | unbound `not_served`: `failed { waiting }`, woken by the publication that serves it | | a remote hold the provider cannot serve | refused `broken`; the dependent polls by backoff | refused and kept at the provider, re-run at its publication; `failed { waiting }` | -| design §3, placement | "served here" decides "here" | the address decides; a remote type without placement keeps today's rule | +| design §3, placement | "served here" decides "here"; `placement` optional on `remote()` | the address decides; every remote type declares its placement (owner, 2026-10-29) | | the start barrier, `pendingLoad` | holds wait for all of `load` | holds wait only for the publication; the barrier is (a) or (b) | | S2, a registry reader newer than its provider | `broken` | `failed { waiting, version }` | | S3, a registry without a value | `transient`, woken at the first value only | `failed { waiting }`, woken at every transition to usable | @@ -1318,12 +1316,9 @@ A needs W's B, B needs S's C; W's hold for C waits for S's whole `load`, which w holding host closes), never because its provider is missing, went away or does not fit. - **Placement by address** (round 1): a `local` type goes here; a `remote` type with a placement goes to the address its placement (computed from the consumer's own contract) gives: here if that is this host's address, whether served - here or not, else over the route for that kind or the parent (design §3). A `remote` type **without placement** keeps - today's rule (round 2: Fable, Opus, Astra; no feature removed): here if served here, otherwise over the parent. A - request of that kind made before the host has a parent link waits here `not_served` and is **re-placed when the - host's first parent link connects** (round 3: Opus N5, Sol 3: a tab loads before it connects); on a host that never - connects (the root), it stays here. Opus would rather require a placement on every `remote()`; that removes a - supported declaration, so the draft keeps the fallback. A hold arriving for an address that is not this host's is relayed or + here or not, else over the route for that kind or the parent (design §3). Every `remote` type declares + its placement (owner, 2026-10-29: the optional placement of slice 3a is removed, and with it the fallback "here if + served, otherwise the parent" and its re-placement at the first parent link). A hold arriving for an address that is not this host's is relayed or refused (`invalid/placement`) as today; a relay never captures a hold because it serves the same name. - **Then the selected provider is checked** (served, version or accepted version and fingerprint, `authorize`, the provider's own placement check). Version checks never look at a same-name registration elsewhere. @@ -1433,8 +1428,7 @@ through, or recovers by its own backoff. | Event | Re-evaluates | | --- | --- | | **a target changes state**: a dependency's object goes live, loading, fails, gets a new cause or ends; a hold's `bound`, `loading`, `failed` arrive, also after a re-hold on a new link | the requests bound to it, and their dependents (§3.4) | -| **a name becomes usable, an install ends, or the first parent link connects, on a host**: a publication (`load`'s setups, a reload's commit, an install's commit) serves a type, a registry, a version; a registry composes or maps a value at a version; an install ends without a commit (rejected) | the unbound requests indexed under those names; the provider's `installing` holds; at the first parent link, the -no-placement remote requests waiting here; a refused hold re-runs its start and sends `loading`, then `bound` or `failed` | +| **a name becomes usable, or an install ends, on a host**: a publication (`load`'s setups, a reload's commit, an install's commit) serves a type, a registry, a version; a registry composes or maps a value at a version; an install ends without a commit (rejected) | the unbound requests indexed under those names; the provider's `installing` holds; a refused hold re-runs its start and sends `loading`, then `bound` or `failed` | No new message kind for wakes: `loading`, `bound`, `failed` and `retry` exist. The wire's `failed` carries the new cause kind `waiting`; `loading` and a `waiting` `failed` carry the provider's attempt id for the probe (§7). diff --git a/packages/chord/docs/next/dependencies/index.html b/packages/chord/docs/next/dependencies/index.html index 3cbb8f533..656a4f6fa 100644 --- a/packages/chord/docs/next/dependencies/index.html +++ b/packages/chord/docs/next/dependencies/index.html @@ -327,12 +327,10 @@ address. A request placed here whose provider is missing is unbound here compares the consumer with the provider actually selected, never with a same-name registration on the consumer's own host (today #assertRoute does that before placement, so worker W1 serving Notes v1 rejects a Notes v2 request placed on W2, which serves v2).

    -

    One case keeps today's rule: a remote type declared without placement (allowed by remote(), used in tests and -for server types a tab reaches through its parent). It goes here if it is served here, otherwise over the parent. A -request of that kind made before the host has a parent link (a tab loads before it connects) waits here as not served -and is placed again when the host's first parent link connects (Opus, Sol); on a host that never connects, the root, -it stays here. Opus would rather require a placement on every remote type, which is simpler; it removes a supported -declaration, and the owner asked for no feature to go, so the draft keeps the fallback.

    +

    Every remote type declares its placement (owner, 2026-10-29). Until then remote() allowed leaving it out, a +leftover of slice 3a, which built remote objects over one link before host kinds existed; such a type went "here if +served, otherwise the parent", which needed its own fallback and a re-placement when a tab that loads before it +connects gets its first parent link (Opus, Sol). Opus and Fable proposed making it required; the owner agreed.

    What the check finds

    Once placed, the provider found there is checked: served, a fitting version (the current one or an accepted older one, with the fingerprint computed from the provider's own history), authorize, and the provider's own placement check. @@ -1932,8 +1930,8 @@ owner's decisions.

    • design §3, placement -"served here" decides "here" -the address decides; a remote type without placement keeps today's rule +"served here" decides "here"; placement optional on remote() +the address decides; every remote type declares its placement (owner, 2026-10-29) the start barrier, pendingLoad @@ -2103,12 +2101,9 @@ holding host closes), never because its provider is missing, went away or does n

  • Placement by address (round 1): a local type goes here; a remote type with a placement goes to the address its placement (computed from the consumer's own contract) gives: here if that is this host's address, whether served -here or not, else over the route for that kind or the parent (design §3). A remote type without placement keeps -today's rule (round 2: Fable, Opus, Astra; no feature removed): here if served here, otherwise over the parent. A -request of that kind made before the host has a parent link waits here not_served and is re-placed when the -host's first parent link connects (round 3: Opus N5, Sol 3: a tab loads before it connects); on a host that never -connects (the root), it stays here. Opus would rather require a placement on every remote(); that removes a -supported declaration, so the draft keeps the fallback. A hold arriving for an address that is not this host's is relayed or +here or not, else over the route for that kind or the parent (design §3). Every remote type declares +its placement (owner, 2026-10-29: the optional placement of slice 3a is removed, and with it the fallback "here if +served, otherwise the parent" and its re-placement at the first parent link). A hold arriving for an address that is not this host's is relayed or refused (invalid/placement) as today; a relay never captures a hold because it serves the same name.

  • Then the selected provider is checked (served, version or accepted version and fingerprint, authorize, the @@ -2307,12 +2302,8 @@ unchanged dependency held only by this object must not end in between; today' the requests bound to it, and their dependents (§3.4) -a name becomes usable, an install ends, or the first parent link connects, on a host: a publication (load's setups, a reload's commit, an install's commit) serves a type, a registry, a version; a registry composes or maps a value at a version; an install ends without a commit (rejected) -the unbound requests indexed under those names; the provider's installing holds; at the first parent link, the - - -no-placement remote requests waiting here; a refused hold re-runs its start and sends loading, then bound or failed - +a name becomes usable, or an install ends, on a host: a publication (load's setups, a reload's commit, an install's commit) serves a type, a registry, a version; a registry composes or maps a value at a version; an install ends without a commit (rejected) +the unbound requests indexed under those names; the provider's installing holds; a refused hold re-runs its start and sends loading, then bound or failed

    No new message kind for wakes: loading, bound, failed and retry exist. The wire's failed carries the new cause diff --git a/packages/chord/docs/next/dependencies/model.md b/packages/chord/docs/next/dependencies/model.md index 172f2a688..2b838eb79 100644 --- a/packages/chord/docs/next/dependencies/model.md +++ b/packages/chord/docs/next/dependencies/model.md @@ -63,12 +63,9 @@ A needs W's B, B needs S's C; W's hold for C waits for S's whole `load`, which w holding host closes), never because its provider is missing, went away or does not fit. - **Placement by address** (round 1): a `local` type goes here; a `remote` type with a placement goes to the address its placement (computed from the consumer's own contract) gives: here if that is this host's address, whether served - here or not, else over the route for that kind or the parent (design §3). A `remote` type **without placement** keeps - today's rule (round 2: Fable, Opus, Astra; no feature removed): here if served here, otherwise over the parent. A - request of that kind made before the host has a parent link waits here `not_served` and is **re-placed when the - host's first parent link connects** (round 3: Opus N5, Sol 3: a tab loads before it connects); on a host that never - connects (the root), it stays here. Opus would rather require a placement on every `remote()`; that removes a - supported declaration, so the draft keeps the fallback. A hold arriving for an address that is not this host's is relayed or + here or not, else over the route for that kind or the parent (design §3). Every `remote` type declares + its placement (owner, 2026-10-29: the optional placement of slice 3a is removed, and with it the fallback "here if + served, otherwise the parent" and its re-placement at the first parent link). A hold arriving for an address that is not this host's is relayed or refused (`invalid/placement`) as today; a relay never captures a hold because it serves the same name. - **Then the selected provider is checked** (served, version or accepted version and fingerprint, `authorize`, the provider's own placement check). Version checks never look at a same-name registration elsewhere. @@ -178,8 +175,7 @@ through, or recovers by its own backoff. | Event | Re-evaluates | | --- | --- | | **a target changes state**: a dependency's object goes live, loading, fails, gets a new cause or ends; a hold's `bound`, `loading`, `failed` arrive, also after a re-hold on a new link | the requests bound to it, and their dependents (§3.4) | -| **a name becomes usable, an install ends, or the first parent link connects, on a host**: a publication (`load`'s setups, a reload's commit, an install's commit) serves a type, a registry, a version; a registry composes or maps a value at a version; an install ends without a commit (rejected) | the unbound requests indexed under those names; the provider's `installing` holds; at the first parent link, the -no-placement remote requests waiting here; a refused hold re-runs its start and sends `loading`, then `bound` or `failed` | +| **a name becomes usable, or an install ends, on a host**: a publication (`load`'s setups, a reload's commit, an install's commit) serves a type, a registry, a version; a registry composes or maps a value at a version; an install ends without a commit (rejected) | the unbound requests indexed under those names; the provider's `installing` holds; a refused hold re-runs its start and sends `loading`, then `bound` or `failed` | No new message kind for wakes: `loading`, `bound`, `failed` and `retry` exist. The wire's `failed` carries the new cause kind `waiting`; `loading` and a `waiting` `failed` carry the provider's attempt id for the probe (§7). From 599127ea3629d3704b389db1b93e57f4eb2ee45c Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 07:24:20 +0000 Subject: [PATCH 012/175] docs(chord): decision: hostAddress instead of placement --- packages/chord/docs/next/decisions.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index d4ba29fe0..b7b7a89ca 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -1771,6 +1771,12 @@ validation and copies the v2 kernel added around Delta. setups a registry without a value cannot get one before a reload, so a reader then fails at once `transient` (`unavailable/undeclared`) and is retried when the registry gets its first value (R18), as `self.get` of a type not served here fails at once (refined 2026-10-27 after the second spec review: no `readyTimeout` wait at start); once live, a later failure of the registry changes nothing (last good value). +- **`hostAddress` instead of `placement` (owner, 2026-10-29: "yes, rename"):** the field of a `remote` contract that + computes where its object lives is `hostAddress: (params) => Address` (it returns the address of the host that serves + the object, the vocabulary of `createHost({ address })`, routes and holds); supersedes the name of "`placement`, not + `host`" (2026-10-21). The error reason `invalid/placement` is `invalid/host_address`; the internal module that decides + where a request goes (`host/placement.ts`, `Placement`) is `host/destination.ts`, `Destination`. Earlier entries' and + slice specs' "placement" reads "host address"; they keep their wording as history. - **Every remote type declares its placement (owner, 2026-10-29: "yes, let's remove it"):** `placement` is required in `remote()` (a type error without it, and a declaration error at runtime); a remote type without placement, a leftover of slice 3a (remote objects over one link, before host kinds in 4a), is gone, and with it the fallback "here if served From 943f30c90953b2e0d5eed4407db22c06ad5224d9 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 07:25:19 +0000 Subject: [PATCH 013/175] docs(chord): design: hostAddress instead of placement --- packages/chord/docs/next/design.md | 35 +++++++++++++++--------------- packages/chord/docs/next/slices.md | 2 +- 2 files changed, 19 insertions(+), 18 deletions(-) diff --git a/packages/chord/docs/next/design.md b/packages/chord/docs/next/design.md index 2772ee9e8..754c96cc2 100644 --- a/packages/chord/docs/next/design.md +++ b/packages/chord/docs/next/design.md @@ -52,7 +52,7 @@ versions and adapters · 28 building: slices · 29 telemetry · 30 cost model · | link | a connection between two hosts | | host kind / address | typed description of addressable hosts / one host of a kind (`Session({ id })`) | | facet | a plugin's declarations for one host kind: the types it serves, its contributions; no state or behaviour of its own | -| object type | a contract: name, params (which instance), host placement, members, version | +| object type | a contract: name, params (which instance), host address, members, version | | object | one instance per (type, params, principal key); `local`: per (type, params) | | principal | who calls; stamped by links, typed by the app | | reference | what `get` returns; keeps its object alive; live and hydrated when awaited | @@ -99,7 +99,7 @@ mode sandbox (a worker thread inside a worker), remote environment daemons (thei the TUI sandbox behind its launcher (the launcher only passes frames). A Durable Object deployment is a mesh routed by the edge (section 24 and the research note). -### Host kinds, addresses and placement +### Host kinds and addresses ```ts export const Server = hostKind("server"); @@ -108,37 +108,38 @@ export const Session = hostKind("session", { id: str() }); // params like a export const Notes = remote("notes", { version: 1, params: { session: str() }, // flat record of required scalars - placement: ({ session }) => Session({ id: session }), // a typed function of the params + hostAddress: ({ session }) => Session({ id: session }), // a typed function of the params variants: { … }, }); -export const Catalog = remote("catalog", { placement: () => Server(), … }); +export const Catalog = remote("catalog", { hostAddress: () => Server(), … }); ``` - `hostKind(name, params?)` declares a kind of addressable host; its params follow the params policy of contracts (section 5). `Session({ id: "S1" })` is an **address**: `{ kind, params, key }`, the key built like an identity key (`session|S1`); two addresses are equal when their keys are. No host-name strings appear in app code. -- **`placement`** (owner, 2026-10-21) is a typed function of the params returning an address, required on every - `remote` contract (owner, 2026-10-29); `local` contracts have none. It is not part of the contract fingerprint; a changed rule surfaces as a refusal (below). +- **`hostAddress`** (owner, 2026-10-21 as `placement`, renamed 2026-10-29) is a typed function of the params returning + the address of the host that serves the object, required on every `remote` contract (owner, 2026-10-29); `local` + contracts have none. It is not part of the contract fingerprint; a changed rule surfaces as a refusal (below). - **`params`** (owner, 2026-10-21; was `input`) select the instance among objects of its type and are part of its identity (section 5). - **The caller computes the address** and sends it in the `hold`; a host that relays routes by that address alone and never needs the contract or parses params. **The provider computes it again** from the params and refuses an object - that is not its own: broken `invalid/placement` (misrouted holds are never forwarded). During a rollout the refusal + that is not its own: broken `invalid/host_address` (misrouted holds are never forwarded). During a rollout the refusal is handled like a mismatch (provider on another revision: park; same revision: fail; slice 7). - **Where a reference made on a host, or a hold arriving on one of its links, goes** (first match; slice 4a): | Case | Goes | | --- | --- | - | the type is served here and its placement (computed here from the params) is this host's address, or the host has no address | here | + | the type is served here and its host address (computed here from the params) is this host's address, or the host has no address | here | | a reference made here (`host.get`/`hold`, `self.get`, `ctx.get`) to a served type placed on another host | on, as if not served here (next rows): worker S1 reaches `Notes { session: "S2" }` through the parent | - | an arriving hold for a served type whose params place it on another host | broken `invalid/placement`; the address the hold names only routes it | + | an arriving hold for a served type whose params place it on another host | broken `invalid/host_address`; the address the hold names only routes it | | the hold names an address of a kind this host has a route for | over that route's link (the address's params checked against the kind, then `authorize`) | | otherwise (a remote type) | over the parent link; a hold that arrived *on* the parent link is never sent back to it: `unavailable` | | no parent link: the host never had one | `unavailable` | | no parent link: it is down (a drop) | waits on the parent path, `loading`, until the next link (design audit, 2026-10-27) | - A `placement` or `authorize` that throws fails the hold broken with that error; an address whose params fail the - kind's check is broken `invalid/placement`. + A `hostAddress` or `authorize` that throws fails the hold broken with that error; an address whose params fail the + kind's check is broken `invalid/host_address`. ### The low-level host API @@ -343,7 +344,7 @@ host.route(Session, { ### Relays (slice 4a; decision "Relay flow control per origin", owner, 2026-10-24) -A host that receives a hold it cannot serve (the placement table above: a route or the parent) **relays** it: it holds +A host that receives a hold it cannot serve (the table above: a route or the parent) **relays** it: it holds it again on that link (upstream) with the arriving hold's principal and incarnation, and forwards between the two without parsing payloads. The provider keeps the window, the pending queues and the calls per origin; the relay never drops, folds or resets. @@ -474,7 +475,7 @@ drops, folds or resets. ## 5. Object types ```ts -export const Notes = remote("notes", { version, placement, params, variants }); +export const Notes = remote("notes", { version, hostAddress, params, variants }); export const NoteStore = local("notes.store", { version, params, members }); ``` @@ -520,7 +521,7 @@ host.serve(Notes, ({ session }, principal, self) => fingerprint, never a description. The provider binds when both equal its current ones, or (slice 7) when the version is one it accepts and the fingerprint equals the one computed from its own history (section 27). Every contract change, additive ones included, bumps `version`. Anything else is `contract_mismatch` (or `updating` during - a rollout, section 21). The fingerprint covers params, variants and members, not placement. A consumer whose contract + a rollout, section 21). The fingerprint covers params, variants and members, not the host address. A consumer whose contract does not fit gets a mismatch on its own binding; the provider object and its other bindings are not affected. - **Variants:** a member name has the same kind (state or method) in every variant that has it, and a method the same signature (arguments and result) in every variant; state schemas may differ per variant (redaction). Enforced at @@ -547,7 +548,7 @@ host.serve(Notes, ({ session }, principal, self) => binding is held only by the served type's registration and by calls in flight (GC-tested; no global strong cache). - **Errors** (L1, slice 3a): `ChordError` with a code and, for some codes, a reason, a closed union that is part of the link protocol (adding one bumps the protocol version; a receiver rejects unknown reasons; a test pins the list): - `invalid` (`args`, `params`, `member`, `placement`, `reentrant`, `read_after_bulk`, `cycle`), `contract_mismatch` + `invalid` (`args`, `params`, `member`, `host_address`, `reentrant`, `read_after_bulk`, `cycle`), `contract_mismatch` (`updating`), `forbidden` (`revoked`), `gone`, `not_ready`, `released` (`ended`, `closed`, `settled`, `version`, `discarded`; none for a holder's own release), `too_large` (`part`), `unavailable` (`locked`, `park_timeout`, `busy`, `storage_failed`, `undeclared`: a registry no plugin serves here, slice 8), `aborted`. A failed check names TypeBox's instance path ("unknown field", "missing", else TypeBox's message; keys containing `/` or `~` make the path ambiguous, documented). Plugins throw `PluginError(code, message, @@ -751,7 +752,7 @@ location are dropped. | object `failed` | reject at once with the cause | `failed` with cause and `retryAt` | - A failed reference stays alive: while held, Chord retries per the backoff; `ref.retry()` tries at once (a Retry - button; over a link the `retry { hold }` message, which also re-runs a refused hold's start: authorize, placement, + button; over a link the `retry { hold }` message, which also re-runs a refused hold's start: authorize, the host address, binding; design audit 2026-10-27); success goes `loading`, then `live`. A provider tells a pending hold's consumer each failure (`failed`, with the rest of its backoff window) and the next attempt (`loading`). - **Parked calls** (slice 3b): calls made while the reference is loading are queued in order, each waiting at most the @@ -2229,7 +2230,7 @@ registry keys with a `conflict` flag; (`native/` directories, `defineNative`, lints); weakening type checking; "magic" input-field matching for delete; a factory trial; a migrate dry run; principal field patterns and re-stamping for revocation; `storeKey`; a consumer contract check during staging; reload votes; per-plugin rollback within a host; `defaultProjectTrust: "always"`; loader variants A and B; a topology layer above the host API; a stream type in -Chord; callbacks as method arguments; `opaque` members; numeric middleware priorities; placement as data naming an +Chord; callbacks as method arguments; `opaque` members; numeric middleware priorities; host addresses as data naming an input field; host-name strings in app code; host values (start data and platform bindings are core objects); consumer-side linger; registry composition as a staging check; a JSONL document storage; `tx.get` for documents; validating state and documents on commits or reads (migration results are validated once, R8); copying values on same-host calls; op-rewriting projections and a view algebra; keeping natives across reloads by name; Chord ending the process on a storage failure; an `optional` diff --git a/packages/chord/docs/next/slices.md b/packages/chord/docs/next/slices.md index cf346eb9e..2ddb88f60 100644 --- a/packages/chord/docs/next/slices.md +++ b/packages/chord/docs/next/slices.md @@ -44,7 +44,7 @@ reload has to reload real registries). | 2 | **Failures and natives** | `self.open`, activation failures and recovery, backoff, the drain, timeouts | | 3a | **Remote objects over one link** | remote contracts, principals, variants, a memory link, the message format, holds, calls, one message per commit (no relay, no drops) | | 3b | **Surviving drops** | re-holds, parking, `ref.state`, `host.hold`, `ref.retry()`, `linger` | -| 4a | **Links and relays** | host kinds, placement, routes, two-sided links, relays and principals, IPC and WebSocket, flow control, heartbeat | +| 4a | **Links and relays** | host kinds, host addresses (`placement` then), routes, two-sided links, relays and principals, IPC and WebSocket, flow control, heartbeat | | 4b | **Authority and housekeeping** | revocation, cycles across hosts, `host.references`, `host.activity`, size limits, compression | | 5 | **Routines and jobs** | `self.routines`, `self.job` | | 6a | **The store** | the store (backends, tables, queries, writes, commit queue, outbox, documents engine, migrations, export), persisted documents, `self.table`, keyed watches and retirement | From d0687bce8582a1cc7077585b71e12b24913d9aa9 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 07:29:25 +0000 Subject: [PATCH 014/175] docs(chord): only hosts with an address serve remote types; holds always carry their address --- packages/chord/docs/next/decisions.md | 4 ++++ packages/chord/docs/next/design.md | 9 +++++---- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index b7b7a89ca..e87d429d7 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -1771,6 +1771,10 @@ validation and copies the v2 kernel added around Delta. setups a registry without a value cannot get one before a reload, so a reader then fails at once `transient` (`unavailable/undeclared`) and is retried when the registry gets its first value (R18), as `self.get` of a type not served here fails at once (refined 2026-10-27 after the second spec review: no `readyTimeout` wait at start); once live, a later failure of the registry changes nothing (last good value). +- **Only hosts with an address serve remote types (owner, 2026-10-29: "b"):** a host serving a `remote` type needs an + address and a principal; `load` refuses a facet that serves one on a host without an address (until now a host with + a principal but no address served remote types and placed every one of them "here"). So "here" is decided by the + address alone, as the host address of the type. Holds carry their `address` always (required on the wire). - **`hostAddress` instead of `placement` (owner, 2026-10-29: "yes, rename"):** the field of a `remote` contract that computes where its object lives is `hostAddress: (params) => Address` (it returns the address of the host that serves the object, the vocabulary of `createHost({ address })`, routes and holds); supersedes the name of "`placement`, not diff --git a/packages/chord/docs/next/design.md b/packages/chord/docs/next/design.md index 754c96cc2..66a76765f 100644 --- a/packages/chord/docs/next/design.md +++ b/packages/chord/docs/next/design.md @@ -88,8 +88,9 @@ worker worker worker addressable by a typed address, started on demand | server | yes, one | its own declaration | runs | | worker | yes, many | stamped by the parent | started on demand by the app | -Clients serve only `local` types (they are not addressable): a host without a principal refuses to serve a remote type -(`load` throws; a remote object is owned by a principal of the host serving it; owner, 2026-10-24). No client providers for now (owner, +Clients serve only `local` types (they are not addressable): a host without a principal or without an address refuses +to serve a remote type (`load` throws; a remote object is owned by a principal of the host serving it, owner, +2026-10-24; and placed by its host address, owner, 2026-10-29). No client providers for now (owner, 2026-10-28): client-side capabilities (a camera, WebMCP tools) go through an owner's requests that clients answer (section 20, the Questions service). A client's local calls have no principal; what a client calls on the server is stamped with the client's authenticated principal. @@ -130,7 +131,7 @@ export const Catalog = remote("catalog", { hostAddress: () => Server(), … }); | Case | Goes | | --- | --- | - | the type is served here and its host address (computed here from the params) is this host's address, or the host has no address | here | + | the type is served here and its host address (computed here from the params) is this host's address | here | | a reference made here (`host.get`/`hold`, `self.get`, `ctx.get`) to a served type placed on another host | on, as if not served here (next rows): worker S1 reaches `Notes { session: "S2" }` through the parent | | an arriving hold for a served type whose params place it on another host | broken `invalid/host_address`; the address the hold names only routes it | | the hold names an address of a kind this host has a route for | over that route's link (the address's params checked against the kind, then `authorize`) | @@ -1797,7 +1798,7 @@ validation, no clone of snapshots or ops. | Kind | Direction | Fields | | --- | --- | --- | | `hello` | both, first | `protocol`, `incarnation`, `calls` (the call window N) | - | `hold` | to provider | `origin?`, `hold`, `type`, `version`, `fingerprint`, `params` (part), `address?`, `principal?`, `incarnation?`, `chain?` (activation chain, Q5), `limit?` (decoded size limit, 4b) | + | `hold` | to provider | `origin?`, `hold`, `type`, `version`, `fingerprint`, `params` (part), `address`, `principal?`, `incarnation?`, `chain?` (activation chain, Q5), `limit?` (decoded size limit, 4b) | | `revoke` | parent to child (route link) | the revocation value (4b) | | `release`, `retry` | to provider | `origin?`, `hold` | | `call` | to provider | `origin?`, `call`, `hold`, `method` (index), `args?` (part; absent: no arguments), `segments?` | From 5ec24188207cfd7c2d19251db1803383caa1e160 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 07:37:37 +0000 Subject: [PATCH 015/175] feat(chord): hostAddress instead of placement, required on remote types; only hosts with an address serve remote types The contract field placement is hostAddress, required on every remote() (type and declaration error); the fallback for remote types without one is gone. invalid/placement is invalid/host_address. host/placement.ts (Placement) is host/destination.ts (Destination); here is decided by address alone, and a host without an address refuses to serve a remote type. A hold always carries its address on the wire (the binary codec's ADDRESS flag is gone, the text codec requires it). Tests and examples declare their host addresses. --- packages/chord/src/next/contracts/README.md | 10 +- packages/chord/src/next/contracts/contract.ts | 8 +- packages/chord/src/next/contracts/declare.ts | 20 ++-- .../chord/src/next/contracts/host-kind.ts | 2 +- packages/chord/src/next/data/README.md | 4 +- packages/chord/src/next/data/errors.ts | 2 +- packages/chord/src/next/data/params.ts | 6 +- packages/chord/src/next/data/validator.ts | 4 +- packages/chord/src/next/host/README.md | 23 ++-- .../host/{placement.ts => destination.ts} | 63 +++++----- packages/chord/src/next/host/host.ts | 42 ++++--- packages/chord/src/next/host/route.ts | 2 +- packages/chord/src/next/link/README.md | 6 +- packages/chord/src/next/link/binary.ts | 25 ++-- packages/chord/src/next/link/message.ts | 4 +- packages/chord/src/next/link/text.ts | 24 ++-- packages/chord/src/next/remote/README.md | 4 +- packages/chord/src/next/remote/hold.ts | 9 +- .../chord/src/next/remote/remote-object.ts | 8 +- packages/chord/test/next/bench/compression.ts | 12 +- packages/chord/test/next/bench/links.ts | 7 +- packages/chord/test/next/bench/relay.ts | 2 +- .../chord/test/next/contracts/accepts.test.ts | 4 + .../test/next/contracts/contract.test.ts | 3 +- .../chord/test/next/contracts/declare.test.ts | 109 ++++++++++++++---- .../chord/test/next/contracts/evolve.test.ts | 11 +- .../test/next/contracts/fingerprint.test.ts | 22 +++- .../test/next/contracts/host-kind.test.ts | 28 +++-- .../chord/test/next/contracts/params.test.ts | 29 ++++- packages/chord/test/next/data/args.test.ts | 3 +- packages/chord/test/next/data/errors.test.ts | 2 +- packages/chord/test/next/data/failure.test.ts | 2 +- packages/chord/test/next/data/params.test.ts | 3 +- .../chord/test/next/examples/08-remote.ts | 14 ++- packages/chord/test/next/examples/09-calls.ts | 12 +- packages/chord/test/next/examples/10-drops.ts | 12 +- .../chord/test/next/examples/12-routes.ts | 7 +- packages/chord/test/next/examples/13-relay.ts | 2 +- .../test/next/examples/14-flow-control.ts | 2 +- .../chord/test/next/examples/15-revocation.ts | 2 +- .../next/examples/16-presence-and-activity.ts | 8 +- .../examples/17-limits-and-compression.ts | 14 ++- packages/chord/test/next/examples/19-jobs.ts | 12 +- .../chord/test/next/host/activity.test.ts | 9 +- packages/chord/test/next/host/attempt.test.ts | 19 ++- packages/chord/test/next/host/ctx.test.ts | 19 ++- packages/chord/test/next/host/host.test.ts | 63 +++++++--- .../test/next/host/member-source.test.ts | 11 +- packages/chord/test/next/host/object.test.ts | 26 ++++- .../chord/test/next/host/presence.test.ts | 54 +++++++-- .../chord/test/next/host/reference.test.ts | 6 +- .../chord/test/next/host/revocation.test.ts | 19 ++- packages/chord/test/next/host/route.test.ts | 16 ++- .../chord/test/next/host/routines.test.ts | 13 ++- packages/chord/test/next/jobs/job.test.ts | 13 ++- packages/chord/test/next/link/channel.test.ts | 10 +- packages/chord/test/next/link/message.test.ts | 72 ++++++++++-- .../chord/test/next/link/websocket.test.ts | 4 +- .../chord/test/next/remote/consumer.test.ts | 4 + packages/chord/test/next/remote/hold.test.ts | 37 ++++-- .../chord/test/next/remote/host-link.test.ts | 8 +- .../chord/test/next/remote/provider.test.ts | 63 ++++++++-- packages/chord/test/next/remote/relay.test.ts | 7 +- .../test/next/remote/remote-object.test.ts | 26 ++++- packages/chord/test/next/support.ts | 20 +++- 65 files changed, 801 insertions(+), 306 deletions(-) rename packages/chord/src/next/host/{placement.ts => destination.ts} (56%) diff --git a/packages/chord/src/next/contracts/README.md b/packages/chord/src/next/contracts/README.md index a6e2a06ba..bd79e9bc0 100644 --- a/packages/chord/src/next/contracts/README.md +++ b/packages/chord/src/next/contracts/README.md @@ -12,7 +12,7 @@ const Note = obj({ text: str() }); export const NotesV1 = remote("notes", { version: 1, params: { session: str() }, - placement: ({ session }) => Session({ id: session }), // where an object lives; not in the fingerprint + hostAddress: ({ session }) => Session({ id: session }), // where an object lives; not in the fingerprint variants: { editor: { state: { notes: arr(Note) }, methods: { add: method(Note, str()), clear: method() } }, reader: { state: { notes: arr(Note) } }, @@ -22,17 +22,17 @@ export const Notes = evolve(NotesV1, (e) => e.args("add", obj({ text: str(), tag type NotesProvider = Implementation; // { variant: "editor"; members: { notes, add(args, context), clear(context) } } | { variant: "reader"; ... } ``` -- `remote(name, { version, params, placement?, variants })`: per-principal variants, each with `state` (schemas) and `methods` +- `remote(name, { version, params, hostAddress, variants })`: per-principal variants, each with `state` (schemas) and `methods` (`method(args?, result?)`; an omitted side is `null`, and a method whose arguments are `null` omits them: `clear(context)`). `local(name, { version, params, members })`: one flat map, where a schema is state and `method(...)` a method. -- `placement(params) => Address` (remote only, optional): the address of the host that serves the object for these +- `hostAddress(params) => Address` (remote only, required; a declaration error without it): the address of the host that serves the object for these params (`host/`'s routing table). The caller computes it and sends it with its hold, so a relay routes by the address alone. It is not part of the normal form or the fingerprint and `evolve` carries it. - `hostKind(name, params?)`: a kind of addressable host (`Server = hostKind("server")`, `Session = hostKind("session", { id: str() })`); `params` follow the params policy below. Calling it makes an `Address { kind, params, key }`, its key built like an identity key (`session|S1`); two addresses are equal when their keys are. The params are checked - where the address is routed or served (`invalid/placement`), not when it is made. No host-name strings in app code. + where the address is routed or served (`invalid/host_address`), not when it is made. No host-name strings in app code. - Schema helpers: `str`, `num`, `int`, `bool`, `nul`, `lit`, `oneOf`, `union`, `obj`, `arr`, `map`, `opt`, `bytes`, `unknown`, and `Static`. They return ordinary TypeBox schemas. - `evolve(previous, (e) => e...)`: the next version from structured edits (`state`, `addState`, `removeState`, `args`, @@ -65,7 +65,7 @@ Rules: - `checkAccepts(contract, accepts)`: validates an accepts table without calling a mapping; returns the accepted versions (version and fingerprint). -- `contractData(contract)`: the Chord-owned data behind a handle (its `placement`, its versions with their +- `contractData(contract)`: the Chord-owned data behind a handle (its `hostAddress`, its versions with their `VariantData`, `MethodData` and schema snapshots, its `ParamsData`); a local contract's members are the variant `LOCAL`. Type-level helpers: `ShapeOf`, `CurrentVariants`, `ContractShape`, `Params` and `Method`. - `currentVariants(contract)`: the current version's variants (a local contract has the one variant `LOCAL`); diff --git a/packages/chord/src/next/contracts/contract.ts b/packages/chord/src/next/contracts/contract.ts index cf113e96f..24a4fb47c 100644 --- a/packages/chord/src/next/contracts/contract.ts +++ b/packages/chord/src/next/contracts/contract.ts @@ -97,8 +97,8 @@ export type ContractData = { readonly kind: Kind; readonly name: string; readonly params: ParamsData; - /** A remote contract's placement: the address of the host serving an object for its params. */ - readonly placement: Placement | undefined; + /** A remote contract's host address: the address of the host serving an object for its params; none for local. */ + readonly hostAddress: HostAddress | undefined; /** Every version of the history, oldest first; the last is current. */ readonly versions: readonly VersionData[]; /** The kind of every member name ever declared in the history. */ @@ -107,8 +107,8 @@ export type ContractData = { export type MemberKind = "state" | "method" | "keyed"; -/** A placement as Chord calls it: with checked params. Plugin code: what it returns is checked where it is routed. */ -export type Placement = (params: ParamsOf) => Address; +/** A `hostAddress` as Chord calls it: with checked params. Plugin code: what it returns is checked where routed. */ +export type HostAddress = (params: ParamsOf) => Address; export type VersionData = { readonly version: number; diff --git a/packages/chord/src/next/contracts/declare.ts b/packages/chord/src/next/contracts/declare.ts index 2e3856c28..8bd3f5533 100644 --- a/packages/chord/src/next/contracts/declare.ts +++ b/packages/chord/src/next/contracts/declare.ts @@ -4,11 +4,11 @@ import { type Contract, createContract, + type HostAddress, type Kind, LOCAL, type MemberKind, type MethodData, - type Placement, type VariantData, } from "./contract.ts"; import { materialize } from "./fingerprint.ts"; @@ -79,7 +79,7 @@ export function remote< readonly version: V; readonly params: P & NoInfer>; /** The address of the host serving an object for these params; never part of the fingerprint. */ - readonly placement?: (params: { [F in keyof P]: Static }) => Address; + readonly hostAddress: (params: { [F in keyof P]: Static }) => Address; readonly variants: D & NoInfer>; }, ): Contract<{ @@ -90,16 +90,14 @@ export function remote< }> { // Declaration policy: JavaScript callers reach every check below. checkName(name, "A contract name"); - checkKeys(declaration, ["version", "params", "placement", "variants"], name); + checkKeys(declaration, ["version", "params", "hostAddress", "variants"], name); const variants = entriesOf(declaration.variants, `${name}.variants`); if (variants.length === 0) throw new Error(`${name}.variants: at least one variant is required`); - const { placement } = declaration; - if (placement !== undefined && typeof placement !== "function") { - throw new Error(`${name}.placement: expected a function of the params`); - } + const { hostAddress } = declaration; + if (typeof hostAddress !== "function") throw new Error(`${name}.hostAddress: expected a function of the params`); const read = variants.map(([variant, members]) => [variant, readVariant(members, `${name}.${variant}`)] as const); - // The placement gets the checked params of this contract. - return declare("remote", name, declaration, new Map(read), placement as Placement | undefined); + // The host address gets the checked params of this contract. + return declare("remote", name, declaration, new Map(read), hostAddress as HostAddress); } /** @@ -165,7 +163,7 @@ function declare( name: string, declaration: { readonly version: number; readonly params: unknown }, variants: ReadonlyMap, - placement?: Placement, + hostAddress?: HostAddress, ): C { const { version } = declaration; if (!Number.isSafeInteger(version) || version < 1) throw new Error(`${name}.version: expected a positive integer`); @@ -173,7 +171,7 @@ function declare( checkMembers(variants, kinds, (variant, member, memberKind) => memberPath(name, kind, variant, member, memberKind)); const params = readParams(entriesOf(declaration.params, `${name}.params`), `${name}.params`); const versions = [materialize(kind, params, version, variants, [])]; - return createContract({ kind, name, params, placement, versions, kinds }); + return createContract({ kind, name, params, hostAddress, versions, kinds }); } /** diff --git a/packages/chord/src/next/contracts/host-kind.ts b/packages/chord/src/next/contracts/host-kind.ts index 3f0e61055..b71152d55 100644 --- a/packages/chord/src/next/contracts/host-kind.ts +++ b/packages/chord/src/next/contracts/host-kind.ts @@ -1,5 +1,5 @@ // Host kinds and addresses (§3): a kind of addressable host with params like a contract's, and the address of one -// host of a kind, which a remote contract's placement computes from its params. +// host of a kind, which a remote contract's `hostAddress` computes from its params. import { checkName } from "./names.ts"; import { joinKey, type Params, type ParamsData, type RequiredFields, readParams } from "./params.ts"; diff --git a/packages/chord/src/next/data/README.md b/packages/chord/src/next/data/README.md index 1b8817496..c3caba867 100644 --- a/packages/chord/src/next/data/README.md +++ b/packages/chord/src/next/data/README.md @@ -33,7 +33,7 @@ try { - `checkParams(contract, raw, context)`: one compiled closed-object check of a get's params and their key; `identityKey(contract, key, principalKey)` (no principal key for `local`); `checkAddress(kind, params, raw, - context)`: the same check of an address's params against its host kind's (`invalid/placement`), and its key. + context)`: the same check of an address's params against its host kind's (`invalid/host_address`), and its key. - `bindProvider(contract, accepts)` once per registration, then `checkArgs(provider, method, raw, boundVersion, context)` per call. It returns `{ kind: "method", args, result? }` (current-shape arguments, and the result mapping back to the caller's version) or `{ kind: "call", args, call }` (old arguments for the provider's replacement). @@ -51,7 +51,7 @@ Rules: them. Results and state are never checked. Nothing in L1 is frozen. - A method without arguments (argument schema `null`) omits the argument in references and implementations; `null` is its internal and wire form, which `checkArgs` checks. Unknown fields fail. -- A failed check is `invalid/args`, `invalid/params` or `invalid/placement` with `details.path`, a JSON Pointer from TypeBox's first error. +- A failed check is `invalid/args`, `invalid/params` or `invalid/host_address` with `details.path`, a JSON Pointer from TypeBox's first error. TypeBox does not escape `~` and `/`, so keys containing them make the path ambiguous. - `checkArgs` refuses an unknown method with `invalid/member` and an unaccepted version or a refused method with `contract_mismatch`. It never invokes a `call` replacement. diff --git a/packages/chord/src/next/data/errors.ts b/packages/chord/src/next/data/errors.ts index 623c05fe2..acb8a7d4e 100644 --- a/packages/chord/src/next/data/errors.ts +++ b/packages/chord/src/next/data/errors.ts @@ -8,7 +8,7 @@ export const PROTOCOL_VERSION = 1; /** Every code with the reasons it may carry. `invalid` requires a reason; every other code may omit it. */ export const CHORD_ERROR_REASONS = { - invalid: ["args", "params", "member", "placement", "reentrant", "read_after_bulk", "cycle"], + invalid: ["args", "params", "member", "host_address", "reentrant", "read_after_bulk", "cycle"], contract_mismatch: ["updating"], forbidden: ["revoked"], gone: [], diff --git a/packages/chord/src/next/data/params.ts b/packages/chord/src/next/data/params.ts index 17ce090e1..be971496f 100644 --- a/packages/chord/src/next/data/params.ts +++ b/packages/chord/src/next/data/params.ts @@ -32,11 +32,11 @@ export function identityKey( } /** - * An address from a kind's name and params as a hold names it or a placement returned it: the params checked against - * the kind's with one compiled closed-object check (`invalid/placement` on failure), the key computed. + * An address from a kind's name and params as a hold names it or a `hostAddress` returned it: the params checked + * against the kind's with one compiled closed-object check (`invalid/host_address` on failure), the key computed. */ export function checkAddress(kind: string, params: ParamsData, raw: unknown, context: Context): Address { - check(validatorFor(params, context), raw, "placement", context); + check(validatorFor(params, context), raw, "host_address", context); // The check above establishes a flat record of the kind's scalar fields. const values = raw as { readonly [field: string]: string | number | boolean }; return { kind, params: values, key: addressKey(kind, params, values) }; diff --git a/packages/chord/src/next/data/validator.ts b/packages/chord/src/next/data/validator.ts index 6c3369a6b..4a16599de 100644 --- a/packages/chord/src/next/data/validator.ts +++ b/packages/chord/src/next/data/validator.ts @@ -34,7 +34,7 @@ const validators = new WeakMap(); export function check( validator: Validator, value: unknown, - reason: "args" | "params" | "placement", + reason: "args" | "params" | "host_address", context: Context, ): void { context.telemetry.metrics?.counter("chord.data.validations").add(1); @@ -48,7 +48,7 @@ export function check( function failedCheck( schema: TSchema, value: unknown, - reason: "args" | "params" | "placement", + reason: "args" | "params" | "host_address", context: Context, ): ChordError { context.telemetry.metrics?.counter("chord.data.walks").add(1); diff --git a/packages/chord/src/next/host/README.md b/packages/chord/src/next/host/README.md index da13258bc..b5c9175fa 100644 --- a/packages/chord/src/next/host/README.md +++ b/packages/chord/src/next/host/README.md @@ -79,7 +79,7 @@ Addressable hosts and routes (slice 4a): a server routes holds of worker types t export const Server = hostKind("server"); export const Session = hostKind("session", { id: str() }); export const Notes = remote("notes", { version: 1, params: { session: str() }, - placement: ({ session }) => Session({ id: session }), variants: { … } }); + hostAddress: ({ session }) => Session({ id: session }), variants: { … } }); const server = createHost

    ({ address: Server(), principal: { kind: "server" }, onError }); server.route(Session, { // before load, one route per kind @@ -125,7 +125,8 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded: - `createHost({ address?, principal?, principalKey?, telemetry?, onError, closeTimeout?, readyTimeout?, heartbeat?, revokedBy?, compression? })`; `R` is the app's revocation type (JSON). `address` (`Session({ id: "S1" })`, see `contracts/`'s `hostKind`) makes the host addressable: it serves only objects placed there; without one it is a client. A host with a `principal` acts as it for its own requests; a host - without one (a client) has none and serves only local types (`serve` of a remote type throws at `load`). + without one (a client) has none. A host serves remote types only with both a principal and an address (`serve` of a + remote type throws at `load` otherwise). `principalKey` maps a principal to its part of an identity (L1's canonical JSON by default; the same function on every host). `closeTimeout` (30 s) bounds each native's close and the drain of an ending object's calls; `readyTimeout` (60 s) bounds every activation @@ -166,7 +167,7 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded: (values, ops, results, arguments; never params, never arguments sent up its parent link) and inflates deflated parts it owns in place. Child links refuse compressed parts (a protocol error: clients never send them, and a small part could inflate to 64 MiB). Without one (a browser) it sends uncompressed and inflates asynchronously - (`DecompressionStream`), only parts it owns once their id and placement are known and their wire size passed, + (`DecompressionStream`), only parts it owns once their id and destination are known and their wire size passed, within the smaller of its link's decoded limit and a hold's stamped one, holding later messages of the link in order; acks go out after a message was applied, so a slow inflate is backpressure like a slow consumer. - **Cycles across hosts (Q5):** every object instance gets an instance id (uuidv7) at its first attempt; a new object @@ -235,17 +236,17 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded: the object activates or retries, while there is no parent link or it is down; `failed` while the object is failed and no attempt runs; `live`. Throws at once for invalid params (`invalid/params`) and a local type not served here (`unavailable`); before `load` resolved - these are decided once it did, and the reference fails `broken` with that error. A placement or `authorize` that - throws or refuses, or a placement whose params fail its kind's check (`invalid/placement`), fails the reference + these are decided once it did, and the reference fails `broken` with that error. A `hostAddress` or `authorize` that + throws or refuses, or a host address whose params fail its kind's check (`invalid/host_address`), fails the reference `broken`. A local type never crosses a link. Where a reference made here (`host.get`/`hold`, `self.get`, `ctx.get`) - or a hold arriving on a link goes, first match (`Placement`; the address is the contract's `placement` of the params, + or a hold arriving on a link goes, first match (`Destination`; the address is the contract's `hostAddress` of the params, computed by the caller; a hold carries it to be routed, and a host serving the type computes it again): | Case | Goes | | --- | --- | - | the type is served here and its address is this host's, or the host or the type has none | here | + | the type is served here and its address is this host's | here | | a reference made here to a served type whose address is another host's | on, as if not served here (rows below): worker S1 reaches `Notes { session: "S2" }` through the parent | - | an arriving hold for a served type whose params place it on another host | broken `invalid/placement` (misrouted; never forwarded) | + | an arriving hold for a served type whose params place it on another host | broken `invalid/host_address` (misrouted; never forwarded) | | the address is of a kind this host has a route for | over that route's link (the params checked against the kind, `authorize` first; `forbidden` when it refuses) | | otherwise (a remote type) | over the parent link, open yet or not; a hold that arrived *on* the parent link is never sent back: `unavailable` | | an arriving hold and no parent link | `unavailable` | @@ -458,7 +459,7 @@ cycles, throwing factories), `06-natives.ts` (opening in order, `not_ready`, clo `08-remote.ts` (two hosts, variants per principal over a shared store, replicas), `09-calls.ts` (calls over a link, bytes, errors, `ctx.principal`, cancellation, release), `10-drops.ts` (a hold before any link, `ref.state`, parked calls, a drop and the re-hold), `11-linger-and-retry.ts` (`linger`, failed references, `ref.retry()`), -`12-routes.ts` (addresses, placement, a route to an in-process worker, a guest's `ctx.get` through it), +`12-routes.ts` (addresses, host addresses, a route to an in-process worker, a guest's `ctx.get` through it), `13-relay.ts` (tab, server and a forked worker over WebSocket and IPC, the worker's start data through its core facet, §18), `14-flow-control.ts` (a slow tab: folding at the provider, and at the worker per origin through a relay while a fast tab keeps receiving), `15-revocation.ts` (a grant revoked: its tabs' links closed, its objects on @@ -488,8 +489,8 @@ delta)` (the keep-alive counts), `cycle(chain, id, closing)` (the `invalid/cycle Inside the directory: `ChordHost` (types, objects by identity, facets, the batch, the backoff map, the endings new instances wait for, its links and the heartbeat timer, its routes and the parent path; `reference` checks a reference's route, `bind` binds it to an object here or a remote object over a path, `acquire` gets one for a -caller), `Placement` (where a reference or an arriving hold goes: here, a route's path or the parent's; whether a hold -served here is placed here), `Route` (one per routed kind: per address its `Path`, at most one running +caller), `Destination` (where a reference or an arriving hold goes: here, a route's path or the parent's; whether a +hold served here is placed here), `Route` (one per routed kind: per address its `Path`, at most one running `resolve`, the backoff per address; an address is kept while it has holders, a link or a running `resolve`), `ChordObject` (one per identity: its state across attempts, holders, retries, cycle checks with its instance id and the chain it was created for, lingering, ending and revocation, the count of its running and finishing job runs, diff --git a/packages/chord/src/next/host/placement.ts b/packages/chord/src/next/host/destination.ts similarity index 56% rename from packages/chord/src/next/host/placement.ts rename to packages/chord/src/next/host/destination.ts index e6011168f..d92fbb0f1 100644 --- a/packages/chord/src/next/host/placement.ts +++ b/packages/chord/src/next/host/destination.ts @@ -1,6 +1,6 @@ -// Placement (§3, section 1's table): where a reference made on a host, or a hold arriving on one of its links, goes: -// here, over the route of its placement's address, or over the parent link; and the check that an object served here -// is placed here. +// Destination (§3, section 1's table): where a reference made on a host, or a hold arriving on one of its links, goes: +// here, over the route of its host address, or over the parent link; and the check that an object served here lives +// here. import type { Context } from "../../context/index.ts"; import type { Address, Contract, ParamsOf } from "../contracts/index.ts"; @@ -12,13 +12,13 @@ import type { HoldRequest, HostLink, Path } from "../remote/internal.ts"; import type { Served } from "./object.ts"; import type { Route } from "./route.ts"; -/** Where something goes: here, or over a path (with the address it was placed at, if its type has placement). */ +/** Where something is placed: here, or over a path (with the host address it was computed or named with). */ export type Placed = | { readonly kind: "here" } - | { readonly kind: "path"; readonly path: Path; readonly address: WireAddress | undefined }; + | { readonly kind: "path"; readonly path: Path; readonly address: WireAddress }; -/** A host's placement: its address, its served types, its routes and its parent path, as the host keeps them. */ -export class Placement { +/** A host's destinations: its address, its served types, its routes and its parent path, as the host keeps them. */ +export class Destination { readonly #address: Address | undefined; readonly #types: ReadonlyMap; readonly #routes: ReadonlyMap; @@ -37,15 +37,16 @@ export class Placement { } /** - * Where a reference to `Type` with `params` made as `principal` goes: here when its type is served here and placed - * here (or this host has no address, or the type no placement); otherwise over the route of its placement's kind, - * or over the parent link (placement decides where an object lives: a worker reaches another worker's object of a - * type it serves too). Throws `invalid/placement` for a placement whose params do not fit its kind, `forbidden` when - * the route refuses, and what a placement or `authorize` throws. + * Where a reference to `Type` with `params` made as `principal` goes: here for a local type (served here: the host + * refused it otherwise) or a remote type served here whose host address is this host's; otherwise over the route of + * its host address's kind, or over the parent link (the host address decides where an object lives: a worker + * reaches another worker's object of a type it serves too). Throws `invalid/host_address` for a host address whose + * params do not fit its kind, `forbidden` when the route refuses, and what a `hostAddress` or `authorize` throws. */ reference(Type: Contract, params: ParamsOf, principal: Principal | undefined, context: Context): Placed { - const address = placementOf(Type, params); - if (this.#types.has(Type.name) && this.#placedHere(address)) return HERE; + if (Type.kind === "local") return HERE; + const address = hostAddressOf(Type, params); + if (this.#types.has(Type.name) && this.#isHere(address)) return HERE; return { kind: "path", path: this.#pathTo(address, principal, context) ?? this.#parent, address }; } @@ -66,40 +67,40 @@ export class Placement { } /** - * Throws `invalid/placement` for a hold of `Type` with `params` (checked) served here but placed on another host - * (misrouted; never forwarded). The placement is computed here from the params: the address a hold names only - * routes it. + * Throws `invalid/host_address` for a hold of `Type` with `params` (checked) served here but whose host address is + * another host (misrouted; never forwarded). The host address is computed here from the params: the address a hold + * names only routes it. */ assertHere(Type: Contract, params: ParamsOf): void { - const address = placementOf(Type, params); - if (this.#placedHere(address)) return; - throw invalid("placement", `${Type.name}: placed on ${address!.kind}, this host is ${this.#address!.key}`); + const address = hostAddressOf(Type, params); + if (this.#isHere(address)) return; + // A host serving a remote type has an address (`serve` refuses one without). + throw invalid("host_address", `${Type.name}: placed on ${address.kind}, this host is ${this.#address!.key}`); } /** * The path to `address` when this host routes its kind (its params checked, `authorize` run for `principal`); - * `undefined` without an address or a route for its kind. + * `undefined` without a route for its kind. */ - #pathTo(address: WireAddress | undefined, principal: Principal | undefined, context: Context): Path | undefined { - if (address === undefined) return undefined; + #pathTo(address: WireAddress, principal: Principal | undefined, context: Context): Path | undefined { return this.#routes.get(address.kind)?.path(address.params, principal, context); } - /** Whether an object placed at `address` lives here: this host has no address, the type no placement, or equal. */ - #placedHere(address: WireAddress | undefined): boolean { + /** Whether an object whose host address is `address` lives here: this host has that address. */ + #isHere(address: WireAddress): boolean { const own = this.#address; - return own === undefined || address === undefined || sameAddress(own, address); + return own !== undefined && sameAddress(own, address); } } const HERE: Placed = { kind: "here" }; -/** The address `Type`'s placement computes for `params`; `undefined` for a type without placement. */ -function placementOf(Type: Contract, params: ParamsOf): WireAddress | undefined { - const placement = contractData(Type).placement; - if (placement === undefined) return undefined; +/** The address a remote `Type`'s `hostAddress` computes for `params`. */ +function hostAddressOf(Type: Contract, params: ParamsOf): WireAddress { + // Every remote contract has one (declaration policy, `remote`). + const hostAddress = contractData(Type).hostAddress!; // Plugin code: what it returns is checked where it is routed or served. - const { kind, params: values } = placement(params) as { readonly kind: string; readonly params: unknown }; + const { kind, params: values } = hostAddress(params) as { readonly kind: string; readonly params: unknown }; return { kind, params: values }; } diff --git a/packages/chord/src/next/host/host.ts b/packages/chord/src/next/host/host.ts index 5b360e4ba..5612e562a 100644 --- a/packages/chord/src/next/host/host.ts +++ b/packages/chord/src/next/host/host.ts @@ -48,11 +48,11 @@ import { Activity, type ActivityCounts } from "./activity.ts"; import { notLive } from "./attempt.ts"; import { Backoff } from "./backoff.ts"; import { type Caller, type Ctx, withLinkedAbortSignal } from "./ctx.ts"; +import { Destination, type Placed } from "./destination.ts"; import { type Facet, type FacetContribution, type ServedEntry, setUp } from "./facet.ts"; import { contributionOf } from "./members.ts"; import type { CloseFailed } from "./native.ts"; import { ChordObject, type Holder, type LocalContract, type RemoteContract, type Served } from "./object.ts"; -import { type Placed, Placement } from "./placement.ts"; import { type Presence, References } from "./presence.ts"; import { Binding, type CheckedParams, type Held, type Reference, type Target } from "./reference.ts"; import { RevocationKeys, type RevokedBy } from "./revocation.ts"; @@ -191,8 +191,8 @@ export interface Host

    }[], context: Context): Promise; /** * A reference once its object is live; waits for `load` first. It acts as the host's principal and goes where its - * type's placement puts it: here, over a route, or over the parent link (`Placement`). A broken failure rejects at - * once; through transient ones it waits until its deadline, then rejects with the last failure's error. + * type's host address puts it: here, over a route, or over the parent link (`Destination`). A broken failure + * rejects at once; through transient ones it waits until its deadline, then rejects with the last failure's error. */ get(Type: C, params: ParamsOf, context: Context, options?: GetOptions): Promise>; /** @@ -316,7 +316,7 @@ export class ChordHost

    {}); readonly #routes = new Map(); - readonly #placement: Placement; + readonly #destination: Destination; /** The downstream origins this host relays holds for, by their number on upstream links (never reused). */ readonly relays = new Map(); #nextOrigin = 1; @@ -344,7 +344,7 @@ export class ChordHost

    | undefined); this.timeouts = { close: options.closeTimeout ?? 30_000, ready: options.readyTimeout ?? 60_000 }; this.#heartbeat = { ping: options.heartbeat?.ping ?? 15_000, close: options.heartbeat?.close ?? 30_000 }; - this.#placement = new Placement(this.address, this.#types, this.#routes, this.#parent); + this.#destination = new Destination(this.address, this.#types, this.#routes, this.#parent); // Chord's own: `openStore` made it. this.store = options.store as ChordStore | undefined; this.store?.attach({ failed: (error) => this.#storageFailed(error), batch: this.batch }); @@ -576,13 +576,13 @@ export class ChordHost

    , principal: Principal | undefined): Path { // A retry has no context of its own: its admission checks count on none. - const placed = this.#placement.reference(Type, params, principal, BACKGROUND_CONTEXT); + const placed = this.#destination.reference(Type, params, principal, BACKGROUND_CONTEXT); if (placed.kind === "path") return placed.path; throw new ChordError({ code: "unavailable", message: `${Type.name} is served here now` }); } @@ -713,8 +713,8 @@ export class ChordHost

    (ADDRESS | PRINCIPAL | INCARNATION | CHAIN | LIMIT)) throw new Error("binary: unknown hold flags"); - const address = flags & ADDRESS ? { address: { kind: reader.string(), params: reader.json() } } : {}; + if (flags > (PRINCIPAL | INCARNATION | CHAIN | LIMIT)) throw new Error("binary: unknown hold flags"); const principal = flags & PRINCIPAL ? { principal: reader.json() as { kind: string } } : {}; const incarnation = flags & INCARNATION ? { incarnation: uuid(reader.raw(16)) } : {}; const chain = flags & CHAIN ? { chain: reader.list(() => uuid(reader.raw(16))) } : {}; @@ -221,7 +216,7 @@ function decodeFields(kind: (typeof KINDS)[number], reader: Reader): Message { version, fingerprint, params, - ...address, + address, ...principal, ...incarnation, ...chain, diff --git a/packages/chord/src/next/link/message.ts b/packages/chord/src/next/link/message.ts index a8c022571..84bab8629 100644 --- a/packages/chord/src/next/link/message.ts +++ b/packages/chord/src/next/link/message.ts @@ -62,8 +62,8 @@ const MESSAGE = union([ version: int({ min: 1 }), fingerprint: str({ pattern: "^[0-9a-f]{16}$" }), params: part, - // The address the caller computed (absent for a type without placement). - address: opt(address), + // The address the caller computed with the type's `hostAddress`: it routes the hold. + address, // Whom the hold acts for and the origin's incarnation; read only on a parent link. principal: opt(principal), incarnation: opt(incarnation), diff --git a/packages/chord/src/next/link/text.ts b/packages/chord/src/next/link/text.ts index c9b7b8e0d..e20c741bb 100644 --- a/packages/chord/src/next/link/text.ts +++ b/packages/chord/src/next/link/text.ts @@ -75,16 +75,10 @@ function ownTuple(message: Message): string { const { address, principal, incarnation, chain, limit } = message; const head = json(["H", message.hold, message.type, message.version, message.fingerprint]).slice(0, -1); // Optional trailing fields: an absent one in the middle is null, absent ones at the end are left out. - const optional = [ - address === undefined ? null : [address.kind, address.params], - principal ?? null, - incarnation ?? null, - chain ?? null, - limit ?? null, - ]; + const optional = [principal ?? null, incarnation ?? null, chain ?? null, limit ?? null]; while (optional.length > 0 && optional.at(-1) === null) optional.pop(); const rest = optional.map((field) => `,${json(field)}`).join(""); - return `${head},${partText(message.params)}${rest}]`; + return `${head},${partText(message.params)},${json([address.kind, address.params])}${rest}]`; } case "release": return json(["R", message.hold]); @@ -188,14 +182,10 @@ function fromFields(fields: unknown[], rest: Uint8Array): Message { arity(3); return { kind: "hello", protocol: field(0), incarnation: field(1), calls: field(2) }; case "H": { - arity(5, 6, 7, 8, 9, 10); - const [address, principal, incarnation, chain, limit] = [5, 6, 7, 8, 9].map((index) => f[index] ?? null); - let wireAddress = {}; - if (address !== null) { - const pair = list(address); - if (pair.length !== 2) throw new Error("text: an address is a pair"); - wireAddress = { address: { kind: pair[0], params: pair[1] } }; - } + arity(6, 7, 8, 9, 10); + const address = list(f[5]); + if (address.length !== 2) throw new Error("text: an address is a pair"); + const [principal, incarnation, chain, limit] = [6, 7, 8, 9].map((index) => f[index] ?? null); return { kind: "hold", hold: field(0), @@ -203,7 +193,7 @@ function fromFields(fields: unknown[], rest: Uint8Array): Message { version: field(2), fingerprint: field(3), params: textPart(f[4]), - ...wireAddress, + address: { kind: address[0] as string, params: address[1] }, ...(principal === null ? {} : { principal: principal as { kind: string } }), ...(incarnation === null ? {} : { incarnation: incarnation as string }), ...(chain === null ? {} : { chain: chain as string[] }), diff --git a/packages/chord/src/next/remote/README.md b/packages/chord/src/next/remote/README.md index cd27471a5..16522f71b 100644 --- a/packages/chord/src/next/remote/README.md +++ b/packages/chord/src/next/remote/README.md @@ -83,7 +83,7 @@ within each link's limits, compression at the origin, activation chains and revo `state` with A's and B's entries: the derive sees both at once, never A new with B old. - `Hold` (one per `hold` served here, a `Holder` of its object): takes its object once `load` resolved; the hold's `params` part is parsed then (malformed: a protocol error). A refused type, a mismatch, invalid params, a wrong - placement or a snapshot that throws is a broken `failed` that ends it (the provider forgets the id). While pending + host address or a snapshot that throws is a broken `failed` that ends it (the provider forgets the id). While pending it follows its object's `state`: each failure, transient or broken, is sent as `failed` with `retryIn`, the rest of the object's backoff window (absent when broken), the current one at once when the object is failed already; the next attempt is announced with `loading`; the hold stays, so a `retry` (from any holder) renews even a broken object @@ -146,7 +146,7 @@ within each link's limits, compression at the origin, activation chains and revo until `retry`; the message's other entries apply. Its call arguments go up a parent link uncompressed (the parent may have accepted it as a child). It keeps its binding's caller: two references with different principals are two remote objects. Its holds carry the params as a - part and the address its placement computed. A `bound` (a first binding, a re-hold; also on a live hold) + part and the address its `hostAddress` computed. A `bound` (a first binding, a re-hold; also on a live hold) replaces the replicas in the message's batch: members it had get the snapshot as one frame `[["r", value]]` and a fresh decoder, members outside a new variant vanish (reads throw `invalid/member`, watches get one `failed` call and stay open, derives over them fail; a later binding with the member brings it back as `[["r", value]]`, as on the diff --git a/packages/chord/src/next/remote/hold.ts b/packages/chord/src/next/remote/hold.ts index 59ee856c9..b018c201b 100644 --- a/packages/chord/src/next/remote/hold.ts +++ b/packages/chord/src/next/remote/hold.ts @@ -72,7 +72,7 @@ type HoldState = /** The members it follows, in the order they were subscribed. */ readonly followed: Followed[]; } - /** Refused where it arrived (placement, `authorize`, a mismatch, invalid params, a cycle): `retry` starts it again. */ + /** Refused where it arrived (host address, `authorize`, a mismatch, invalid params, a cycle): `retry` restarts it. */ | { readonly kind: "refused"; readonly request: HoldRequest } | { readonly kind: "released" }; @@ -86,8 +86,8 @@ const BINDING: HoldState = { kind: "binding" }; /** * One hold served here: a reference to its object acting as its caller (the hold's principal). While pending, each * failure of its object is sent as `failed` (with the rest of its backoff window) and the next attempt as `loading`; a - * hold that cannot be taken (a refused type, a mismatch, invalid params or placement) is a broken `failed` that holds - * nothing until a `retry` starts it again. Once live it binds when its link's window has a slot, then sends a part per batch that changed its members + * hold that cannot be taken (a refused type, a mismatch, invalid params or a wrong host address) is a broken `failed` + * that holds nothing until a `retry` starts it again. Once live it binds when its link's window has a slot, then sends a part per batch that changed its members * (or keeps their ops pending while the window is full), and `gone` when its object ends. */ export class Hold implements Holder { @@ -139,7 +139,8 @@ export class Hold implements Holder { * Admits the hold once `load` resolved: its params within the link's part limit (`too_large/part`), then relayed * (it is replaced by a relayed hold), or served here, its params opened and parsed (a malformed part is a protocol * error), its object taken and followed (`moved`). A hold that cannot go anywhere, a type that does not match, - * invalid params, a wrong placement or a cycle (Q5) is a broken `failed`; the hold is kept, refused, for a `retry`. + * invalid params, a wrong host address or a cycle (Q5) is a broken `failed`; the hold is kept, refused, for a + * `retry`. */ async start(request: HoldRequest): Promise { const { link } = this.#provider; diff --git a/packages/chord/src/next/remote/remote-object.ts b/packages/chord/src/next/remote/remote-object.ts index 1765b7d1d..05ea9cea9 100644 --- a/packages/chord/src/next/remote/remote-object.ts +++ b/packages/chord/src/next/remote/remote-object.ts @@ -86,7 +86,7 @@ export class RemoteObject implements Target { host: AnyHost, contract: Contract, { params, key }: CheckedParams, - address: WireAddress | undefined, + address: WireAddress, caller: Caller, path: Path, dependent: ChordObject | undefined, @@ -100,7 +100,7 @@ export class RemoteObject implements Target { version, fingerprint, params: jsonPart(params), - ...(address === undefined ? {} : { address }), + address, }; this.caller = caller; this.#params = params; @@ -216,7 +216,7 @@ export class RemoteObject implements Target { /** * `ref.retry()`: a failed object's provider is asked for an attempt (a hold it refused is started there again). One * that failed for a part past the link's limits is placed anew and held with a new id, now or on its path's next - * link (a placement that throws fails it broken again). + * link (a `hostAddress` that throws fails it broken again). */ retry(): void { if (this.#state.kind !== "failed") return; @@ -227,7 +227,7 @@ export class RemoteObject implements Target { this.#hold?.side.retry(this.#hold.id); } - /** Places a reference whose part was past the limits anew, on its path now (a placement that throws fails it). */ + /** Places a reference whose part was past the limits anew, on its path now (a throwing `hostAddress` fails it). */ #place(): void { let path: Path; try { diff --git a/packages/chord/test/next/bench/compression.ts b/packages/chord/test/next/bench/compression.ts index 7b74ad8f8..15e7024ce 100644 --- a/packages/chord/test/next/bench/compression.ts +++ b/packages/chord/test/next/bench/compression.ts @@ -4,13 +4,15 @@ // Run by hand from packages/chord (not in CI): // node --conditions=source --experimental-strip-types --expose-gc test/next/bench/compression.ts import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; -import { arr, int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { arr, hostKind, int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; import { defineState } from "../../../src/next/documents/index.ts"; import { createHost, defineFacet } from "../../../src/next/host/index.ts"; import { type Compression, type Link, memoryLinkPair } from "../../../src/next/link/index.ts"; import { DeflatedPart, inflateAsync } from "../../../src/next/link/internal.ts"; import { nodeCompression } from "../../../src/next-node/compression.ts"; +const Server = hostKind("server"); + const context = BACKGROUND_CONTEXT; type P = { kind: "owner" } | { kind: "server" }; @@ -23,6 +25,7 @@ const Store = local("bench.store", { version: 1, params: {}, members: { doc: obj const View = remote("bench.view", { version: 1, params: {}, + hostAddress: () => Server(), variants: { owner: { state: { doc: obj({ entries: arr(Entry) }) }, methods: { export: method(undefined, str()) } } }, }); @@ -115,7 +118,12 @@ for (const [name, json] of [ /** A server compressing with `server` and a client with `client` (none: it inflates asynchronously). */ async function linked(server: Compression | undefined, client: Compression | undefined) { - const serverHost = createHost

    ({ principal: { kind: "server" }, compression: server, onError: console.error }); + const serverHost = createHost

    ({ + address: Server(), + principal: { kind: "server" }, + compression: server, + onError: console.error, + }); await serverHost.load([{ name: "@bench/app", facet }], context); const clientHost = createHost

    ({ compression: client, onError: console.error }); await clientHost.load([], context); diff --git a/packages/chord/test/next/bench/links.ts b/packages/chord/test/next/bench/links.ts index 3bbbf6d31..39e7e7c55 100644 --- a/packages/chord/test/next/bench/links.ts +++ b/packages/chord/test/next/bench/links.ts @@ -8,13 +8,15 @@ import { GCProfiler } from "node:v8"; import { InMemoryTelemetryContext } from "@earendil-works/pi-telemetry"; import { BACKGROUND_CONTEXT, type Context, withTelemetry } from "../../../src/context/index.ts"; -import { arr, bytes, int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { arr, bytes, hostKind, int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; import { defineState } from "../../../src/next/documents/index.ts"; import { createHost, defineFacet, type Host, type RefState } from "../../../src/next/host/index.ts"; import { type Codec, type Link, memoryLinkPair } from "../../../src/next/link/index.ts"; import { binaryCodec, type Message, textCodec } from "../../../src/next/link/internal.ts"; import type { NativeWatchable } from "../../../src/watch.ts"; +const Server = hostKind("server"); + type P = { kind: "owner" } | { kind: "server" }; const Entry = obj({ id: int(), text: str(), author: str(), at: int() }); @@ -37,6 +39,7 @@ const Store = local("bench.store", { const View = remote("bench.view", { version: 1, params: { name: str() }, + hostAddress: () => Server(), variants: { owner: { state: { doc: Shape } } }, }); @@ -44,6 +47,7 @@ const View = remote("bench.view", { const Files = remote("bench.files", { version: 1, params: {}, + hostAddress: () => Server(), variants: { owner: { methods: { ping: method(obj({ n: int() }), int()), put: method(obj({ data: bytes() }), int()) } }, }, @@ -138,6 +142,7 @@ const document = (): Entry[] => Array.from({ length: 500 }, (_, id) => entry(id, async function setup(name: string, entries: Entry[], codec: Codec, metered = true) { const context = metered ? counting : BACKGROUND_CONTEXT; const server = createHost

    ({ + address: Server(), principal: { kind: "server" }, telemetry: metered ? telemetry : undefined, onError: console.error, diff --git a/packages/chord/test/next/bench/relay.ts b/packages/chord/test/next/bench/relay.ts index 4f2c382b0..c699e9282 100644 --- a/packages/chord/test/next/bench/relay.ts +++ b/packages/chord/test/next/bench/relay.ts @@ -39,7 +39,7 @@ const Store = local("bench.store", { const View = remote("bench.view", { version: 1, params: { session: str() }, - placement: ({ session }) => Session({ id: session }), + hostAddress: ({ session }) => Session({ id: session }), variants: { owner: { state: { doc: Shape }, methods: { stream: method(str()), ping: method(int(), int()) } } }, }); diff --git a/packages/chord/test/next/contracts/accepts.test.ts b/packages/chord/test/next/contracts/accepts.test.ts index 83cbfcca9..7091d9d7e 100644 --- a/packages/chord/test/next/contracts/accepts.test.ts +++ b/packages/chord/test/next/contracts/accepts.test.ts @@ -3,6 +3,7 @@ import { type AcceptsTable, arr, evolve, + hostKind, int, local, method, @@ -13,12 +14,15 @@ import { } from "../../../src/next/contracts/index.ts"; import { checkAccepts } from "../../../src/next/contracts/internal.ts"; +const Server = hostKind("server"); + const Note = obj({ text: str() }); const NoteV2 = obj({ text: str(), tag: str() }); const NotesV1 = remote("notes", { version: 1, params: { session: str() }, + hostAddress: () => Server(), variants: { editor: { state: { notes: arr(Note), count: int() }, methods: { add: method(Note, str()), clear: method() } }, reader: { state: { notes: arr(Note) }, methods: { add: method(Note, str()) } }, diff --git a/packages/chord/test/next/contracts/contract.test.ts b/packages/chord/test/next/contracts/contract.test.ts index c9a6f12ab..e7c409022 100644 --- a/packages/chord/test/next/contracts/contract.test.ts +++ b/packages/chord/test/next/contracts/contract.test.ts @@ -19,11 +19,12 @@ import { import { checkAccepts, currentVariants, LOCAL, takesNoArgs } from "../../../src/next/contracts/internal.ts"; import { bindProvider, checkParams } from "../../../src/next/data/internal.ts"; import type { Watchable } from "../../../src/next/state/index.ts"; -import { expectCollected } from "../support.ts"; +import { expectCollected, Server } from "../support.ts"; const Notes = remote("notes", { version: 1, params: { session: str(), page: int() }, + hostAddress: () => Server(), variants: { editor: { state: { notes: arr(str()) }, methods: { add: method(obj({ text: str() }), str()), clear: method() } }, reader: { state: { notes: arr(str()) } }, diff --git a/packages/chord/test/next/contracts/declare.test.ts b/packages/chord/test/next/contracts/declare.test.ts index 016fb2f90..25e4e0811 100644 --- a/packages/chord/test/next/contracts/declare.test.ts +++ b/packages/chord/test/next/contracts/declare.test.ts @@ -1,10 +1,25 @@ import Type from "typebox"; import { describe, expect, it } from "vitest"; -import { arr, evolve, int, keyed, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { + arr, + evolve, + hostKind, + int, + keyed, + local, + method, + obj, + remote, + str, +} from "../../../src/next/contracts/index.ts"; import { currentVariants, LOCAL } from "../../../src/next/contracts/internal.ts"; const Note = obj({ text: str() }); +const Server = hostKind("server"); + +const hostAddress = () => Server(); + /** Declares through an untyped door, as a JavaScript caller would. */ const untyped = { remote: remote as (name: unknown, declaration: unknown) => unknown, @@ -16,6 +31,7 @@ describe("declaration checks", () => { const Notes = remote("notes", { version: 1, params: { session: str() }, + hostAddress, variants: { editor: { state: { notes: arr(Note) }, methods: { add: method(Note, str()), clear: method() } }, reader: { state: { notes: arr(Note) } }, @@ -31,77 +47,109 @@ describe("declaration checks", () => { expect(Info).toMatchObject({ kind: "local", name: "app.session-info", version: 3 }); }); + // Each case declares a valid `hostAddress` unless it checks that one, so the check it names is the one reached. it.each([ - ["empty name", "", { version: 1, params: {}, variants: { a: {} } }, 'A contract name "": expected lowercase'], + [ + "empty name", + "", + { version: 1, params: {}, hostAddress, variants: { a: {} } }, + 'A contract name "": expected lowercase', + ], [ "name not a string", 1, - { version: 1, params: {}, variants: { a: {} } }, + { version: 1, params: {}, hostAddress, variants: { a: {} } }, "A contract name 1: expected lowercase", ], - ["an uppercase name", "Notes", { version: 1, params: {}, variants: { a: {} } }, "expected lowercase"], - ["a name with a colon", "a:b", { version: 1, params: {}, variants: { a: {} } }, "expected lowercase"], - ["version zero", "n", { version: 0, params: {}, variants: { a: {} } }, "n.version: expected a positive integer"], + [ + "an uppercase name", + "Notes", + { version: 1, params: {}, hostAddress, variants: { a: {} } }, + "expected lowercase", + ], + [ + "a name with a colon", + "a:b", + { version: 1, params: {}, hostAddress, variants: { a: {} } }, + "expected lowercase", + ], + [ + "version zero", + "n", + { version: 0, params: {}, hostAddress, variants: { a: {} } }, + "n.version: expected a positive integer", + ], [ "fractional version", "n", - { version: 1.5, params: {}, variants: { a: {} } }, + { version: 1.5, params: {}, hostAddress, variants: { a: {} } }, "n.version: expected a positive integer", ], [ "unsafe version", "n", - { version: 2 ** 53, params: {}, variants: { a: {} } }, + { version: 2 ** 53, params: {}, hostAddress, variants: { a: {} } }, "n.version: expected a positive integer", ], [ "unknown declaration key", "n", - { version: 1, params: {}, variants: { a: {} }, place: "x" }, + { version: 1, params: {}, hostAddress, variants: { a: {} }, place: "x" }, 'n: unknown key "place"', ], [ - "a placement that is no function", + "a host address that is no function", "n", - { version: 1, params: {}, variants: { a: {} }, placement: "x" }, - "n.placement: expected a function of the params", + { version: 1, params: {}, variants: { a: {} }, hostAddress: "x" }, + "n.hostAddress: expected a function of the params", ], - ["no variants", "n", { version: 1, params: {}, variants: {} }, "n.variants: at least one variant is required"], - ["missing params", "n", { version: 1, variants: { a: {} } }, "n.params: expected an object"], + [ + "no host address", + "n", + { version: 1, params: {}, variants: { a: {} } }, + "n.hostAddress: expected a function of the params", + ], + [ + "no variants", + "n", + { version: 1, params: {}, hostAddress, variants: {} }, + "n.variants: at least one variant is required", + ], + ["missing params", "n", { version: 1, hostAddress, variants: { a: {} } }, "n.params: expected an object"], [ "unknown variant key", "n", - { version: 1, params: {}, variants: { a: { members: {} } } }, + { version: 1, params: {}, hostAddress, variants: { a: { members: {} } } }, 'n.a: unknown key "members"', ], [ "a schema under methods", "n", - { version: 1, params: {}, variants: { a: { methods: { m: str() } } } }, + { version: 1, params: {}, hostAddress, variants: { a: { methods: { m: str() } } } }, "n.a.methods.m: expected method(...)", ], [ "a method under state", "n", - { version: 1, params: {}, variants: { a: { state: { s: method() } } } }, + { version: 1, params: {}, hostAddress, variants: { a: { state: { s: method() } } } }, "n.a.state.s: a method belongs in methods", ], [ "bad state schema", "n", - { version: 1, params: {}, variants: { a: { state: { s: { not: {} } } } } }, + { version: 1, params: {}, hostAddress, variants: { a: { state: { s: { not: {} } } } } }, 'n.a.state.s: unsupported keyword "not"', ], [ "open arguments", "n", - { version: 1, params: {}, variants: { a: { methods: { m: method(Type.Object({})) } } } }, + { version: 1, params: {}, hostAddress, variants: { a: { methods: { m: method(Type.Object({})) } } } }, "n.a.methods.m.args: argument objects must be closed", ], [ "bad result", "n", - { version: 1, params: {}, variants: { a: { methods: { m: method(undefined, { not: {} }) } } } }, + { version: 1, params: {}, hostAddress, variants: { a: { methods: { m: method(undefined, { not: {} }) } } } }, 'n.a.methods.m.result: unsupported keyword "not"', ], ])("remote refuses %s", (_case, name, declaration, message) => { @@ -110,7 +158,7 @@ describe("declaration checks", () => { it.each([ ["unknown key", { version: 1, params: {}, members: {}, variants: {} }, 'i: unknown key "variants"'], - ["a placement", { version: 1, params: {}, members: {}, placement: () => null }, 'i: unknown key "placement"'], + ["a host address", { version: 1, params: {}, members: {}, hostAddress }, 'i: unknown key "hostAddress"'], [ "bad member schema", { version: 1, params: {}, members: { s: { not: {} } } }, @@ -125,6 +173,13 @@ describe("declaration checks", () => { expect(() => untyped.local("i", declaration)).toThrow(message); }); + it("refuse a remote declaration without a host address at the type level", () => { + expect(() => + // @ts-expect-error: every remote type declares its host address + remote("n", { version: 1, params: {}, variants: { a: {} } }), + ).toThrow("n.hostAddress: expected a function of the params"); + }); + it("does not check state for closed objects", () => { expect(() => local("i", { version: 1, params: {}, members: { s: Type.Object({ a: str() }) } })).not.toThrow(); }); @@ -136,6 +191,7 @@ describe("D11: members across variants", () => { remote("n", { version: 1, params: {}, + hostAddress, // @ts-expect-error: `x` is a method in variant b variants: { a: { state: { x: str() } }, b: { methods: { x: method() } } }, }), @@ -147,6 +203,7 @@ describe("D11: members across variants", () => { remote("n", { version: 1, params: {}, + hostAddress, // @ts-expect-error: `x` is both state and a method variants: { a: { state: { x: str() }, methods: { x: method() } } }, }), @@ -158,6 +215,7 @@ describe("D11: members across variants", () => { remote("n", { version: 1, params: {}, + hostAddress, // @ts-expect-error: `m` takes a string in a and an integer in b variants: { a: { methods: { m: method(str()) } }, b: { methods: { m: method(int()) } } }, }), @@ -166,6 +224,7 @@ describe("D11: members across variants", () => { remote("n", { version: 1, params: {}, + hostAddress, variants: { a: { methods: { m: method(str(), str()) } }, b: { methods: { m: method(str({ max: 3 }), str()) } }, @@ -179,6 +238,7 @@ describe("D11: members across variants", () => { remote("n", { version: 1, params: {}, + hostAddress, variants: { editor: { state: { notes: arr(obj({ text: str(), author: str() })) }, methods: { m: method(str()) } }, reader: { state: { notes: arr(obj({ text: str() })) }, methods: { m: method(str()) } }, @@ -190,9 +250,9 @@ describe("D11: members across variants", () => { describe("reserved member names", () => { it.each(["then", "release", "variant", "state", "retry"])("refuses %s in remote, local and evolve", (name) => { - expect(() => remote("n", { version: 1, params: {}, variants: { v: { state: { [name]: str() } } } })).toThrow( - `n.v.state.${name}: "${name}" is reserved: a reference has it`, - ); + expect(() => + remote("n", { version: 1, params: {}, hostAddress, variants: { v: { state: { [name]: str() } } } }), + ).toThrow(`n.v.state.${name}: "${name}" is reserved: a reference has it`); expect(() => local("n", { version: 1, params: {}, members: { [name]: method() } })).toThrow( `n.members.${name}: "${name}" is reserved: a reference has it`, ); @@ -220,6 +280,7 @@ describe("keyed members (slice 6a §12)", () => { remote("app.notes", { version: 1, params: {}, + hostAddress, // @ts-expect-error keyed members are only in local contracts variants: { editor: { state: { notes: keyed(Note) } } }, }), diff --git a/packages/chord/test/next/contracts/evolve.test.ts b/packages/chord/test/next/contracts/evolve.test.ts index abc2263e3..85c1d20eb 100644 --- a/packages/chord/test/next/contracts/evolve.test.ts +++ b/packages/chord/test/next/contracts/evolve.test.ts @@ -7,6 +7,7 @@ import { arr, bool, evolve, + hostKind, type Implementation, int, local, @@ -18,11 +19,14 @@ import { str, } from "../../../src/next/contracts/index.ts"; +const Server = hostKind("server"); + const Note = obj({ text: str() }); const NotesV1 = remote("notes", { version: 1, params: { session: str() }, + hostAddress: () => Server(), variants: { editor: { state: { notes: arr(Note), count: int() }, methods: { add: method(Note, str()), clear: method() } }, reader: { state: { notes: arr(Note) }, methods: { add: method(Note, str()) } }, @@ -164,7 +168,12 @@ describe("kinds and fixed parts", () => { }); it("a new base is a full declaration with an explicit version and no older versions", () => { - const base = remote("notes", { version: 5, params: { session: str(), tenant: str() }, variants: { editor: {} } }); + const base = remote("notes", { + version: 5, + params: { session: str(), tenant: str() }, + hostAddress: () => Server(), + variants: { editor: {} }, + }); expect(contractData(base).versions.map(({ version }) => version)).toEqual([5]); expect(evolve(base, (e) => e).version).toBe(6); }); diff --git a/packages/chord/test/next/contracts/fingerprint.test.ts b/packages/chord/test/next/contracts/fingerprint.test.ts index d28e360fb..f159b7856 100644 --- a/packages/chord/test/next/contracts/fingerprint.test.ts +++ b/packages/chord/test/next/contracts/fingerprint.test.ts @@ -1,12 +1,15 @@ import Type, { type TSchema } from "typebox"; import { describe, expect, it } from "vitest"; -import { evolve, int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { evolve, hostKind, int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; import { NORMAL_FORM_VERSION } from "../../../src/next/contracts/internal.ts"; import type { Params } from "../../../src/next/contracts/params.ts"; +const Server = hostKind("server"); + /** Fingerprint of a one-variant remote contract with state `s` and params `params`. */ function fingerprintOf(state: TSchema, params: Params = {}): string { - return remote("n", { version: 1, params, variants: { a: { state: { s: state } } } }).fingerprint; + return remote("n", { version: 1, params, hostAddress: () => Server(), variants: { a: { state: { s: state } } } }) + .fingerprint; } describe("fingerprints", () => { @@ -15,6 +18,7 @@ describe("fingerprints", () => { const Notes = remote("notes", { version: 1, params: { session: str() }, + hostAddress: () => Server(), variants: { editor: { state: { notes: obj({ text: str() }) }, methods: { add: method(obj({ text: str() }), str()) } }, reader: { state: { notes: obj({ text: str() }) } }, @@ -58,7 +62,8 @@ describe("fingerprints", () => { }); it("give the same v1 from a separate declaration, and evolve keeps v1's", () => { - const declare = () => remote("n", { version: 1, params: {}, variants: { a: { state: { s: str() } } } }); + const declare = () => + remote("n", { version: 1, params: {}, hostAddress: () => Server(), variants: { a: { state: { s: str() } } } }); const v1 = declare(); const v2 = evolve(v1, (e) => e.state("a", "s", int())); expect(declare().fingerprint).toBe(v1.fingerprint); @@ -67,14 +72,19 @@ describe("fingerprints", () => { }); it("differ between local and remote", () => { - const asRemote = remote("n", { version: 1, params: {}, variants: { members: { state: { s: str() } } } }); + const asRemote = remote("n", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { members: { state: { s: str() } } }, + }); const asLocal = local("n", { version: 1, params: {}, members: { s: str() } }); expect(asLocal.fingerprint).not.toBe(asRemote.fingerprint); }); it("do not depend on the name or version", () => { - const a = remote("a", { version: 1, params: {}, variants: { v: {} } }); - const b = remote("b", { version: 7, params: {}, variants: { v: {} } }); + const a = remote("a", { version: 1, params: {}, hostAddress: () => Server(), variants: { v: {} } }); + const b = remote("b", { version: 7, params: {}, hostAddress: () => Server(), variants: { v: {} } }); expect(a.fingerprint).toBe(b.fingerprint); }); }); diff --git a/packages/chord/test/next/contracts/host-kind.test.ts b/packages/chord/test/next/contracts/host-kind.test.ts index 3414ffd97..93026d54d 100644 --- a/packages/chord/test/next/contracts/host-kind.test.ts +++ b/packages/chord/test/next/contracts/host-kind.test.ts @@ -30,7 +30,7 @@ describe("host kinds", () => { expect(hostKind("@pi/session.v2_~-").kind).toBe("@pi/session.v2_~-"); }); - it("check an address's params where it is routed, with the kind's compiled check: invalid/placement", () => { + it("check an address's params where it is routed, with the kind's compiled check: invalid/host_address", () => { const params = kindParams(Session); expect(checkAddress("session", params, { id: "S1" }, context).key).toBe("session|S1"); for (const bad of [{ id: 1 }, {}, { id: "S1", extra: 1 }, null]) { @@ -40,7 +40,7 @@ describe("host kinds", () => { } catch (error) { thrown = error; } - expect(isChordError(thrown, "invalid") && thrown.reason).toBe("placement"); + expect(isChordError(thrown, "invalid") && thrown.reason).toBe("host_address"); } }); @@ -54,18 +54,22 @@ describe("host kinds", () => { }); }); -describe("placement", () => { +describe("hostAddress", () => { const variants = { v: { state: { n: int() } } }; it("is not part of the fingerprint and is carried by evolve", () => { - const placement = ({ session }: { session: string }) => Session({ id: session }); - const placed = remote("app.placed", { version: 1, params: { session: str() }, placement, variants }); - const unplaced = remote("app.placed", { version: 1, params: { session: str() }, variants }); - expect(placed.fingerprint).toBe(unplaced.fingerprint); - expect(contractData(placed).placement).toBe(placement); - expect(contractData(unplaced).placement).toBeUndefined(); - const next = evolve(placed, (v) => v.addState("v", "m", int())); - expect(contractData(next).placement).toBe(placement); + const hostAddress = ({ session }: { session: string }) => Session({ id: session }); + const onSession = remote("app.placed", { version: 1, params: { session: str() }, hostAddress, variants }); + const onServer = remote("app.placed", { + version: 1, + params: { session: str() }, + hostAddress: () => Server(), + variants, + }); + expect(onSession.fingerprint).toBe(onServer.fingerprint); + expect(contractData(onSession).hostAddress).toBe(hostAddress); + const next = evolve(onSession, (v) => v.addState("v", "m", int())); + expect(contractData(next).hostAddress).toBe(hostAddress); }); it("is typed by the contract's params", () => { @@ -73,7 +77,7 @@ describe("placement", () => { version: 1, params: { session: str() }, // @ts-expect-error: the params have no `id` - placement: ({ id }) => Session({ id }), + hostAddress: ({ id }) => Session({ id }), variants, }); }); diff --git a/packages/chord/test/next/contracts/params.test.ts b/packages/chord/test/next/contracts/params.test.ts index 159ff72ee..96187b2b0 100644 --- a/packages/chord/test/next/contracts/params.test.ts +++ b/packages/chord/test/next/contracts/params.test.ts @@ -1,6 +1,20 @@ import Type from "typebox"; import { describe, expect, it } from "vitest"; -import { arr, bool, bytes, int, num, obj, oneOf, opt, remote, str } from "../../../src/next/contracts/index.ts"; +import { + arr, + bool, + bytes, + hostKind, + int, + num, + obj, + oneOf, + opt, + remote, + str, +} from "../../../src/next/contracts/index.ts"; + +const Server = hostKind("server"); const variants = { a: { state: { s: str() } } }; @@ -15,16 +29,17 @@ describe("params policy", () => { c: bool(), d: oneOf("chat", "agent"), }, + hostAddress: () => Server(), variants, }), ).not.toThrow(); }); it("refuses optional fields, in the types and at declaration", () => { - // @ts-expect-error: params fields are required - expect(() => remote("n", { version: 1, params: { a: opt(str()) }, variants })).toThrow( - "n.params.a: a param is a required str(), int(), bool() or oneOf(...) of strings", - ); + expect(() => + // @ts-expect-error: params fields are required + remote("n", { version: 1, params: { a: opt(str()) }, hostAddress: () => Server(), variants }), + ).toThrow("n.params.a: a param is a required str(), int(), bool() or oneOf(...) of strings"); }); it.each([ @@ -36,6 +51,8 @@ describe("params policy", () => { ["defaults", Type.String({ default: "x" })], ])("refuses %s, in the types and at declaration", (_case, schema) => { // @ts-expect-error: not a params field schema - expect(() => remote("n", { version: 1, params: { p: schema }, variants })).toThrow("n.params.p: a param is"); + expect(() => remote("n", { version: 1, params: { p: schema }, hostAddress: () => Server(), variants })).toThrow( + "n.params.p: a param is", + ); }); }); diff --git a/packages/chord/test/next/data/args.test.ts b/packages/chord/test/next/data/args.test.ts index 90dc8cae6..896861fc0 100644 --- a/packages/chord/test/next/data/args.test.ts +++ b/packages/chord/test/next/data/args.test.ts @@ -15,7 +15,7 @@ import { } from "../../../src/next/contracts/index.ts"; import { isChordError } from "../../../src/next/data/index.ts"; import { bindProvider, type CheckedArgs, checkArgs, type Value } from "../../../src/next/data/internal.ts"; -import { counted, expectCollected } from "../support.ts"; +import { counted, expectCollected, Server } from "../support.ts"; const Note = obj({ text: str() }); const NoteV2 = obj({ text: str(), tag: str() }); @@ -23,6 +23,7 @@ const NoteV2 = obj({ text: str(), tag: str() }); const NotesV1 = remote("notes", { version: 1, params: { session: str() }, + hostAddress: () => Server(), variants: { editor: { state: { notes: arr(Note) }, diff --git a/packages/chord/test/next/data/errors.test.ts b/packages/chord/test/next/data/errors.test.ts index c5d955a99..bebc4e8e4 100644 --- a/packages/chord/test/next/data/errors.test.ts +++ b/packages/chord/test/next/data/errors.test.ts @@ -78,7 +78,7 @@ describe("protocol table", () => { const pinned = { version: 1, reasons: { - invalid: ["args", "params", "member", "placement", "reentrant", "read_after_bulk", "cycle"], + invalid: ["args", "params", "member", "host_address", "reentrant", "read_after_bulk", "cycle"], contract_mismatch: ["updating"], forbidden: ["revoked"], gone: [], diff --git a/packages/chord/test/next/data/failure.test.ts b/packages/chord/test/next/data/failure.test.ts index a5ddbacf2..b833f4d8a 100644 --- a/packages/chord/test/next/data/failure.test.ts +++ b/packages/chord/test/next/data/failure.test.ts @@ -20,7 +20,7 @@ describe("failure causes", () => { }); it("the kind comes from the stage, never from the error code", () => { - const chord = invalid("placement", "wrong host"); + const chord = invalid("host_address", "wrong host"); expect(openFailure(chord, "open")).toEqual({ kind: "transient", error: chord }); expect(openFailure(broken(chord), "open")).toEqual({ kind: "broken", error: chord }); }); diff --git a/packages/chord/test/next/data/params.test.ts b/packages/chord/test/next/data/params.test.ts index 55b9a8ebc..5848b0378 100644 --- a/packages/chord/test/next/data/params.test.ts +++ b/packages/chord/test/next/data/params.test.ts @@ -5,11 +5,12 @@ import { bool, int, local, oneOf, remote, str } from "../../../src/next/contract import { isChordError } from "../../../src/next/data/index.ts"; import { checkParams, identityKey } from "../../../src/next/data/internal.ts"; import { validatorFor } from "../../../src/next/data/validator.ts"; -import { counted, expectCollected } from "../support.ts"; +import { counted, expectCollected, Server } from "../support.ts"; const Session = remote("session", { version: 1, params: { id: str({ min: 1 }), mode: oneOf("chat", "agent"), page: int({ min: 0 }), open: bool() }, + hostAddress: () => Server(), variants: { owner: {} }, }); diff --git a/packages/chord/test/next/examples/08-remote.ts b/packages/chord/test/next/examples/08-remote.ts index c88d81e15..fb538aecb 100644 --- a/packages/chord/test/next/examples/08-remote.ts +++ b/packages/chord/test/next/examples/08-remote.ts @@ -3,7 +3,7 @@ // Run from packages/chord: // node --conditions=source --experimental-strip-types test/next/examples/08-remote.ts import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; -import { arr, int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { arr, hostKind, int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; import { defineState } from "../../../src/next/documents/index.ts"; import { createHost, defineFacet } from "../../../src/next/host/index.ts"; import { memoryLinkPair } from "../../../src/next/link/index.ts"; @@ -17,10 +17,14 @@ type P = { kind: "owner" } | { kind: "guest"; session: string } | { kind: "serve const Note = obj({ text: str(), author: str() }); const PublicNote = obj({ text: str() }); +// The server's host kind: every remote contract names the address of the host serving its objects. +const Server = hostKind("server"); + // A remote contract: one object per (type, params, principal key), in the variant its factory chooses. const Notes = remote("example.notes", { version: 1, params: { session: str() }, + hostAddress: () => Server(), variants: { editor: { state: { notes: arr(Note), count: int() }, methods: { add: method(obj({ text: str() })) } }, reader: { state: { notes: arr(PublicNote), count: int() } }, @@ -68,8 +72,12 @@ const facet = defineFacet

    ((host) => { }); }); -// 1. The server: it acts as `server` for its own requests. -const server = createHost

    ({ principal: { kind: "server" }, onError: (error) => console.error("server:", error) }); +// 1. The server, at its address: it acts as `server` for its own requests. +const server = createHost

    ({ + address: Server(), + principal: { kind: "server" }, + onError: (error) => console.error("server:", error), +}); await server.load([{ name: "@example/notes", facet }], context); // 2. A client (a tab): no principal of its own. The entry point authenticated it as a guest of S1 and stamps that on diff --git a/packages/chord/test/next/examples/09-calls.ts b/packages/chord/test/next/examples/09-calls.ts index 01dfbbe14..5c099fd5d 100644 --- a/packages/chord/test/next/examples/09-calls.ts +++ b/packages/chord/test/next/examples/09-calls.ts @@ -3,7 +3,7 @@ // Run from packages/chord: // node --conditions=source --experimental-strip-types test/next/examples/09-calls.ts import { BACKGROUND_CONTEXT, withCancel } from "../../../src/context/index.ts"; -import { bytes, int, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { bytes, hostKind, int, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; import { isChordError, PluginError } from "../../../src/next/data/index.ts"; import { createHost, defineFacet } from "../../../src/next/host/index.ts"; import { memoryLinkPair } from "../../../src/next/link/index.ts"; @@ -12,9 +12,13 @@ const context = BACKGROUND_CONTEXT; type P = { kind: "owner" } | { kind: "server" }; +// The server's host kind: a remote contract's `hostAddress` names the host serving its objects. +const Server = hostKind("server"); + const Files = remote("example.files", { version: 1, params: {}, + hostAddress: () => Server(), variants: { owner: { methods: { @@ -58,7 +62,11 @@ const facet = defineFacet

    ((host) => { })); }); -const server = createHost

    ({ principal: { kind: "server" }, onError: (error) => console.error("server:", error) }); +const server = createHost

    ({ + address: Server(), + principal: { kind: "server" }, + onError: (error) => console.error("server:", error), +}); await server.load([{ name: "@example/files", facet }], context); const [up, down] = memoryLinkPair(); server.accept(up, { principal: { kind: "owner" } }, context); diff --git a/packages/chord/test/next/examples/10-drops.ts b/packages/chord/test/next/examples/10-drops.ts index 701961c45..f0ca424f7 100644 --- a/packages/chord/test/next/examples/10-drops.ts +++ b/packages/chord/test/next/examples/10-drops.ts @@ -3,7 +3,7 @@ // Run from packages/chord: // node --conditions=source --experimental-strip-types test/next/examples/10-drops.ts import { BACKGROUND_CONTEXT, type Context } from "../../../src/context/index.ts"; -import { arr, int, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { arr, hostKind, int, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; import { defineState } from "../../../src/next/documents/index.ts"; import { createHost, defineFacet, type Host } from "../../../src/next/host/index.ts"; import { type Link, memoryLinkPair } from "../../../src/next/link/index.ts"; @@ -12,9 +12,13 @@ const context = BACKGROUND_CONTEXT; type P = { kind: "owner" } | { kind: "server" }; +// The server's host kind: a remote contract's `hostAddress` names the host serving its objects. +const Server = hostKind("server"); + const Notes = remote("example.notes", { version: 1, params: { session: str() }, + hostAddress: () => Server(), variants: { editor: { state: { notes: arr(str()) }, @@ -47,7 +51,11 @@ const facet = defineFacet

    ((host) => { ); }); -const server = createHost

    ({ principal: { kind: "server" }, onError: (error) => console.error("server:", error) }); +const server = createHost

    ({ + address: Server(), + principal: { kind: "server" }, + onError: (error) => console.error("server:", error), +}); await server.load([{ name: "@example/notes", facet }], context); const client = createHost

    ({ onError: (error) => console.error("client:", error) }); await client.load([], context); diff --git a/packages/chord/test/next/examples/12-routes.ts b/packages/chord/test/next/examples/12-routes.ts index 7074bd979..7f6516887 100644 --- a/packages/chord/test/next/examples/12-routes.ts +++ b/packages/chord/test/next/examples/12-routes.ts @@ -1,4 +1,4 @@ -// Routes: host kinds and addresses, a contract's placement, a server routing a session's objects to an in-process +// Routes: host kinds and addresses, a contract's host address, a server routing a session's objects to an in-process // worker (one `resolve` per address, `authorize` per principal), a guest's `ctx.get` on the server reaching the worker // as the guest, and a worker that drops and is resolved again. // Run from packages/chord: @@ -26,7 +26,7 @@ const Session = hostKind("session", { id: str() }); const Notes = remote("example.notes", { version: 1, params: { session: str() }, - placement: ({ session }) => Session({ id: session }), + hostAddress: ({ session }) => Session({ id: session }), variants: { editor: { state: { notes: arr(str()) }, @@ -36,10 +36,11 @@ const Notes = remote("example.notes", { }, }); -// The lobby lives on the server (no placement): its handler gets notes for whoever calls it. +// The lobby lives on the server: its handler gets notes for whoever calls it. const Lobby = remote("example.lobby", { version: 1, params: {}, + hostAddress: () => Server(), variants: { visitor: { methods: { peek: method(obj({ session: str() }), str()) } } }, }); diff --git a/packages/chord/test/next/examples/13-relay.ts b/packages/chord/test/next/examples/13-relay.ts index 6203d3f52..f132e76f8 100644 --- a/packages/chord/test/next/examples/13-relay.ts +++ b/packages/chord/test/next/examples/13-relay.ts @@ -30,7 +30,7 @@ const SessionInfo = local("example.session-info", { version: 1, params: {}, memb const Notes = remote("example.notes", { version: 1, params: { session: str() }, - placement: ({ session }) => Session({ id: session }), + hostAddress: ({ session }) => Session({ id: session }), variants: { owner: { state: { info: Init, notes: arr(str()) }, methods: { add: method(obj({ text: str() }), int()) } }, }, diff --git a/packages/chord/test/next/examples/14-flow-control.ts b/packages/chord/test/next/examples/14-flow-control.ts index 4eba6e1ee..184133150 100644 --- a/packages/chord/test/next/examples/14-flow-control.ts +++ b/packages/chord/test/next/examples/14-flow-control.ts @@ -28,7 +28,7 @@ const Reply = local("example.reply", { const View = remote("example.view", { version: 1, params: { session: str() }, - placement: ({ session }) => Session({ id: session }), + hostAddress: ({ session }) => Session({ id: session }), variants: { owner: { state: { reply: obj({ text: str() }) } } }, }); diff --git a/packages/chord/test/next/examples/15-revocation.ts b/packages/chord/test/next/examples/15-revocation.ts index 1a004ffa6..9ee2a3648 100644 --- a/packages/chord/test/next/examples/15-revocation.ts +++ b/packages/chord/test/next/examples/15-revocation.ts @@ -33,7 +33,7 @@ const Session = hostKind("session", { id: str() }); const Notes = remote("example.notes", { version: 1, params: { session: str() }, - placement: ({ session }) => Session({ id: session }), + hostAddress: ({ session }) => Session({ id: session }), variants: { all: { state: { notes: arr(str()) }, methods: { add: method(obj({ text: str() })), follow: method() } }, }, diff --git a/packages/chord/test/next/examples/16-presence-and-activity.ts b/packages/chord/test/next/examples/16-presence-and-activity.ts index 9091557e0..ba69803cf 100644 --- a/packages/chord/test/next/examples/16-presence-and-activity.ts +++ b/packages/chord/test/next/examples/16-presence-and-activity.ts @@ -6,7 +6,7 @@ // Run from packages/chord: // node --conditions=source --experimental-strip-types test/next/examples/16-presence-and-activity.ts import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; -import { int, method, remote, str } from "../../../src/next/contracts/index.ts"; +import { hostKind, int, method, remote, str } from "../../../src/next/contracts/index.ts"; import { createHost, defineFacet } from "../../../src/next/host/index.ts"; import { memoryLinkPair } from "../../../src/next/link/index.ts"; @@ -14,15 +14,19 @@ const context = BACKGROUND_CONTEXT; type P = { kind: "user"; name: string } | { kind: "server" }; +// The server's host kind: a remote contract's `hostAddress` names the host serving its objects. +const Server = hostKind("server"); + /** A shared document view: who else is looking (`viewers`), and a slow method. */ const Doc = remote("example.doc", { version: 1, params: { id: str() }, + hostAddress: () => Server(), variants: { v: { state: { viewers: int() }, methods: { think: method(undefined, int()) } } }, }); // 1. The server serves `Doc`; its factory derives the viewer count from `host.references(Doc)` (on the facet's host). -const server = createHost

    ({ principal: { kind: "server" }, onError: console.error }); +const server = createHost

    ({ address: Server(), principal: { kind: "server" }, onError: console.error }); const facet = defineFacet

    ((host) => { host.serve( Doc, diff --git a/packages/chord/test/next/examples/17-limits-and-compression.ts b/packages/chord/test/next/examples/17-limits-and-compression.ts index 970dce626..da3dbe0a9 100644 --- a/packages/chord/test/next/examples/17-limits-and-compression.ts +++ b/packages/chord/test/next/examples/17-limits-and-compression.ts @@ -9,7 +9,7 @@ // Run from packages/chord: // node --conditions=source --experimental-strip-types test/next/examples/17-limits-and-compression.ts import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; -import { arr, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { arr, hostKind, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; import { isChordError } from "../../../src/next/data/index.ts"; import { defineState } from "../../../src/next/documents/index.ts"; import { createHost, defineFacet, type Host } from "../../../src/next/host/index.ts"; @@ -29,10 +29,14 @@ const Store = local("example.store", { members: { entries: arr(Entry), append: method(Entry) }, }); +// The server's host kind: a remote contract's `hostAddress` names the host serving its objects. +const Server = hostKind("server"); + /** What a tab holds: the conversation's entries, and appending. */ const Conversation = remote("example.conversation", { version: 1, params: { id: str() }, + hostAddress: () => Server(), variants: { v: { state: { entries: arr(Entry) }, methods: { append: method(Entry) } } }, }); @@ -78,7 +82,12 @@ function counted(link: Link, bytes: { received: number }): Link { * `limits`; the tab's own link to the server has `tabLimits`. */ async function setUp(compression: Compression | undefined, limits?: PartLimits, tabLimits?: PartLimits) { - const server = createHost

    ({ principal: { kind: "server" }, compression, onError: console.error }); + const server = createHost

    ({ + address: Server(), + principal: { kind: "server" }, + compression, + onError: console.error, + }); await server.load([{ name: "@example/conversation", facet }], context); const tab = createHost

    ({ onError: console.error }); await tab.load([], context); @@ -168,6 +177,7 @@ for (const compression of [undefined, nodeCompression]) { // child, would refuse a compressed part. Its 100 KB argument arrives as JSON, within the server's default limits. { const server = createHost

    ({ + address: Server(), principal: { kind: "server" }, compression: nodeCompression, onError: console.error, diff --git a/packages/chord/test/next/examples/19-jobs.ts b/packages/chord/test/next/examples/19-jobs.ts index 06964dc63..94c9646ca 100644 --- a/packages/chord/test/next/examples/19-jobs.ts +++ b/packages/chord/test/next/examples/19-jobs.ts @@ -6,7 +6,7 @@ // Run from packages/chord: // node --conditions=source --experimental-strip-types test/next/examples/19-jobs.ts import { BACKGROUND_CONTEXT, type Context } from "../../../src/context/index.ts"; -import { arr, int, local, method, obj, remote, type Static, str } from "../../../src/next/contracts/index.ts"; +import { arr, hostKind, int, local, method, obj, remote, type Static, str } from "../../../src/next/contracts/index.ts"; import { defineState } from "../../../src/next/documents/index.ts"; import { createHost, defineFacet, sleep } from "../../../src/next/host/index.ts"; import { defineJob } from "../../../src/next/jobs/index.ts"; @@ -56,10 +56,14 @@ const ChatStore = local("example.chat-store", { }, }); +// The server's host kind: a remote contract's `hostAddress` names the host serving its objects. +const Server = hostKind("server"); + /** Per-principal views: the owner runs shells and sees their output, a guest sees which shells run. */ const Chat = remote("example.chat", { version: 1, params: {}, + hostAddress: () => Server(), variants: { owner: { state: { entries: arr(str()), shells: arr(Shell) }, @@ -170,7 +174,11 @@ const facet = defineFacet

    ((host) => { }); // 2. A server and two tabs, the owner's and a guest's. -const server = createHost

    ({ principal: { kind: "server" }, onError: (error) => console.error(error) }); +const server = createHost

    ({ + address: Server(), + principal: { kind: "server" }, + onError: (error) => console.error(error), +}); await server.load([{ name: "@example/chat", facet }], context); async function tab(principal: P) { const host = createHost

    ({ onError: (error) => console.error(error) }); diff --git a/packages/chord/test/next/host/activity.test.ts b/packages/chord/test/next/host/activity.test.ts index 405089d6d..cdfa929b7 100644 --- a/packages/chord/test/next/host/activity.test.ts +++ b/packages/chord/test/next/host/activity.test.ts @@ -6,9 +6,14 @@ import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; import { bool, int, local, method, remote } from "../../../src/next/contracts/index.ts"; import { createHost, defineFacet } from "../../../src/next/host/index.ts"; import { constant } from "../../../src/next/state/index.ts"; -import { fakeTimers, flushed, framesOf, linked, loaded, Notes, serveNotes, transported } from "../support.ts"; +import { fakeTimers, flushed, framesOf, linked, loaded, Notes, Server, serveNotes, transported } from "../support.ts"; -const Kept = remote("app.kept", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); +const Kept = remote("app.kept", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: { n: int() } } }, +}); describe("host.activity", () => { it("counts holds arriving over links and served here, not this host's own references", async () => { diff --git a/packages/chord/test/next/host/attempt.test.ts b/packages/chord/test/next/host/attempt.test.ts index 4066748e7..d1227bc4a 100644 --- a/packages/chord/test/next/host/attempt.test.ts +++ b/packages/chord/test/next/host/attempt.test.ts @@ -21,7 +21,17 @@ import { TableChanges } from "../../../src/next/store/table-changes.ts"; import { evolveTable } from "../../../src/next/store/table-evolve.ts"; import { TableHandle } from "../../../src/next/store/table-handle.ts"; import { prepareTables } from "../../../src/next/store/table-migration.ts"; -import { failures, fakeTimers, flushed, loaded, memoryStore, metered, rejection, tableOwner } from "../support.ts"; +import { + failures, + fakeTimers, + flushed, + loaded, + memoryStore, + metered, + rejection, + Server, + tableOwner, +} from "../support.ts"; const List = defineState("list", { schema: obj({ items: arr(str()) }), version: 1, initial: () => ({ items: [] }) }); const Page = local("app.page", { version: 1, params: { id: int() }, members: { label: str(), slow: method() } }); @@ -565,6 +575,7 @@ describe("remote variants", () => { const Two = remote("app.two", { version: 1, params: {}, + hostAddress: () => Server(), variants: { full: { state: { n: int() }, methods: { bump: method() } }, view: { state: { n: int() } } }, }); const owner = { kind: "owner" } as const; @@ -572,7 +583,11 @@ describe("remote variants", () => { /** A host acting as `owner` whose factory for `Two` returns `implementation`. */ async function two(implementation: unknown) { const errors: HostError[] = []; - const host = createHost({ principal: owner, onError: (error) => errors.push(error) }); + const host = createHost({ + address: Server(), + principal: owner, + onError: (error) => errors.push(error), + }); const facet = defineFacet((h) => h.serve(Two, () => implementation as never)); await host.load([{ name: "@test/app", facet }], context); return { host, errors }; diff --git a/packages/chord/test/next/host/ctx.test.ts b/packages/chord/test/next/host/ctx.test.ts index df259c227..bf4744b76 100644 --- a/packages/chord/test/next/host/ctx.test.ts +++ b/packages/chord/test/next/host/ctx.test.ts @@ -5,7 +5,16 @@ import { withLinkedAbortSignal } from "../../../src/next/host/ctx.ts"; import type { ChordHost } from "../../../src/next/host/host.ts"; import type { Ctx, Host, Reference } from "../../../src/next/host/index.ts"; import { constant } from "../../../src/next/state/index.ts"; -import { flushed, incarnationOf, linked, loaded, rawParent, rejection, type TestPrincipal } from "../support.ts"; +import { + flushed, + incarnationOf, + linked, + loaded, + rawParent, + rejection, + Server, + type TestPrincipal, +} from "../support.ts"; const Item = local("app.item", { version: 1, params: { id: int() }, members: { id: int() } }); const Desk = local("app.desk", { @@ -111,7 +120,12 @@ describe("ctx", () => { }); it("a ctx.get still waiting when its handler settles gives up: its reference is released at once", async () => { - const Far = remote("app.far", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + const Far = remote("app.far", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: { n: int() } } }, + }); const Starter = local("app.starter", { version: 1, params: {}, members: { start: method() } }); let pending: Promise | undefined; const { host } = await loaded((h) => @@ -146,6 +160,7 @@ describe("local handlers act as the host (§4, audit)", () => { const Gate = remote("app.gate", { version: 1, params: {}, + hostAddress: () => Server(), variants: { v: { methods: { who: method(undefined, str()), through: method(undefined, str()) } } }, }); const who = (ctx: Ctx) => `${ctx.principal?.kind ?? "none"} ${ctx.incarnation}`; diff --git a/packages/chord/test/next/host/host.test.ts b/packages/chord/test/next/host/host.test.ts index 64a291b43..0074d240e 100644 --- a/packages/chord/test/next/host/host.test.ts +++ b/packages/chord/test/next/host/host.test.ts @@ -31,6 +31,8 @@ import { partValue, RawPeer, rejection, + SERVER_ADDRESS, + Server, Session, serveNotes, type TestPrincipal, @@ -482,13 +484,24 @@ describe("getters", () => { forEachCodec(() => { describe("principals and where a reference goes", () => { - const Who = remote("app.who", { version: 1, params: { id: int() }, variants: { v: { state: { who: str() } } } }); - const Outer = remote("app.outer", { version: 1, params: {}, variants: { v: { state: { who: str() } } } }); + const Who = remote("app.who", { + version: 1, + params: { id: int() }, + hostAddress: () => Server(), + variants: { v: { state: { who: str() } } }, + }); + const Outer = remote("app.outer", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: { who: str() } } }, + }); const Inner = local("app.inner", { version: 1, params: { id: int() }, members: { who: str() } }); const Desk = local("app.desk", { version: 1, params: {}, members: { who: method(undefined, str()) } }); const Gate = remote("app.gate", { version: 1, params: {}, + hostAddress: () => Server(), variants: { v: { methods: { who: method(undefined, str()) } } }, }); type P = TestPrincipal; @@ -518,6 +531,12 @@ forEachCodec(() => { await expect(loading).rejects.toThrow("app.who: a host without a principal serves only local types"); }); + it("a host without an address serves no remote type, whatever its principal", async () => { + const host = createHost

    ({ principal: { kind: "server" }, onError: () => {} }); + const loading = host.load([{ name: "@test/app", facet: defineFacet

    ((h) => serveAll(h, [])) }], context); + await expect(loading).rejects.toThrow("app.who: a host without an address serves only local types"); + }); + it("ctx.get acts as ctx.principal: a guest's call gets the guest's view; a local handler's acts as the host", async () => { const { server, client } = await linked((h) => serveAll(h, []), { child: { kind: "guest", session: "S1" } }); expect(await (await client.get(Gate, {}, context)).who(context)).toBe("guest"); @@ -533,6 +552,7 @@ forEachCodec(() => { it("identity includes the principal key: equal keys share one object", async () => { const factories: string[] = []; const host = createHost

    ({ + address: Server(), principal: { kind: "guest", session: "S1" }, principalKey: (principal) => principal.kind, onError: () => {}, @@ -549,6 +569,7 @@ forEachCodec(() => { version: Who.version, fingerprint: Who.fingerprint, params: jsonPart({ id: 1 }), + address: SERVER_ADDRESS, }); await peer.next("bound"); expect(factories).toEqual(["1 guest"]); @@ -562,6 +583,7 @@ forEachCodec(() => { version: Who.version, fingerprint: Who.fingerprint, params: jsonPart({ id: 1 }), + address: SERVER_ADDRESS, }); await owner.next("bound"); expect(factories).toEqual(["1 guest", "1 owner"]); @@ -569,7 +591,12 @@ forEachCodec(() => { it("remote objects with empty params are not activated by load; they live while referenced", async () => { const factories: string[] = []; - const Empty = remote("app.empty", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + const Empty = remote("app.empty", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: { n: int() } } }, + }); const { server } = await linked((h) => h.serve(Empty, (_, principal) => { factories.push(principal.kind); @@ -661,7 +688,7 @@ describe("acquiring: hold and get", () => { const offline = { code: "unavailable", message: "app.svc(1): offline" }; it("invalid params and a local type not served here: at once", async () => { - const host = createHost({ ...noErrors, principal: { kind: "server" } }); + const host = createHost({ address: Server(), ...noErrors, principal: { kind: "server" } }); await host.load([{ name: "@test/app", facet: defineFacet((h) => void serveNotes(h)) }], context); const params = { code: "invalid", reason: "params" }; expect(() => host.hold(Notes, { session: 1 } as never, context)).toThrow("params"); @@ -814,8 +841,8 @@ forEachCodec(() => { return { host, child: new RawPeer(b), parent, errors }; } - /** Sends a hold of `PlacedNotes` (or `type`) for `session`, naming `address` when given. */ - function holdOf(peer: RawPeer, session: string, address?: string, type: string = PlacedNotes.name, hold = 0) { + /** Sends a hold of `PlacedNotes` (or `type`) for `session`, naming the session `address`. */ + function holdOf(peer: RawPeer, session: string, address: string, type: string = PlacedNotes.name, hold = 0) { peer.send({ kind: "hold", hold, @@ -823,13 +850,13 @@ forEachCodec(() => { version: PlacedNotes.version, fingerprint: PlacedNotes.fingerprint, params: jsonPart({ session }), - ...(address === undefined ? {} : { address: { kind: "session", params: { id: address } } }), + address: { kind: "session", params: { id: address } }, }); } const misplaced = { code: "invalid", - reason: "placement", + reason: "host_address", message: "app.placed-notes: placed on session, this host is session|S1", }; @@ -839,7 +866,7 @@ forEachCodec(() => { expect((await child.next("bound")).holds[0]).toMatchObject({ hold: 0 }); }); - it("refuses a hold served here whose params place it elsewhere, whatever address it names: invalid/placement (race 5)", async () => { + it("refuses a hold served here whose params place it elsewhere, whatever address it names: invalid/host_address (race 5)", async () => { const { child } = await worker(); holdOf(child, "S2", "S1"); expect(await child.next("failed")).toMatchObject({ @@ -847,7 +874,7 @@ forEachCodec(() => { cause: { kind: "broken", error: { kind: "chord", error: misplaced } }, }); // A refused hold is kept (a retry starts it again): the next hold takes the next id. - holdOf(child, "S2", undefined, PlacedNotes.name, 1); + holdOf(child, "S2", "S2", PlacedNotes.name, 1); expect(await child.next("failed")).toMatchObject({ hold: 1, cause: { kind: "broken", error: { kind: "chord", error: misplaced } }, @@ -860,7 +887,7 @@ forEachCodec(() => { expect((await child.next("bound")).holds[0]).toMatchObject({ hold: 0 }); }); - it("sends a reference made here whose placement is another host over the parent link, though it serves the type", async () => { + it("sends a reference made here whose host address is another host over the parent link, though it serves the type", async () => { const { host, parent } = await worker({ parent: true }); const elsewhere = host.hold(PlacedNotes, { session: "S2" }, context); const hold = await parent!.next("hold"); @@ -871,7 +898,7 @@ forEachCodec(() => { it("sends a hold it cannot serve or route over the parent link, unparsed", async () => { const { child, parent } = await worker({ parent: true }); - holdOf(child, "S9", undefined, "app.elsewhere"); + holdOf(child, "S9", "S9", "app.elsewhere"); const relayed = await parent!.next("hold"); expect(relayed).toMatchObject({ type: "app.elsewhere" }); expect(partValue(relayed.params)).toEqual({ session: "S9" }); @@ -879,7 +906,7 @@ forEachCodec(() => { it("never sends a hold from the parent back to it: unavailable (race 8)", async () => { const { parent } = await worker({ parent: true }); - holdOf(parent!, "S9", undefined, "app.elsewhere"); + holdOf(parent!, "S9", "S9", "app.elsewhere"); expect(await parent!.next("failed")).toMatchObject({ hold: 0, cause: { @@ -892,25 +919,25 @@ forEachCodec(() => { it("refuses a hold it cannot serve or route without a parent link: unavailable", async () => { const { child } = await worker(); - holdOf(child, "S9", undefined, "app.elsewhere"); + holdOf(child, "S9", "S9", "app.elsewhere"); expect(await child.next("failed")).toMatchObject({ cause: { kind: "broken", error: { kind: "chord", error: { code: "unavailable" } } }, }); }); - it("fails a reference broken with what its placement throws", async () => { + it("fails a reference broken with what its hostAddress throws", async () => { const Throwing = remote("app.throwing", { version: 1, params: {}, - placement: () => { - throw new Error("placement threw"); + hostAddress: () => { + throw new Error("hostAddress threw"); }, variants: { v: { state: { n: int() } } }, }); const client = createHost(noErrors); await client.load([], context); expect(await rejection(client.get(Throwing, {}, context))).toMatchObject({ - message: "app.throwing: placement threw", + message: "app.throwing: hostAddress threw", }); expect(client.hold(Throwing, {}, context).state.value).toMatchObject({ kind: "failed" }); }); diff --git a/packages/chord/test/next/host/member-source.test.ts b/packages/chord/test/next/host/member-source.test.ts index 4a804cdb7..628b84e2c 100644 --- a/packages/chord/test/next/host/member-source.test.ts +++ b/packages/chord/test/next/host/member-source.test.ts @@ -22,6 +22,7 @@ import { loaded, memoryStore, metered, + Server, transported, valuesOf, } from "../support.ts"; @@ -324,7 +325,12 @@ describe("re-binding when a new attempt goes live", () => { describe("subscribers", () => { it("a hold over a link on a remote object re-exposing a local member gets frames again after the local retry", async () => { const Count = local("app.count", { version: 1, params: {}, members: { n: obj({ n: int() }), add: method() } }); - const View = remote("app.view", { version: 1, params: {}, variants: { v: { state: { n: obj({ n: int() }) } } } }); + const View = remote("app.view", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: { n: obj({ n: int() }) } } }, + }); const Counter = defineState("counter", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); const selves: Self[] = []; const { server, client } = await linked((h) => { @@ -406,10 +412,11 @@ describe("a member the new attempt's variant lacks", () => { const Two = remote("app.two", { version: 1, params: {}, + hostAddress: () => Server(), variants: { full: { state: { n: int() } }, view: { state: { m: int() } } }, }); const owner = { kind: "owner" } as const; - const host = createHost({ principal: owner, onError: () => {} }); + const host = createHost({ address: Server(), principal: owner, onError: () => {} }); const selves: Self[] = []; const facet = defineFacet((h) => h.serve(Two, (_, _principal, self) => { diff --git a/packages/chord/test/next/host/object.test.ts b/packages/chord/test/next/host/object.test.ts index d492240f0..ab1675fb2 100644 --- a/packages/chord/test/next/host/object.test.ts +++ b/packages/chord/test/next/host/object.test.ts @@ -26,6 +26,8 @@ import { rejection, relayed, restartable, + SERVER_ADDRESS, + Server, Session, StoreRestartable, storeRestartable, @@ -705,14 +707,19 @@ describe("linger", () => { /** Every object attempted in this file, weakly: the objects of the cycles among them. */ const attempted = instancesOf(ChordObject, "attempt"); -/** On the server (no placement): its activation holds `Far` on the session's worker. */ -const Near = remote("app.near", { version: 1, params: { session: str() }, variants: { v: { state: { n: int() } } } }); +/** On the server: its activation holds `Far` on the session's worker. */ +const Near = remote("app.near", { + version: 1, + params: { session: str() }, + hostAddress: () => Server(), + variants: { v: { state: { n: int() } } }, +}); /** On the session's worker: its activation holds `Near` back on the server, or, with `local`, `Middle`. */ const Far = remote("app.far", { version: 1, params: { session: str() }, - placement: ({ session }) => Session({ id: session }), + hostAddress: ({ session }) => Session({ id: session }), variants: { v: { state: { n: int() } } }, }); @@ -720,6 +727,7 @@ const Far = remote("app.far", { const Middle = remote("app.middle", { version: 1, params: { session: str() }, + hostAddress: () => Server(), variants: { v: { state: { n: int() } } }, }); @@ -730,7 +738,7 @@ const Hub = local("app.hub", { version: 1, params: { session: str() }, members: const Loop = remote("app.loop", { version: 1, params: { here: str(), there: str() }, - placement: ({ here }) => Session({ id: here }), + hostAddress: ({ here }) => Session({ id: here }), variants: { v: { state: { n: int() } } }, }); @@ -910,7 +918,15 @@ function farHold(hold: number): Extract { /** A hold of `Near` for S1, as a worker sends it to the server. */ function nearHold(hold: number): Extract { const { name: type, version, fingerprint } = Near; - return { kind: "hold", hold, type, version, fingerprint, params: jsonPart({ session: "S1" }) }; + return { + kind: "hold", + hold, + type, + version, + fingerprint, + params: jsonPart({ session: "S1" }), + address: SERVER_ADDRESS, + }; } describe("a failed activation's cleanup (audit)", () => { diff --git a/packages/chord/test/next/host/presence.test.ts b/packages/chord/test/next/host/presence.test.ts index 6260f9360..a43b523bf 100644 --- a/packages/chord/test/next/host/presence.test.ts +++ b/packages/chord/test/next/host/presence.test.ts @@ -22,6 +22,8 @@ import { PlacedNotes, RawPeer, relayed, + SERVER_ADDRESS, + Server, serveNotes, type TestPrincipal, testCodec, @@ -84,9 +86,10 @@ describe("host.references", () => { const Shown = remote("app.shown", { version: 1, params: { session: str() }, + hostAddress: () => Server(), variants: { v: { state: { here: int() } } }, }); - const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const server = createHost({ address: Server(), principal: { kind: "server" }, onError: () => {} }); await server.load( [ { @@ -130,9 +133,10 @@ describe("host.references", () => { const Viewed = remote("app.viewed", { version: 1, params: { session: str() }, + hostAddress: () => Server(), variants: { v: { state: { count: int(), viewers: int() } } }, }); - const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const server = createHost({ address: Server(), principal: { kind: "server" }, onError: () => {} }); const facet = defineFacet((host) => { serveNotes(host); host.serve(Viewed, ({ session }, _principal, self) => { @@ -151,7 +155,15 @@ describe("host.references", () => { server.accept(end, { principal: { kind: "owner" }, codec: testCodec() }, context); const peer = new RawPeer(b); const { name: type, version, fingerprint } = Viewed; - peer.send({ kind: "hold", hold: 0, type, version, fingerprint, params: jsonPart({ session: "S1" }) }); + peer.send({ + kind: "hold", + hold: 0, + type, + version, + fingerprint, + params: jsonPart({ session: "S1" }), + address: SERVER_ADDRESS, + }); return peer; }; const drop = { armed: false }; @@ -176,9 +188,10 @@ describe("host.references", () => { const Viewed = remote("app.viewed", { version: 1, params: {}, + hostAddress: () => Server(), variants: { v: { state: { viewers: int() } } }, }); - const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const server = createHost({ address: Server(), principal: { kind: "server" }, onError: () => {} }); const facet = defineFacet((host) => { host.serve( Viewed, @@ -195,13 +208,13 @@ describe("host.references", () => { server.accept(closingOnSend(a, drop), { principal: { kind: "owner" }, codec: testCodec() }, context); const peer = new RawPeer(b); const { name: type, version, fingerprint } = Viewed; - peer.send({ kind: "hold", hold: 0, type, version, fingerprint, params: "{}" }); + peer.send({ kind: "hold", hold: 0, type, version, fingerprint, params: "{}", address: SERVER_ADDRESS }); await peer.next("bound"); peer.send({ kind: "ack" }); await transported(); // The second hold's presence entry changes the first's state: sending it closes the link. drop.armed = true; - peer.send({ kind: "hold", hold: 1, type, version, fingerprint, params: "{}" }); + peer.send({ kind: "hold", hold: 1, type, version, fingerprint, params: "{}", address: SERVER_ADDRESS }); await transported(); expect(peer.closed).toBe(true); expect(server.references(Viewed).value).toEqual([]); @@ -210,8 +223,13 @@ describe("host.references", () => { }); it("completes a release's bookkeeping before its departure publishes: a link closing then leaves no lingering", async () => { - const Viewed = remote("app.viewed", { version: 1, params: {}, variants: { v: { state: { viewers: int() } } } }); - const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const Viewed = remote("app.viewed", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: { viewers: int() } } }, + }); + const server = createHost({ address: Server(), principal: { kind: "server" }, onError: () => {} }); const facet = defineFacet((host) => { host.serve( Viewed, @@ -229,7 +247,7 @@ describe("host.references", () => { const peer = new RawPeer(b); const { name: type, version, fingerprint } = Viewed; for (const hold of [0, 1]) { - peer.send({ kind: "hold", hold, type, version, fingerprint, params: "{}" }); + peer.send({ kind: "hold", hold, type, version, fingerprint, params: "{}", address: SERVER_ADDRESS }); await peer.next("bound"); peer.send({ kind: "ack" }); } @@ -248,6 +266,7 @@ describe("host.references", () => { it("removes a bulk departure with one list and one splice op per entry, in one frame (a revoke of many holds)", async () => { const server = createHost({ + address: Server(), principal: { kind: "server" }, revokedBy: (principal) => (principal.kind === "guest" ? [{ session: principal.session }] : []), onError: () => {}, @@ -258,7 +277,15 @@ describe("host.references", () => { const peer = new RawPeer(b); const { name: type, version, fingerprint } = Notes; for (let hold = 0; hold < 4; hold++) { - peer.send({ kind: "hold", hold, type, version, fingerprint, params: jsonPart({ session: "S1" }) }); + peer.send({ + kind: "hold", + hold, + type, + version, + fingerprint, + params: jsonPart({ session: "S1" }), + address: SERVER_ADDRESS, + }); } await transported(); const references = server.references(Notes); @@ -285,7 +312,12 @@ describe("host.references", () => { }); it("has no entries for lingering objects", async () => { - const Kept = remote("app.kept", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + const Kept = remote("app.kept", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: { n: int() } } }, + }); const { server, client } = await linked((h) => h.serve(Kept, () => ({ variant: "v", members: { n: constant(1) } }), { linger: 60_000 }), ); diff --git a/packages/chord/test/next/host/reference.test.ts b/packages/chord/test/next/host/reference.test.ts index adeeb7288..2f2375b1c 100644 --- a/packages/chord/test/next/host/reference.test.ts +++ b/packages/chord/test/next/host/reference.test.ts @@ -29,6 +29,7 @@ import { memoryStore, Notes, rejection, + Server, settledHeap, } from "../support.ts"; @@ -338,12 +339,13 @@ describe("remote references", () => { const Two = remote("app.two", { version: 1, params: {}, + hostAddress: () => Server(), variants: { full: { state: { n: int() }, methods: { bump: method() } }, view: { state: { m: int() } } }, }); const Holder = local("app.holder", { version: 1, params: {}, members: { n: int() } }); const seen: unknown[] = []; const owner = { kind: "owner" } as const; - const host = createHost({ principal: owner, onError: () => {} }); + const host = createHost({ address: Server(), principal: owner, onError: () => {} }); const facet = defineFacet((h) => { h.serve(Two, () => ({ variant: "view", members: { m: constant(1) } })); h.serve(Holder, (_, self) => { @@ -369,6 +371,7 @@ describe("remote reference types", () => { const Two = remote("app.two", { version: 1, params: {}, + hostAddress: () => Server(), variants: { full: { state: { n: int() }, methods: { bump: method() } }, view: { state: { m: int() } } }, }); type Ref = Reference; @@ -384,6 +387,7 @@ describe("remote reference types", () => { const Two = remote("app.two", { version: 1, params: {}, + hostAddress: () => Server(), variants: { full: { state: { n: int() }, methods: { bump: method() } }, view: { state: { m: int() } } }, }); expectTypeOf["variant"]>().toEqualTypeOf<"full" | "view" | undefined>(); diff --git a/packages/chord/test/next/host/revocation.test.ts b/packages/chord/test/next/host/revocation.test.ts index a80fba200..ed5bab268 100644 --- a/packages/chord/test/next/host/revocation.test.ts +++ b/packages/chord/test/next/host/revocation.test.ts @@ -19,6 +19,8 @@ import { rawParent, rawWorkers, relayed, + SERVER_ADDRESS, + Server, Session, serveNotes, type TestPrincipal, @@ -50,9 +52,10 @@ const Barrier = local("app.barrier", { version: 1, params: {}, members: { n: int /** A guest's variant: its `wait` method's index. */ const { wait } = wireNames(Notes, "reader").methods as { wait: number }; -/** A worker of session S1 serving `Notes` (as `serveNotes`), its parent a raw peer whose holds name their principal. */ +/** A worker of session S1 serving `Notes` (as `serveNotes`, at the address `Notes` names), its parent a raw peer whose holds name their principal. */ async function worker(setup: (host: FacetHost) => void = (h) => void serveNotes(h)) { const host = createHost({ + address: Server(), principal: { kind: "worker", session: "S1" }, revokedBy, onError: () => {}, @@ -71,6 +74,7 @@ function holdAs(peer: RawPeer, hold: number, principal: TestPrincipal, Type: typ version: Type.version, fingerprint: Type.fingerprint, params: jsonPart({ session: "S1" }), + address: SERVER_ADDRESS, principal, }); } @@ -79,6 +83,7 @@ describe("revocation keys", () => { it("are computed once per principal object, compared as canonical JSON", async () => { let calls = 0; const host = createHost({ + address: Server(), principal: { kind: "server" }, revokedBy: () => { calls++; @@ -103,6 +108,7 @@ describe("revocation keys", () => { it("keep no principal alive (a WeakMap keyed by the principal the host cached them for)", async () => { const cached: WeakRef[] = []; const host = createHost({ + address: Server(), principal: { kind: "worker", session: "S1" }, revokedBy: (principal) => { // The principal decoded on this host, the cache's key: not the sender's object. @@ -172,7 +178,12 @@ forEachCodec(() => { }); it("abandons an activation in progress (race 1): its open's context is aborted, its holders get gone", async () => { - const Slow = remote("app.slow", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + const Slow = remote("app.slow", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: { n: int() } } }, + }); const aborted: unknown[] = []; const { host, parent } = await worker((h) => h.serve(Slow, (_, _principal, self) => { @@ -196,6 +207,7 @@ forEachCodec(() => { version: Slow.version, fingerprint: Slow.fingerprint, params: "{}", + address: SERVER_ADDRESS, principal: S1, }); await transported(); @@ -217,6 +229,7 @@ forEachCodec(() => { // A worker connects before it loads (W6); a load can wait for an activation (the start barrier). const opened = Promise.withResolvers(); const host = createHost({ + address: Server(), principal: { kind: "worker", session: "S1" }, revokedBy, onError: () => {}, @@ -313,6 +326,7 @@ forEachCodec(() => { describe("links and revoke messages", () => { it("closes the child links whose principal it revokes, never the parent link", async () => { const host = createHost({ + address: Server(), principal: { kind: "server" }, revokedBy, onError: () => {}, @@ -336,6 +350,7 @@ forEachCodec(() => { it("ignores a revoke from a child link (race 6)", async () => { const host = createHost({ + address: Server(), principal: { kind: "server" }, revokedBy, onError: () => {}, diff --git a/packages/chord/test/next/host/route.test.ts b/packages/chord/test/next/host/route.test.ts index eb3d6413f..756ae9429 100644 --- a/packages/chord/test/next/host/route.test.ts +++ b/packages/chord/test/next/host/route.test.ts @@ -12,6 +12,7 @@ import { PlacedNotes, rejection, relayed, + Server, Session, serveNotes, type TestPrincipal, @@ -101,18 +102,18 @@ forEachCodec(() => { }); }); - it("fails a hold broken invalid/placement for an address whose params do not fit the kind", async () => { + it("fails a hold broken invalid/host_address for an address whose params do not fit the kind", async () => { const Misplaced = remote("app.misplaced", { version: 1, params: { n: int() }, - // An address of the right kind with params of the wrong type, as a buggy placement could compute. - placement: ({ n }) => ({ kind: "session", params: { id: n }, key: "" }) as never, + // An address of the right kind with params of the wrong type, as a buggy `hostAddress` could compute. + hostAddress: ({ n }) => ({ kind: "session", params: { id: n }, key: "" }) as never, variants: { v: { state: { n: int() } } }, }); const { tab, resolves } = await relayed(); expect(await rejection(tab.get(Misplaced, { n: 1 }, context))).toMatchObject({ code: "invalid", - reason: "placement", + reason: "host_address", }); expect(resolves).toEqual([]); }); @@ -322,11 +323,16 @@ forEachCodec(() => { const Who = remote("app.who", { version: 1, params: {}, + hostAddress: () => Server(), variants: { v: { methods: { app: method(undefined, str()) } } }, }); const resolved: (string | undefined)[] = []; let worker: Host | undefined; - const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const server = createHost({ + address: Server(), + principal: { kind: "server" }, + onError: () => {}, + }); server.route( Session, { diff --git a/packages/chord/test/next/host/routines.test.ts b/packages/chord/test/next/host/routines.test.ts index e3bb69a6e..17dfe5eb1 100644 --- a/packages/chord/test/next/host/routines.test.ts +++ b/packages/chord/test/next/host/routines.test.ts @@ -36,6 +36,7 @@ import { loaded, Notes, rawParent, + Server, serveNotes, type TestPrincipal, transported, @@ -789,9 +790,10 @@ describe("on", () => { const Halves = remote("app.halves", { version: 1, params: {}, + hostAddress: () => Server(), variants: { v: { state: { half: int() }, methods: { set: method(int()) } } }, }); - const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const server = createHost({ address: Server(), principal: { kind: "server" }, onError: () => {} }); const serve = defineFacet((h) => h.serve(Halves, (_, _principal, self) => { const doc = self.state(Doc); @@ -844,7 +846,7 @@ describe("on", () => { }); it("follows ref.state: a reference over a link going loading and live again", async () => { - const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const server = createHost({ address: Server(), principal: { kind: "server" }, onError: () => {} }); await server.load([{ name: "@test/app", facet: defineFacet((h) => void serveNotes(h)) }], context); const Viewer = local("app.viewer", { version: 1, params: { id: int() }, members: {} }); const states: string[] = []; @@ -1069,7 +1071,12 @@ describe("routines and the end", () => { }); it("act as a remote object's principal", async () => { - const Watcher = remote("app.watcher", { version: 1, params: {}, variants: { v: { state: {} } } }); + const Watcher = remote("app.watcher", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: {} } }, + }); const seen: unknown[] = []; const { client } = await linked( (h) => diff --git a/packages/chord/test/next/jobs/job.test.ts b/packages/chord/test/next/jobs/job.test.ts index 4fd9aadcb..d82d82b3f 100644 --- a/packages/chord/test/next/jobs/job.test.ts +++ b/packages/chord/test/next/jobs/job.test.ts @@ -34,6 +34,7 @@ import { loaded, metered, rejection, + Server, type TestPrincipal, transported, } from "../support.ts"; @@ -1057,11 +1058,12 @@ describe("keep-alive", () => { const View = remote("app.view", { version: 1, params: {}, + hostAddress: () => Server(), variants: { v: { state: { running: arr(unknown()) } } }, }); let job: JobHandle<{ text: string }, string, undefined> | undefined; let ran = false; - const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const server = createHost({ address: Server(), principal: { kind: "server" }, onError: () => {} }); const facet = defineFacet((h) => h.serve(View, (_, _principal, self) => { job = self.job(Echo, { @@ -1157,13 +1159,19 @@ describe("keep-alive", () => { describe("object end", () => { type Revocation = { session: string }; - const Jobs = remote("app.jobs", { version: 1, params: {}, variants: { v: { state: {} } } }); + const Jobs = remote("app.jobs", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: {} } }, + }); it("revocation (R11): runs' signals fire with the end's error, the drain waits for them, no finish", async () => { let job: JobHandle<{ text: string }, string, undefined> | undefined; const finished: string[] = []; let reason: unknown; const server = createHost({ + address: Server(), principal: { kind: "server" }, revokedBy: (p) => (p.kind === "guest" ? [{ session: p.session }] : []), onError: () => {}, @@ -1209,6 +1217,7 @@ describe("object end", () => { const gate = Promise.withResolvers(); const events: string[] = []; const server = createHost({ + address: Server(), principal: { kind: "server" }, revokedBy: (p) => (p.kind === "guest" ? [{ session: p.session }] : []), onError: () => {}, diff --git a/packages/chord/test/next/link/channel.test.ts b/packages/chord/test/next/link/channel.test.ts index f5abbe8eb..4179830af 100644 --- a/packages/chord/test/next/link/channel.test.ts +++ b/packages/chord/test/next/link/channel.test.ts @@ -20,11 +20,18 @@ import { RawPeer, rawLinked, rawProvider, + SERVER_ADDRESS, + Server, testCodec, transported, } from "../support.ts"; -const Thing = remote("app.thing", { version: 1, params: {}, variants: { only: { state: { n: int() } } } }); +const Thing = remote("app.thing", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { only: { state: { n: int() } } }, +}); const hold = { kind: "hold", @@ -33,6 +40,7 @@ const hold = { version: 1, fingerprint: "0000000000000000", params: "{}", + address: SERVER_ADDRESS, } as const; const problems = (errors: readonly HostError[]) => diff --git a/packages/chord/test/next/link/message.test.ts b/packages/chord/test/next/link/message.test.ts index 6adf76b4d..968612a2a 100644 --- a/packages/chord/test/next/link/message.test.ts +++ b/packages/chord/test/next/link/message.test.ts @@ -21,12 +21,23 @@ function partText(part: Part): Part { } const incarnation = "0190f5a2-7c3e-7d4a-9b1c-2d3e4f5a6b7c"; + +/** The address a hold names: `Server()` on the wire. */ +const server = { kind: "server", params: {} }; const error = { kind: "chord", error: { code: "released", message: "gone" } } as const; /** One message of every kind, with the optional fields both present and absent; parts are JSON text. */ const EVERY_KIND: Message[] = [ { kind: "hello", protocol: 1, incarnation, calls: 256 }, - { kind: "hold", hold: 300, type: "app.notes", version: 2, fingerprint: "0123456789abcdef", params: '{"id":1}' }, + { + kind: "hold", + hold: 300, + type: "app.notes", + version: 2, + fingerprint: "0123456789abcdef", + params: '{"id":1}', + address: server, + }, { kind: "hold", hold: 1, @@ -45,6 +56,7 @@ const EVERY_KIND: Message[] = [ version: 1, fingerprint: "0123456789abcdef", params: "{}", + address: server, incarnation, }, { kind: "release", hold: 0 }, @@ -106,7 +118,16 @@ const EVERY_KIND: Message[] = [ { kind: "ack" }, { kind: "ack", origin: 1 }, { kind: "closed", origin: 300 }, - { kind: "hold", origin: 2, hold: 0, type: "t", version: 1, fingerprint: "0123456789abcdef", params: "{}" }, + { + kind: "hold", + origin: 2, + hold: 0, + type: "t", + version: 1, + fingerprint: "0123456789abcdef", + params: "{}", + address: server, + }, { kind: "release", origin: 2, hold: 3 }, { kind: "call", origin: 2, call: 1, hold: 3, method: 0, args: "[1]" }, { kind: "abort", origin: 2, call: 1 }, @@ -130,10 +151,20 @@ const EVERY_KIND: Message[] = [ version: 1, fingerprint: "0123456789abcdef", params: "{}", + address: server, chain: [incarnation, incarnation], limit: 65_536, }, - { kind: "hold", hold: 4, type: "t", version: 1, fingerprint: "0123456789abcdef", params: "{}", limit: 0 }, + { + kind: "hold", + hold: 4, + type: "t", + version: 1, + fingerprint: "0123456789abcdef", + params: "{}", + address: server, + limit: 0, + }, { kind: "revoke", revocation: { grant: "g1" } }, { kind: "ping" }, { kind: "pong" }, @@ -181,7 +212,15 @@ forEachCodec(() => { const codec = testCodec(); const deflated = () => new DeflatedPart(new Uint8Array([0, 1, 2, 255])); const messages: Message[] = [ - { kind: "hold", hold: 0, type: "t", version: 1, fingerprint: "0123456789abcdef", params: deflated() }, + { + kind: "hold", + hold: 0, + type: "t", + version: 1, + fingerprint: "0123456789abcdef", + params: deflated(), + address: server, + }, { kind: "call", call: 0, hold: 0, method: 0, args: deflated() }, { kind: "result", call: 0, value: deflated(), segments: [new Uint8Array([7])] }, { @@ -253,6 +292,7 @@ forEachCodec(() => { version: 1, fingerprint: "0123456789abcdef", params: jsonPart({ "\uFEFFkey": "\uFEFFvalue" }), + address: server, }; const decoded = codec.decode(codec.encode(message)) as Extract; expect(plain(decoded)).toEqual(message); @@ -283,9 +323,10 @@ describe("the binary codec", () => { version: 1, fingerprint: "0123456789abcdef", params: "{}", + address: server, }); - // kind, id, type, version, fingerprint, params, the optional fields' flags - expect(hold).toHaveLength(1 + 1 + 2 + 1 + 8 + 3 + 1); + // kind, id, type, version, fingerprint, params, the address's kind and params, the optional fields' flags + expect(hold).toHaveLength(1 + 1 + 2 + 1 + 8 + 3 + 7 + 3 + 1); expect([...binaryCodec.encode({ kind: "ack" })]).toEqual([13]); // An origin: the kind byte's high bit, then the origin as a varint. expect([...binaryCodec.encode({ kind: "ack", origin: 300 })]).toEqual([13 | 0x80, 0xac, 0x02]); @@ -345,6 +386,7 @@ describe("the binary codec", () => { version: 1, fingerprint: "0123456789abcdef", params: "{}", + address: server, }); const forged = hold.slice(); forged[forged.length - 1] = 32; @@ -446,10 +488,12 @@ describe("the text codec", () => { version: 1, fingerprint: "0123456789abcdef", params: "{}", + address: server, } as const; - expect(text(hold)).toBe('["H",0,"t",1,"0123456789abcdef",[0,{}]]'); - expect(text({ ...hold, incarnation })).toBe(`["H",0,"t",1,"0123456789abcdef",[0,{}],null,null,"${incarnation}"]`); - expect(text({ ...hold, limit: 5 })).toBe('["H",0,"t",1,"0123456789abcdef",[0,{}],null,null,null,null,5]'); + const head = '["H",0,"t",1,"0123456789abcdef",[0,{}],["server",{}]'; + expect(text(hold)).toBe(`${head}]`); + expect(text({ ...hold, incarnation })).toBe(`${head},null,"${incarnation}"]`); + expect(text({ ...hold, limit: 5 })).toBe(`${head},null,null,null,5]`); }); it("puts byte segments after the tuple and a newline, their lengths the tuple's last field", () => { @@ -474,6 +518,15 @@ describe("the text codec", () => { expect(() => textCodec.decode(new TextEncoder().encode('["@",1,["@",2,["a"]]]'))).toThrow("wraps one message"); expect(() => textCodec.decode(new TextEncoder().encode('["@",1]'))).toThrow("wraps one message"); }); + + it("refuses a hold without its address: every hold names one", () => { + const decode = (tuple: string) => textCodec.decode(new TextEncoder().encode(tuple)); + expect(() => decode('["H",0,"t",1,"0123456789abcdef",[0,{}]]')).toThrow('a "H" message with 5 fields'); + expect(() => decode('["H",0,"t",1,"0123456789abcdef",[0,{}],null]')).toThrow("expected a list"); + expect(() => decode('["H",0,"t",1,"0123456789abcdef",[0,{}],["server"]]')).toThrow("an address is a pair"); + const hold = { kind: "hold", hold: 0, type: "t", version: 1, fingerprint: "0123456789abcdef", params: "{}" }; + expect(() => checkMessage(hold, context)).toThrow("message schema"); + }); }); describe("describeMessage", () => { @@ -534,6 +587,7 @@ describe("the binary codec's short strings", () => { version: 1, fingerprint: "0123456789abcdef", params: jsonPart(type), + address: server, }; expect(plain(binaryCodec.decode(binaryCodec.encode(message)))).toEqual(message); }); diff --git a/packages/chord/test/next/link/websocket.test.ts b/packages/chord/test/next/link/websocket.test.ts index 821c1468e..c67212b48 100644 --- a/packages/chord/test/next/link/websocket.test.ts +++ b/packages/chord/test/next/link/websocket.test.ts @@ -4,7 +4,7 @@ import { WebSocket, WebSocketServer } from "ws"; import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; import { createHost, defineFacet } from "../../../src/next/host/index.ts"; import { type WebSocketLike, webSocketLink } from "../../../src/next/link/index.ts"; -import { Notes, serveNotes, type TestPrincipal, transported } from "../support.ts"; +import { Notes, Server, serveNotes, type TestPrincipal, transported } from "../support.ts"; /** A socket playing a WebSocket by hand. */ class FakeSocket implements WebSocketLike { @@ -93,7 +93,7 @@ describe("WebSocket links over a fake socket", () => { /** A `ws` server on a free loopback port whose connections a server host accepts, and a URL to it. */ async function loopback(options: { readonly maxPayload?: number } = {}) { - const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const server = createHost({ address: Server(), principal: { kind: "server" }, onError: () => {} }); await server.load([{ name: "@test/app", facet: defineFacet((h) => void serveNotes(h)) }], context); const wss = new WebSocketServer({ port: 0, host: "127.0.0.1", maxPayload: options.maxPayload }); await new Promise((resolve) => wss.once("listening", resolve)); diff --git a/packages/chord/test/next/remote/consumer.test.ts b/packages/chord/test/next/remote/consumer.test.ts index 32e4fb839..5f7005a68 100644 --- a/packages/chord/test/next/remote/consumer.test.ts +++ b/packages/chord/test/next/remote/consumer.test.ts @@ -23,6 +23,8 @@ import { rawParent, rawProvider, rejection, + SERVER_ADDRESS, + Server, serveNotes, testCodec, transported, @@ -287,6 +289,7 @@ forEachCodec(() => { version: Notes.version, fingerprint: Notes.fingerprint, params: jsonPart({ session: "S1" }), + address: SERVER_ADDRESS, }); const [bound] = (await peer.next("bound")).holds; expect(bound!.variant).toBe(EDITOR.variant); @@ -415,6 +418,7 @@ forEachCodec(() => { const Echo = remote("app.echo", { version: 1, params: {}, + hostAddress: () => Server(), variants: { v: { methods: { echo: method(map(int()), map(int())) } } }, }); const { client } = await linked((h) => diff --git a/packages/chord/test/next/remote/hold.test.ts b/packages/chord/test/next/remote/hold.test.ts index ab1c07b8a..03d27f74c 100644 --- a/packages/chord/test/next/remote/hold.test.ts +++ b/packages/chord/test/next/remote/hold.test.ts @@ -20,6 +20,8 @@ import { linked, RawPeer, rawLinked, + SERVER_ADDRESS, + Server, type TestPrincipal, testCodec, transported, @@ -50,6 +52,7 @@ const N = defineState("n", { schema: obj({ n: int() }), version: 1, initial: () const Halves = remote("app.halves", { version: 1, params: {}, + hostAddress: () => Server(), variants: { v: { state: { n: obj({ n: int() }), half: int() }, methods: { set: method(int()), wait: method() } } }, }); @@ -91,18 +94,24 @@ async function halves() { function hold(peer: RawPeer, id: number, Type: { name: string; version: number; fingerprint: string }) { const { name: type, version, fingerprint } = Type; - peer.send({ kind: "hold", hold: id, type, version, fingerprint, params: "{}" }); + peer.send({ kind: "hold", hold: id, type, version, fingerprint, params: "{}", address: SERVER_ADDRESS }); } /** The provider object behind a hold, reached through the host's own lookup (to end it by hand). */ function providerObject(server: unknown) { const host = server as ChordHost; const { name: type, version, fingerprint } = Halves; - return host.serving({ type, version, fingerprint, params: "{}" }, {}, { kind: "owner" }, context); + const request = { type, version, fingerprint, params: "{}", address: SERVER_ADDRESS }; + return host.serving(request, {}, { kind: "owner" }, context); } /** A remote type whose activation never completes: its open waits for its abort. */ -const Pending = remote("app.pending", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); +const Pending = remote("app.pending", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: { n: int() } } }, +}); function servePending(h: Parameters[0]>[0]) { h.serve(Pending, (_, _principal, self) => { @@ -221,7 +230,12 @@ forEachCodec(() => { fakeTimers(); it("a new hold of an object failed transiently is sent the failure at once", async () => { - const Flaky = remote("app.flaky", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + const Flaky = remote("app.flaky", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: { n: int() } } }, + }); const { peer } = await rawLinked((h) => h.serve(Flaky, (_, _principal, self) => { self.open( @@ -309,6 +323,7 @@ forEachCodec(() => { const Slow = local("app.slow", { version: 1, params: {}, members: { n: int() } }); const { server, refs } = await (async () => { const server = createHost({ + address: Server(), principal: { kind: "server" }, readyTimeout: 2_000, onError: () => {}, @@ -343,7 +358,12 @@ forEachCodec(() => { }); /** A remote type whose open throws `offline` (broken when `failing.broken`) while `failing.left` is above 0. */ -const Flaky = remote("app.flaky", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); +const Flaky = remote("app.flaky", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: { n: int() } } }, +}); function serveFlaky(h: Parameters[0]>[0], failing: { left: number; broken: boolean }) { h.serve(Flaky, (_, _principal, self) => { @@ -441,11 +461,13 @@ forEachCodec(() => { const Tally = remote("app.tally", { version: 1, params: {}, + hostAddress: () => Server(), variants: { v: { state: { n: obj({ n: int() }) }, methods: { bump: method() } } }, }); const Fresh = remote("app.fresh", { version: 1, params: {}, + hostAddress: () => Server(), variants: { v: { state: { n: obj({ n: int() }) }, methods: { bump: method() } } }, }); @@ -521,6 +543,7 @@ forEachCodec(() => { const Retiring = remote("app.retiring", { version: 1, params: {}, + hostAddress: () => Server(), variants: { v: { state: { doc: obj({ n: int() }), big: bool(), t: int() }, @@ -536,7 +559,7 @@ const retiredAll = [RETIRING.state.doc!, RETIRING.state.big!, RETIRING.state.t!] /** A server serving `Retiring` to a raw peer that acks only when told; `call` calls a method of hold 0. */ async function retiring() { - const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const server = createHost({ address: Server(), principal: { kind: "server" }, onError: () => {} }); await server.load( [ { @@ -583,7 +606,7 @@ async function retiring() { expect((await peer.next("result")).call).toBe(id); }; const { name: type, version, fingerprint } = Retiring; - peer.send({ kind: "hold", hold: 0, type, version, fingerprint, params: "{}" }); + peer.send({ kind: "hold", hold: 0, type, version, fingerprint, params: "{}", address: SERVER_ADDRESS }); await peer.next("bound"); peer.ack(); return { peer, call }; diff --git a/packages/chord/test/next/remote/host-link.test.ts b/packages/chord/test/next/remote/host-link.test.ts index 7eb0a8551..c5205ea12 100644 --- a/packages/chord/test/next/remote/host-link.test.ts +++ b/packages/chord/test/next/remote/host-link.test.ts @@ -22,6 +22,7 @@ import { rawWorkers, rejection, relayed, + Server, Session, serveNotes, type TestPrincipal, @@ -33,6 +34,7 @@ import { const ServerWho = remote("app.server-who", { version: 1, params: {}, + hostAddress: () => Server(), variants: { v: { state: { who: str() }, methods: { echo: method(str(), str()) } } }, }); @@ -40,7 +42,7 @@ const ServerWho = remote("app.server-who", { const Asker = remote("app.asker", { version: 1, params: { session: str() }, - placement: ({ session }) => Session({ id: session }), + hostAddress: ({ session }) => Session({ id: session }), variants: { v: { state: { who: str() }, methods: { ask: method(str(), str()) } } }, }); @@ -48,7 +50,7 @@ const Asker = remote("app.asker", { const Neighbour = remote("app.neighbour", { version: 1, params: { session: str(), other: str() }, - placement: ({ session }) => Session({ id: session }), + hostAddress: ({ session }) => Session({ id: session }), variants: { v: { state: { count: int() } } }, }); @@ -141,6 +143,7 @@ forEachCodec(() => { const Gateway = remote("app.gateway", { version: 1, params: {}, + hostAddress: () => Server(), variants: { v: { state: { n: int() }, methods: { open: method(str(), int()) } } }, }); // With params: not activated by `load`, which would wait for the raw worker. @@ -342,6 +345,7 @@ async function compressing(options: { }) { const errors: HostError[] = []; const server = createHost({ + address: Server(), principal: { kind: "server" }, compression: options.server, onError: (error) => errors.push(error), diff --git a/packages/chord/test/next/remote/provider.test.ts b/packages/chord/test/next/remote/provider.test.ts index b1be51b9e..a93e54c62 100644 --- a/packages/chord/test/next/remote/provider.test.ts +++ b/packages/chord/test/next/remote/provider.test.ts @@ -21,6 +21,8 @@ import { RAW_PEER, RawPeer, rawLinked, + SERVER_ADDRESS, + Server, serveNotes, type TestPrincipal, testCodec, @@ -39,7 +41,16 @@ function hold( ) { const { name: type, version, fingerprint } = { ...Type, ...override }; const from = origin === undefined ? {} : { origin }; - peer.send({ kind: "hold", ...from, hold: id, type, version, fingerprint, params: jsonPart(params) }); + peer.send({ + kind: "hold", + ...from, + hold: id, + type, + version, + fingerprint, + params: jsonPart(params), + address: SERVER_ADDRESS, + }); } /** Sends a call of the method at index `method` of the bound variant, with `args` (none when left out). */ @@ -50,7 +61,12 @@ function call(peer: RawPeer, id: number, holdId: number, method: number, args?: } /** A remote type kept per id. */ -const Kept = remote("app.kept", { version: 1, params: { id: int() }, variants: { v: { state: { n: int() } } } }); +const Kept = remote("app.kept", { + version: 1, + params: { id: int() }, + hostAddress: () => Server(), + variants: { v: { state: { n: int() } } }, +}); /** The wire's indexes of `Notes`'s reader variant. */ const READER = wireNames(Notes, "reader"); @@ -120,7 +136,11 @@ forEachCodec(() => { }); it("arriving before load resolved wait for it", async () => { - const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const server = createHost({ + address: Server(), + principal: { kind: "server" }, + onError: () => {}, + }); const [a, b] = memoryLinkPair(); server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); const peer = new RawPeer(b); @@ -177,7 +197,12 @@ forEachCodec(() => { }); it("before bound are refused not_ready (race 3)", async () => { - const Slow = remote("app.slow", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + const Slow = remote("app.slow", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: { n: int() } } }, + }); const { peer } = await rawLinked((h) => h.serve(Slow, (_, _principal, self) => { self.open( @@ -333,7 +358,11 @@ forEachCodec(() => { it("a hold released while it waits for load takes nothing once load resolves", async () => { const events: string[] = []; - const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const server = createHost({ + address: Server(), + principal: { kind: "server" }, + onError: () => {}, + }); const [a, b] = memoryLinkPair(); server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); const peer = new RawPeer(b); @@ -354,6 +383,7 @@ forEachCodec(() => { const Slow = remote("app.slow", { version: 1, params: { id: int() }, + hostAddress: () => Server(), variants: { v: { state: { n: int() } } }, }); const aborted: string[] = []; @@ -381,6 +411,7 @@ forEachCodec(() => { const Closing = remote("app.closing", { version: 1, params: { id: int() }, + hostAddress: () => Server(), variants: { v: { state: { n: int() } } }, }); const events: string[] = []; @@ -429,6 +460,7 @@ forEachCodec(() => { const Counter = remote("app.counter", { version: 1, params: { id: int() }, + hostAddress: () => Server(), variants: { v: { state: { n: int(), seen: int() }, methods: { bump: method() } } }, }); const N = defineState("n", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); @@ -488,6 +520,7 @@ forEachCodec(() => { const Grid = remote("app.grid", { version: 1, params: {}, + hostAddress: () => Server(), variants: { v: { state: { grid: obj({ cells: map(int()) }), one: int() }, @@ -520,7 +553,7 @@ function serveGrid(h: FacetHost): void { /** A server serving `Notes` and `Grid` to a raw peer that acks only when told, the link's context counting. */ async function slowConsumer() { const counting = await metered(); - const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const server = createHost({ address: Server(), principal: { kind: "server" }, onError: () => {} }); await server.load( [ { @@ -729,7 +762,12 @@ forEachCodec(() => { }); it("carry constants and failed members, and an empty variant's binding, in a first bound (race 24)", async () => { - const Empty = remote("app.empty", { version: 1, params: {}, variants: { v: {} } }); + const Empty = remote("app.empty", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: {} }, + }); const N = defineState("n", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); const { peer } = await rawLinked((h) => { h.serve(Empty, () => ({ variant: "v", members: {} })); @@ -836,6 +874,7 @@ forEachCodec(() => { const Halves = remote("app.halves", { version: 1, params: {}, + hostAddress: () => Server(), variants: { v: { state: { one: int(), half: int() }, methods: { set: method(int()) } } }, }); @@ -968,7 +1007,15 @@ forEachCodec(() => { /** A hold of `Notes` for S1 as a message. */ function notesHold(id: number): Extract { const { name: type, version, fingerprint } = Notes; - return { kind: "hold", hold: id, type, version, fingerprint, params: jsonPart({ session: "S1" }) }; + return { + kind: "hold", + hold: id, + type, + version, + fingerprint, + params: jsonPart({ session: "S1" }), + address: SERVER_ADDRESS, + }; } const UTF8 = new TextEncoder(); diff --git a/packages/chord/test/next/remote/relay.test.ts b/packages/chord/test/next/remote/relay.test.ts index 701b2adaf..73d5c3b62 100644 --- a/packages/chord/test/next/remote/relay.test.ts +++ b/packages/chord/test/next/remote/relay.test.ts @@ -21,6 +21,7 @@ import { rawWorkers, rejection, relayed, + Server, serveNotes, type TestPrincipal, testCodec, @@ -379,7 +380,11 @@ forEachCodec(() => { describe("a parent link that is down (audit)", () => { it("holds a hold arriving meanwhile on the parent path: it waits loading and binds on the next parent link", async () => { - const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const server = createHost({ + address: Server(), + principal: { kind: "server" }, + onError: () => {}, + }); await server.load( [{ name: "@test/app", facet: defineFacet((h) => void serveNotes(h)) }], context, diff --git a/packages/chord/test/next/remote/remote-object.test.ts b/packages/chord/test/next/remote/remote-object.test.ts index 03baaa01c..eeb3fc7a9 100644 --- a/packages/chord/test/next/remote/remote-object.test.ts +++ b/packages/chord/test/next/remote/remote-object.test.ts @@ -35,6 +35,7 @@ import { rawParent, rawProvider, rejection, + Server, serveNotes, type TestPrincipal, testCodec, @@ -48,7 +49,12 @@ const Counter = defineState("counter", { schema: obj({ n: int() }), version: 1, forEachCodec(() => { describe("a hold over the parent link", () => { it("reject a broken failure at once", async () => { - const Broken = remote("app.broken", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + const Broken = remote("app.broken", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: { n: int() } } }, + }); const { client } = await linked((h) => h.serve(Broken, () => { throw new Error("factory threw"); @@ -197,6 +203,7 @@ forEachCodec(() => { const Flaky = remote("app.flaky", { version: 1, params: { id: int() }, + hostAddress: () => Server(), variants: { v: { state: { n: int() } } }, }); const opens: string[] = []; @@ -250,6 +257,7 @@ forEachCodec(() => { const Halves = remote("app.halves", { version: 1, params: {}, + hostAddress: () => Server(), variants: { v: { state: { half: int() }, methods: { set: method(int()) } } }, }); const { server, client } = await linked((h) => @@ -363,7 +371,12 @@ forEachCodec(() => { fakeTimers(); it("fails at once with the real cause when its load holds it (no wait for a timeout)", async () => { - const Flaky = remote("app.flaky", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + const Flaky = remote("app.flaky", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: { n: int() } } }, + }); const View = local("app.view", { version: 1, params: {}, members: { n: int() } }); const { server, client: first } = await linked((h) => h.serve(Flaky, (_, _principal, self) => { @@ -406,6 +419,7 @@ forEachCodec(() => { const Shifting = remote("app.shifting", { version: 1, params: {}, + hostAddress: () => Server(), variants: { a: { state: { x: int(), y: int() }, methods: { m: method() } }, b: { state: { y: int(), z: int() } }, @@ -592,10 +606,15 @@ forEachCodec(() => { const NotesV2 = remote("app.notes", { version: 2, params: { session: str() }, + hostAddress: () => Server(), variants: { editor: { state: { count: int() } } }, }); const { server, client, links } = await linked((h) => void serveNotes(h)); - const upgraded = createHost({ principal: { kind: "server" }, onError: () => {} }); + const upgraded = createHost({ + address: Server(), + principal: { kind: "server" }, + onError: () => {}, + }); const facet = defineFacet((h) => h.serve(NotesV2, () => ({ variant: "editor", members: { count: constant(0) } })), ); @@ -644,6 +663,7 @@ forEachCodec(() => { const Flaky = remote("app.flaky", { version: 1, params: {}, + hostAddress: () => Server(), variants: { only: { state: {}, methods: { ping: method() } } }, }); let factories = 0; diff --git a/packages/chord/test/next/support.ts b/packages/chord/test/next/support.ts index 9796d534f..06fb043ac 100644 --- a/packages/chord/test/next/support.ts +++ b/packages/chord/test/next/support.ts @@ -401,6 +401,7 @@ export async function linked( const errors = { server: [] as HostError[], client: [] as HostError[] }; const { telemetry, readyTimeout } = options; const server = createHost({ + address: Server(), principal: { kind: "server" }, telemetry, readyTimeout, @@ -547,6 +548,7 @@ export async function rawLinked( ): Promise<{ readonly server: Host; readonly peer: RawPeer; readonly errors: HostError[] }> { const errors: HostError[] = []; const server = createHost({ + address: Server(), principal: { kind: "server" }, readyTimeout: options.readyTimeout, compression: options.compression, @@ -580,8 +582,19 @@ const NOTES_VARIANTS = { reader: { state: { notes: arr(obj({ text: str() })), count: int() }, methods: { wait: method() } }, }; +/** The host kind of the test servers, where every test type but `PlacedNotes` lives. */ +export const Server = hostKind("server"); + +/** `Server()` as a hold names it on the wire: the address of every test type served on the server. */ +export const SERVER_ADDRESS = { kind: "server", params: {} }; + /** The remote type of the link tests: notes per session, an editor variant for owners, a reader one for guests. */ -export const Notes = remote("app.notes", { version: 1, params: { session: str() }, variants: NOTES_VARIANTS }); +export const Notes = remote("app.notes", { + version: 1, + params: { session: str() }, + hostAddress: () => Server(), + variants: NOTES_VARIANTS, +}); /** The kind of the relay tests' workers: one per session. */ export const Session = hostKind("session", { id: str() }); @@ -590,7 +603,7 @@ export const Session = hostKind("session", { id: str() }); export const PlacedNotes = remote("app.placed-notes", { version: 1, params: { session: str() }, - placement: ({ session }) => Session({ id: session }), + hostAddress: ({ session }) => Session({ id: session }), variants: NOTES_VARIANTS, }); @@ -630,7 +643,7 @@ export function serveNotes( }, }; }); - // The two types differ in their name and placement only. + // The two types differ in their name and host address only. host.serve(Type as typeof Notes, ({ session }, principal, self) => { probe.factories.push(`${session} ${principal.kind}`); const store = self.get(NoteStore, { session }); @@ -807,6 +820,7 @@ export async function relayed( const resolves: string[] = []; const { revokedBy, compression = {} } = options; const server = createHost({ + address: Server(), principal: { kind: "server" }, revokedBy: options.serverRevokedBy ?? revokedBy, compression: compression.server, From 373fbcc706686225c5c4f25cbecf95bae3d7811e Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 07:37:56 +0000 Subject: [PATCH 016/175] docs(chord): dependency model: hostAddress --- packages/chord/docs/next/dependencies/book.md | 41 ++++++++++--------- .../chord/docs/next/dependencies/index.html | 39 +++++++++--------- .../chord/docs/next/dependencies/model.md | 14 +++---- 3 files changed, 48 insertions(+), 46 deletions(-) diff --git a/packages/chord/docs/next/dependencies/book.md b/packages/chord/docs/next/dependencies/book.md index 15665a434..1bafbc4a2 100644 --- a/packages/chord/docs/next/dependencies/book.md +++ b/packages/chord/docs/next/dependencies/book.md @@ -166,22 +166,22 @@ and a reload that removes a type sends `gone`. Both go. A request goes to one place, and the address decides which. For a `local` type, the place is this host. For a `remote` type, the consumer computes the address from the params with its own contract's -placement function (design §3): `Notes { session: "S1" }` is placed on `Session { id: "S1" }`. If that is this host's +`hostAddress` function (design §3; `placement` until 2026-10-29): `Notes { session: "S1" }` is placed on `Session { id: "S1" }`. If that is this host's address, the request goes here; otherwise over the route for that kind, or over the parent link. Today the code checks something else first: whether the type is registered here. A type placed on this host but not registered yet is sent over the parent link, which is wrong twice. Sol's example: the server has address `Server`, -the catalog type is placed on `Server`, and no facet serves it yet. Today's `Placement.reference` sends the server's +the catalog type is placed on `Server`, and no facet serves it yet. Today's `Placement.reference` (`host/placement.ts`) sends the server's own request over a parent link the server never had. And when the catalog plugin is installed later, nothing wakes that request, because it was never registered as waiting here. -:::flow wide | Placement by address. The destination follows from the address alone; only then is the provider found there checked. +:::flow wide | The destination by address. The destination follows from the address alone; only then is the provider found there checked. row "Where" accent -box addr "the request's address" "from the consumer's own placement function" accent +box addr "the request's address" "from the consumer's own `hostAddress`" accent box here "here" "a local type, or this host's address" box away "a route or the parent" "every other address" blue row "Then" muted -box check "check the provider here" "served? fitting version? authorize, own placement" +box check "check the provider here" "served? fitting version? authorize, own host address" box hold "hold over the path" "the provider host checks it the same way" blue arrow addr -> here arrow addr -> away @@ -189,13 +189,14 @@ arrow here -> check arrow away -> hold ::: -The rule, from the reviews (Sol B1, Astra F1): placement never looks at what is registered when the type names an +The rule, from the reviews (Sol B1, Astra F1): the destination never depends on what is registered when the type names an address. A request placed here whose provider is missing is **unbound here**, waiting for the name. A version check compares the consumer with the provider actually selected, never with a same-name registration on the consumer's own -host (today `#assertRoute` does that before placement, so worker W1 serving `Notes` v1 rejects a `Notes` v2 request +host (today `#assertRoute` does that before choosing the destination, so worker W1 serving `Notes` v1 rejects a `Notes` v2 request placed on W2, which serves v2). -Every `remote` type declares its placement (owner, 2026-10-29). Until then `remote()` allowed leaving it out, a +Every `remote` type declares its `hostAddress` (owner, 2026-10-29, who also renamed `placement` to it). Until then +`remote()` allowed leaving it out, a leftover of slice 3a, which built remote objects over one link before host kinds existed; such a type went "here if served, otherwise the parent", which needed its own fallback and a re-placement when a tab that loads before it connects gets its first parent link (Opus, Sol). Opus and Fable proposed making it required; the owner agreed. @@ -203,7 +204,7 @@ connects gets its first parent link (Opus, Sol). Opus and Fable proposed making ## What the check finds Once placed, the provider found there is checked: served, a fitting version (the current one or an accepted older one, -with the fingerprint computed from the provider's own history), `authorize`, and the provider's own placement check. +with the fingerprint computed from the provider's own history), `authorize`, and the provider's own host-address check. The request is then **bound** (to an object here, or to a hold over a path), or **unbound** with a reason: | The check finds | The request | Shows | @@ -211,7 +212,7 @@ The request is then **bound** (to an object here, or to a hold over a path), or | a fitting provider | bound | its object's state | | this host has not published its facets yet | unbound `unpublished` | `loading` | | nobody serves the name at its address | unbound `not_served` | `failed { waiting }` | -| no fitting version, a fingerprint that differs at the same version, a placement the provider computes elsewhere | unbound `version` | `failed { waiting }` | +| no fitting version, a fingerprint that differs at the same version, a host address the provider computes elsewhere | unbound `version` | `failed { waiting }` | | the registry is served, but its first `compose`, or the reader's version mapping, threw | unbound `compose` | `failed { waiting }` | | the provider host is installing, and the hold does not fit its current code | unbound `installing`, at the provider | `loading` | | the path has no link (a drop, a route resolving) | waits on the path | `loading` | @@ -598,9 +599,9 @@ has natives (§21), as for any reload. ## A type that moved -A plugin can move a type from the worker to the server by changing its placement. A consumer's request carries the +A plugin can move a type from the worker to the server by changing its `hostAddress`. A consumer's request carries the address its own code computed, so the provider appearing on the server cannot wake it: the request is not waiting -there. The old host answers `not_served`; the consumer's own reload, which brings the new placement, makes a new +there. The old host answers `not_served`; the consumer's own reload, which brings the new `hostAddress`, makes a new request. The model adds no redirect: moving state between hosts is a deliberate migration, not something Chord guesses. ## Rollouts across hosts @@ -1216,7 +1217,7 @@ scenarios against five drafts; the catalog in the chapters before this one is a | R18, the boundary | "live" means every dependency was live at the end of activation | the dependency check is the boundary: after it, a dependency leaving `live` is treated as after live | | `self.get` of a missing `local` type | throws in the factory: `broken` | unbound `not_served`: `failed { waiting }`, woken by the publication that serves it | | a remote hold the provider cannot serve | refused `broken`; the dependent polls by backoff | refused and kept at the provider, re-run at its publication; `failed { waiting }` | -| design §3, placement | "served here" decides "here"; `placement` optional on `remote()` | the address decides; every remote type declares its placement (owner, 2026-10-29) | +| design §3, the destination | "served here" decides "here"; `placement` optional on `remote()` | the address decides; every remote type declares its `hostAddress`, renamed from `placement` (owner, 2026-10-29) | | the start barrier, `pendingLoad` | holds wait for all of `load` | holds wait only for the publication; the barrier is (a) or (b) | | S2, a registry reader newer than its provider | `broken` | `failed { waiting, version }` | | S3, a registry without a value | `transient`, woken at the first value only | `failed { waiting }`, woken at every transition to usable | @@ -1314,14 +1315,14 @@ A needs W's B, B needs S's C; W's hold for C waits for S's whole `load`, which w code (`host.get`, `host.hold`, `host.registry`), a link's peer (`hold`). **It ends only when its holder ends it** (the attempt ends, the call settles, `release()`, a getter gives up, the peer's link or origin closes, revocation, the holding host closes), never because its provider is missing, went away or does not fit. -- **Placement by address** (round 1): a `local` type goes here; a `remote` type with a placement goes to the address - its placement (computed from the consumer's own contract) gives: here if that is this host's address, whether served +- **The destination by address** (round 1): a `local` type goes here; a `remote` type goes to the address its + `hostAddress` (computed from the consumer's own contract) gives: here if that is this host's address, whether served here or not, else over the route for that kind or the parent (design §3). Every `remote` type declares - its placement (owner, 2026-10-29: the optional placement of slice 3a is removed, and with it the fallback "here if + its `hostAddress` (owner, 2026-10-29: the optional `placement` of slice 3a is removed and the field renamed, and with it the fallback "here if served, otherwise the parent" and its re-placement at the first parent link). A hold arriving for an address that is not this host's is relayed or - refused (`invalid/placement`) as today; a relay never captures a hold because it serves the same name. + refused (`invalid/host_address`) as today; a relay never captures a hold because it serves the same name. - **Then the selected provider is checked** (served, version or accepted version and fingerprint, `authorize`, the - provider's own placement check). Version checks never look at a same-name registration elsewhere. + provider's own host-address check). Version checks never look at a same-name registration elsewhere. - **The publication gate:** a request made on this host, or arriving on its links, before the host's first publication (§6) is not placed yet; it shows `loading` and is placed by the publication. The gate lasts only for the synchronous setups, never for activation, so it cannot deadlock. @@ -1332,7 +1333,7 @@ A needs W's B, B needs S's C; W's hold for C waits for S's whole `load`, which w | a fitting provider | bound (to the object here, or to the hold over a path) | | not published yet | unbound `unpublished`: `loading` | | nobody serves the name at its address | unbound `not_served`: `failed { waiting }` | - | no fitting version, a fingerprint that differs at the same version, a placement the provider computes elsewhere (code skew) | unbound `version`: `failed { waiting }` | + | no fitting version, a fingerprint that differs at the same version, a host address the provider computes elsewhere (code skew) | unbound `version`: `failed { waiting }` | | the registry is served, but its first `compose` (or the reader's version mapping) threw | unbound `compose`: `failed { waiting }` | | the provider host is installing a revision and the hold does not fit its current code | unbound `installing` at the provider: `loading` | | the path has no link (a drop, a route resolving) | on the path: `loading` | @@ -1646,7 +1647,7 @@ ring that never existed, `/tmp/model/sol-probe4.log`). R18 (dependency failures `waiting`, no backoff, the deadline paused while parked, the dependency check as boundary, wakes on every change, an ended dependency breaks), S2 and S3 (`waiting`, woken at every usable value), slice 8's -last-good rule (dropped when a contribution leaves), §3's placement table (by address; the publication gate), §6 (the +last-good rule (dropped when a contribution leaves), §3's destination table (by address; the publication gate), §6 (the simultaneous-cycle gap: (A) one narrowed deadline or (B) the probe), §7 (`host.hold`, getters, calls), §21 (no `gone` on removal; a dropped version unbinds only its requests; installs never await activations; rollout without revision fields; the reload handover), the start barrier (R17, W6), `FailureCause` (+ `waiting`), `loading`/`failed` with the diff --git a/packages/chord/docs/next/dependencies/index.html b/packages/chord/docs/next/dependencies/index.html index 656a4f6fa..81c7ae1fe 100644 --- a/packages/chord/docs/next/dependencies/index.html +++ b/packages/chord/docs/next/dependencies/index.html @@ -314,26 +314,27 @@ and a reload that removes a type sends gone. Both go.

    The address decides where

    A request goes to one place, and the address decides which. For a local type, the place is this host. For a remote type, the consumer computes the address from the params with its own contract's -placement function (design §3): Notes { session: "S1" } is placed on Session { id: "S1" }. If that is this host's +hostAddress function (design §3; placement until 2026-10-29): Notes { session: "S1" } is placed on Session { id: "S1" }. If that is this host's address, the request goes here; otherwise over the route for that kind, or over the parent link.

    Today the code checks something else first: whether the type is registered here. A type placed on this host but not registered yet is sent over the parent link, which is wrong twice. Sol's example: the server has address Server, -the catalog type is placed on Server, and no facet serves it yet. Today's Placement.reference sends the server's +the catalog type is placed on Server, and no facet serves it yet. Today's Placement.reference (host/placement.ts) sends the server's own request over a parent link the server never had. And when the catalog plugin is installed later, nothing wakes that request, because it was never registered as waiting here.

    -
    WHERETHENthe request's addressfrom the consumer's own placementfunctionherea local type, or this host's addressa route or the parentevery other addresscheck the provider hereserved? fitting version? authorize,own placementhold over the paththe provider host checks it the sameway
    Figure 2. Placement by address. The destination follows from the address alone; only then is the provider found there checked.
    -

    The rule, from the reviews (Sol B1, Astra F1): placement never looks at what is registered when the type names an +

    WHERETHENthe request's addressfrom the consumer's own `hostAddress`herea local type, or this host's addressa route or the parentevery other addresscheck the provider hereserved? fitting version? authorize,own host addresshold over the paththe provider host checks it the sameway
    Figure 2. The destination by address. The destination follows from the address alone; only then is the provider found there checked.
    +

    The rule, from the reviews (Sol B1, Astra F1): the destination never depends on what is registered when the type names an address. A request placed here whose provider is missing is unbound here, waiting for the name. A version check compares the consumer with the provider actually selected, never with a same-name registration on the consumer's own -host (today #assertRoute does that before placement, so worker W1 serving Notes v1 rejects a Notes v2 request +host (today #assertRoute does that before choosing the destination, so worker W1 serving Notes v1 rejects a Notes v2 request placed on W2, which serves v2).

    -

    Every remote type declares its placement (owner, 2026-10-29). Until then remote() allowed leaving it out, a +

    Every remote type declares its hostAddress (owner, 2026-10-29, who also renamed placement to it). Until then +remote() allowed leaving it out, a leftover of slice 3a, which built remote objects over one link before host kinds existed; such a type went "here if served, otherwise the parent", which needed its own fallback and a re-placement when a tab that loads before it connects gets its first parent link (Opus, Sol). Opus and Fable proposed making it required; the owner agreed.

    What the check finds

    Once placed, the provider found there is checked: served, a fitting version (the current one or an accepted older one, -with the fingerprint computed from the provider's own history), authorize, and the provider's own placement check. +with the fingerprint computed from the provider's own history), authorize, and the provider's own host-address check. The request is then bound (to an object here, or to a hold over a path), or unbound with a reason:

    @@ -359,7 +360,7 @@ The request is then bound (to an object here, or to a hold over - + @@ -721,9 +722,9 @@ object then; Sol and Astra caught it: the object, its documents, its jobs and it fine. Only the requests on version 1 become unbound version. The object's own plugin reload still restarts it if it has natives (§21), as for any reload.

    A type that moved

    -

    A plugin can move a type from the worker to the server by changing its placement. A consumer's request carries the +

    A plugin can move a type from the worker to the server by changing its hostAddress. A consumer's request carries the address its own code computed, so the provider appearing on the server cannot wake it: the request is not waiting -there. The old host answers not_served; the consumer's own reload, which brings the new placement, makes a new +there. The old host answers not_served; the consumer's own reload, which brings the new hostAddress, makes a new request. The model adds no redirect: moving state between hosts is a deliberate migration, not something Chord guesses.

    Rollouts across hosts

    A rollout installs a revision on the server first, then on the workers, then on the tabs (design §21). Between those @@ -1929,9 +1930,9 @@ owner's decisions.

    - + - + @@ -2099,15 +2100,15 @@ code (host.get, host.hold, host.registry) attempt ends, the call settles, release(), a getter gives up, the peer's link or origin closes, revocation, the holding host closes), never because its provider is missing, went away or does not fit.

    -

  • Placement by address (round 1): a local type goes here; a remote type with a placement goes to the address -its placement (computed from the consumer's own contract) gives: here if that is this host's address, whether served +

  • The destination by address (round 1): a local type goes here; a remote type goes to the address its +hostAddress (computed from the consumer's own contract) gives: here if that is this host's address, whether served here or not, else over the route for that kind or the parent (design §3). Every remote type declares -its placement (owner, 2026-10-29: the optional placement of slice 3a is removed, and with it the fallback "here if +its hostAddress (owner, 2026-10-29: the optional placement of slice 3a is removed and the field renamed, and with it the fallback "here if served, otherwise the parent" and its re-placement at the first parent link). A hold arriving for an address that is not this host's is relayed or -refused (invalid/placement) as today; a relay never captures a hold because it serves the same name.

    +refused (invalid/host_address) as today; a relay never captures a hold because it serves the same name.

  • Then the selected provider is checked (served, version or accepted version and fingerprint, authorize, the -provider's own placement check). Version checks never look at a same-name registration elsewhere.

    +provider's own host-address check). Version checks never look at a same-name registration elsewhere.

  • The publication gate: a request made on this host, or arriving on its links, before the host's first publication (§6) is not placed yet; it shows loading and is placed by the publication. The gate lasts only for the @@ -2134,7 +2135,7 @@ synchronous setups, never for activation, so it cannot deadlock.

    • - + @@ -2565,7 +2566,7 @@ once it is live; the run count keeps the object as today.

    12. What changes in the design

    R18 (dependency failures waiting, no backoff, the deadline paused while parked, the dependency check as boundary, wakes on every change, an ended dependency breaks), S2 and S3 (waiting, woken at every usable value), slice 8's -last-good rule (dropped when a contribution leaves), §3's placement table (by address; the publication gate), §6 (the +last-good rule (dropped when a contribution leaves), §3's destination table (by address; the publication gate), §6 (the simultaneous-cycle gap: (A) one narrowed deadline or (B) the probe), §7 (host.hold, getters, calls), §21 (no gone on removal; a dropped version unbinds only its requests; installs never await activations; rollout without revision fields; the reload handover), the start barrier (R17, W6), FailureCause (+ waiting), loading/failed with the diff --git a/packages/chord/docs/next/dependencies/model.md b/packages/chord/docs/next/dependencies/model.md index 2b838eb79..bc3d606c8 100644 --- a/packages/chord/docs/next/dependencies/model.md +++ b/packages/chord/docs/next/dependencies/model.md @@ -61,14 +61,14 @@ A needs W's B, B needs S's C; W's hold for C waits for S's whole `load`, which w code (`host.get`, `host.hold`, `host.registry`), a link's peer (`hold`). **It ends only when its holder ends it** (the attempt ends, the call settles, `release()`, a getter gives up, the peer's link or origin closes, revocation, the holding host closes), never because its provider is missing, went away or does not fit. -- **Placement by address** (round 1): a `local` type goes here; a `remote` type with a placement goes to the address - its placement (computed from the consumer's own contract) gives: here if that is this host's address, whether served +- **The destination by address** (round 1): a `local` type goes here; a `remote` type goes to the address its + `hostAddress` (computed from the consumer's own contract) gives: here if that is this host's address, whether served here or not, else over the route for that kind or the parent (design §3). Every `remote` type declares - its placement (owner, 2026-10-29: the optional placement of slice 3a is removed, and with it the fallback "here if + its `hostAddress` (owner, 2026-10-29: the optional `placement` of slice 3a is removed and the field renamed, and with it the fallback "here if served, otherwise the parent" and its re-placement at the first parent link). A hold arriving for an address that is not this host's is relayed or - refused (`invalid/placement`) as today; a relay never captures a hold because it serves the same name. + refused (`invalid/host_address`) as today; a relay never captures a hold because it serves the same name. - **Then the selected provider is checked** (served, version or accepted version and fingerprint, `authorize`, the - provider's own placement check). Version checks never look at a same-name registration elsewhere. + provider's own host-address check). Version checks never look at a same-name registration elsewhere. - **The publication gate:** a request made on this host, or arriving on its links, before the host's first publication (§6) is not placed yet; it shows `loading` and is placed by the publication. The gate lasts only for the synchronous setups, never for activation, so it cannot deadlock. @@ -79,7 +79,7 @@ A needs W's B, B needs S's C; W's hold for C waits for S's whole `load`, which w | a fitting provider | bound (to the object here, or to the hold over a path) | | not published yet | unbound `unpublished`: `loading` | | nobody serves the name at its address | unbound `not_served`: `failed { waiting }` | - | no fitting version, a fingerprint that differs at the same version, a placement the provider computes elsewhere (code skew) | unbound `version`: `failed { waiting }` | + | no fitting version, a fingerprint that differs at the same version, a host address the provider computes elsewhere (code skew) | unbound `version`: `failed { waiting }` | | the registry is served, but its first `compose` (or the reader's version mapping) threw | unbound `compose`: `failed { waiting }` | | the provider host is installing a revision and the hold does not fit its current code | unbound `installing` at the provider: `loading` | | the path has no link (a drop, a route resolving) | on the path: `loading` | @@ -393,7 +393,7 @@ ring that never existed, `/tmp/model/sol-probe4.log`). R18 (dependency failures `waiting`, no backoff, the deadline paused while parked, the dependency check as boundary, wakes on every change, an ended dependency breaks), S2 and S3 (`waiting`, woken at every usable value), slice 8's -last-good rule (dropped when a contribution leaves), §3's placement table (by address; the publication gate), §6 (the +last-good rule (dropped when a contribution leaves), §3's destination table (by address; the publication gate), §6 (the simultaneous-cycle gap: (A) one narrowed deadline or (B) the probe), §7 (`host.hold`, getters, calls), §21 (no `gone` on removal; a dropped version unbinds only its requests; installs never await activations; rollout without revision fields; the reload handover), the start barrier (R17, W6), `FailureCause` (+ `waiting`), `loading`/`failed` with the From aee933269ba79b84e7b7454097c1d4c426021d82 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 07:44:21 +0000 Subject: [PATCH 017/175] docs(chord): dependency model: hostAddress wording --- packages/chord/docs/next/dependencies/book.md | 13 +++++++------ packages/chord/docs/next/dependencies/index.html | 13 +++++++------ 2 files changed, 14 insertions(+), 12 deletions(-) diff --git a/packages/chord/docs/next/dependencies/book.md b/packages/chord/docs/next/dependencies/book.md index 1bafbc4a2..2269bea41 100644 --- a/packages/chord/docs/next/dependencies/book.md +++ b/packages/chord/docs/next/dependencies/book.md @@ -169,11 +169,12 @@ host. For a `remote` type, the consumer computes the address from the params wit `hostAddress` function (design §3; `placement` until 2026-10-29): `Notes { session: "S1" }` is placed on `Session { id: "S1" }`. If that is this host's address, the request goes here; otherwise over the route for that kind, or over the parent link. -Today the code checks something else first: whether the type is registered here. A type placed on this host but not -registered yet is sent over the parent link, which is wrong twice. Sol's example: the server has address `Server`, -the catalog type is placed on `Server`, and no facet serves it yet. Today's `Placement.reference` (`host/placement.ts`) sends the server's -own request over a parent link the server never had. And when the catalog plugin is installed later, nothing wakes -that request, because it was never registered as waiting here. +Until 2026-10-29 the code checked something else first: whether the type is registered here. A type placed on this host but not +registered yet was sent over the parent link, which is wrong twice. Sol's example: the server has address `Server`, +the catalog type is placed on `Server`, and no facet serves it yet. That code, `Placement.reference` (`host/placement.ts`, now `host/destination.ts`), sent the server's +own request over a parent link the server never had, and when the catalog plugin was installed later, nothing woke +that request, because it was never registered as waiting here. The rename removed the first half; slice 7 builds the +wait. :::flow wide | The destination by address. The destination follows from the address alone; only then is the provider found there checked. row "Where" accent @@ -1076,7 +1077,7 @@ The Rule column cites sections of the working document, reproduced in appendix C | # | Scenario | Under the model | Today | Rule | | --- | --- | --- | --- | --- | | 1 | `self.get(NoteStore)` in `Notes { S1 }`'s factory on W1; `NoteStore` is served and activating. | `Notes` makes its request, it binds to the `NoteStore` object, `Notes` parks `loading`. `NoteStore` goes live: a target state change wakes `Notes`, which opens its natives and goes live. No timer ran. | The same wait, bounded by `readyTimeout`; past it `Notes` fails `transient` and backs off (`attempt.ts` `waitFor`). | §3.1, §3.4, §4 | -| 2 | `self.get(Catalog)` from W1's `ForwardStore`; the parent link is up, S has published, `Catalog` is served there. | The request goes over the parent (placement `Server()`); S binds the hold; `bound` arrives; `ForwardStore` was `loading`, now goes on and live. | The same, after S's whole barrier (`Hold.start` awaits `pendingLoad`). | §1, §2 | +| 2 | `self.get(Catalog)` from W1's `ForwardStore`; the parent link is up, S has published, `Catalog` is served there. | The request goes over the parent (`hostAddress: () => Server()`); S binds the hold; `bound` arrives; `ForwardStore` was `loading`, now goes on and live. | The same, after S's whole barrier (`Hold.start` awaits `pendingLoad`). | §1, §2 | | 3 | The same, but `Catalog`'s plugin is not installed on S. | S indexes the hold `not_served` under "p2.catalog" and answers `failed { waiting, not_served }`; `ForwardStore`'s reference shows `failed { waiting }`, the object parks `failed { waiting, "waiting for p2.catalog: not served" }`. A reload on S serving `Catalog` re-runs the hold: `loading`, `bound`; the reference goes live, the dependent wakes. No backoff, no new hold. | The hold is refused `broken`; `ForwardStore` fails `transient` and re-holds every 1-60 s, refused each time (`remote/hold.ts` `#refuse`, `object.ts` backoff). | §1, §3.1, §4, §5 | | 4 | `self.get(SessionInfo)` on W1 when the core facet is missing (same host, not served). | The request is unbound `not_served` on W1 itself; the dependent parks `failed { waiting }`; a reload serving the type binds it and wakes the dependent. | `#assertRoute` throws in the factory: the object is `broken` until `retry()` or a reload (`host.ts` `reference`). | §1, §3.1, §4 | | 5 | A tab's `NotesView` declares `Notes { S1 }`; S routes `Session`, `resolve` forks W1, W1 connects and publishes. | The hold waits at T on its path (`no_link` while S resolves: `loading`), is relayed to W1, waits there `unpublished` until W1's setups, then binds; `bound` comes back through the relay. `NotesView` was `loading` the whole time and goes live. | The same chain of waits, each bounded by `readyTimeout`; a slow worker start past it fails the tab object `transient` and polls. | §1, §6.2 | diff --git a/packages/chord/docs/next/dependencies/index.html b/packages/chord/docs/next/dependencies/index.html index 81c7ae1fe..cefc248a9 100644 --- a/packages/chord/docs/next/dependencies/index.html +++ b/packages/chord/docs/next/dependencies/index.html @@ -316,11 +316,12 @@ and a reload that removes a type sends gone. Both go.

    host. For a remote type, the consumer computes the address from the params with its own contract's hostAddress function (design §3; placement until 2026-10-29): Notes { session: "S1" } is placed on Session { id: "S1" }. If that is this host's address, the request goes here; otherwise over the route for that kind, or over the parent link.

    -

    Today the code checks something else first: whether the type is registered here. A type placed on this host but not -registered yet is sent over the parent link, which is wrong twice. Sol's example: the server has address Server, -the catalog type is placed on Server, and no facet serves it yet. Today's Placement.reference (host/placement.ts) sends the server's -own request over a parent link the server never had. And when the catalog plugin is installed later, nothing wakes -that request, because it was never registered as waiting here.

    +

    Until 2026-10-29 the code checked something else first: whether the type is registered here. A type placed on this host but not +registered yet was sent over the parent link, which is wrong twice. Sol's example: the server has address Server, +the catalog type is placed on Server, and no facet serves it yet. That code, Placement.reference (host/placement.ts, now host/destination.ts), sent the server's +own request over a parent link the server never had, and when the catalog plugin was installed later, nothing woke +that request, because it was never registered as waiting here. The rename removed the first half; slice 7 builds the +wait.

    WHERETHENthe request's addressfrom the consumer's own `hostAddress`herea local type, or this host's addressa route or the parentevery other addresscheck the provider hereserved? fitting version? authorize,own host addresshold over the paththe provider host checks it the sameway
    Figure 2. The destination by address. The destination follows from the address alone; only then is the provider found there checked.

    The rule, from the reviews (Sol B1, Astra F1): the destination never depends on what is registered when the type names an address. A request placed here whose provider is missing is unbound here, waiting for the name. A version check @@ -1358,7 +1359,7 @@ does is needed to recover; and an ended holder leaves nothing behind in any inde

    - + From a54f250fb40ba9664c62fb013b4aeede70fbe4df Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 08:00:54 +0000 Subject: [PATCH 018/175] docs(chord): decision: getters wait through waiting until their readyTimeout --- packages/chord/docs/next/decisions.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index e87d429d7..065822097 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -1771,6 +1771,14 @@ validation and copies the v2 kernel added around Delta. setups a registry without a value cannot get one before a reload, so a reader then fails at once `transient` (`unavailable/undeclared`) and is retried when the registry gets its first value (R18), as `self.get` of a type not served here fails at once (refined 2026-10-27 after the second spec review: no `readyTimeout` wait at start); once live, a later failure of the registry changes nothing (last good value). +- **Getters wait through `waiting` (owner, 2026-10-29: "(b) with a default seems fine; UI stays responsive anyways + because it usually holds instead of get"; dependency model, decision 2):** `host.get`, `ctx.get` and `host.registry` + wait while the reference is `loading`, `failed { transient }` or `failed { waiting }`, bounded by their + `readyTimeout` (default 60 s; callers choose their own), and reject at the deadline with the last cause; only + `failed { broken }` rejects at once. Calls stay as they are: on a `loading` reference they park, bounded; on a + `failed` one (any cause) they reject at once. A UI holds (`host.hold`) and shows `ref.state`; code that needs the + object now gets it. Reviewers: Fable (b, calls included), Opus (a, plus a `whenLive` helper), Sol leaning (c, callers + park until their own context aborts), Astra pending. - **Only hosts with an address serve remote types (owner, 2026-10-29: "b"):** a host serving a `remote` type needs an address and a principal; `load` refuses a facet that serves one on a host without an address (until now a host with a principal but no address served remote types and placed every one of them "here"). So "here" is decided by the From 249f24bfc5008244493272b3736b3591aaf4956a Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 08:03:10 +0000 Subject: [PATCH 019/175] docs(chord): getters decision: reviewer votes --- packages/chord/docs/next/decisions.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 065822097..b05ae7047 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -1777,8 +1777,10 @@ validation and copies the v2 kernel added around Delta. `readyTimeout` (default 60 s; callers choose their own), and reject at the deadline with the last cause; only `failed { broken }` rejects at once. Calls stay as they are: on a `loading` reference they park, bounded; on a `failed` one (any cause) they reject at once. A UI holds (`host.hold`) and shows `ref.state`; code that needs the - object now gets it. Reviewers: Fable (b, calls included), Opus (a, plus a `whenLive` helper), Sol leaning (c, callers - park until their own context aborts), Astra pending. + object now gets it. Reviewers (`/tmp/model/stw-*.md`): Fable (b, calls included), Opus (a, plus a `whenLive` + helper), Sol and Astra (c: callers park until their own context aborts, with caller deadlines and caps on pending + calls; (b)'s default deadline is that cap). Sol's p2 note: cleanup in a `finally` after an abort needs its own + bounded context, not the aborted one. - **Only hosts with an address serve remote types (owner, 2026-10-29: "b"):** a host serving a `remote` type needs an address and a principal; `load` refuses a facet that serves one on a host without an address (until now a host with a principal but no address served remote types and placed every one of them "here"). So "here" is decided by the From 6fed3cfb99fbe22540030a4d0f099a873ee8e87a Mon Sep 17 00:00:00 2001 From: "robopipi[bot]" <336283572+robopipi[bot]@users.noreply.github.com> Date: Thu, 29 Oct 2026 09:24:53 +0100 Subject: [PATCH 020/175] fix(coding-agent): let the SDK stream resolve auth for summaries (#11556) In sessions created by `createAgentSession`, compaction, branch summaries and bug-report summaries resolved the stored OAuth credential in `AgentSession._getSummarizationRequestAuth`, then passed the access token on as `options.apiKey`. The SDK's stream function calls `modelRuntime.streamSimple`, which resolves auth again. Because it was given an explicit `apiKey`, `resolveProviderAuth` took the API-key override branch instead of using stored OAuth. Built-in providers accept the token either way. A provider whose API-key path differs from its OAuth path, such as one that refuses explicit keys, failed every summary while normal prompts worked. `createAgentSession` now marks its stream function with `markRuntimeStreamFunction`. This records it in a module-level `WeakMap` keyed by the function, mapped to its `ModelRuntime`. `_getSummarizationRequestAuth` skips pre-resolution when the agent's stream function is marked for the session's own runtime, so that runtime resolves auth once, as it does for normal prompts. The check runs after virtual-model routing. Bare `streamSimple` and custom `streamFn`s still receive resolved credentials, so the fixes for #6324 and #7579 are unchanged. If a caller replaces `agent.streamFunction`, the old behavior applies. **Testing** - Added `test/suite/regressions/11520-sdk-summary-oauth-auth.test.ts`. It builds an SDK session on a provider that has stored OAuth and also an API-key method, runs a branch summary, and asserts two things: no `getAuth` call carries an `apiKey` override, and the provider receives the OAuth access token. It fails without the change and passes with it. - `npm run check` passes and `./test.sh` passes. I also ran the related tests: 6324 branch-summary auth, compaction model overrides, auto-compaction cancellation, SDK stream options, virtual models, and the compaction and branch-summary tests. - Manual reproduction: an SDK script with an `openai` OAuth credential that logs `ModelRuntime.getAuth` during `navigateTree(..., { summarize: true })`. Before the change there were two calls, the second with `apiKey = `. After the change there is one call, with no override. Fixes #11520 Requested by @badlogic Co-authored-by: triagent --- packages/coding-agent/CHANGELOG.md | 1 + .../coding-agent/src/core/agent-session.ts | 16 +++ packages/coding-agent/src/core/sdk.ts | 4 +- .../11520-sdk-summary-oauth-auth.test.ts | 122 ++++++++++++++++++ 4 files changed, 142 insertions(+), 1 deletion(-) create mode 100644 packages/coding-agent/test/suite/regressions/11520-sdk-summary-oauth-auth.test.ts diff --git a/packages/coding-agent/CHANGELOG.md b/packages/coding-agent/CHANGELOG.md index 6f09e904e..7133939a1 100644 --- a/packages/coding-agent/CHANGELOG.md +++ b/packages/coding-agent/CHANGELOG.md @@ -15,6 +15,7 @@ ### Fixed +- Fixed compaction, branch summaries, and bug-report summaries in SDK sessions passing the stored OAuth token back to the model runtime as an explicit API key. Provider auth that treats explicit keys differently from stored OAuth now sees the OAuth login, as normal prompts do ([#11520](https://github.com/earendil-works/pi/issues/11520)) - Fixed a redundant system prompt update (tools, rules) when an extension set the same active tools in a different order, for example a reversed `pi.setActiveTools()` list. Tools that stay active now keep their order and newly activated tools are appended ([#11495](https://github.com/earendil-works/pi/issues/11495)) - Fixed `pi.exec()` crashing pi with `RangeError: Invalid string length` when a command printed more output than fits in a string. The default output cap is the smaller of 256 MiB and the runtime string limit; past the cap, the command is terminated and the result has `truncated: true` and a nonzero exit code ([#11360](https://github.com/earendil-works/pi/issues/11360)) - Fixed `pi.exec()` never returning when a command ignored SIGTERM after a timeout or abort. pi now sends SIGKILL 5 seconds later, as intended diff --git a/packages/coding-agent/src/core/agent-session.ts b/packages/coding-agent/src/core/agent-session.ts index c1529f019..dbe36c365 100644 --- a/packages/coding-agent/src/core/agent-session.ts +++ b/packages/coding-agent/src/core/agent-session.ts @@ -29,6 +29,7 @@ import { type BeforeToolCallResult, type PrepareNextTurnContext, runToolCall, + type StreamFn, type ThinkingLevel, } from "@earendil-works/pi-agent-core"; import { contentText, getCurrentSystemMessage, getCurrentTools, retryDelayMs } from "@earendil-works/pi-ai"; @@ -158,6 +159,18 @@ import { type VirtualModelStateData, } from "./virtual-models.ts"; +/** + * Stream functions that resolve auth through a given ModelRuntime themselves, such as the one + * createAgentSession installs. Summaries sent through them must not pass pre-resolved credentials: + * the runtime would take them as an explicit API key instead of the stored OAuth login (#11520). + */ +const runtimeStreamFunctions = new WeakMap(); + +/** Mark `streamFn` as resolving auth through `modelRuntime` on its own. */ +export function markRuntimeStreamFunction(streamFn: StreamFn, modelRuntime: ModelRuntime): void { + runtimeStreamFunctions.set(streamFn, modelRuntime); +} + // ============================================================================ // Skill Block Parsing // ============================================================================ @@ -590,6 +603,9 @@ export class AgentSession { if (this.agent.streamFunction === streamSimple) { return { ...(await this._getRequiredRequestAuth(model, signal)), thinkingLevel }; } + if (runtimeStreamFunctions.get(this.agent.streamFunction) === this._modelRuntime) { + return { model, thinkingLevel }; + } try { const result = await this._modelRuntime.getAuth(model, { signal }); diff --git a/packages/coding-agent/src/core/sdk.ts b/packages/coding-agent/src/core/sdk.ts index 0bf874fff..dbab1d1d9 100644 --- a/packages/coding-agent/src/core/sdk.ts +++ b/packages/coding-agent/src/core/sdk.ts @@ -4,7 +4,7 @@ import type { ModelsSimpleStreamOptions } from "@earendil-works/pi-ai"; import { clampThinkingLevel, type Message, type Model, streamSimple } from "@earendil-works/pi-ai/compat"; import { getAgentDir } from "../config.ts"; import { resolvePath } from "../utils/paths.ts"; -import { AgentSession } from "./agent-session.ts"; +import { AgentSession, markRuntimeStreamFunction } from "./agent-session.ts"; import { formatNoModelsAvailableMessage } from "./auth-guidance.ts"; import { CacheWarmer } from "./cache-warmer.ts"; import { getClientIdentity } from "./client-identity.ts"; @@ -436,6 +436,8 @@ export async function createAgentSession(options: CreateAgentSessionOptions = {} thinkingBudgets: settingsManager.getThinkingBudgets(), maxRetryDelayMs: settingsManager.getProviderRetrySettings().maxRetryDelayMs, }); + // This stream resolves auth through modelRuntime, so summaries must not pre-resolve it (#11520). + markRuntimeStreamFunction(agent.streamFunction, modelRuntime); // Restore missing settings metadata for older sessions. if (hasExistingSession) { diff --git a/packages/coding-agent/test/suite/regressions/11520-sdk-summary-oauth-auth.test.ts b/packages/coding-agent/test/suite/regressions/11520-sdk-summary-oauth-auth.test.ts new file mode 100644 index 000000000..02639a701 --- /dev/null +++ b/packages/coding-agent/test/suite/regressions/11520-sdk-summary-oauth-auth.test.ts @@ -0,0 +1,122 @@ +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + type Api, + createAssistantMessageEventStream, + type Model, + type SimpleStreamOptions, +} from "@earendil-works/pi-ai"; +import { afterEach, describe, expect, it } from "vitest"; +import { AuthStorage } from "../../../src/core/auth-storage.ts"; +import { ModelRuntime } from "../../../src/core/model-runtime.ts"; +import { createAgentSession } from "../../../src/core/sdk.ts"; +import { SessionManager } from "../../../src/core/session-manager.ts"; +import { SettingsManager } from "../../../src/core/settings-manager.ts"; +import { assistantMsg, createTestResourceLoader, userMsg } from "../../utilities.ts"; + +// Issue #11520: summaries in SDK sessions passed the resolved OAuth token back to the +// ModelRuntime as an explicit API key, so provider auth took the API-key override branch. +describe("issue #11520 SDK summary OAuth auth", () => { + const tempDirs: string[] = []; + + afterEach(() => { + while (tempDirs.length > 0) rmSync(tempDirs.pop()!, { recursive: true, force: true }); + }); + + it("lets the runtime resolve stored OAuth for branch summaries", async () => { + const cwd = mkdtempSync(join(tmpdir(), "pi-11520-")); + tempDirs.push(cwd); + const credentials = AuthStorage.inMemory({ + "oauth-only-provider": { + type: "oauth", + access: "oauth-access", + refresh: "refresh", + expires: Date.now() + 3_600_000, + }, + }); + const modelRuntime = await ModelRuntime.create({ credentials, modelsPath: null, allowModelNetwork: false }); + const model: Model = { + id: "oauth-model", + name: "OAuth Model", + api: "openai-completions", + provider: "oauth-only-provider", + baseUrl: "https://example.test/v1", + reasoning: false, + input: ["text"], + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, + contextWindow: 128000, + maxTokens: 4096, + }; + const requests: SimpleStreamOptions[] = []; + modelRuntime.registerProvider(model.provider, { + name: "OAuth Only", + baseUrl: model.baseUrl, + api: model.api, + // With an API key method as well, an explicit key would bypass the stored OAuth login. + apiKey: "UNUSED_ENV_KEY", + oauth: { + name: "Subscription", + login: async () => ({ access: "oauth-access", refresh: "refresh", expires: Date.now() + 60_000 }), + refreshToken: async (credential) => credential, + getApiKey: (credential) => credential.access, + }, + models: [model], + streamSimple: (requestModel, _context, options) => { + requests.push(options ?? {}); + const stream = createAssistantMessageEventStream(); + stream.end({ + role: "assistant", + content: [{ type: "text", text: "branch summary text" }], + api: requestModel.api, + provider: requestModel.provider, + model: requestModel.id, + usage: { + input: 1, + output: 1, + cacheRead: 0, + cacheWrite: 0, + totalTokens: 2, + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, + }, + stopReason: "stop", + timestamp: Date.now(), + }); + return stream; + }, + }); + const authOverrides: Array = []; + const getAuth = modelRuntime.getAuth.bind(modelRuntime); + modelRuntime.getAuth = (async (...args: Parameters) => { + authOverrides.push(args[1]?.apiKey); + return getAuth(...args); + }) as ModelRuntime["getAuth"]; + + const sessionManager = SessionManager.inMemory(cwd); + const { session } = await createAgentSession({ + cwd, + agentDir: cwd, + model, + modelRuntime, + settingsManager: SettingsManager.inMemory(), + sessionManager, + resourceLoader: createTestResourceLoader(), + }); + try { + const targetId = sessionManager.appendMessage(userMsg("first branch")); + sessionManager.appendMessage(assistantMsg("first reply")); + sessionManager.appendMessage(userMsg("abandoned branch work")); + sessionManager.appendMessage(assistantMsg("abandoned reply")); + + const result = await session.navigateTree(targetId, { summarize: true }); + + expect(result.summaryEntry?.summary).toContain("branch summary text"); + expect(requests).toHaveLength(1); + expect(requests[0].apiKey).toBe("oauth-access"); + expect(authOverrides.length).toBeGreaterThan(0); + expect(authOverrides.every((apiKey) => apiKey === undefined)).toBe(true); + } finally { + session.dispose(); + } + }); +}); From a5609aae81ea44439f5cb695d623c914c2f71bd8 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 08:31:15 +0000 Subject: [PATCH 021/175] docs(chord): decision: no cycle probe; parked_long diagnostic and authoring rules --- packages/chord/docs/next/decisions.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index b05ae7047..85c34607d 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -1771,6 +1771,21 @@ validation and copies the v2 kernel added around Delta. setups a registry without a value cannot get one before a reload, so a reader then fails at once `transient` (`unavailable/undeclared`) and is retried when the registry gets its first value (R18), as `self.get` of a type not served here fails at once (refined 2026-10-27 after the second spec review: no `readyTimeout` wait at start); once live, a later failure of the registry changes nothing (last good value). +- **Cycles across hosts: no probe, a diagnostic (owner, 2026-10-29: "yes"; dependency model, decision 6):** cycles stay + found as today: on one host by the walk over waiting edges at `self.get` (exact), across hosts by the activation chain + in holds (objects that start one after another). The remaining gap, a ring across hosts whose objects start at the + same moment, is not detected: under the dependency model, which has no dependency deadline, such a ring stays parked. + It is made visible: an attempt parked longer than its `readyTimeout` on a remote attempt that is itself only + `loading` is reported once per parking episode as `HostError` `parked_long { object, waitingFor, since }`; nothing + fails or retries. A cycle is a plugin bug (owner: "if no plugin bug, we don't need it"); the guide's rules for plugin + authors and agents: (1) a factory declares only what the object needs, never a type that needs it back; (2) + back-references and occasional lookups go into methods and routines (`ctx.get`); (3) data flows up through state and + registries, not mutual `self.get`; (4) host-lifetime objects on the server never declare worker types in their + factory. Rejected: the probe (edge-chasing searches with a check along the recorded trail, `/tmp/model/review*`, + seven review rounds; complete only with a re-search timer or a full search lifecycle, the most intricate code in + Chord for a plugin bug) and the slice 4b timeout narrowed (it polls chains across hosts again, misses a ring of + waiting attempts, and retries in lockstep without finding the ring). Supersedes slice 4b review decision (2)'s + timeout once the dependency model is built (slice 7); folded into design §6 with it. - **Getters wait through `waiting` (owner, 2026-10-29: "(b) with a default seems fine; UI stays responsive anyways because it usually holds instead of get"; dependency model, decision 2):** `host.get`, `ctx.get` and `host.registry` wait while the reference is `loading`, `failed { transient }` or `failed { waiting }`, bounded by their From 39b16b2281d6f76d33b4d96abf100ca452e89480 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 08:31:59 +0000 Subject: [PATCH 022/175] docs(chord): the dependency model is decided --- packages/chord/docs/next/decisions.md | 10 ++ .../chord/docs/next/dependencies/model.md | 161 ++++-------------- 2 files changed, 45 insertions(+), 126 deletions(-) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 85c34607d..73b18c6a1 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -1771,6 +1771,16 @@ validation and copies the v2 kernel added around Delta. setups a registry without a value cannot get one before a reload, so a reader then fails at once `transient` (`unavailable/undeclared`) and is retried when the registry gets its first value (R18), as `self.get` of a type not served here fails at once (refined 2026-10-27 after the second spec review: no `readyTimeout` wait at start); once live, a later failure of the registry changes nothing (last good value). +- **The dependency model is decided (owner, 2026-10-29: "just build this"):** `packages/chord/docs/next/dependencies/model.md` + (explainer `book.md`, `index.html`) as decided, with its remaining four points taken as recommended: (1) the barrier + is (a), `load` resolves at the first outcomes of this host's host-lifetime objects (local blockers still in their + first attempt are waited for; waits on other hosts or failed dependencies end the first pass); (3) `waiting` is a new + `FailureCause` kind; (4) a pure swap whose new factory needs a missing dependency parks the object, no overlap of old + and new code; (5) a registry's last good value is kept only while every contribution it was built from is still in + the table. Supersedes R18's dependency deadline and backoff, S2's `broken` and S3's backoff, §7's throw for a missing + local type and getters' rejection, §21's `gone` on removal and its rollout `updating` deadline, the start barrier's + admission gate (holds wait only for the publication). Built as slice 7a (`slice-7a.md`); design.md is folded with + the build. - **Cycles across hosts: no probe, a diagnostic (owner, 2026-10-29: "yes"; dependency model, decision 6):** cycles stay found as today: on one host by the walk over waiting edges at `self.get` (exact), across hosts by the activation chain in holds (objects that start one after another). The remaining gap, a ring across hosts whose objects start at the diff --git a/packages/chord/docs/next/dependencies/model.md b/packages/chord/docs/next/dependencies/model.md index bc3d606c8..5843e7bab 100644 --- a/packages/chord/docs/next/dependencies/model.md +++ b/packages/chord/docs/next/dependencies/model.md @@ -1,4 +1,4 @@ -# Dependency resolution in Chord v2: one model (draft 7, redesign session, 2026-10-29) +# Dependency resolution in Chord v2: one model (decided, 2026-10-29) Status: proposal for the owner, not decided. Built in slice 7 with reload and rollout. Sources: the owner's direction ("park until resolved"; "no difference between local and remote resolution: a link going is like a plugin @@ -178,7 +178,7 @@ through, or recovers by its own backoff. | **a name becomes usable, or an install ends, on a host**: a publication (`load`'s setups, a reload's commit, an install's commit) serves a type, a registry, a version; a registry composes or maps a value at a version; an install ends without a commit (rejected) | the unbound requests indexed under those names; the provider's `installing` holds; a refused hold re-runs its start and sends `loading`, then `bound` or `failed` | No new message kind for wakes: `loading`, `bound`, `failed` and `retry` exist. The wire's `failed` carries the new cause -kind `waiting`; `loading` and a `waiting` `failed` carry the provider's attempt id for the probe (§7). +kind `waiting`. ## 5. Removal, versions, rollout @@ -231,121 +231,33 @@ kind `waiting`; `loading` and a `waiting` `failed` carry the provider's attempt - p2 today resumes its harness without waiting for all host objects (`apps/p2/src/hosts/worker.ts`), because its worker objects that need server types are reachable only after `control.ready()`. -## 7. Cycles +## 7. Cycles (decided 2026-10-29: no probe) -On one host, cycles are found exactly as today: at `self.get`, following the waiting edges of activating and parked -attempts. Across hosts, a hold carries the activation chain it was made in (today), which finds every cycle whose -objects start one after another. What remains is the documented gap (design §6): objects on two hosts that start at -the same moment and need each other. The owner decided on 2026-10-24 (slice 4b review decisions (2)) that this is a -documented limitation that times out and retries, with no probe message. Without dependency deadlines that timeout no -longer exists, so the gap needs one of two answers (**Owner**): +On one host, cycles are found as today: at `self.get`, following the waiting edges of activating and parked attempts +(exact). Across hosts, a hold carries the activation chain it was made in, which finds every ring whose objects start +one after another. The remaining gap, a ring across hosts whose objects start at the same moment, is not detected: +without a dependency deadline it stays parked. It is made visible: an attempt parked longer than its `readyTimeout` on +a remote attempt that is itself only `loading` is reported once per parking episode as `HostError` `parked_long +{ object, waitingFor, since }`; nothing fails or retries. A cycle is a plugin bug; the guide gives plugin authors and +agents four rules: (1) a factory declares only what the object needs, never a type that needs it back; (2) +back-references and occasional lookups go into methods and routines (`ctx.get`); (3) data flows up through state and +registries, not mutual `self.get`; (4) host-lifetime objects on the server never declare worker types in their factory. -- **(A) The 4b decision, narrowed.** One deadline stays: an attempt parked on a dependency held over a link whose target - attempt is known and still activating or parked fails `transient` after its `readyTimeout` and retries with backoff; - its next attempt usually starts after the other side and its chain finds the cycle. It has a cost that decides - against it: the target being *parked* is the common case of a chain across hosts waiting for a missing plugin (a - tab's view on a worker object that waits for an uninstalled type), and those dependents would time out and poll - again, as today, so "local = remote" would not hold across hosts. Narrowing it to targets that are `loading` (not - `failed { waiting }`) avoids that but misses a real ring: A and B on two hosts wait on each other, B also on a - missing X; B is `failed { waiting }`, so A is too; when X appears, both stay `failed { waiting }` on each other, with - no deadline and no chain to find it. And even the timeout does not find what it is meant to (round 6, Fable): an - object's chain is fixed when it is created and its instance id survives retries, so a retried attempt sends the same - hold with the same chain; two simultaneous starts time out and retry in lockstep and are never found. -- **(B) The probe (recommended)**, below: no deadline at all, one new message kind, the most intricate rule set in the - model. Five - review rounds found a counterexample in each of the first four drafts; draft 6's version answers all of them, but it - has not been reviewed again, and slice 7 would have to prove it with tests before the deadline could go. - -### (B) The probe - -Replaces the activation chain in holds. On one host the `self.get` walk stays (it is synchronous and exact: a ring -with no remote edge is found there, where (B1)'s timer does not run; Opus, round 7); the probe serves the rings that -cross a link. It replaces draft 1's bounded chain too -draft 1's bounded chain (a ring of 33 is never found with chains cut at 32; chains keep stale ancestry). Draft 5 replaces the -second round by a check along the recorded path (round 4, Sol: two rounds through different stale paths can report a -ring that never existed, `/tmp/model/sol-probe4.log`). - -1. **Attempts, edges, tokens.** Every attempt has its own id (uuidv7 per attempt: a retry or reload is a new - attempt). A **wait edge** runs from a parked attempt to the parked or activating attempt it waits for, through a - bound request (unbound waits and registry reads are never edges; they are not deadlocks). Each edge carries an **edge - token** (round 5, Astra, Sol): a number from a counter per attempt, never reused, replaced by a fresh one on any - interruption or change of that edge, so two requests to one object are two edges: a new target attempt (also a - local target's retry, without a re-bind), any `loading` on a bound or pending hold (a relay's upstream re-hold - included), an unbinding, a rebinding, a re-hold after a reconnect. An attempt is parked for one interval only (one - that passed its dependency check never parks again) and ids are never reused. A provider tells its consumers the attempt they wait for on - every non-live answer: `loading { hold, attempt }` and a `waiting` `failed { hold, cause, attempt }`. -2. **A search per edge change.** Whenever a parked attempt gets a new edge or a new token on one, it starts a - search: `probe { kind: "search", round, initiator, trail }` along each of its wait edges, `round` the next number of - a counter the attempt keeps, `trail` the list of `(attempt, edge token)` hops it crossed, starting with - the initiator's own. A ring closes at its last - edge, and that edge's source starts the search that finds it. Coalesced per batch. Locally a call on the target - attempt, forwarded through a queue (no recursion); over a link the message `probe { hold, to, ... }`, consumer to - provider, relayed by hold id with the origin. **Probes are honoured on parent and route links and dropped on child - links** (`accept`), whatever `trusted` says. -3. **Forwarding.** An attempt receiving a search drops it if `to` is not its current attempt or it is not parked. If - the initiator is itself, the search returned (4). If the attempt is already in the trail, it drops it (a ring that does not contain the initiator is - found by its own members' searches; Opus). Otherwise it forwards it along every wait edge, appending `(itself, the edge's - token)` to the trail, only if the round number is higher than the one it stored for that - initiator (older or equal rounds are dropped: the initiator superseded them), and stores it. The seen map holds one - number per initiator, up to 256 initiators, and goes with the attempt. -4. **The check.** A returned search is accepted while its initiating attempt is still parked, whatever round it - carries (round 7, Astra: a re-search must not invalidate an older search still under way); the check decides. The - initiator then sends `probe { kind: "check", round, trail }` along the recorded trail: the initiator first verifies its own hop; each hop passes it on only if it is still - that attempt, still parked, still has the recorded edge token, pointing at the next hop's attempt; otherwise the - check dies (also on a `to` mismatch). The check follows the trail by position and bypasses the seen - map. The initiator verifies its own hop again when the check returns. If the check returns, the initiator fails `broken invalid/cycle`, the path named for display from the trail, - and today's propagation breaks the other activating objects on the ring and the dependents off it. 5. **No silent loss.** Dedupe can drop the copy of a search that travelled the ring in favour of a copy whose path - later breaks (Astra, round 5); then the check dies and nothing is left to find the ring. Round 6 tried `dead` - notices back to the initiator; Sol and Astra showed that reliable notices need recovery state per search that - eviction may not drop and that a long-parked attempt would otherwise keep for every initiator that ever passed (a - leak), i.e. a full search lifecycle with cancellation. Two ways (**Owner**, part of decision 6): - - **(B1) Re-search with backoff (recommended).** An attempt parked on a remote attempt starts a fresh search again - after a backoff (1 s doubling to 60 s, reset by an edge change), whether it shows `loading` or `failed { waiting }`; - the timer stops when the attempt leaves parking. No `dead` messages, no recovery state, eviction - only costs work; a new search starts within one backoff period, so every stable ring that crosses a link is - eventually found (no wall-clock bound: delivery is only fair; a ring on one host is the walk's). The timer runs as long as the attempt stays parked on a remote attempt: a view on a worker object that - waits for a missing plugin re-searches every 60 s until a reload (Fable: cheap, but for as long as it waits). The timer sends probes only: it never - fails an object, re-holds or polls a provider. Cost: one probe per parked remote dependent per period, which - crosses the downstream wait graph and dies at its leaves (an unbound wait is not an edge). - - **(B2) Events only.** Each search is owned until it settles: branches report completion or abort back along their - trails, a dropped probe answers `dead { initiator, round }` with a reverse route, an edge records the searches it - carries until they settle, and an initiator that ends or starts a new round cancels its old one along the same - routes. Complete without a timer, bounded by the searches in flight, and the most intricate protocol in Chord; not - yet specified to the reviewers' satisfaction. - Rejected: enumerating every simple trail without dedupe (Sol's correctness reference): no ring copy is ever - suppressed, but work is exponential on chains of diamonds. -- **Why a reported ring existed:** every edge of the trail kept its token from the moment the search crossed it - to the moment the check crossed it; the search crossed all of them before the check started, so at the moment the - check started every edge of the ring existed at once. A token changes on every interruption and is never reused, so an edge cannot - vanish and come back unnoticed. Sol's counterexample fails at the check: `C1` is no longer `B`'s target. -- **Why every ring is found** (under an eventually stable wait graph and fair delivery; endless edge churn can keep - invalidating searches; with rule 5's (B1) or (B2) for a search suppressed behind a path that broke): a ring is closed by its last edge; its source starts a search when the edge appears; every - other edge exists then and every attempt on the ring waits transitively on the source, so none goes live before the - search and the check return. Rings of any length, local and remote mixed. On a DAG searches die at the leaves; on a - diamond the second arrival is dropped. -- **What it does not promise:** (1) a ring that existed when the check started may be broken up by a concurrent reload, - retry, unbinding or revocation just after; it is still reported (it did exist). Its broken objects recover as every - broken object does: `ref.retry()` or a reload of their plugin. (2) Linear work under every schedule: with more than - 256 initiators passing one attempt at once, an evicted entry lets a search be forwarded again, so work can grow with - the number of simple paths (never endlessly: a search never revisits an attempt in its trail). (3) Constant message size: a probe carries its trail, so its size grows with the length of the - path it crossed (a returning search's trail is as long as its ring); no cap cuts it, since the trail is the proof. -- **Cost:** one search per edge change, crossing each downstream edge once, one check per returned search; with (B1), - one more search per backoff period for every attempt parked on a remote attempt, also while an acyclic chain waits - for a missing plugin, plus the trail bytes. The bounded memory is the seen map's; probes in flight are bounded by - the searches started. -- **Transition:** today's dependency deadline for activations waiting across links stays until the probe's tests pass - in slice 7 (long rings, diamonds, a ring closed after a reconnect with unchanged attempt ids, Sol's phantom schedule, - 300 concurrent initiators, edges added and removed, reloads, relays, forged probes); then it goes in the same slice. +Rejected (owner, 2026-10-29): the probe (edge-chasing searches with a check along the recorded trail; seven review +rounds, `/tmp/model/review*-*.md`; complete only with a re-search timer or a full search lifecycle, the most intricate +code in Chord, for a plugin bug) and the slice 4b timeout narrowed (it polls chains across hosts again, misses a ring of +`waiting` attempts, and retries in lockstep without finding the ring). ## 8. Getters, holds, calls: one rule - **`host.hold`** returns a reference at once; only invalid params throw (also before `load`). A missing local type gives `failed { waiting }` after the publication, `loading` before it. -- **`host.get`, `ctx.get` and `host.registry`:** resolve when live; wait while `loading`, or `failed { transient }` - (its own retry is under way), up to their `readyTimeout`; reject at once on `failed { waiting }` or `{ broken }`. On +- **`host.get`, `ctx.get` and `host.registry`** (owner, 2026-10-29, decision 2): resolve when live; wait while `loading`, + `failed { transient }` or `failed { waiting }`, up to their `readyTimeout` (default 60 s, the caller's choice), then + reject with the last cause; reject at once only on `failed { broken }`. On success, rejection or timeout their temporary wait goes; `host.registry` returns the `RegistryValue` with its - ordinary watch ownership (no `release()`). **Owner:** recommended (nobody waits 60 s for a plugin that is not - there; code that wants to wait holds instead), or wait through `waiting` too. + ordinary watch ownership (no `release()`). A UI holds and shows `ref.state`; code that needs the object now gets it, with the + patience it passes. - **Calls** on a `loading` reference park, each bounded by the reference's `readyTimeout` from when it was made; on a `failed` one they reject at once with the cause; sent calls are never resent. @@ -384,8 +296,8 @@ ring that never existed, `/tmp/model/sol-probe4.log`). | `readyTimeout` of a parked call | kept | an action must not run long after it was asked for | | identity backoff 1-60 s | only for the object's own `transient` failures | nothing else says the disk or network is back | | dependency backoff, polling of refused holds | removed | recovery is an event | -| dependency deadline across links | (A): only for a parked attempt whose remote target attempt stays activating or parked; (B): until the probe's tests pass, then removed | the simultaneous-cycle gap | -| probe re-search backoff (B1) | new, only with (B1): 1 s doubling to 60 s while parked on a remote attempt | finds a ring whose search was suppressed; sends probes only, never fails or re-holds | +| dependency deadline across links | removed | the simultaneous-cycle gap is a plugin bug, reported by `parked_long` | +| `parked_long` report | new: once per parking episode after `readyTimeout` on a remote attempt that is only `loading` | a diagnostic; nothing fails or retries | | route backoff, heartbeat, `closeTimeout`, routine and `finish` backoffs, `linger` | kept | not dependency waits | | rollout install timeout (not built) | a watchdog on the install only | never a deadline for dependencies | @@ -394,22 +306,19 @@ ring that never existed, `/tmp/model/sol-probe4.log`). R18 (dependency failures `waiting`, no backoff, the deadline paused while parked, the dependency check as boundary, wakes on every change, an ended dependency breaks), S2 and S3 (`waiting`, woken at every usable value), slice 8's last-good rule (dropped when a contribution leaves), §3's destination table (by address; the publication gate), §6 (the -simultaneous-cycle gap: (A) one narrowed deadline or (B) the probe), §7 (`host.hold`, getters, calls), §21 (no `gone` on +simultaneous-cycle gap: no probe, `parked_long`), §7 (`host.hold`, getters, calls), §21 (no `gone` on removal; a dropped version unbinds only its requests; installs never await activations; rollout without revision -fields; the reload handover), the start barrier (R17, W6), `FailureCause` (+ `waiting`), `loading`/`failed` with the -attempt id, one message kind (`probe`, tagged `search` or `check`). Unchanged: what an object's own code reload does (§21), R18 after live, +fields; the reload handover), the start barrier (R17, W6), `FailureCause` (+ `waiting`), `parked_long`. Unchanged: what an object's own code reload does (§21), R18 after live, `retryOn` outside parking, `self.follow`'s R4, the call window and flow control, revocation, no client providers. -## 13. Open for the owner +## 13. Decisions (owner, 2026-10-29) -1. The barrier: (a) or (b) (§6.3). -2. Getters on `waiting`: reject at once (recommended) or wait up to `readyTimeout` (§8). -3. `waiting` as a new cause kind (recommended) or not (§2). -4. A pure swap whose new factory declares a dependency that is not there: the object parks (its gate closes, holders - see `loading` or `failed { waiting }`, calls park or reject; recommended) or stays on the old code until it - resolves (old and new code would overlap without bound, against §21's "a host is at R or R−1"). -5. **The last good value** (§9): kept only while every contribution it was built from is still in the table (draft; - Astra), or kept as slice 8 decided, with natives in a registry looked up at use through their owner (Sol). -6. **Cycles across hosts** (§7): (B) the probe, with (B1) a re-search backoff for probes only (recommended) or (B2) a - full search lifecycle without any timer; or (A) the slice 4b decision narrowed to one deadline, which polls chains - across hosts again, misses a ring of `waiting` attempts, and retries in lockstep without finding the cycle it is for. +1. The barrier: (a), `load` resolves at the first outcomes of this host's host-lifetime objects (§6.3). +2. Getters wait through `waiting` until their `readyTimeout`; calls on `failed` still reject (§8). +3. `waiting` is a new `FailureCause` kind (§2). +4. A pure swap whose new factory declares a dependency that is not there parks the object (its gate closes; holders + see `loading` or `failed { waiting }`; calls park or reject); no overlap of old and new code (§21). +5. The last good registry value is kept only while every contribution it was built from is still in the table (§9). +6. Cycles across hosts: no probe; `parked_long` and four authoring rules (§7). + +1, 3, 4 and 5 are the recommendations, taken when the owner said "just build this" (2, 6 decided in discussion). From babe30a7e6c185346c85859da30b58183e5964f0 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 08:32:56 +0000 Subject: [PATCH 023/175] docs(chord): slice 7a spec: the dependency model; slice 7 split into 7a and 7b --- packages/chord/docs/next/slice-7a.md | 130 +++++++++++++++++++++++++++ packages/chord/docs/next/slices.md | 5 +- 2 files changed, 133 insertions(+), 2 deletions(-) create mode 100644 packages/chord/docs/next/slice-7a.md diff --git a/packages/chord/docs/next/slice-7a.md b/packages/chord/docs/next/slice-7a.md new file mode 100644 index 000000000..a5612ce33 --- /dev/null +++ b/packages/chord/docs/next/slice-7a.md @@ -0,0 +1,130 @@ +# Slice 7a: the dependency model (spec) + +Status: **ready to build** (owner, 2026-10-29: the model decided, "just build this"). Slice 7 is split: 7a builds the +dependency model on what exists (one `load` per host, links, routes, relays); 7b builds reload and rollout on top of it +(removal, dropped versions, installs, own-code reloads, loader D). + +Read in full before writing code: `dependencies/model.md` (the model as decided; its section numbers are cited below as +M§), `design.md`, `decisions.md` (the entries of 2026-10-28/29: "No client providers", "Every remote type declares its +placement", "`hostAddress` instead of `placement`", "Only hosts with an address serve remote types", "Getters wait +through `waiting`", "Cycles across hosts: no probe, a diagnostic", "The dependency model is decided"), `slices.md` +(slice 7's carry-over: 7a takes none of it unless named here; 7b takes the rest), the explainer +`dependencies/book.md` for the scenarios (its catalogs are test material). + +## 1. Requests and the unbound index (M§1) + +- Every reference is a request with one holder (attempt, call, outside code, a link's peer). It ends only when its + holder ends it; never because its provider is missing or does not fit. +- The destination by address is built (`host/destination.ts`, slice 7a's predecessor commit `5ec241882`). Add: version + checks only against the selected provider (no `#assertRoute` comparison with a same-name registration on the + consumer's host); a local type not served here is no longer a throw at `self.get`/`host.hold`/getters but an unbound + request. +- **The publication gate:** a request made on this host, or a hold arriving on its links, before the host's first + publication is not placed: `loading`, placed by the publication. `pendingLoad` no longer gates `Hold.start` or + `#hold` beyond the publication (it ends the startup deadlock: Sol's probe `/tmp/dep-graph-sol-barrier.log` becomes a + regression test that passes with fake timers never advanced). +- **Unbound reasons** (M§1's table): `unpublished` (loading), `not_served` and `version` (`failed { waiting }`), + `compose` for registries (`failed { waiting }`), `no_link` on a path (loading); `authorize` refused or invalid params + stay `broken`, never indexed. Indexes per host by name (local requests; refused holds that arrived over links, kept as + today's refused state in `remote/hold.ts`); entries removed when the request ends (release, call settled, getter gave + up, `closed { origin }`, link close, revocation). GC tests. + +## 2. States (M§2) + +- `FailureCause` gains `{ kind: "waiting"; error }` (no `retryAt`). The wire's `failed` cause schema carries it; the + pinned list and its test change; `PROTOCOL_VERSION` stays 1. +- The mapping of M§2's table: `loading` for activating, unpublished, no link (and `installing`, 7b); `failed + { waiting }` for not served, version, compose, and a dependent whose dependency failed. +- A remote refusal for not served / version is `failed { waiting }` instead of `broken`; the provider keeps the refused + hold (it does today) and the consumer never retries it by itself (no backoff). + +## 3. Parked attempts (M§3) + +- Order: factory, persisted singleton hydration (as today), requests made; the attempt is **parked** while any + dependency (`self.get` local or remote, `self.registry`, keyed pins) is not live and hydrated. `self.follow` is not a + dependency (its attach is an own step). +- What it shows: `loading` while blockers are only `loading`; `failed { waiting, "waiting for X: " }` (nearest + blocker plus root cause, 500-character rule) when one failed; `broken` when one failed with `invalid/cycle` (off-ring + dependents) or ended. +- **No timer while parked:** no dependency deadline, no identity backoff for dependency failures; the attempt's + `readyTimeout` deadline is paused while parked (as eager migrations pause it) and resumes with the budget left. The + identity backoff stays for the object's own `transient` failures only. +- **The dependency check is the boundary:** once all are live, the own steps run (natives, follow attaches, member + checks, derives) under the deadline; a dependency leaving `live` afterwards is treated as after live (R18): the + attempt goes on. +- **Wakes on every state change** of a dependency (live, loading, failed, a new cause compared by value, ended): + re-evaluate what the object shows, leave parking when all are live; never re-run the factory while parked; coalesce + per object after the batch that caused them; remember changes during own steps or teardown. +- **Retry passes on:** `ref.retry()` on a `failed { waiting }` object retries its `broken` and `transient` blockers hop + by hop (the existing `retry` message), once per operation (an id dedupes diamonds); `waiting` blockers pass it on. + A `retryOn` value call while parked `waiting` does nothing. + +## 4. Wakes (M§4) + +Two events: a target changes state (the requests bound to it and their dependents); a name becomes usable on a host +(in 7a: `load`'s publication serving types and registries, a registry's value turning usable at a version): the +unbound requests indexed under it, and refused holds, which re-run their start and send `loading`, then `bound` or +`failed`. No new message kind. + +## 5. Startup (M§6) + +- `load` runs the setups, publishes them in one batch (types, registries served and composed once, contributions), + wakes after that batch, then activates host-lifetime objects. +- **The barrier, decision (a):** `load` resolves when every host-lifetime object of the load has a first outcome: + live, failed by its own cause, or parked with no blocker that is a local object still without its first outcome + (`local` = executing on this host, keyed pins still loading here included). One `firstOutcome` signal per object, + separate from the readiness dependents wait for; counted once; an object of the cohort that ends before its first + outcome settles as ended; `host.close` settles a waiting `load`. + +## 6. Getters, holds, calls (M§8, decision "Getters wait through `waiting`") + +- `host.hold` returns a reference at once; only invalid params throw (also before `load`); a missing local type gives + `loading` before the publication, `failed { waiting }` after it, before and after `load` alike. +- `host.get`, `ctx.get`, `host.registry`: resolve when live; wait through `loading`, `failed { transient }` and + `failed { waiting }` until their `readyTimeout`, then reject with the last cause and release; reject at once only on + `failed { broken }`. +- Calls: park on `loading` (bounded, as today); reject at once on `failed`; never resent. + +## 7. Registries (M§9) + +A reader's request: bound while R is served and has a usable value at its version; otherwise unbound `not_served`, +`version` (S2's newer or unmapped reader: `waiting`, not `broken`) or `compose` (first compose or the version mapping +threw), `loading` before the publication. Woken at every transition to usable (`#valued`'s latch goes). The last good +value is kept only while every contribution it was built from is still in the table (decision 5): when one leaves, +the last good value is dropped and `.value` throws until `compose` succeeds. + +## 8. Keep-alive, revocation, cycles (M§7, M§10) + +- A request over a link counts once in `host.activity.references` from arrival to end, bound or unbound; + `host.references` lists bound references only. +- Revocation ends the requests of a revoked principal, bound and unbound, at every hop. +- `gone` only for revocation (a closing provider host sends nothing, as today). +- **Cycles:** the walk at `self.get` on one host, through parked attempts too; the activation chains across hosts as + today. Remove the dependency deadline (no fallback). Add `HostError` `parked_long { object, waitingFor, since }`: + once per parking episode, when an attempt has been parked longer than its `readyTimeout` on a remote attempt that is + itself only `loading`. The guide (host README) states the four authoring rules of M§7. + +## 9. Not in 7a (7b) + +Removal and re-serving of types (local re-binding of a `Binding`, a `failed` on a bound hold unbinding it, window +slots), a dropped accepted version, `installing` and install ends, own-code reloads and the parked-attempt handover, +pure swaps parking (decision 4), rollout, loader D. 7a's code must leave those paths possible, not build them. + +## 10. Tests and cost + +- One `describe` per mechanism; the four catalogs in `dependencies/book.md` (chapters XIV-XVII) are the scenario list: + every row that 7a's scope reaches becomes a test (memory links; relays; fake timers where a test proves no timer + runs). Must include: the startup deadlock without `readyTimeout`; a hold before the publication placed here, not via + the parent; a missing local type on `self.get`, `host.hold`, `host.get` (waits, then rejects with the cause); a + remote refusal kept and re-run when the provider's late `load` publishes; a parked three-level chain across hosts + woken in order; wakes on every state change with causes updated; retry passing on through a diamond once; the + barrier's first-outcome rule (CodeMode → Harness waits; a remote blocker ends the pass); a cohort object ending + before its outcome; registries' waiting reader, wake after a provider arrives, last good dropped when a contribution + leaves; keep-alive counting unbound holds; revocation of unbound holds; `parked_long` reported once; GC tests for the + indexes and a parked chain released from the top. +- Benchmarks (the bar of "The performance and memory bar"): calls, activation churn, a watched derive, links against + HEAD, pinned, alternating, a HEAD copy at a path of the same length; results in `benchmarks.md`. +- Examples: `31-parked-dependencies.ts` (a missing provider arrives; causes shown; no timers), `32-startup.ts` (two + hosts that would have deadlocked). +- Docs: fold the model into `design.md` (§3, §6, §7, §17, §21's 7a parts) in the same change as the build; mark + `dependencies/model.md` as built for 7a. diff --git a/packages/chord/docs/next/slices.md b/packages/chord/docs/next/slices.md index 2ddb88f60..97883b776 100644 --- a/packages/chord/docs/next/slices.md +++ b/packages/chord/docs/next/slices.md @@ -50,7 +50,8 @@ reload has to reload real registries). | 6a | **The store** | the store (backends, tables, queries, writes, commit queue, outbox, documents engine, migrations, export), persisted documents, `self.table`, keyed watches and retirement | | 6b | **Durable and jobs on the store** | persisted jobs, Durable's `StoreStorage`, the Chord source over Durable's views (`self.follow`) | | 8 | **Registries and `host.root`** | registries, `self.writable`, `retryOn`, Durable's `registryReader`, `host.root` (`chord(fn, context)` and the harness object are p3's base code) | -| 7 | **Reload and rollout** | plugin loading, staging, swaps and restarts, revisions, `accepts` in use | +| 7a | **The dependency model** | requests that outlive providers, parked attempts without timers, wakes, the publication gate, getters, registries as dependencies, `parked_long` (`slice-7a.md`, `dependencies/model.md`) | +| 7b | **Reload and rollout** | plugin loading (loader D), staging, swaps and restarts, removal and re-serving, revisions, `accepts` in use | | 9 | **Telemetry** | spans, events, metrics across hosts | The playground (`apps/chord-playground`) moves to the new code once slice 4 exists. Round-3 cuts (C3-C16) and @@ -162,7 +163,7 @@ deleted (the slice's spec and code are the record). Moved here from `carry-over. - **`map(bytes())` is refused at declaration** ("unsupported keyword `~refine`"): TypeBox's `Record` copies the bytes schema's check as an enumerable key, which the snapshot walk refuses. An L1 bug found in slice 3a, not fixed yet. -### Slice 7: reload and rollout +### Slice 7b: reload and rollout (7a: the dependency model, `slice-7a.md`) - **A native restart drains first** (owner, 2026-10-26): park new calls, await running calls (bounded by `closeTimeout`), then close the natives and restart; a call outliving the drain is cut off by the seal fence. Running From 339847eb14995b738fced67f0b80649d2ad0f52a Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 11:40:57 +0000 Subject: [PATCH 024/175] feat(chord): Chord v2 slice 7a, the dependency model A reference is a request that outlives its provider: a missing provider or a version that does not fit leaves it unbound, waiting for the name (failed { waiting }, a new FailureCause kind), local and remote alike. An attempt parks until its dependencies are live, with no timer and no backoff, woken on every state change of a dependency; its readyTimeout runs only for its own steps. Requests wait only for the host's publication (the startup deadlock is gone); load resolves at the first outcomes of its host-lifetime objects. Getters wait through waiting until their readyTimeout. Registry readers are waiting dependencies, woken at every usable value; a last good value is dropped when a contribution it was built from leaves. Unbound holds count for keep-alive and are revoked; retry passes on to failed blockers once per operation (operation id and visited list on the wire); the dependency deadline is gone and parked_long reports a long wait on a remote attempt. --- packages/chord/docs/next/benchmarks.md | 40 ++ .../chord/docs/next/dependencies/model.md | 10 +- packages/chord/docs/next/design.md | 258 +++++++---- packages/chord/docs/next/slices.md | 10 + packages/chord/src/next/data/failure.ts | 70 ++- packages/chord/src/next/data/internal.ts | 2 +- packages/chord/src/next/host/README.md | 197 ++++++--- packages/chord/src/next/host/attempt.ts | 285 ++++++++---- packages/chord/src/next/host/backoff.ts | 2 +- packages/chord/src/next/host/dependencies.ts | 142 ++++++ packages/chord/src/next/host/destination.ts | 53 +-- packages/chord/src/next/host/host.ts | 351 +++++++++------ packages/chord/src/next/host/internal.ts | 7 +- packages/chord/src/next/host/object.ts | 274 +++++++----- packages/chord/src/next/host/ref-state.ts | 12 +- packages/chord/src/next/host/reference.ts | 159 +++++-- packages/chord/src/next/host/route.ts | 5 + packages/chord/src/next/host/unbound.ts | 62 +++ packages/chord/src/next/link/binary.ts | 35 +- packages/chord/src/next/link/message.ts | 22 +- packages/chord/src/next/link/text.ts | 22 +- packages/chord/src/next/registries/README.md | 32 +- packages/chord/src/next/registries/record.ts | 43 +- packages/chord/src/next/registries/value.ts | 105 ++++- packages/chord/src/next/remote/README.md | 36 +- packages/chord/src/next/remote/consumer.ts | 8 +- packages/chord/src/next/remote/hold.ts | 181 +++++--- packages/chord/src/next/remote/host-link.ts | 6 +- packages/chord/src/next/remote/provider.ts | 18 +- packages/chord/src/next/remote/relay.ts | 6 +- .../chord/src/next/remote/remote-object.ts | 60 ++- packages/chord/test/next/data/failure.test.ts | 66 ++- .../chord/test/next/examples/05-errors.ts | 5 +- .../chord/test/next/examples/07-failures.ts | 3 +- .../next/examples/31-parked-dependencies.ts | 148 +++++++ .../chord/test/next/examples/32-startup.ts | 104 +++++ .../chord/test/next/host/activity.test.ts | 13 + packages/chord/test/next/host/attempt.test.ts | 97 +++- .../chord/test/next/host/dependencies.test.ts | 294 +++++++++++++ packages/chord/test/next/host/host.test.ts | 413 +++++++++++++++++- .../test/next/host/keyed-members.test.ts | 68 ++- packages/chord/test/next/host/object.test.ts | 180 ++++---- .../chord/test/next/host/reference.test.ts | 4 +- .../chord/test/next/host/retry-on.test.ts | 21 + .../chord/test/next/host/revocation.test.ts | 140 +++++- .../chord/test/next/host/routines.test.ts | 3 +- packages/chord/test/next/host/unbound.test.ts | 171 ++++++++ packages/chord/test/next/link/message.test.ts | 41 +- .../chord/test/next/registries/record.test.ts | 29 +- .../chord/test/next/registries/value.test.ts | 173 +++++++- packages/chord/test/next/remote/hold.test.ts | 171 +++++++- .../chord/test/next/remote/provider.test.ts | 16 +- .../test/next/remote/remote-object.test.ts | 94 ++-- .../test/next/store/table-migration.test.ts | 27 +- 54 files changed, 3832 insertions(+), 962 deletions(-) create mode 100644 packages/chord/src/next/host/dependencies.ts create mode 100644 packages/chord/src/next/host/unbound.ts create mode 100644 packages/chord/test/next/examples/31-parked-dependencies.ts create mode 100644 packages/chord/test/next/examples/32-startup.ts create mode 100644 packages/chord/test/next/host/dependencies.test.ts create mode 100644 packages/chord/test/next/host/unbound.test.ts diff --git a/packages/chord/docs/next/benchmarks.md b/packages/chord/docs/next/benchmarks.md index 916c70838..c72316e7e 100644 --- a/packages/chord/docs/next/benchmarks.md +++ b/packages/chord/docs/next/benchmarks.md @@ -633,3 +633,43 @@ p95, and swapping host files moves it without a pattern (code layout, not work o `writable.set` 0.87 µs, the same value 0.011-0.016 µs and 0 B (the seal check now comes first: a set of an ended attempt's writable throws whatever its value); one of 10 contributions set 0.71-0.78 µs, of 100 1.43-1.61 µs (one run in six 3.4 µs on either variant tried: JIT). + +## Slice 7a: the dependency model (2026-10-29) + +Node 24.4.1, one CPU (`taskset -c 6`), runs alternating. HEAD is `babe30a7e`, extracted at a path of the same length +(`/home/badlogic/pi-h7`); the build in the worktree (`/home/badlogic/pi-7a`). + +**Paths under every object** (`test/next/bench/calls.ts`), median per operation over nine bursts, heap allocated per +operation, three alternating runs each, HEAD → build: + +| Case | HEAD | build | +| --- | --- | --- | +| local method call (its ctx) | 2.81-2.82 µs, 6,894 B | 2.80-2.82 µs, 6,894 B | +| commit through a watched derive | 10.43-10.70 µs, 15,310 B | 10.38-10.53 µs, 15,310 B | +| activation churn: get, live, release (an object with params and one dependency) | 58.5-59.9 µs, 43,716-43,719 B | 53.7-55.4 µs, 41,540-41,559 B | +| retained per live object with one dependency (10,000 kept, an ad-hoc probe) | 7,398-7,406 B | 6,931-6,936 B | + +An activation whose dependencies are live at the check does not park: one check of its bindings, then its own steps a +microtask later (as HEAD's wait did). The first build parked every attempt with a dependency (a waiter list, a +`parked_long` record, a microtask per wake even outside parking) and cost 62.0-63.6 µs and 580 B more per activation; +now a wake outside parking is a flag, the first-outcome promise is made only for whoever waits (the start barrier), +and an object keeps no `Set` of waits and no settlement promise per attempt (R18's), hence the lower numbers. + +**Links** (`links.ts`, three runs each): wire sizes, encodes, decodes, heap per message and retained heap per hold +(176.0 → 175.4-176.0 KiB) unchanged. Hold to live over two hops (mean of 100), binary: 1,000-item list 0.781-0.817 → +0.807-0.811 ms, 500 KB document 1.250-1.278 → 1.289-1.310 ms (+1-3%: the hold counted at its arrival, the index +check of the publication); text 1.545-1.580 → 1.521-1.573 and 3.299-3.379 → 3.317-3.393 ms. Commits to frames, +streamed appends, call round trips and reconnects within the runs' spread. + +**Existing benchmarks**, HEAD → build: the derive diamond (`derives.ts`, three runs) 9.4-10.1 → 9.5-9.6 µs median, +p95 19.3-19.9 → 19.3-19.4 µs. Registries (`registries.ts`, three runs): `writable.set` 0.82-0.97 → 0.77-0.87 µs; one of +10 contributions set 0.72-2.71 → 0.70-1.06 µs; of 100 1.50-3.46 → 1.44-3.40 µs (one slow run per side: JIT); +allocation within 20 B. A registry's last good composition keeps the list of contributions it was built from: the +list is replaced (not spliced) when a contributor comes or goes, so a composition costs no copy. + +**Fix round 1** (2026-10-29: revocation over paths, retry operations on the wire, the provider's own host-address +check for same-host remote requests, mapped values' provenance), `calls.ts`, three alternating runs, HEAD → build: +local call 2.81 → 2.79-2.82 µs (6,894 B both), watched derive commit 10.45-10.70 → 10.35-10.47 µs, activation churn +58.6-59.8 → 53.9-54.2 µs (43,716-43,720 → 41,559 B); retained per live object with one dependency 7,399-7,405 → +6,930-6,938 B. The address check runs only for a remote type whose consumer and provider `hostAddress` differ; a +retry's operation id is made per `ref.retry()`, and an object keeps its handled ids only once it was retried. diff --git a/packages/chord/docs/next/dependencies/model.md b/packages/chord/docs/next/dependencies/model.md index 5843e7bab..367c0612c 100644 --- a/packages/chord/docs/next/dependencies/model.md +++ b/packages/chord/docs/next/dependencies/model.md @@ -1,6 +1,9 @@ # Dependency resolution in Chord v2: one model (decided, 2026-10-29) -Status: proposal for the owner, not decided. Built in slice 7 with reload and rollout. Sources: the owner's direction +Status: decided (owner, 2026-10-29); **built for slice 7a** (§1-4 and §6-11 without what installs and reloads add: +§1's `installing`, §4's install-end and reload wakes, §3.9's reload handover and §5's removal, versions and rollout +are slice 7b's; folded into `design.md` §3, §5, §6, §7, §17, §21's start barrier, §22; `slice-7a.md`). The draft +history below is kept as it was. Sources: the owner's direction ("park until resolved"; "no difference between local and remote resolution: a link going is like a plugin unloading"), the reviews (`/tmp/dep-mechanism-*.md`, `/tmp/dep-graph-*.md`, `/tmp/barrier-*.md`, `/tmp/model/review{1,2}-*.md`), the code at pi `358f7d46a`. **Owner** marks an open decision. @@ -138,6 +141,7 @@ through, or recovers by its own backoff. | any `loading`, none failed | `loading` | | any `failed` | `failed { waiting, "waiting for X: " }`, `retryAt` undefined; the message keeps the nearest blocker and the root cause within the 500-character rule | | one failed with `invalid/cycle` (a dependent off the ring) | `broken` with the same error: it would wait on a ring that never heals by itself | + | one failed with `forbidden` (an `authorize` refused it: authority, §3.6) | `broken` with the same error: only `retry()` or a reload helps (slice 7a fix round 1) | | one `ended` (revoked, or its host closed) | `broken` with that error: the request cannot come back; `retry()` or a reload makes new requests (a guard: a dependency ends only with its holder's principal or host, so the dependent ends too; Fable) | 2. **The dependency check is the activation boundary:** once every dependency is `live`, the attempt runs its own @@ -152,7 +156,9 @@ through, or recovers by its own backoff. failed, a new cause compared by value, ended; Opus: otherwise a ring of waiting attempts passes its causes round and round), updating what it shows; it leaves parking when all are live. The factory never re-runs while parked. Wakes are coalesced per object and run after the batch that caused them; a change during the - attempt's own steps or its teardown is remembered. + attempt's own steps or its teardown is remembered. (As built in 7a, with the parent's note 5: a change while the + parked attempt checks or loads its pinned keys is remembered and checked again; after the dependency check, during + the own steps, a change is ignored, §3.2's R18 rule.) 5. **Kept while parked:** its documents (the object's), its declarations and requests, its seal token; no natives. Ended with the object (its last holder; a host-lifetime object when its plugin is removed or the host closes). 6. **`broken` only for the object's own code** (a throwing factory, a derive throwing at activation, hydration, diff --git a/packages/chord/docs/next/design.md b/packages/chord/docs/next/design.md index 66a76765f..3f07b01cc 100644 --- a/packages/chord/docs/next/design.md +++ b/packages/chord/docs/next/design.md @@ -124,23 +124,50 @@ export const Catalog = remote("catalog", { hostAddress: () => Server(), … }); - **`params`** (owner, 2026-10-21; was `input`) select the instance among objects of its type and are part of its identity (section 5). - **The caller computes the address** and sends it in the `hold`; a host that relays routes by that address alone and - never needs the contract or parses params. **The provider computes it again** from the params and refuses an object - that is not its own: broken `invalid/host_address` (misrouted holds are never forwarded). During a rollout the refusal - is handled like a mismatch (provider on another revision: park; same revision: fail; slice 7). -- **Where a reference made on a host, or a hold arriving on one of its links, goes** (first match; slice 4a): + never needs the contract or parses params. **The provider computes it again** from the params: a hold naming this + host whose params the provider's code places elsewhere (code skew) waits unbound `version` (`failed { waiting }`, + `invalid/host_address`, dependency model §1; slice 7a); a hold naming another host's address is relayed, never served + here (misrouted holds are never captured by a relay that serves the same name). +- **Every reference is a request** (dependency model, slice 7a): it has one holder (an attempt, a call, outside code, + a link's peer) and ends only when its holder ends it (release, a call settling, a getter giving up, the peer's link + or origin closing, revocation, the host closing), never because its provider is missing, went away or does not fit. +- **Where a reference made on a host, or a hold arriving on one of its links, goes**, by address alone (slice 4a; + slice 7a: no same-name check on the consumer's host): | Case | Goes | | --- | --- | - | the type is served here and its host address (computed here from the params) is this host's address | here | - | a reference made here (`host.get`/`hold`, `self.get`, `ctx.get`) to a served type placed on another host | on, as if not served here (next rows): worker S1 reaches `Notes { session: "S2" }` through the parent | - | an arriving hold for a served type whose params place it on another host | broken `invalid/host_address`; the address the hold names only routes it | - | the hold names an address of a kind this host has a route for | over that route's link (the address's params checked against the kind, then `authorize`) | + | a `local` type | here | + | a `remote` type whose host address (computed here from the params) is this host's address, served here or not | here | + | an arriving hold naming this host's address | here | + | the address (computed, or named by the hold) is of a kind this host has a route for | over that route's link (the address's params checked against the kind, then `authorize`) | | otherwise (a remote type) | over the parent link; a hold that arrived *on* the parent link is never sent back to it: `unavailable` | | no parent link: the host never had one | `unavailable` | | no parent link: it is down (a drop) | waits on the parent path, `loading`, until the next link (design audit, 2026-10-27) | - A `hostAddress` or `authorize` that throws fails the hold broken with that error; an address whose params fail the - kind's check is broken `invalid/host_address`. + A `hostAddress` or `authorize` that throws fails the request broken with that error; an address whose params fail + the kind's check is broken `invalid/host_address`. +- **The publication gate** (slice 7a): a request made on this host, or a hold arriving on its links, before the + host's first publication (`load`'s batch of declarations) is not placed yet: it is `loading`, and the publication + places it. The gate lasts only for the synchronous setups, never for activation, so it cannot deadlock. +- **Then the selected provider is checked** (here: served, the version and fingerprint, the params, the provider's + own host-address check; over a path: at the provider host). Version checks never look at a same-name registration + elsewhere. What the check finds: + + | The check finds | The request | + | --- | --- | + | a fitting provider | bound (to the object here, or to the hold over a path) | + | not published yet | unbound `unpublished`: `loading` | + | nobody serves the name at its address (or serves it as the other kind) | unbound `not_served`: `failed { waiting }` (`unavailable` "X is not served here") | + | no fitting version, a fingerprint that differs at the same version, a host address the provider computes elsewhere | unbound `version`: `failed { waiting }` (`contract_mismatch`, `invalid/host_address`) | + | a registry served, but its first `compose` (or the reader's version mapping) threw | unbound `compose`: `failed { waiting }` (§17) | + | the path has no link (a drop, a route resolving) | on the path: `loading` | + | `authorize` refused, invalid params, a cycle | `failed { broken }`, never indexed | + + The unbound reasons that wait for a name are indexed under it on the host that decided them: the consumer's host + for a request made there, the provider's host for a hold that arrived over a link (kept refused there; a `retry` + re-runs its start). A publication serving a name (in 7a: `load`'s) wakes its requests, which run their check again + (a refused hold sends `loading`, then `bound` or `failed`); no new message kind. Index entries go when their request + ends (GC-tested). The consumer never re-holds a request by itself: no backoff, no polling. ### The low-level host API @@ -203,16 +230,16 @@ parent link closes. `resolve` returns the link before the worker has loaded (the | Call | Meaning | | --- | --- | | `createHost

    ({ address?, principal?, ...options })` | with an address: addressable, serves only objects placed there, acts as `principal` on its own host; without: a client. `P` is the app's principal type | -| `load([{ name, facet }], context)` | facets with their plugin names (the plugin's `package.json` name, npm style; duplicates and invalid names are refused); runs every facet's setup, then waits for the start barrier (section 21); a throwing setup rejects and closes the host; runs once. Loaders are adapters; base code passes its core facet the same way | +| `load([{ name, facet }], context)` | facets with their plugin names (the plugin's `package.json` name, npm style; duplicates and invalid names are refused); runs every facet's setup, publishes all of them in one batch (types and registries served, registries composed once, contributions; requests waiting for the publication are placed after it), then waits for the start barrier (section 21: the first outcomes of its host-lifetime objects, slice 7a); a throwing setup rejects and closes the host; runs once. Loaders are adapters; base code passes its core facet the same way | | `connect(link, { principal, trusted?, codec? }, context)` | the link to the parent, at most one open; once it closed (a drop), `connect` takes the next, and every remote reference over the parent is held again on it. `principal` = who the parent is | | `accept(link, { principal, trusted?, codec? }, context)` | a link from a child; holds and calls on it act as `principal` | | `route(Kind, { authorize, resolve }, context)` | how to reach hosts of a kind (below); called before `load`, once per kind; work arriving on its links runs under `context` | -| `get(Type, params, context, { readyTimeout? })` | a reference once its object is live (section 7) | +| `get(Type, params, context, { readyTimeout? })` | a reference once its object is live, waiting through `loading`, `failed { transient }` and `failed { waiting }` up to its `readyTimeout` (section 7) | | `hold(Type, params, context, { readyTimeout? })` | a reference at once that follows its object until released (section 7) | | `close(context)` | closes the links (children, routes, then the parent), then ends every object at once and tears them down, the last to go live first (reason `close`); rejects with an `AggregateError` of the native closes that threw | | `revoke(r, context)` | revocation (section 4; slice 4b) | | `root(fn, context, { principal? })` | a call scope for code outside objects (a Durable hook, a CLI): `fn` gets a handler's `ctx`, sealed when `fn` settles, `context` aborts or the host closes (slice 8 D3, section 8) | -| `registry(R, context)` | registry `R`'s composed value for code outside objects; waits for `load`, rejects when it has none (slice 8, section 17) | +| `registry(R, context, { readyTimeout? })` | registry `R`'s composed value for code outside objects once it is usable; waits for the publication, then while it has none up to its `readyTimeout`, then rejects with the last error (slice 8, 7a, section 17) | **Options of `createHost`:** `address`, `principal`, `principalKey` (section 4; the same function on every host), `telemetry` (section 29), `onError`, `closeTimeout`, `readyTimeout`, `heartbeat: { ping?, close? }`; later `revokedBy` @@ -221,7 +248,7 @@ and closed by the app with `store.close(ctx)` after `host.close`; the host never background, as a tagged `HostError`: `listener_error` (a watch listener threw), `delivery_failed` (a frame delivery threw), `activation_failed { object, failure }`, `close_failed { object, native, error }`, `close_timeout { object, native }`, `drain_timeout { object }`, `protocol_error { error }`, `resolve_failed { address, error }`; later routine -errors (slice 5), `derive_failed { object, member?, error }` (a derive's compute threw, once per failure episode, slice 6a), the registries' `registry_conflict { registry, message, plugins }`, `registry_failed { registry, error }`, `contribution_failed { registry, plugin, type?, error }` and `contribution_refused { registry, plugin, type?, version }` (each once per episode, slice 8), `object_failed { object, failure }` (a live object failing, slice 6a, `broken`; `transient` when a `self.follow` source failed, slice 6b R4; `failure` where the value is a `FailureCause`, `error` otherwise, 2026-10-27), `job_interrupted { object, job, id, restarts }` (a persisted run ended at the restart limit, slice 6b D6) and `storage_failed { error }` +errors (slice 5), `derive_failed { object, member?, error }` (a derive's compute threw, once per failure episode, slice 6a), the registries' `registry_conflict { registry, message, plugins }`, `registry_failed { registry, error }`, `contribution_failed { registry, plugin, type?, error }` and `contribution_refused { registry, plugin, type?, version }` (each once per episode, slice 8), `object_failed { object, failure }` (a live object failing, slice 6a, `broken`; `transient` when a `self.follow` source failed, slice 6b R4; `failure` where the value is a `FailureCause`, `error` otherwise, 2026-10-27), `job_interrupted { object, job, id, restarts }` (a persisted run ended at the restart limit, slice 6b D6), `parked_long { object, waitingFor, since }` (an attempt parked longer than its `readyTimeout` on a remote attempt that is only `loading`, once per parking episode: a diagnostic of a ring across hosts started at the same moment, slice 7a, §6) and `storage_failed { error }` (slice 6a), which every app must handle: Chord never ends the process, the app exits with its own code for its supervisor or reopens. @@ -230,7 +257,7 @@ the app exits with its own code for its supervisor or reopens. | Timeout | Bounds | Host default | Override | | --- | --- | --- | --- | | `closeTimeout` | a native's `close` (then reported as a possible leak; the rest continues); the drain of an ending object's running handlers (callers are rejected at once) | 30 s | per native: `self.open(open, close, { closeTimeout })` | -| `readyTimeout` | an activation attempt (its wait for the previous instance's teardown included); a getter's whole wait; how long a call made while its reference is loading waits (the park deadline) | 60 s | per `get`/`hold`: `{ readyTimeout }` | +| `readyTimeout` | an activation attempt's own steps (its wait for the previous instance's teardown included; paused while the attempt is parked on its dependencies, slice 7a); a getter's whole wait; how long a call made while its reference is loading waits (the park deadline) | 60 s | per `get`/`hold`/`registry`: `{ readyTimeout }` | The park deadline's per-type override is not built (only if a real case needs it). Timeouts of later slices (routine handover 10 s, job `finish` retry backoff 1 s to 60 s, rollout install 60 s, provider-side `linger` per type) @@ -251,7 +278,9 @@ bounded by the app. - **Worker keep-alive** works as in p2: the worker counts, the app's supervisor decides. **`host.activity`** (slice 4b; owner, 2026-10-24) is a JSON `Watchable` of `{ references, calls, jobs, lingering }`: holds arriving over links, method calls running on this host, running jobs, provider-side lingering objects. Counters are kept where the work - happens; a change marks it dirty and it is published at most once per microtask turn with changes. Core code + happens; a change marks it dirty and it is published at most once per microtask turn with changes. A hold placed + here counts from its arrival to its end, bound or not (one waiting for a missing provider keeps the worker alive; + slice 7a); a hold this host relays counts at its destination; `host.references` lists bound references only. Core code reports it plus Durable tasks (the core `Harness`); the server's supervisor stops the worker when all are zero for the idle time (p2: 5 minutes, checked every second). Chord provides the counts, no idle logic. @@ -521,9 +550,10 @@ host.serve(Notes, ({ session }, principal, self) => - **Compatibility (strict, like documents; R6):** a hold carries only the consumer's contract `version` and fingerprint, never a description. The provider binds when both equal its current ones, or (slice 7) when the version is one it accepts and the fingerprint equals the one computed from its own history (section 27). Every - contract change, additive ones included, bumps `version`. Anything else is `contract_mismatch` (or `updating` during - a rollout, section 21). The fingerprint covers params, variants and members, not the host address. A consumer whose contract - does not fit gets a mismatch on its own binding; the provider object and its other bindings are not affected. + contract change, additive ones included, bumps `version`. Anything else leaves the request unbound `version`: + `failed { waiting }` with `contract_mismatch` (slice 7a: not the consumer's fault; a publication that fits wakes it, + §3). The fingerprint covers params, variants and members, not the host address. A consumer whose contract does not + fit waits on its own request; the provider object and its other bindings are not affected. - **Variants:** a member name has the same kind (state or method) in every variant that has it, and a method the same signature (arguments and result) in every variant; state schemas may differ per variant (redaction). Enforced at the type level in `remote(...)` and at declaration at runtime. A remote factory returns one variant, `{ variant, @@ -567,7 +597,7 @@ host.serve(Notes, ({ session }, principal, self) => ### Lifecycle ``` -(get) → activating → live → ending → ended +(get) → activating (parked on its dependencies → own steps) → live → ending → ended └→ failed (backoff) → activating … live → failed (a runtime failure of the owner, slice 6a: broken, by reload or retry only) ``` @@ -584,11 +614,11 @@ host.serve(Notes, ({ session }, principal, self) => - **Activation** (one fresh attempt each time, bounded by `readyTimeout`, its wait for the previous instance's teardown included): the factory runs; singleton documents take `initial()` and, on the object's first activation, persisted documents load and its tables' definitions are checked and migrated (slice 6a; the deadline paused during - eager migration work); references are bound (before the - previous attempt's are released, so a dependency held only by it never ends in between) and waited for until live - and hydrated; natives open one after another; the returned members are checked (section 5); derives compute; - routines start (slice 5). The object goes live when all of that succeeded. Calls wait behind its **gate** until - then. + eager migration work); its requests are made (`self.get`, `self.registry`, keyed pins), and the attempt is + **parked** until every one is live and hydrated (below; the deadline paused); then its own steps run under the + deadline: natives open one after another, followed sources attach, the returned members are checked (section 5), + derives compute; routines start (slice 5). The object goes live when all of that succeeded. Calls wait behind its + **gate** until then. - **The seal fence** (owner, 2026-10-26): every attempt has a seal token (a job run invocation its own, for its state changes and outcome). After the seal, new work through the attempt's handles is refused at use (`self.tx`, `ctx.get`, a job or routine start); commits are checked when they enter the store's commit step (sealed: rejected `released`, @@ -609,18 +639,36 @@ host.serve(Notes, ({ session }, principal, self) => with the cause, state members keep their last values, natives close, documents stay on the object; recovery by reload or `ref.retry()`; live dependents stay live. The members freeze only after the commits admitted before the seal adopted and published once, and a retry's factory waits for that (2026-10-27, slice 6a build reviews). -- **Waiting for dependencies (R18):** an object whose references are not live yet stays `loading` and goes live as - soon as they are (bounded by `readyTimeout`; a link drop while a dependency is still loading is no failure: the - reference re-holds on the next link and the dependent keeps waiting, design audit 6, 2026-10-27). A dependency - going live while the dependent still activates is remembered and retries it at once if that attempt fails. A - dependency that fails (`broken` or `transient`), or does not become - live in time, fails the dependent **`transient`** with the dependency's cause inside, never `broken`: the cause is - the dependency's. The dependent keeps its references while failed, and Chord installs their state as an automatic - retry trigger: the moment the dependency goes live, the dependent is retried at once (the remaining backoff is - cancelled; the backoff stays as a safety net). Factory authors write `self.get` and know nothing of this. When a - dependent's last reference goes, it ends and releases its dependencies. After `live`, a dependency leaving `live` - does not change the dependent: it stays live, and its calls that use the dependency get its error (or park while it - is loading). Only R18 failures keep references; any other failure releases them. +- **Parked on its dependencies** (dependency model §3, slice 7a; supersedes R18's dependency deadline and backoff): a + dependency is what an attempt declares and needs before it goes live: `self.get` (local or remote), `self.registry`, + keyed pins (`at(key)` on another object's keyed member); `self.follow` is the attempt's own step, not a dependency. + While any dependency is not live (and hydrated), the attempt is parked and its object shows: + + | Its dependencies | The object shows | + | --- | --- | + | any `loading`, none failed | `loading` | + | any `failed` | `failed { waiting }`, `retryAt` undefined, error `unavailable` "waiting for X: " (X the first failed one in declaration order, the root cause found through its message's "waiting for …: "; always these two, so a ring of waiting attempts settles after one round; within 500 characters, X's label clipped first, fix round 1) | + | one failed with `invalid/cycle` (a dependent off the ring) or `forbidden` (an `authorize` that refused it: authority, M§3.6) | `broken` with that error: nothing heals it but `ref.retry()` or a reload | + | one `ended` (revoked, its host closed) | `broken` with that error: the request cannot come back | + + **No timer runs while parked:** no dependency deadline, no backoff; the attempt's `readyTimeout` deadline is paused + (as eager migrations pause it) and resumes with the budget left; the identity backoff stays for the object's own + `transient` failures only. **Wakes:** the parked attempt checks again at every change of a dependency (live, + loading, failed, a new cause compared by value, ended, and its first outcome, below), coalesced per object after the + batch that caused them (a change while it checks or loads pinned keys is remembered); it never re-runs its factory. + A parked attempt keeps its documents, declarations, requests and seal token, and holds no natives. **The dependency + check is the boundary:** once all are live (and the pinned keys loaded), the own steps run; a dependency that leaves + `live` afterwards changes nothing (R18 after live: the object goes live; its calls through that reference meet its + state). **Retry passes on:** `ref.retry()` (or a `retry` message) on an object parked `failed { waiting }` passes on + to its failed dependencies, once per retry operation: the operation (a uuidv7) travels in the `retry` message + (`operation`) with the instance ids of the objects it passed through (`through`, added hop by hop, at most 32), and + an object already in `through` drops it, so a ring across hosts stops after one lap however many retries run at + once; an object also remembers the last few ids it handled, so a diamond retries it once. A `broken` or + `transient` one starts an attempt, a parked one passes it on, a request waiting unbound in an index is left as it is + (an event wakes it), a refused hold over a link re-runs its start. A `retryOn` wake while parked does nothing. When + a parked object's last reference goes, it ends at once (it is not live: no `linger`) and releases its requests. + `broken` is only for an object's own code, cycles and authority; no failure keeps references (the R18 retry trigger + is gone: parking replaces it). - **Dependency cycles are rejected (Q5):** every object instance has a uuidv7 instance id, minted at its first attempt; an object that restarts gets a new one, so an old chain never matches (slice 4b, the first user; D4, owner, 2026-10-24). On one host, at @@ -629,23 +677,32 @@ host.serve(Notes, ({ session }, principal, self) => (bounded; ignored from untrusted links): the provider rejects the hold when the instance being activated is in the chain (slice 4b; on one host slice 1; at most 32 entries; locally created objects inherit the activating chain). A cycle fails the activation `broken invalid/cycle` with the cycle as cause ("A → B → A"), and every other activating - object on it the same way; no references are kept. A dependency failing with `invalid/cycle` fails its dependent - **broken** with the same error, an exception to R18 (owner, 2026-10-24: otherwise every backoff retry hits the same - cycle); dependents that are not on the cycle break too, and a reload or `ref.retry()` recovers them. Checked again - after a reload. Cost: O(chain) per declared reference, only during activation. - **Known limitation** (owner, 2026-10-24): a chain is fixed when its object is created, so a cycle entering an object - that is already activating is not detected. Example: A on the server and B on a worker start at the same moment, A - declares B and B declares A; the holds carry `[A]` and `[B]`, neither finds the other, both fail `transient` at - `readyTimeout` and retry with backoff (later attempts usually come in order and detect it). A probe along waiting - holds was not built. + object on it the same way; no references are kept. A dependency failing with `invalid/cycle` makes its dependent + **broken** with the same error (owner, 2026-10-24: it would wait on a ring that never heals); dependents that are not + on the cycle break too, and a reload or `ref.retry()` recovers them. The walk on one host follows the waiting edges + of activating and parked attempts (exact). Checked again after a reload. Cost: O(chain) per declared reference, + only during activation. + **The gap: rings across hosts started at the same moment** (owner, 2026-10-29: no probe): a chain is fixed when its + object is created, so a cycle entering an object already activating is not detected. Example: A on the server and B + on a worker start at the same moment, A declares B and B declares A; the holds carry `[A]` and `[B]`, neither finds + the other, and without a dependency deadline both stay parked. It is made visible: an attempt parked longer than its + `readyTimeout` on a remote attempt that is itself only `loading` (its hold on a link; the clock and `since` start + when the hold gets on a link and stop while the path has no link) + is reported once per parking episode as + `parked_long { object, waitingFor, since }`; nothing fails or retries. A cycle is a plugin bug; **the guide's four + rules** for plugin authors and agents (host README): (1) a factory declares only what the object needs, never a type + that needs it back; (2) back-references and occasional lookups go into methods and routines (`ctx.get`); (3) data + flows up through state and registries, not mutual `self.get`; (4) host-lifetime objects on the server never declare + worker types in their factory. - **Failed:** a broken part fails the whole object: a throwing factory, document hydration or `migrate`, a failed - open, a derive that throws on its first computation during activation, or a dependency that did not become live. - After the object is live, a throwing derive fails only its own member (`broken`, W1). The cause is tagged and decided by Chord from where it - failed: + open, a derive that throws on its first computation during activation. After the object is live, a throwing derive + fails only its own member (`broken`, W1). The cause is tagged and decided by Chord from where it failed: ```ts - cause: { kind: "transient"; error: ChordError } // an open threw, a reference was unreachable: automatic backoff - | { kind: "broken"; error: ChordError } // factory, derive, hydration or migrate threw: no automatic retries + cause: { kind: "transient"; error: ChordError } // the object's own effect failed (an open, a follow): backoff + | { kind: "broken"; error: ChordError } // its own code (factory, derive, hydration, migrate), a cycle, authority + | { kind: "waiting"; error: ChordError } // not its fault (slice 7a): no provider, no fitting version, a + // dependency failed; retryAt undefined, woken by an event ``` Only an `open` may choose `broken` itself (`throw broken(error)`, e.g. invalid configuration). **Transient** causes @@ -718,13 +775,17 @@ location are dropped. | a call | `await ctx.get(Type, params, { readyTimeout? })` | when the handler settles | | outside | `await host.get(Type, params, context, { readyTimeout? })` / `host.hold(Type, params, context, { readyTimeout? })` | `release()` | -- **`host.get`** (and `ctx.get`) resolves with a reference once its object is live and **hydrated** (every state member - of its variant has a value). A broken failure rejects at once; through transient ones it waits up to its - `readyTimeout`, then rejects with the last failure's error; a getter that gives up releases its reference. - **`host.hold`** returns a reference at once that follows its object until released (loading, live, failed and - retried as its object is, loading again while its link is down) and never rejects; invalid params (checked at once, also before `load`: design audit 7, 2026-10-27) and a local type - not served here throw at once (before `load` resolved, these fail the reference broken once it did); its `variant` is - `undefined` until its first binding (slice 3b). There is no `state: false` option (C8) and no `optional` flag. +- **`host.get`** (and `ctx.get`, `host.registry`) resolves with a reference once its object is live and **hydrated** + (every state member of its variant has a value). It waits while the reference is `loading`, `failed { transient }` + or `failed { waiting }`, up to its `readyTimeout` (default 60 s, the caller's choice), then rejects with the last + cause; only `failed { broken }` rejects at once (owner, 2026-10-29, decision "Getters wait through `waiting`"). A + getter that gives up releases its reference. **`host.hold`** returns a reference at once that follows its object + until released (loading, live, failed and retried as its object is, loading again while its link is down) and never + rejects; only invalid params throw (checked at once, also before `load`: design audit 7, 2026-10-27). A type not + served where it goes is `loading` before the host's publication and `failed { waiting }` after it, before and after + `load` alike (§3; slice 7a). Its `variant` is `undefined` until its first binding (slice 3b). A UI holds and shows + `ref.state`; code that needs the object now gets it, with the patience it passes. There is no `state: false` option + (C8) and no `optional` flag. - **Narrowing by variant:** `ref.variant` is the discriminant (C15: no `inVariant`/`expectVariant`); `RefIn` narrows to some variants (strict unions, never weakened). - **Members across attempts** (2026-10-27, slice 6a build reviews): on the same host a reference's state member @@ -741,7 +802,8 @@ location are dropped. ```ts | { kind: "loading" } | { kind: "live" } - | { kind: "failed"; cause: FailureCause; retryAt: number | undefined } // retryAt undefined: broken, no automatic retry + | { kind: "failed"; cause: FailureCause; retryAt: number | undefined } // retryAt: a transient cause's backoff; + // undefined for broken and waiting | { kind: "ended"; error: ChordError } // released by its holder, or its object ended ``` @@ -749,13 +811,16 @@ location are dropped. | --- | --- | --- | | link drop | park until reconnect | readable (`loading`) | | object loading or restarting | park until live | readable (`loading`) | -| rollout: provider ahead or installing | park until installed, with a deadline (slice 7) | readable | +| rollout: the provider host installing a revision the hold does not fit (slice 7b) | park until installed, bounded | readable (`loading`) | +| unbound (not served, another version) or parked on a failed dependency | reject at once with the cause | `failed { waiting }` (slice 7a) | | object `failed` | reject at once with the cause | `failed` with cause and `retryAt` | -- A failed reference stays alive: while held, Chord retries per the backoff; `ref.retry()` tries at once (a Retry - button; over a link the `retry { hold }` message, which also re-runs a refused hold's start: authorize, the host address, - binding; design audit 2026-10-27); success goes `loading`, then `live`. A provider tells a pending - hold's consumer each failure (`failed`, with the rest of its backoff window) and the next attempt (`loading`). +- A failed reference stays alive: while held, Chord retries a `transient` failure per the backoff and wakes a + `waiting` one at the event it waits for; `ref.retry()` tries at once (a Retry button; over a link the `retry + { hold }` message, which also re-runs a refused hold's start: the host address, binding; design audit 2026-10-27; + on a parked object it passes on to the failed dependencies, §6); success goes `loading`, then `live`. A provider + tells a pending hold's consumer each failure (`failed`, with the rest of its backoff window for a transient one), + each new cause of an object parked on a failed dependency, and the next attempt (`loading`). - **Parked calls** (slice 3b): calls made while the reference is loading are queued in order, each waiting at most the reference's `readyTimeout` from the moment it was made (never renewed), then fail `unavailable/park_timeout`; they are sent in order when it goes live (a new call never overtakes them) and rejected with the cause when it fails. @@ -1490,7 +1555,7 @@ export default defineFacet((host, self) => { // `self` (contributions) a - A plugin's per-host setup, **declarations only** (owner, 2026-10-26): `host.serve`, `self.contribute` (slice 8), `host.references`; no natives, no documents (C16 accepted), no routines, no jobs. Everything a plugin does at runtime is done through objects: shared or persisted state and standing behaviour live in a `local` store object with empty - params (host lifetime, its own failure domain, R18 recovers its dependents), as p2 does (its 63 facets take only + params (host lifetime, its own failure domain; its dependents park while it is failed and go on when it recovers), as p2 does (its 63 facets take only `host`). A facet cannot fail after `load`. - `host.serve(Type, factory, { linger? })`: `linger` (milliseconds, 0 by default; slice 3b) keeps a live object with params alive that long after its last reference, so a new reference within that time reuses it with its documents @@ -1524,8 +1589,11 @@ const router = self.registry(Http); // a declared dependency; read rou live or ends, when a dynamic contribution changes). It always delivers a value with its fallback built in (e.g. the first by plugin name wins) and reports conflicts through `report` (owner notices and telemetry events); readers get a plain value and never handle conflicts. If `compose` throws, the failure is reported and its - watches get a `failed` call, while `.value` keeps the last good composition (slice 8); only before any good - composition do readers get the error. Work that started earlier keeps the value it read (a pinned Durable phase keeps + watches get a `failed` call, while `.value` keeps the last good composition (slice 8), **but only while every + contribution it was built from is still in the table** (dependency model decision 5, slice 7a): when one leaves (its + contributor ended, failed or restarted), the last good value is dropped and `.value` throws until `compose` + succeeds, so it never hands out a native whose contributor ended; only before any good composition, or after such a + drop, do readers get the error. Work that started earlier keeps the value it read (a pinned Durable phase keeps its old table). Helpers: `keyedEntries(table, entriesOf, report)` (a map, the first by plugin name wins, a duplicate reported), `orderedEntries(table, itemsOf, orderOf?)`; both keep each item's plugin (named so beside the contracts' `keyed`, slice 8 build). @@ -1538,7 +1606,8 @@ const router = self.registry(Http); // a declared dependency; read rou (`compose`, version, `accepts`); other plugins' tokens only identify it by name and types (contribute and read, never bring their own `compose`). A reload of the defining plugin replaces the declaration atomically and the table recomposes. Without a declaration (the defining plugin is not loaded) the registry has no composed value: readers - get an error like a failed reference; contributions wait in the table. + wait `failed { waiting }` (`unavailable/undeclared`), like a request for a type nobody serves; contributions wait in + the table. - **Versions (Q8, the owner-adapter pattern):** a contribution carries the registry version its plugin was built against; older ones are adapted with the defining facet's pure `accepts` when the table is built. A contribution without a matching version or adapter (older without adapter, or newer) is left out of the table and reported (an @@ -1548,12 +1617,16 @@ const router = self.registry(Http); // a declared dependency; read rou use site catches synchronous throws and rejections when it calls an entry, reports naming the contributor (the composed table keeps each entry's plugin name), and continues with the other entries. Durable already does this for extensions (`extension_error`; a throwing tool becomes an error result); `compose` reports through `report`. -- **Slice 8's rules** (decisions "Slice 8 D1" to "Slice 8 S3", `slice-8.md`): the token is shared, the provider serves it - (`host.serve(R, { compose, accepts })`); `self.registry(R)` in a factory is a declared dependency (activation waits - for a value; with none after `load`, the reader fails `transient` and is retried when one arrives); contributions - never fail a registry (a failed one is left out and reported); a throwing `compose` keeps the last good composition; - older readers get the value mapped back (`accepts[old].value`); `retryOn` retries on value calls only; `host.root(fn, - context, { principal? })` seals its context at settle, abort and `host.close`. +- **Slice 8's rules** (decisions "Slice 8 D1" to "Slice 8 S3", `slice-8.md`; S2's `broken` and S3's backoff superseded + by slice 7a): the token is shared, the provider serves it (`host.serve(R, { compose, accepts })`); `self.registry(R)` + in a factory is a declared dependency: the attempt parks until the registry has a usable value at its version + (composed; mapped by `accepts[old].value` for an older reader), otherwise it shows `failed { waiting }` naming the + registry (`not_served`: `unavailable/undeclared`; `version`: a reader newer than the provider or without a mapping, + `contract_mismatch`; `compose`: a first `compose` or the mapping threw; `loading` before the publication), woken at + every change of whether the value is usable; `host.registry` waits the same way up to its `readyTimeout`; + contributions never fail a registry (a failed one is left out and reported); a throwing `compose` keeps the last + good composition (above); `retryOn` retries on value calls only; `host.root(fn, context, { principal? })` seals its + context at settle, abort and `host.close`. - **Registries may carry natives.** An entry lives exactly as long as its contributor; consumers look up at each use or follow the composed value and keep nothing from an entry beyond a use. - **Contributions that change (owner, 2026-10-19; W4):** registries are host-local, so contributions are usually @@ -1716,7 +1789,9 @@ agent asks everyone present. Its store must live in an object (it predates "nati keeps the hold: a pure object re-subscribes it through the version's mappings and sends `[["r", value]]` for each member whose value changed; a restarted object sends a fresh `bound` for the same hold (a `bound` on a bound hold replaces its replicas). A served hold whose object leaves `live` while its `bound` waits for a slot goes back to - pending (slice 7 builds it: nothing in 4a leaves `live`). A reload that drops the version ends its holds with `gone` (`contract_mismatch`) before + pending (slice 7 builds it: nothing in 4a leaves `live`). *(Pending slice 7b's reconciliation with the dependency + model M§5, which unbinds such holds `version` instead of ending them; until then this item and the rollout's + `updating` deadline below are not current rules.)* A reload that drops the version ends its holds with `gone` (`contract_mismatch`) before the new code serves anything. A consumer whose own code moves to a new version re-holds (a new hold id) and releases the old. - **Old-version calls:** `checkArgs`'s `{ kind: "call" }` result runs the provider's replacement; `{ kind: "method"; @@ -1729,14 +1804,22 @@ agent asks everyone present. Its store must live in an object (it predates "nati - **Consumers elsewhere** see a restarting object like any object that briefly goes away. - No factory trial, no migrate dry run, no consumer contract check during staging: incompatible consumers fail at bind with `contract_mismatch` and a notice naming plugin and type. -- **Start barrier** (host start and first load): load facets (the core facet with its start data among them), wait - until every facet and host-lifetime object is live or failed (an object that must - contribute at start, like the model runtime, is a `local` object with empty params). Then the registries hold the contributions of every - plugin that loaded, and dependent work - resumes: base code calls `harness.resume()` after `load` (the harness is an ordinary host-lifetime object that - opened Durable paused during the barrier; owner, 2026-10-21, refines R17). Nothing - waits for specific contributions: what is missing shows where it is used (section 24). Failed plugins are reported - with their error. Plugin reloads need no barrier. +- **Start barrier** (host start and first load; dependency model §6, decision (a), slice 7a): `load` runs the facets' + setups (the core facet with its start data among them) and **publishes** them in one batch (types served, + registries served and composed once, contributions in their tables); requests that waited for the publication are + placed after that batch (holds and `host.hold` wait for the publication only, never for the barrier: two hosts + whose objects need each other's types start without a deadlock). Then it activates the host-lifetime objects and + resolves when each has its **first outcome**: live, failed by its own cause, or parked with no blocker that is a + local object still without its first outcome (`local`: executing on this host, a keyed pin still loading here + included; a wait on another host or on a failed dependency ends the first pass), or ended (`host.close` settles a + waiting `load`). One first-outcome signal per object, separate from the readiness dependents wait for, counted once, + never reopened by retries. So `CodeMode → Harness` waits for the harness's outcome. **What it promises:** every + host-lifetime object here had its first try, and the contributions of those that went live are in their tables; + **not** every tool that will arrive (a parked or slow contributor, tools found in `onLive`, MCP): what a first turn + needs is p3's explicit admission check. Base code calls `harness.resume()` after `load` (the harness is an ordinary + host-lifetime object that opened Durable paused during the barrier; owner, 2026-10-21, refines R17). Nothing waits + for specific contributions: what is missing shows where it is used (section 24). Failed plugins are reported with + their error. Plugin reloads need no barrier. ### Rollout across hosts @@ -1800,14 +1883,15 @@ validation, no clone of snapshots or ops. | `hello` | both, first | `protocol`, `incarnation`, `calls` (the call window N) | | `hold` | to provider | `origin?`, `hold`, `type`, `version`, `fingerprint`, `params` (part), `address`, `principal?`, `incarnation?`, `chain?` (activation chain, Q5), `limit?` (decoded size limit, 4b) | | `revoke` | parent to child (route link) | the revocation value (4b) | - | `release`, `retry` | to provider | `origin?`, `hold` | + | `release` | to provider | `origin?`, `hold` | + | `retry` | to provider | `origin?`, `hold`, `operation?` (the retry it belongs to, a uuidv7), `through?` (with `operation`: the instance ids of the objects it passed through, at most 32; an object in it drops the retry, so a ring stops after one lap); relays forward both (slice 7a fix rounds 1, 2) | | `call` | to provider | `origin?`, `call`, `hold`, `method` (index), `args?` (part; absent: no arguments), `segments?` | | `abort` | to provider | `origin?`, `call` (named `cancel` until 2026-10-27) | | `ack` | to provider | `origin?`; one per applied `bound`/`state` message | | `closed` | relay to provider | `origin`: the downstream origin ended | | `bound` | to consumer | `origin?`, `holds: [{ hold, variant, members: [{ member, value }], failed }]`; one batch | | `state` | to consumer | `origin?`, `holds: [{ hold, members: [{ member, ops }], failed, retired }]`; one batch (`retired`: members retired since the last message, slice 6a) | - | `failed` | to consumer | `origin?`, `hold`, `cause { kind, error }`, `retryIn?` (absent: broken) | + | `failed` | to consumer | `origin?`, `hold`, `cause { kind, error }` (`kind`: `transient`, `broken` or `waiting`, slice 7a), `retryIn?` (a transient cause's backoff; absent: broken, waiting); each new cause of a parked object is a message of its own | | `loading` | to consumer | `origin?`, `hold`: a new attempt after a failure, or a relay's upstream dropped | | `result` | to consumer | `origin?`, `call`, `value?` (part; absent: `null`), `segments?` | | `error` | to consumer | `origin?`, `call`, `error` (a ChordError's JSON or a plugin's code, message, details) | @@ -2132,8 +2216,8 @@ function doing counted work counts it through its context; call sites never coun site labels. **Testing the core:** every slice spec contains an explicit test list, one group per mechanism with its edge cases, -tested individually and exhaustively: activation and dependency recovery (R18: broken dependency, trigger, backoff as -safety net, a dependency failing while live, cycles), revocation over running jobs (R11), compatibility (R6), params +tested individually and exhaustively: activation and parked dependencies (the dependency model: unbound requests, waits without timers, wakes on every +change, retry passing on, a dependency failing while live, cycles, the start barrier's first outcomes), revocation over running jobs (R11), compatibility (R6), params and arguments for every caller (R7), migration results and old calls on migrated documents (R8, R15), flow control (the window per origin, folding, relays forwarding and returning acks), the incarnation fence (R20), the harness start after the barrier (R17), native restart patterns (R16), and W1's publication list (a throwing derive, a chain A→B→C diff --git a/packages/chord/docs/next/slices.md b/packages/chord/docs/next/slices.md index 97883b776..93c5cfc86 100644 --- a/packages/chord/docs/next/slices.md +++ b/packages/chord/docs/next/slices.md @@ -76,6 +76,7 @@ pattern, no code. Decided cuts are noted where they apply. | 3 | park deadline (= `readyTimeout`, C13) | 3b | | 3 | `host.activity` (keep-alive counts) | 4b | | 3 | `host.root(fn, …)` for code outside objects | 8 | +| 3 | requests that outlive providers: the destination by address, the publication gate, the unbound index, the check against the selected provider only; the start barrier's first outcomes (decision (a)) | 7a | | 4 | principal type `P`, links stamp principals, principal keys in identity | 3a | | 4 | revocation (`revokedBy`, `host.revoke`, propagation to children) | 4b | | 5 | `local` types: params, members, checked arguments (Q15), results unchecked | 1 | @@ -85,6 +86,7 @@ pattern, no code. Decided cuts are noted where they apply. | 6 | lifecycle, one object per identity, lifetime from kind and params (Q-A), factory, activation, ending | 1 | | 6 | failure causes, backoff per identity, dependency recovery (R18) | 2 | | 6 | `retryOn` | 8 (first independently living source: registries) | +| 6 | parked attempts (no dependency deadline or backoff, wakes on every change, retry passing on), `FailureCause` `waiting`, `parked_long` and the authoring rules against cycles | 7a | | 6 | cycle rejection on one host (Q5) | 1 | | 6 | cycle rejection across hosts (activation chain in hold headers) | 4b | | 6 | shared store with per-principal views; worker-wide native owners; natives following a relocation | doc (1, 3, 2) | @@ -92,6 +94,7 @@ pattern, no code. Decided cuts are noted where they apply. | 7 | per-reference watches (released with their reference) | 1 | | 7 | `ctx.get` (call-scoped references) | 3a | | 7 | `ref.state` (tagged), `ref.retry()`, `host.hold`, calls parking, failed references | 3b | +| 7 | getters wait through `waiting`; `host.hold` throws only for invalid params | 7a | | 7 | `ref.variant` as discriminant, `RefIn` (C15: no `inVariant`/`expectVariant`) | 3a | | 7 | `state: false` (C8: cut) | - | | 7 | optional dependencies as registries | doc (8) | @@ -119,6 +122,7 @@ pattern, no code. Decided cuts are noted where they apply. | 15 | `defineJob`, `self.job`, intent/run/finish, state documents, dedupe (Q9), `busy`, `running`, blocked `finish` (C3 decided); persistence, resume, restart limit; `accepts` in use | 5 (persistence: 6b; `accepts` and reload: 7) | | 16 | `defineFacet

    `, facet as declarations only (documents and routines removed in 6a) | 1, 6a | | 17 | registries: `defineRegistry`, contributions, `compose`, versions (Q8), use-site guards | 8 | +| 17 | registry readers as parked dependencies (`waiting`), `host.registry` waiting, the last good value dropped when a contribution leaves | 7a | | 18 | `constant(value)`, start data as a core object (`SessionInfo`) | 1 (core facet pattern: 4a) | | 18 | platform bindings as core objects | doc (8) | | 19 | `host.references` (presence) | 4b | @@ -185,6 +189,12 @@ deleted (the slice's spec and code are the record). Moved here from `carry-over. (`local`, empty params) stays broken until the host restarts; slice 7 maps the records to the new version (or blocks the old runs) on reload. +- **Publications wake per name** (from slice 7a): `UnboundIndex.wakeAll()` wakes every indexed request, right for the + one publication 7a has; a reload's or an install's publication wakes only the requests of the names it serves + (`wake(names)`), or every refused hold on the host would re-run its start. +- **The reload handover of a parked attempt** (from slice 7a): the new attempt's requests bind before the old ones are + released (M§3.9); the mechanism (an attempt releasing its predecessor's references after binding its own) was removed + in 7a with R18's kept references, its only feeder. - **Routines on reload** (from slice 5): the handover per object, bounded by `closeTimeout`; calls do not wait for it. - **Jobs on reload** (from slice 5): `accepts` in use (args and result mapped forwards once per run; `blocked { version }` without an adapter or for a newer version); when an older run's state document may migrate while its diff --git a/packages/chord/src/next/data/failure.ts b/packages/chord/src/next/data/failure.ts index a76190118..2f01fe258 100644 --- a/packages/chord/src/next/data/failure.ts +++ b/packages/chord/src/next/data/failure.ts @@ -1,11 +1,14 @@ -// Failure causes of an open: transient (retry later) or broken (bad configuration). The failing stage decides. +// Failure causes: transient (the object's own effect failed: retry later), broken (its own code, a cycle or authority: +// retry by hand) or waiting (not its fault: no provider, no fitting version, a dependency that failed; woken by an +// event). The failing stage decides, never the error code. -import { asChordError, type ChordError } from "./errors.ts"; +import { asChordError, ChordError } from "./errors.ts"; -/** Why an open failed; the kind comes from the failing stage, never from the error code. */ +/** Why an object, or a request, is failed; the kind comes from the failing stage, never from the error code. */ export type FailureCause = | { readonly kind: "transient"; readonly error: ChordError } - | { readonly kind: "broken"; readonly error: ChordError }; + | { readonly kind: "broken"; readonly error: ChordError } + | { readonly kind: "waiting"; readonly error: ChordError }; /** Marks an open's failure as broken (bad configuration): `throw broken(error)`. */ export function broken(cause: unknown): Error { @@ -18,6 +21,65 @@ export function openFailure(thrown: unknown, label: string): FailureCause { return { kind: "transient", error: asChordError(thrown, label) }; } +/** The longest message a `waiting` cause of a dependent carries (telemetry's rule for error messages). */ +const MESSAGE_LIMIT = 500; + +/** How a dependent's message names its blocker: the nearest one first. */ +const PREFIX = "waiting for "; + +/** The separator after a blocker's label. */ +const SEPARATOR = ": "; + +/** The marker of a clipped label or root. */ +const CLIPPED = "…"; + +/** The most of the 500 characters the root cause takes when both are long: the nearest blocker keeps the rest. */ +const ROOT_LIMIT = 320; + +/** + * The `waiting` cause of a dependent parked on `blocker`, which failed with `error`: `unavailable`, its message always + * "waiting for : " (the root found through any "waiting for …: " of `error`'s message), so a ring + * of waiting attempts passing causes round reaches the same message after one round; within 500 characters, the + * label clipped first, the root only past its share. `error` is its cause. + */ +export function waitingFor(blocker: string, error: ChordError): FailureCause { + let root = rootMessage(error.message); + let label = blocker; + const room = MESSAGE_LIMIT - PREFIX.length - SEPARATOR.length; + if (label.length + root.length > room) { + const rootRoom = Math.max(ROOT_LIMIT, room - label.length); + if (root.length > rootRoom) root = clipped(root, rootRoom); + if (label.length + root.length > room) label = clipped(label, room - root.length); + } + const message = `${PREFIX}${label}${SEPARATOR}${root}`; + return { kind: "waiting", error: new ChordError({ code: "unavailable", message }, { cause: error }) }; +} + +/** `text` within `length` characters, clipped with "…" when longer. */ +function clipped(text: string, length: number): string { + return text.length <= length ? text : `${text.slice(0, length - CLIPPED.length)}${CLIPPED}`; +} + +/** The root cause in a "waiting for X: root" message: past every leading "waiting for …: ". */ +function rootMessage(message: string): string { + let rest = message; + while (rest.startsWith(PREFIX)) { + const end = rest.indexOf(SEPARATOR, PREFIX.length); + if (end < 0) break; + rest = rest.slice(end + SEPARATOR.length); + } + return rest; +} + +/** Whether two failure causes say the same (a wake compares causes by value): kind, code, reason and message. */ +export function sameCause(a: FailureCause, b: FailureCause): boolean { + const { error: left } = a; + const { error: right } = b; + return ( + a.kind === b.kind && left.code === right.code && left.reason === right.reason && left.message === right.message + ); +} + class BrokenOpen extends Error { constructor(cause: unknown) { super(cause instanceof Error ? cause.message : String(cause), { cause }); diff --git a/packages/chord/src/next/data/internal.ts b/packages/chord/src/next/data/internal.ts index d59881ebe..95d03a2cc 100644 --- a/packages/chord/src/next/data/internal.ts +++ b/packages/chord/src/next/data/internal.ts @@ -8,7 +8,7 @@ export { invalid, PROTOCOL_VERSION, } from "./errors.ts"; -export { openFailure } from "./failure.ts"; +export { openFailure, sameCause, waitingFor } from "./failure.ts"; export { checkAddress, checkParams, identityKey } from "./params.ts"; export { canonicalJson, principalKey } from "./principal.ts"; export { check, validatorFor } from "./validator.ts"; diff --git a/packages/chord/src/next/host/README.md b/packages/chord/src/next/host/README.md index b5c9175fa..0bc278cbf 100644 --- a/packages/chord/src/next/host/README.md +++ b/packages/chord/src/next/host/README.md @@ -62,12 +62,13 @@ const host = createHost({ closeTimeout: 30_000, readyTimeout: 60_000, }); -await host.load([{ name: "@app/notes", facet }], ctx); // every host-lifetime object live or failed (reported) -const notes = await host.get(Notes, {}, ctx); // live and hydrated; waits through transient retries +await host.load([{ name: "@app/notes", facet }], ctx); // published, then every host-lifetime object's first outcome +const notes = await host.get(Notes, {}, ctx); // live and hydrated; waits through loading, transient, waiting await notes.add({ text: "milk" }, ctx); notes.release(); const draft = host.hold(Draft, { conversation: "c1" }, ctx); // a reference at once: loading, then live (await draft.state.watch(ctx)).start(async (call) => call.kind === "value" && render(call.value)); // loading, live, failed, ended +// failed { waiting }: "waiting for X: …", not this object's fault; it goes on when X is live if (draft.state.value.kind === "failed") draft.retry(); // a Retry button host.accept(link, { principal: { kind: "guest", session: "S1" } }, ctx); // a child's link, authenticated by its entry point await host.close(ctx); // closes links, then natives with reason "close" @@ -130,7 +131,8 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded: `principalKey` maps a principal to its part of an identity (L1's canonical JSON by default; the same function on every host). `closeTimeout` (30 s) bounds each native's close and the drain of an ending object's calls; `readyTimeout` (60 s) bounds every activation - attempt (its wait for the previous instance's teardown included) and is the default deadline of a `get`. `heartbeat: { ping?, close? }`: a link silent for `ping` ms (15 s) + attempt (its wait for the previous instance's teardown included; paused while it is parked on its dependencies) and + is the default deadline of a `get` and of `host.registry`. `heartbeat: { ping?, close? }`: a link silent for `ping` ms (15 s) sends a `ping`, one silent for `close` ms (30 s), its handshake included, is closed (its references go `loading`); one interval timer per host (`min(ping, close) / 10`), running only while it has links. A handler that blocks the event loop longer than `close` gets its links closed. @@ -139,8 +141,11 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded: principal has `r` among its keys ends at once, whatever references remain (R11): each holder gets `forbidden` with reason `revoked` (a UI can say "access revoked"; a hold over a link sends `gone` with it), running calls are aborted with it (callers rejected), an activation in progress is abandoned; teardown follows as for any end (drain, closes - with reason `end`). Holds of the principal that arrived before the revoke but still wait (for `load`, or relayed for - their route's link) end too (`gone`). + with reason `end`). Requests of the principal that hold nothing end too, bound to nothing yet or never: holds that + arrived before the publication, holds refused here (`failed { waiting }`), relayed holds waiting for their route's + link (each gets `gone`), references made here still unbound, and this host's own remote references of the + principal over its paths, a route still resolving or a link down included (`ended`, released at the provider if + held there): no later link holds them again. Then every route link is sent `revoke { revocation }`, which the child applies with its own `revokedBy` (and passes down its own route links), and every child link whose principal it revokes closes (a relay sends `closed { origin }` upstream for it, after the `revoke`). Never this host's own principal, never the parent link; a @@ -157,8 +162,9 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded: splice op per removal, one for the additions: a revoke of many holds is one list and one frame), otherwise it is built on its next read. A departure discovered while a commit's links send (a transport failing on send) lands in the next batch, so one commit stays one message per link and origin. -- `host.activity`: a JSON watchable `{ references, calls, jobs, lingering }`: holds arriving over links and served - here, method calls running here (local and remote), job runs running or finishing here, objects lingering after their +- `host.activity`: a JSON watchable `{ references, calls, jobs, lingering }`: holds arriving over links and placed + here, from their arrival to their end, bound or not (a hold waiting for a missing provider keeps a worker alive; + `host.references` lists bound references only; a relayed hold counts at its destination), method calls running here (local and remote), job runs running or finishing here, objects lingering after their last reference. Counted where the work happens; the changes of one microtask turn are one publication (`[["r", counts]]`), a slow watcher gets the newest counts; nothing is scheduled while nobody watches or derives from it. Chord counts; the app's supervisor decides. @@ -176,10 +182,12 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded: here, plus its own id; the last 32); honoured from trusted links only, relays forward it as it is. A provider refuses a hold whose chain names the instance it would bind while that activates: broken `invalid/cycle` naming the cycle by label where known ("app.a(1) → … → app.a(1)"); a local `self.get` of an activation in the chain is refused the same - way. A dependency failing with `invalid/cycle` fails its dependent broken with that error (owner, an exception to - R18's transient dependency failures: a retry would meet the same cycle). Known limitation: a chain is fixed when its - object is created, so a cycle entering an object that is already activating (two activations started at the same - moment) is not detected: both fail `transient` at `readyTimeout` and retry with backoff. + way. A dependency failing with `invalid/cycle` makes its dependent broken with that error (it would wait on a ring + that never heals). The gap (owner, 2026-10-29: no probe): a chain is fixed when its object is created, so a ring + across hosts whose objects start at the same moment is not detected, and without a dependency deadline it stays + parked; an attempt parked longer than its `readyTimeout` on a remote attempt that is only `loading` (its hold on a + link: the clock and `since` run only while it is, so a late link is not reported at once) is reported once per + parking episode (`parked_long`), nothing fails. A cycle is a plugin bug: see the authoring rules below. - `onError(error: HostError)` gets everything that fails in the background, by `kind` (`failure` where the value is a `FailureCause`, `error` otherwise; a position is `{ kind: "on" | "onLive" | "native", position }`, from 1): `activation_failed` (`object`, `failure`: every failed attempt), `object_failed` (`object`, `failure`: a live object @@ -192,7 +200,9 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded: route's `resolve` rejected; the address waits its backoff), `routine_failed` (`object`, `routine`: its position, `error`: a routine call threw, or its watch could not be taken; a source's failure calls never reach a routine), `job_failed` (`object`, `job`, `id`, `error`: a run threw, not after an abort), `job_finish_failed` (the same: a `finish` threw; it is retried) and - `job_dropped` (`object`, `job`, `id`: a blocked run dropped without `finish`, by `abort` or its object's end), and + `job_dropped` (`object`, `job`, `id`: a blocked run dropped without `finish`, by `abort` or its object's end), + `parked_long` (`object`, `waitingFor`: the remote blocker's label, `since`: when it parked; once per parking episode, + slice 7a), and the registries' (slice 8, each once per episode) `registry_conflict` (`registry`, `message`, `plugins`: a conflict `compose` resolved), `registry_failed` (`registry`, `error`: `compose` or an older version's `value` adapter threw), `contribution_failed` (`registry`, `plugin`, `type?`, `error`: a watchable contribution failed or ended, or its @@ -207,11 +217,13 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded: batch, anything else is a plain value whatever methods it has (registries carry natives); a registry value closing a cycle of registries (through the tables and the factory's own earlier contributions) is refused `invalid/cycle` at `contribute` (an edge lives as long as its contribution is in a table). Every served registry composes once at the end of `load`'s setups, before any object activates. - `self.registry(R)` is a declared dependency: with no value at activation (no provider, a first `compose` that threw, - a version not served here) the attempt fails `dependency_failed` (`transient`, the registry's error as the cause, - the references kept) at once, and the registry's first value retries it (R18); a version the provider does not - serve fails it `broken` (`contract_mismatch`: only a reload helps). `host.registry(R, ctx)` waits for `load` and - rejects without a value. + `self.registry(R)` is a declared dependency: without a usable value at its version (no provider: + `unavailable/undeclared`; a first `compose` or the version mapping that threw; a version the provider does not serve: + `contract_mismatch`) the attempt parks `failed { waiting }` naming the registry, and every change of whether the value + is usable wakes it. `host.registry(R, ctx, { readyTimeout? })` waits for the publication, then the same way up to its + deadline, and rejects with the last error. The last good composition is kept only while every contribution it was + built from is still in the table (a contributor that fails or ends drops it: `.value` throws until `compose` + succeeds). - **Wake signals** (`self.retryOn(...sources)`, D2): registry values (constants are accepted and never wake; anything else throws: a reference's member, the attempt's writable, `ref.state`). The object follows them from the factory's return (before hydration, references and natives, so a failed open still has them) until the next attempt's @@ -227,37 +239,47 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded: `host.root` at once (an uncooperative `fn` runs on and fails at its next use of Chord); an aborted `ctx` runs no `fn`; `host.close` rejects it at once `released/closed`, as a method's caller when its object ends; after `close` it rejects `released/closed`. Not counted in `host.activity`. -- `host.load([{ name, facet }], ctx)`: once per host; runs every setup, then activates every type with empty params - and resolves when each has had its first outcome, live or failed (the start barrier; later retries never hold it). - A setup that throws rejects it with that error and closes the host as `close` does, its links included. Gets wait - for it. +- `host.load([{ name, facet }], ctx)`: once per host; runs every setup and publishes them in one batch (types and + registries served, registries composed once, contributions); requests made here and holds arriving before that + waited in the unbound index (`loading`) and are placed after the batch (they never wait for the barrier: two hosts + whose objects need each other's types start without a deadlock). Then it activates every type with empty params and + resolves when each has its first outcome (decision (a)): live, failed by its own cause, parked with no blocker that + is an object here still without its first outcome (a keyed pin still loading counts as here; a wait on another host + or on a failed dependency ends the first pass), or ended (`close` settles a waiting `load`); later retries never + hold it. A setup that throws rejects it with that error and closes the host as `close` does, its links included. - `host.hold(Type, params, ctx, { readyTimeout? })`: a reference at once (`Held`: its `variant` is `undefined` until - its first binding), which follows its object until released and never rejects: `loading` while `load` runs, while - the object activates or retries, while there is no parent link or it is down; `failed` while the object is failed - and no attempt runs; `live`. Throws at once for invalid params (`invalid/params`) and a local type not served here - (`unavailable`); before `load` resolved - these are decided once it did, and the reference fails `broken` with that error. A `hostAddress` or `authorize` that - throws or refuses, or a host address whose params fail its kind's check (`invalid/host_address`), fails the reference - `broken`. A local type never crosses a link. Where a reference made here (`host.get`/`hold`, `self.get`, `ctx.get`) - or a hold arriving on a link goes, first match (`Destination`; the address is the contract's `hostAddress` of the params, - computed by the caller; a hold carries it to be routed, and a host serving the type computes it again): + its first binding), which follows its object until released and never rejects: `loading` before the publication, + while the object activates, retries or is parked on dependencies under way, while there is no parent link or it is + down; `failed` while the object is failed and no attempt runs, or while nothing fits it (`waiting`: not served where + it goes, another version; bound when a publication serves it) or its object is parked on a failed dependency; + `live`. Throws at once only for invalid params (`invalid/params`, also before `load`). A `hostAddress` or + `authorize` that throws or refuses, or a host address whose params fail its kind's check (`invalid/host_address`), + fails the reference `broken`. A local type never crosses a link. Where a reference made here (`host.get`/`hold`, + `self.get`, `ctx.get`) or a hold arriving on a link goes, by address alone (`Destination`; the address is the + contract's `hostAddress` of the params, computed by the caller; a hold carries it to be routed, and a host serving + the type computes it again): | Case | Goes | | --- | --- | - | the type is served here and its address is this host's | here | - | a reference made here to a served type whose address is another host's | on, as if not served here (rows below): worker S1 reaches `Notes { session: "S2" }` through the parent | - | an arriving hold for a served type whose params place it on another host | broken `invalid/host_address` (misrouted; never forwarded) | + | a local type; a remote type whose address is this host's, served here or not; a hold naming this host's address | here, checked against what is served here (below) | | the address is of a kind this host has a route for | over that route's link (the params checked against the kind, `authorize` first; `forbidden` when it refuses) | | otherwise (a remote type) | over the parent link, open yet or not; a hold that arrived *on* the parent link is never sent back: `unavailable` | | an arriving hold and no parent link | `unavailable` | - A hold that arrives and goes on is relayed (`remote/`): this host holds it upstream as the arriving hold's caller - and forwards between the two without parsing params, arguments, results or state. + Placed here, a request is checked against the type served here only (never a same-name registration elsewhere): + not served (or served as the other kind) or not at the caller's version and fingerprint, or params the provider's + own `hostAddress` places on another host (code skew; checked for an arriving hold and for a remote request made + here alike), it waits unbound `failed { waiting }` (`unavailable`, `contract_mismatch`, `invalid/host_address`), + indexed under its name on this host (`unbound.ts`) until a publication wakes it (a refused hold also by a `retry`, + which re-runs its start: `loading`, then `bound` or `failed`); invalid params, a cycle and a provider's + `hostAddress` that throws (its own code) are `broken`, never indexed. A hold that arrives and goes on is relayed (`remote/`): this host + holds it upstream as the arriving hold's caller and forwards between the two without parsing params, arguments, + results or state. - `host.get(Type, params, ctx, { readyTimeout? })`: a hold, awaited until live. Each getter has one deadline for its - whole wait, `load` and reconnects included. A broken failure rejects it at once with its error; through transient - failures and drops it waits, and at its deadline rejects with the last failure's error (`unavailable` "did not - become live within …" when none failed yet). A getter whose deadline passes or whose context is cancelled releases - only its own reference. + whole wait, the publication and reconnects included. A broken failure rejects it at once with its error; through + `loading`, `failed { transient }`, `failed { waiting }` and drops it waits, and at its deadline rejects with the last + failure's error (`unavailable` "did not become live within …" when none failed yet). A getter whose deadline passes + or whose context is cancelled releases only its own reference. - `host.serve(Type, factory, { linger? })` in a facet's setup: a live object with params whose last reference is released stays live for `linger` ms (0 by default), with its documents and dependencies; a reference within that time reuses it, otherwise it ends. An object waiting, activating or failed when its last reference goes ends at once; @@ -298,7 +320,9 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded: at `host.close`. When this host holds over a route link it sends whom each hold acts for (the table in `remote/`): a guest's handler that `ctx.get`s a worker type reaches the worker as the guest. - `host.close(ctx)`: aborts every route's `resolve`, closes its links first, children, routes, then the parent (no - `gone` goes out), ends every remote reference (`released/closed`, parked calls rejected with it), then ends every + `gone` goes out), ends every remote reference (`released/closed`, parked calls rejected with it), every request + still unbound (waiting for the publication or a provider) and every waiting `host.registry` (`released/closed`, at + once, not after the slow closes below), then ends every object at once (calls fail `released/ended`, `self.tx` throws; documents stay readable until it is done), aborts pending opens, waits for endings already running, then drains and closes object by object, the last to go live first (a dependent before its dependencies), with reason `close`. Every call returns the same closing. It rejects with an `AggregateError` of the closes that threw. It does not wait @@ -366,7 +390,8 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded: loading or failed; at runtime it carries every member of every variant, and one outside the bound variant fails `invalid/member`. - `ref.state` (a `NativeWatchable`, made on first access): `{ kind: "loading" } | { kind: "live" } | { kind: - "failed"; cause; retryAt }` (`retryAt`: when the object's backoff window ends, `undefined` for a broken failure) `| + "failed"; cause; retryAt }` (`retryAt`: when the object's backoff window ends, `undefined` for a `broken` or + `waiting` cause) `| { kind: "ended"; error }` (for good: `released` without a reason when its holder released it, `released/ended` when its object ended, `released/closed` at host close, `forbidden/revoked`). A watch gets the current state, then every change; it ends `ended` when the reference ends. What a @@ -382,16 +407,18 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded: Calls made after the reference went live never overtake queued ones (a call a handler makes while they are sent queues behind them). A parked call is checked against its deadline when it is about to be sent, and against the variant it is sent on; it also leaves the queue when its caller cancels (rejected with the reason), the reference - fails (the cause's error) or is released (`released`). A `host.hold` released before `load` resolved makes nothing. + fails (the cause's error) or is released (`released`). A `host.hold` released before the publication makes nothing. - A local reference follows its object's state members through member sources (`member-source.ts`, one per object and member): when a new attempt goes live every member re-binds by name to it (watches get one frame from the last value, other objects' derives over the member recompute, a failed one included), as a remote reference's re-hold does. The keys a reference holds on a keyed member (`at(key)`, `watch(key)`) follow the object the same way (`keyed-members.ts`, one source per member and key; the new attempt's member loads them before it goes live). A key - whose first load is still in flight when its object fails or ends is given up: the activation waiting on it fails - `transient` (R18) and retries on the next attempt; the late pin is released once. + whose first load is still in flight when its object fails or ends is given up: the parked attempt waiting on it shows + `failed { waiting }` and loads it again at the next change of that reference (its object's new attempt going live) + or a `ref.retry()`; the late pin is released once. - `ref.retry()`: a failed object (broken too) gets an attempt now, its backoff window skipped; every reference to it - goes `loading`. Ignored while an attempt runs, once live and once released; over a link it is a `retry` message. + goes `loading`. One parked `failed { waiting }` passes it on (above). Ignored while an attempt runs otherwise, once + live and once released; over a link it is a `retry` message carrying its operation. Rules: @@ -402,23 +429,48 @@ Rules: `self.get` in a remote object's factory as that object's principal, in a local object's factory as the host's. Over a route link they travel with the hold; over the parent link the parent acts as this host (upward is a boundary: a worker that acts for a user towards the server passes the user as data and the server object decides). -- **Activation**, one attempt: the factory; singleton documents hydrated; references bound and live; natives opened one - after another in declaration order (an opener may read documents, references and earlier natives, not derives); - members checked; derives computed; live. An attempt not live within `readyTimeout` is abandoned: its openers' - contexts are aborted, what it opened is closed (`end`), an open resolving later is closed at once. +- **Activation**, one attempt: the factory; singleton documents hydrated; its requests made (references bound, + registries read); parked until every dependency is live (below); then its own steps: natives opened one after + another in declaration order (an opener may read documents, references and earlier natives, not derives), followed + sources attached, members checked, derives computed; live. An attempt whose own steps do not finish within + `readyTimeout` (the time parked not counted) is abandoned: its openers' contexts are aborted, what it opened is + closed (`end`), an open resolving later is closed at once. +- **Parked attempts** (dependency model §3, slice 7a; `dependencies.ts`): a dependency is a `self.get` reference + (local or remote), a `self.registry` value or a keyed pin (`ref.member.at(key)`, loaded once the reference is live); + `self.follow` is an own step. While one is not live the attempt is parked and its object shows `loading` (all under + way), `failed { waiting }` with `unavailable` "waiting for X: " (X the first failed one, the root cause + found through X's message; always these two, so a ring of waiting attempts settles after one round; within 500 + characters, X's label clipped first; X's error as `cause`), or fails `broken` with the error of a dependency that + failed for a cycle, for authority (`forbidden`: an `authorize` refused it) or ended. A getter's deadline rejects + with the last cause it showed, also after the dependency fell back to `loading`. No timer runs (no dependency deadline, no backoff; the `readyTimeout` + deadline paused); the factory never runs again; every change of a dependency (its state, a new cause by value, its + first outcome) makes it check again once after the batch that made it; once all are live it goes on to its own + steps, and a dependency leaving `live` after that changes nothing. `ref.retry()` on a parked `failed { waiting }` + object passes on to its failed dependencies once per retry operation (a uuidv7 that travels in the `retry` + message with the instance ids of the objects it passed through: one already in that list drops it, so a ring across + hosts stops after one lap whatever the number of concurrent retries, and the last few ids an object handled + shortcut a diamond: a `broken` or + `transient` one starts an attempt, a parked one passes it on, a request waiting in an index is left alone, a + refused hold over a link re-runs its start); a `retryOn` wake does nothing while parked. A parked object whose last reference goes ends at once. +- **Authoring rules against cycles** (dependency model §7; a cycle is a plugin bug, and a ring across hosts started at + the same moment is only reported, `parked_long`): (1) a factory declares only what the object needs, never a type + that needs it back; (2) back-references and occasional lookups go into methods and routines (`ctx.get`); (3) data + flows up through state and registries, not mutual `self.get`; (4) host-lifetime objects on the server never declare + worker types in their factory. - **Failures** become a `FailureCause` with a ChordError; the kind comes from where it failed: the factory, a singleton's `initial()`, the member check (a native watchable as state included), a first derive compute or a - dependency cycle are `broken`; a throwing open is `transient` unless it threw `broken(error)`; the `readyTimeout` is - `transient`; a dependency that failed (R18) is `transient` with `unavailable` "A: dependency B failed" and B's error - as `cause`. A keyed document's `initial()` runs in a later transaction and fails only that transaction. A cycle - (`a(1) → b(1) → a(1)`) fails every object on it. + dependency cycle are `broken`; a throwing open or a failed `self.follow` attach is `transient` unless it threw + `broken(error)`; the `readyTimeout` is `transient`; a dependency that failed is no failure of the dependent (it + parks `failed { waiting }`, above), one that failed for a cycle, for authority (`forbidden`) or ended makes it + `broken`. A keyed document's + `initial()` runs in a later transaction and fails only that transaction. A cycle (`a(1) → b(1) → a(1)`) fails every + object on it. - **Retries:** one attempt per object at a time. A transient failure is retried after a backoff per identity: 1 s, doubling to 60 s, reset by a success; a new object for an identity inside its window waits for the window's end. - A broken failure is retried only by `ref.retry()` (and reload, slice 7, `retryOn`, slice 8); a new object after its - end tries at once. Only a dependency failure, or a `readyTimeout` while waiting for dependencies, keeps the object's - references; when that dependency goes live (a remote one: bound again after a drop) the object is retried at once. A dependency failure is transient whatever the dependency's kind: a dependent of a broken dependency is - retried with its backoff for as long as it is held. A retry binds its references before the previous attempt's are - released, so a dependency held only by the retrying object never ends in between. + A broken failure is retried only by `ref.retry()` (and reload, slice 7b, `retryOn`, slice 8); a new object after its + end tries at once. A failed attempt releases its references once its natives closed; the next attempt makes new + ones. A dependent of a failed dependency is not retried at all: it is parked and goes on when the dependency is + live (no backoff of its own). - **Ending:** transactions refused and the callers of running calls rejected `released` at once, their handlers' contexts aborted, routines' and job runs' too (their waiters rejected); then the drain (the handlers', routine calls' and runs' own promises, a run's `finish` included, up to `closeTimeout`), the natives closed in reverse @@ -428,8 +480,8 @@ Rules: - Calls check the method against the bound variant first (`invalid/member`), then the arguments against the provider's current version (`invalid/member`, `invalid/args`); on one host arguments and results pass by reference. - **Release** rejects the calls made through the reference with `released` and aborts their handlers' contexts, on one - host as over a link. A get needs the provider's version and fingerprint (`contract_mismatch`); an unserved - type is `unavailable`, a closed host `released`. + host as over a link. A get needs the provider's version and fingerprint (`contract_mismatch`, `waiting`); an + unserved type is `unavailable` (`waiting`); both wait up to the getter's deadline; a closed host is `released`. - Watches taken through a reference end when it is released, even while others keep the object alive. - A failure's error keeps its stack frames' receivers alive until its `stack` is read (V8), its failed attempt among them (natives closed, documents dropped). An object keeps its last failure's error until it goes live or ends; an @@ -470,30 +522,35 @@ compression bomb), `18-routines.ts` (a reconciling `on` with a retry, a periodic job), and (slice 6a) `20-tables.ts` (tables, writes, pages, counts), `21-persisted-documents.ts` (documents across two hosts on one store file), `22-migrations.ts` (`evolveTable` and an eager migration at activation), `23-keyed-watches.ts` (`watch(key)`, `at(key)`, the `retired` flag), `24-keyed-members.ts` (a keyed member pinned per -key by its consumers) and `25-export-import.ts` (an archive in JSONL and an import). +key by its consumers) and `25-export-import.ts` (an archive in JSONL and an import), and (slice 7a) +`31-parked-dependencies.ts` (a missing provider arrives: causes shown, no timers) and `32-startup.ts` (two hosts that +would have deadlocked at startup). ## For Chord's layers (`internal.ts`) For `remote/`: `ChordHost` (`place` decides where an arriving hold goes, here or relayed over a path; `serving` finds the object a hold placed here binds to; `relays` and `relayOrigin`, the origins this host relays holds for by -their number; `pendingLoad`, `closing`, `linkOpened`, `unlink`), `AnyHost` (a host whatever its principal type: `ChordHost

    ` is invariant in `P`), +their number; `published`, `index`/`unindex` (the unbound index), `closing`, `linkOpened`, `unlink`), `AnyHost` (a host whatever its principal type: `ChordHost

    ` is invariant in `P`), `Target` (what a binding is bound to, implemented by `ChordObject` and by `remote/`'s `RemoteObject`), `ChordObject` (`state`, `retry`) and `Holder` (what holds an object: a binding, or a hold over a link: `moved()` when the object's `state` changed, which a new holder also reads itself; `end(error)`), `Binding` (`moved`, `end`), -`CheckedParams`, `Caller` (principal and incarnation of a reference's maker or a hold's origin), `RefState` with `LOADING`, `LIVE`, `RELEASED`, and -`settlement`. Also on `ChordHost` for `remote/`: `revoke` (a parent's `revoke` message), `compression`, `count(kind, +`CheckedParams`, `Caller` (principal and incarnation of a reference's maker or a hold's origin), `RefState` with `LOADING`, `LIVE`, +`RetryOperation` (one `ref.retry()` passing on), `Serving` (what a hold placed here binds to, or why it waits) and +`UnboundRequest` (what the unbound index holds). Also on `ChordHost` for `remote/`: `revoke` (a parent's `revoke` message), `compression`, `count(kind, delta)` (the keep-alive counts), `cycle(chain, id, closing)` (the `invalid/cycle` error naming a chain by labels), `unplaced` (the path of remote references failed for a part past the limits) and `pathOf(Type, params, principal)` (placing one anew on `retry`). Inside the directory: `ChordHost` (types, objects by identity, facets, the batch, the backoff map, the endings new -instances wait for, its links and the heartbeat timer, its routes and the parent path; `reference` checks a -reference's route, `bind` binds it to an object here or a remote object over a path, `acquire` gets one for a -caller), `Destination` (where a reference or an arriving hold goes: here, a route's path or the parent's; whether a -hold served here is placed here), `Route` (one per routed kind: per address its `Path`, at most one running +instances wait for, its links and the heartbeat timer, its routes and the parent path, the publication and the +unbound index; `reference` makes a request's binding, `bind` binds it to an object here or a remote object over a +path or leaves it waiting, `acquire` gets one for a caller), `UnboundIndex` (requests that cannot bind yet, by name), +`checkDependencies` and `ParkedLong` (what a parked attempt's dependencies say; the `parked_long` report), +`Destination` (where a reference or an arriving hold goes, by address: here, a route's path or the parent's; whether +a hold served here is placed here), `Route` (one per routed kind: per address its `Path`, at most one running `resolve`, the backoff per address; an address is kept while it has holders, a link or a running `resolve`), -`ChordObject` (one per identity: its state across attempts, holders, retries, cycle checks with its instance id and -the chain it was created for, lingering, ending and revocation, the count of its running and finishing job runs, +`ChordObject` (one per identity: its state across attempts (what a parked attempt shows), its first outcome, holders, +retries passed on while parked, cycle checks with its instance id and the chain it was created for, lingering, ending and revocation, the count of its running and finishing job runs, which keeps it like a reference: `countRuns`, its contributions while live and its wake signals), `RetryWakes` (an object's `retryOn` subscriptions), `RootScopes` (the open `host.root` scopes, sealed once each), `Activity` (`host.activity`: counters, one publication per microtask turn), `References` (one type's presence list), `RevocationKeys` (each principal's keys, diff --git a/packages/chord/src/next/host/attempt.ts b/packages/chord/src/next/host/attempt.ts index 259b6a0cf..1fd546056 100644 --- a/packages/chord/src/next/host/attempt.ts +++ b/packages/chord/src/next/host/attempt.ts @@ -8,7 +8,7 @@ import type { JsonValue } from "../../json.ts"; import type { NativeWatchable, Watchable, WatchEnd } from "../../watch.ts"; import type { Contract, Implementation, ParamsOf } from "../contracts/index.ts"; import { contractData, type ShapeOf } from "../contracts/internal.ts"; -import { ChordError, type FailureCause, isChordError, type Principal } from "../data/index.ts"; +import { ChordError, type FailureCause, type Principal } from "../data/index.ts"; import { asChordError, checkArgs, checkParams, invalid, openFailure } from "../data/internal.ts"; import type { HandleOf, JsonObject, StateDefinition, Tx } from "../documents/index.ts"; import { @@ -22,7 +22,12 @@ import { import type { JobDefinition, JobHandle, JobImplementation } from "../jobs/index.ts"; import { type AnyJobImplementation, Job } from "../jobs/internal.ts"; import type { Contribution, Registry, RegistryValue } from "../registries/index.ts"; -import { type RegistryContribution, type RegistryEdge, RegistryValueSource } from "../registries/internal.ts"; +import { + type RegistryContribution, + type RegistryEdge, + RegistryValueSource, + type RegistryWaiter, +} from "../registries/internal.ts"; import type { Writable } from "../state/index.ts"; import { Derive, type DeriveSource, WritableSource } from "../state/internal.ts"; import type { Columns } from "../store/columns.ts"; @@ -33,6 +38,7 @@ import { type Table, TableHandle } from "../store/table-handle.ts"; import { prepareTables } from "../store/table-migration.ts"; import { CallContext, type Caller, type Ctx, withLinkedAbortSignal } from "./ctx.ts"; import { Deadline } from "./deadline.ts"; +import { checkDependencies, type Dependency, ParkedLong, waitingOn } from "./dependencies.ts"; import { type FollowAttach, FollowSource } from "./follow.ts"; import type { HostError } from "./host.ts"; import { attachDerives, checkImplementation, contributionOf, type KeyedBinding, type Members } from "./members.ts"; @@ -46,7 +52,8 @@ import { type NativeSlot, } from "./native.ts"; import type { ChordObject } from "./object.ts"; -import type { Binding, CheckedParams, Reference, Target } from "./reference.ts"; +import type { Failed } from "./ref-state.ts"; +import type { Binding, CheckedParams, Reference, RetryOperation } from "./reference.ts"; import { type RetrySource, retrySource } from "./retry-on.ts"; import { type OnHandler, @@ -84,8 +91,9 @@ export interface Self

    extends RoutineDeclaratio get(Type: C, params: ParamsOf): Reference; /** * Registry `R`'s composed value as this plugin's version of `R` sees it: a declared dependency, as `get` is - * (activation waits until it has a value; with none it fails `transient` and is retried when one arrives, R18). - * Read `.value` at use; once live, a later failure of the registry keeps its last good value. + * (the attempt parks until it has a usable value: `failed { waiting }` while it has none, woken when it has). Read + * `.value` at use; once live, a later failure of the registry keeps its last good value while every contribution it + * was built from is still in the table. */ registry(R: Registry): RegistryValue; /** @@ -150,17 +158,14 @@ export type Factory = Shape ? (params: ParamsOf, self: keyof ParamsOf extends never ? StoreSelf

    : Self

    ) => Implementation> : (params: ParamsOf, principal: P, self: Self

    ) => Implementation>; -/** How an activation fails on a declared registry without a value (`#readRegistries`). */ -type UnreadRegistry = Extract; - /** No declarations of a kind: the shared empty list. */ const NONE: readonly never[] = []; /** An attempt's registry reads, contributions and wake signals, and its writables (slice 8). */ type RegistryDeclarations = { - /** The registry values it declared (`self.registry`), registered as read during activation (`reads`). */ + /** The registry values it declared (`self.registry`): dependencies, as its references are. */ readonly registries: RegistryValueSource[]; - /** Releases its registry reads (with its references). */ + /** Releases its registry reads (with its references): a read keeps an older version's mapping. */ readonly reads: (() => void)[]; /** Its wake signals (`self.retryOn`), handed to its object right after the factory. */ readonly retrySources: RetrySource[]; @@ -171,26 +176,29 @@ type RegistryDeclarations = { contributions: readonly RegistryContribution[]; }; -/** How an attempt's activation ended. `dependency_failed` (R18) keeps the attempt's references. */ +/** How an attempt's activation ended. */ type Activation = | { readonly kind: "live" } | { readonly kind: "failed"; readonly cause: FailureCause } - | { readonly kind: "dependency_failed"; readonly cause: FailureCause } | { readonly kind: "ended"; readonly error: ChordError }; +/** How an activation that did not go live ended. */ +type NotLive = Exclude; + /** The error of an object, or a getter, that waited `ms` without the object going live. */ export function notLive(label: string, ms: number): ChordError { return new ChordError({ code: "unavailable", message: `${label}: did not become live within ${ms} ms` }); } /** - * One attempt of an object. Activation: factory, documents hydrated (the object's slots), references bound (before the - * previous attempt's are released) and live, natives opened one after another, members checked, derives computed; - * bounded by the host's `readyTimeout`. Ending: sealed at once (calls aborted, its transactions refused and the one - * holding the line discarded, its job invocations cut off), then the drain, the closes in reverse, derives detached; - * its object releases its references. Documents and job runs stay on the object. + * One attempt of an object. Activation: factory, documents hydrated (the object's slots), references bound, then parked + * until every dependency is live (no timer: the deadline paused), then its own steps: natives opened one after + * another, followed sources attached, members checked, derives computed; bounded by the host's `readyTimeout`. + * Ending: sealed at once (calls aborted, its transactions refused and the one holding the line discarded, its job + * invocations cut off), then the drain, the closes in reverse, derives detached; its object releases its references. + * Documents and job runs stay on the object. */ -export class Attempt implements StoreSelf, KeyDeclarer { +export class Attempt implements StoreSelf, KeyDeclarer, RegistryWaiter { readonly #object: ChordObject; /** The object's context: the natives' closes run under it. */ readonly #context: Context; @@ -228,18 +236,21 @@ export class Attempt implements StoreSelf, KeyDeclarer { * the call path costs no more than before the cleanups. */ readonly #running = new Map, (() => void) | undefined>(); - /** The previous attempt while it still holds its references (R18): released once this one has bound its own. */ - #previous: Attempt | undefined; /** What activation returns once the attempt was abandoned: a timeout, a cycle through its object, or its end. */ - #abandoned: Exclude | undefined; + #abandoned: NotLive | undefined; + /** Its dependencies while it is parked on them (the dependency check); `undefined` otherwise. */ + #parked: readonly Dependency[] | undefined; + /** A dependency changed since the last check: the parked attempt checks again (one check per batch of changes). */ + #moved = false; + /** What a parked attempt waiting for a change waits on; resolved after the change's batch. */ + #changed: PromiseWithResolvers | undefined; /** A followed source's end while this attempt activated: it fails as it goes live (if its activation returned). */ #endedEarly: ChordError | undefined; /** Its registry reads, contributions, wake signals and writables, made by the first: most attempts have none. */ #registryDeclarations: RegistryDeclarations | undefined; - constructor(object: ChordObject, previous: Attempt | undefined, context: Context) { + constructor(object: ChordObject, context: Context) { this.#object = object; - this.#previous = previous; this.#context = context; this.#opening = withAbortSignal(this.#abort.signal, context); } @@ -453,7 +464,7 @@ export class Attempt implements StoreSelf, KeyDeclarer { } /** Abandons the activation with `outcome` (once): its waits stop and the openers' contexts are aborted. */ - abandon(outcome: Exclude): void { + abandon(outcome: NotLive): void { if (this.#abandoned !== undefined) return; this.#abandoned = outcome; this.#abort.abort(outcome.kind === "ended" ? outcome.error : outcome.cause.error); @@ -498,6 +509,11 @@ export class Attempt implements StoreSelf, KeyDeclarer { }); } + /** Whether it is parked on its dependencies (the dependency check runs). */ + get isParked(): boolean { + return this.#parked !== undefined; + } + /** Its contributions, resolved by its activation: in the registries' tables while its object is live. */ get contributions(): readonly RegistryContribution[] { return this.#registryDeclarations?.contributions ?? NONE; @@ -624,9 +640,8 @@ export class Attempt implements StoreSelf, KeyDeclarer { if (declarations !== undefined) for (const writable of declarations.writables) writable.detach(ended); } - /** Releases its references and registry reads, and the previous attempt's if this one never bound its own. */ + /** Releases its references and registry reads. */ releaseReferences(): void { - this.#releasePrevious(); for (const { binding } of this.#references) binding.release(); const declarations = this.#registryDeclarations; if (declarations !== undefined) for (const release of declarations.reads.splice(0)) release(); @@ -635,8 +650,7 @@ export class Attempt implements StoreSelf, KeyDeclarer { /** The activation steps; each catches its own failures with their kind (the table in slice 2's spec). */ async #activate(deadline: Deadline): Promise { let implementation: unknown; - let dependencies: Target[]; - let unread: UnreadRegistry | undefined; + const dependencies: Dependency[] = []; try { implementation = this.#declare(); // Right after the factory, before anything can fail: a failed attempt keeps its wake signals. @@ -650,49 +664,158 @@ export class Attempt implements StoreSelf, KeyDeclarer { // A persisted job's runs, loaded once per object: rebuilt before anything runs. for (const job of this.#jobs.values()) await job.load(this.#opening); const { host } = this.#object; - dependencies = this.#references.map(({ binding, contract, params }) => - host.bind(binding, contract, params, this.#opening), - ); - for (const dependency of dependencies) this.#object.assertNoCycle(dependency, this.#opening); - if (this.#registryDeclarations !== undefined) unread = this.#readRegistries(this.#registryDeclarations); + for (const { binding, contract, params } of this.#references) { + host.bind(binding, contract, params, this.#opening); + dependencies.push({ kind: "reference", binding }); + } + // Once all are bound: a dependency made here may have bound back to this object meanwhile. + for (const dependency of dependencies) { + const target = dependency.kind === "reference" ? dependency.binding.target : undefined; + if (target !== undefined) this.#object.assertNoCycle(target, this.#opening); + } + const declarations = this.#registryDeclarations; + for (const value of declarations?.registries ?? NONE) { + // Read from here until its references are released: an older version's mapping is kept meanwhile. + declarations?.reads.push(value.read()); + dependencies.push({ kind: "registry", value }); + } } catch (error) { return this.#broken(error); + } + // Parked only when something is not live yet (most activations find their dependencies live). + if ((dependencies.length > 0 && checkDependencies(dependencies).kind !== "live") || this.#pinsKeys()) { + const parked = await this.#park(dependencies, deadline); + if (parked !== undefined) return parked; + } else { + // Its own steps start in a later turn: an object ended in the turn it was made opens nothing. + await undefined; + } + return this.#ownSteps(implementation); + } + + /** + * Parks while a dependency is not live (dependency model §3), with no timer: the deadline paused, `parked_long`'s + * report the only clock. Checks at every change of a dependency (after its batch) and shows what it found on its + * object; once all are live, the keys the factory pinned on them are loaded (a failed load parks it `waiting` + * until the next change or a retry). Resolves `undefined` once all are live and pinned, else with how it ends: + * abandoned, or broken (a cycle, an ended dependency). + */ + async #park(dependencies: readonly Dependency[], deadline: Deadline): Promise { + const object = this.#object; + deadline.pause(); + this.#parked = dependencies; + object.waitsFor(dependencies); + // Waiters on a registry are woken when its value turns usable or its failure changes. + const waits: (() => void)[] = []; + for (const each of dependencies) if (each.kind === "registry") waits.push(each.value.wait(this)); + const long = new ParkedLong(object); + try { + for (;;) { + this.#moved = false; + const check = checkDependencies(dependencies); + if (check.kind === "broken") + return this.#abandoned ?? { kind: "failed", cause: { kind: "broken", error: check.error } }; + if (check.kind === "live") { + const pinned = await this.#pinKeys(); + if (this.#abandoned !== undefined) return this.#abandoned; + // Changed while the keys loaded: checked again. + if (pinned === undefined && !this.#moved) { + // Its own steps follow: `loading` until its outcome (a failure shows as the failure, not this). + object.unparked(this); + return undefined; + } + if (pinned !== undefined) object.parked(this, pinned, true); + long.stop(); + } else { + object.parked(this, check.shown, check.firstPass); + long.waitingOn(check.remoteLoading); + } + if (!(await this.#nextChange())) return this.#abandoned!; + } } finally { - // Bound first: a dependency held only by the previous attempt never ends in between. - this.#releasePrevious(); + long.stop(); + for (const release of waits) release(); + this.#parked = undefined; + object.waitsFor(NONE); + deadline.resume(); } - // No value after `load`'s setups: none comes before a first composition (which retries it, R18) or a reload. - if (unread !== undefined) return this.#abandoned ?? unread; + } + + /** + * Waits for the next change of a dependency (at once when one came during the check); `false` when the attempt was + * abandoned meanwhile (its end, the only thing that stops the wait). + */ + async #nextChange(): Promise { + if (this.#moved) return true; + this.#changed = Promise.withResolvers(); try { - await Promise.all(dependencies.map((dependency) => this.#object.waitFor(dependency, this.#opening))); - } catch (error) { - const abandoned = this.#abandoned; - // A timeout while waiting keeps the references (R18): a dependency that goes live later retries this object - // at once (a remote one whose link dropped and is held again). - if (abandoned?.kind === "failed" && abandoned.cause.kind === "transient") { - return { kind: "dependency_failed", cause: abandoned.cause }; - } - if (abandoned !== undefined) return abandoned; - // `waitFor` rejects with a ChordError naming the dependency, its error as the cause; or with a cycle's - // `invalid/cycle`, which fails this object broken too (proposal, slice 4b: an exception to R18, since every - // retry would meet the same cycle), keeping no references. - const failed = error as ChordError; - if (failed.code === "invalid" && failed.reason === "cycle") { - return { kind: "failed", cause: { kind: "broken", error: failed } }; - } - return { kind: "dependency_failed", cause: { kind: "transient", error: failed } }; + await awaitWithContext(this.#changed.promise, this.#opening); + return true; + } catch { + return false; + } finally { + this.#changed = undefined; } - try { - // The keys the factory pinned on its dependencies' keyed members, loaded before this object goes live. - for (const { binding } of this.#references) { + } + + /** + * Loads the keys the factory pinned on its dependencies' keyed members (`at(key)`); `undefined` once all are, else + * what the parked attempt shows for the first that failed to load (`waiting` on its reference). + */ + async #pinKeys(): Promise { + for (const { binding } of this.#references) { + try { const pinning = binding.pinKeys(this.#opening); if (pinning !== undefined) await pinning; + } catch (error) { + return waitingOn(binding.label, asChordError(error, binding.label)); } - } catch (error) { - // A pin load that failed its provider is a dependency failure (rule 6, R18). - const cause: FailureCause = { kind: "transient", error: asChordError(error, this.#object.label) }; - return this.#abandoned ?? { kind: "dependency_failed", cause }; } + return undefined; + } + + /** + * A dependency of this parked attempt changed (its state, its cause, its first outcome): it checks again after the + * batch that changed it, once however many changed. A change while it checks or loads pinned keys is remembered + * (`#moved`); one outside parking costs a flag. + */ + dependencyMoved(): void { + if (this.#moved) return; + this.#moved = true; + if (this.#changed !== undefined) queueMicrotask(() => this.#changed?.resolve()); + } + + /** A registry it is parked on closed with its host: the parked attempt ends with `error`. */ + closed(error: ChordError): void { + this.abandon({ kind: "ended", error }); + } + + /** Whether its factory pinned keys on its dependencies' keyed members (`at(key)`), which load while it is parked. */ + #pinsKeys(): boolean { + for (const { binding } of this.#references) if (binding.pinsKeys) return true; + return false; + } + + /** + * `ref.retry()` on its parked object (`failed { waiting }`): passed on to every failed dependency (a broken or + * transient one retries, a parked one passes it on, a request waiting in an index is left alone), as `operation`; + * then it checks again (a key that failed to load is loaded again). + */ + retryDependencies(operation: RetryOperation): void { + for (const dependency of this.#parked ?? NONE) { + if (dependency.kind === "reference" && dependency.binding.refState.kind === "failed") { + dependency.binding.retry(operation); + } + } + this.dependencyMoved(); + } + + /** + * The steps after the dependency check (its own: natives, followed sources, the member check, derives, the keys + * references pin on its keyed members), under the deadline. A dependency that leaves `live` meanwhile changes + * nothing (R18 after live). + */ + async #ownSteps(implementation: unknown): Promise { for (const native of this.#natives) { // An attempt abandoned before this open (ended while it waited) closed its natives already: none opens late. if (this.#abandoned !== undefined) return this.#abandoned; @@ -711,14 +834,16 @@ export class Attempt implements StoreSelf, KeyDeclarer { return this.#abandoned ?? { kind: "failed", cause }; } } + let members: Members; try { const checked = checkImplementation(this.#object.served.contract, implementation); - this.#members = checked.members; + members = checked.members; + this.#members = members; this.#variant = checked.variant; if (this.#registryDeclarations !== undefined) this.#registryDeclarations.contributions = this.#resolveContributions(this.#registryDeclarations); const { host, label } = this.#object; - attachDerives(this.#derives, this.#members.state, (member) => (error) => { + attachDerives(this.#derives, members.state, (member) => (error) => { // During activation a throwing derive fails the attempt, which `activation_failed` reports alone. if (!this.#isLive) return; host.report({ kind: "derive_failed", object: label, ...(member === undefined ? {} : { member }), error }); @@ -729,40 +854,15 @@ export class Attempt implements StoreSelf, KeyDeclarer { } try { // The keys references pin on the object's keyed members, loaded on this attempt's before it goes live. - const preparing = this.#object.prepareKeys(this.#members.keyed, this.#opening); + const preparing = this.#object.prepareKeys(members.keyed, this.#opening); if (preparing !== undefined) await preparing; } catch (error) { const cause: FailureCause = { kind: "transient", error: asChordError(error, this.#object.label) }; - return this.#abandoned ?? { kind: "dependency_failed", cause }; + return this.#abandoned ?? { kind: "failed", cause }; } return this.#abandoned ?? { kind: "live" }; } - /** - * Registers this attempt's object as a reader of every registry it declared (woken by a first value, R18, until its - * references are released); returns how the activation fails on the first with no value, `undefined` when every - * one has a value. A version the provider does not serve (S2) is final until a reload: broken, the - * `contract_mismatch` itself the cause. Otherwise (no provider, a first `compose` that threw) a value may still come: - * `transient` (R18, the references kept), naming the registry, its error as the cause. - */ - #readRegistries({ registries, reads }: RegistryDeclarations): UnreadRegistry | undefined { - let unread: UnreadRegistry | undefined; - for (const value of registries) { - reads.push(value.read(this.#object)); - const start = value.start(); - if (unread !== undefined || start.kind === "value" || start.last !== undefined) continue; - const { error } = start; - if (isChordError(error, "contract_mismatch")) { - unread = { kind: "failed", cause: { kind: "broken", error } }; - continue; - } - const message = `${this.#object.label}: registry ${value.name} has no value`; - const cause = new ChordError({ code: "unavailable", message }, { cause: error }); - unread = { kind: "dependency_failed", cause: { kind: "transient", error: cause } }; - } - return unread; - } - /** Its declared contributions, as Chord follows them (its references are live now), by its plugin and type. */ #resolveContributions({ declared }: RegistryDeclarations): readonly RegistryContribution[] { const { served, host } = this.#object; @@ -847,11 +947,6 @@ export class Attempt implements StoreSelf, KeyDeclarer { return this.#abandoned ?? { kind: "failed", cause }; } - #releasePrevious(): void { - this.#previous?.releaseReferences(); - this.#previous = undefined; - } - /** The registry edges its declared contributions would add (another registry's value contributed). */ #pendingEdges(): RegistryEdge[] { const edges: RegistryEdge[] = []; diff --git a/packages/chord/src/next/host/backoff.ts b/packages/chord/src/next/host/backoff.ts index bae0a91df..383b04b42 100644 --- a/packages/chord/src/next/host/backoff.ts +++ b/packages/chord/src/next/host/backoff.ts @@ -58,7 +58,7 @@ export class Backoff { return entry?.timer === undefined ? undefined : entry.until; } - /** Ends the window at once (its object is retried now, R18); the delay stays for the next failure. */ + /** Ends the window at once (its object is retried now: `ref.retry()`, a wake); the delay stays for the next failure. */ skip(identity: string): void { const entry = this.#entries.get(identity); clearTimeout(entry?.timer); diff --git a/packages/chord/src/next/host/dependencies.ts b/packages/chord/src/next/host/dependencies.ts new file mode 100644 index 000000000..73381e42e --- /dev/null +++ b/packages/chord/src/next/host/dependencies.ts @@ -0,0 +1,142 @@ +// Parked attempts (dependency model §3, §7): the dependency check, what an attempt's declared dependencies (`self.get` +// references and `self.registry` values) say at one moment: all live, or what the parked attempt shows while one is not +// (`loading`, `failed { waiting }` naming the nearest blocker and the root cause, or `broken` for a cycle, authority or +// an ended dependency), whether this host's first pass can count it (no blocker here still without its first outcome, decision +// (a)), and the remote blocker that is only loading; and the `parked_long` report a long wait on that blocker makes. + +import { type ChordError, isChordError } from "../data/index.ts"; +import { asChordError, waitingFor } from "../data/internal.ts"; +import type { RegistryValueSource } from "../registries/internal.ts"; +import type { ChordObject } from "./object.ts"; +import { type Failed, LOADING, type RefState } from "./ref-state.ts"; +import type { Binding } from "./reference.ts"; + +/** One declared dependency: a reference (`self.get`) or a registry's value (`self.registry`). */ +export type Dependency = + | { readonly kind: "reference"; readonly binding: Binding } + | { readonly kind: "registry"; readonly value: RegistryValueSource }; + +/** + * What the dependencies say. `live`: every one is live (hydrated: a reference's members have values). `parked`: what + * the attempt shows (`loading`, or `failed { waiting }` naming its first failed blocker), whether its first pass is + * over (`firstPass`: no blocker, loading or failed, is an object here still without its first outcome), and a remote + * blocker whose hold is on a link and only loading (its label), for `parked_long`. `broken`: one failed for a cycle + * (`invalid/cycle`), for authority (`forbidden`) or ended: it never comes back by itself. + */ +export type Check = + | { readonly kind: "live" } + | { + readonly kind: "parked"; + readonly shown: RefState; + readonly firstPass: boolean; + readonly remoteLoading: string | undefined; + } + | { readonly kind: "broken"; readonly error: ChordError }; + +const LIVE: Check = { kind: "live" }; + +/** Checks `dependencies` in declaration order (the first failed one is the one the attempt names). */ +export function checkDependencies(dependencies: readonly Dependency[]): Check { + let failed: Failed | undefined; + let firstPass = true; + let remoteLoading: string | undefined; + let loading = false; + for (const dependency of dependencies) { + if (dependency.kind === "registry") { + const { value } = dependency; + const start = value.start(); + if (start.kind === "value" || start.last !== undefined) continue; + failed ??= waitingOn(`registry ${value.name}`, asChordError(start.error, `registry ${value.name}`)); + continue; + } + const { binding } = dependency; + const state = binding.refState; + switch (state.kind) { + case "live": + continue; + case "ended": + return { kind: "broken", error: state.error }; + case "failed": { + const { error } = state.cause; + if (healsNever(error)) return { kind: "broken", error }; + failed ??= waitingOn(binding.label, error); + // Parked `waiting` on something here still in its first attempt: no first outcome yet (fix round 1). + if (binding.target?.blocksFirstPass === true) firstPass = false; + continue; + } + case "loading": { + loading = true; + const target = binding.target; + if (target?.blocksFirstPass === true) firstPass = false; + if (target?.heldOnLink === true) remoteLoading ??= target.label; + } + } + } + if (failed === undefined && !loading) return LIVE; + return { kind: "parked", shown: failed ?? LOADING, firstPass, remoteLoading }; +} + +/** + * Whether a dependency failed with `error` never heals by itself, so its dependent breaks too: a cycle + * (`invalid/cycle`: it would wait on a ring) or authority (`forbidden`: an `authorize` that refused, M§3.6; only + * `retry()` or a reload helps). + */ +function healsNever(error: ChordError): boolean { + return (isChordError(error, "invalid") && error.reason === "cycle") || isChordError(error, "forbidden"); +} + +/** A parked attempt's state for a blocker `label` that failed with `error`. */ +export function waitingOn(label: string, error: ChordError): Failed { + return { kind: "failed", cause: waitingFor(label, error), retryAt: undefined }; +} + +/** + * The `parked_long` report of one parking episode (dependency model §7): once, when the attempt has waited + * `readyTimeout` on a remote blocker that is only loading with its hold on a link (a ring across hosts started at the + * same moment stays parked so; nothing fails or retries). The clock, and the report's `since`, start when such a + * blocker is there and stop when none is (a link that drops, a path still without one): only the time a remote + * attempt answered for the wait counts. Its one timer runs only meanwhile. + */ +export class ParkedLong { + readonly #object: ChordObject; + /** When the attempt began to wait on a remote blocker that is only loading on a link, while it does. */ + #since: number | undefined; + #timer: ReturnType | undefined; + /** The remote blocker it would name now. */ + #blocker: string | undefined; + #reported = false; + + constructor(object: ChordObject) { + this.#object = object; + } + + /** What the check found: `blocker`, a remote attempt that is only loading on a link, or none. */ + waitingOn(blocker: string | undefined): void { + this.#blocker = blocker; + if (blocker === undefined) { + this.stop(); + return; + } + if (this.#reported || this.#timer !== undefined) return; + this.#since ??= Date.now(); + const due = this.#since + this.#object.host.timeouts.ready - Date.now(); + this.#timer = setTimeout(() => this.#report(), Math.max(0, due)); + } + + /** The episode ended, or no remote blocker is loading on a link now: the clock and its timer stop. */ + stop(): void { + clearTimeout(this.#timer); + this.#timer = undefined; + this.#since = undefined; + } + + #report(): void { + this.#timer = undefined; + const waitingFor = this.#blocker; + const since = this.#since; + if (this.#reported || waitingFor === undefined || since === undefined) return; + this.#reported = true; + const object = this.#object; + object.host.report({ kind: "parked_long", object: object.label, waitingFor, since }); + } +} diff --git a/packages/chord/src/next/host/destination.ts b/packages/chord/src/next/host/destination.ts index d92fbb0f1..8a372b257 100644 --- a/packages/chord/src/next/host/destination.ts +++ b/packages/chord/src/next/host/destination.ts @@ -1,6 +1,6 @@ -// Destination (§3, section 1's table): where a reference made on a host, or a hold arriving on one of its links, goes: -// here, over the route of its host address, or over the parent link; and the check that an object served here lives -// here. +// Destination (§3, section 1's table; dependency model §1): where a reference made on a host, or a hold arriving on one +// of its links, goes, by address alone: here when the address is this host's (served here or not), else over the +// route of its host address's kind, or over the parent link; and the check that an object served here lives here. import type { Context } from "../../context/index.ts"; import type { Address, Contract, ParamsOf } from "../contracts/index.ts"; @@ -9,7 +9,6 @@ import { ChordError, type Principal } from "../data/index.ts"; import { invalid } from "../data/internal.ts"; import type { WireAddress } from "../link/internal.ts"; import type { HoldRequest, HostLink, Path } from "../remote/internal.ts"; -import type { Served } from "./object.ts"; import type { Route } from "./route.ts"; /** Where something is placed: here, or over a path (with the host address it was computed or named with). */ @@ -17,47 +16,42 @@ export type Placed = | { readonly kind: "here" } | { readonly kind: "path"; readonly path: Path; readonly address: WireAddress }; -/** A host's destinations: its address, its served types, its routes and its parent path, as the host keeps them. */ +/** A host's destinations: its address, its routes and its parent path, as the host keeps them. */ export class Destination { readonly #address: Address | undefined; - readonly #types: ReadonlyMap; readonly #routes: ReadonlyMap; readonly #parent: Path; - constructor( - address: Address | undefined, - types: ReadonlyMap, - routes: ReadonlyMap, - parent: Path, - ) { + constructor(address: Address | undefined, routes: ReadonlyMap, parent: Path) { this.#address = address; - this.#types = types; this.#routes = routes; this.#parent = parent; } /** - * Where a reference to `Type` with `params` made as `principal` goes: here for a local type (served here: the host - * refused it otherwise) or a remote type served here whose host address is this host's; otherwise over the route of - * its host address's kind, or over the parent link (the host address decides where an object lives: a worker - * reaches another worker's object of a type it serves too). Throws `invalid/host_address` for a host address whose - * params do not fit its kind, `forbidden` when the route refuses, and what a `hostAddress` or `authorize` throws. + * Where a reference to `Type` with `params` made as `principal` goes: here for a local type, and for a remote type + * whose host address is this host's, served here or not (the check against what is served is the host's); otherwise + * over the route of its host address's kind, or over the parent link (the host address decides where an object + * lives: a worker reaches another worker's object of a type it serves too). Throws `invalid/host_address` for a host + * address whose params do not fit its kind, `forbidden` when the route refuses, and what a `hostAddress` or + * `authorize` throws. */ reference(Type: Contract, params: ParamsOf, principal: Principal | undefined, context: Context): Placed { if (Type.kind === "local") return HERE; const address = hostAddressOf(Type, params); - if (this.#types.has(Type.name) && this.#isHere(address)) return HERE; + if (this.#isHere(address)) return HERE; return { kind: "path", path: this.#pathTo(address, principal, context) ?? this.#parent, address }; } /** - * Where a hold arriving on `link` acting as `principal` goes: here when its type is served here; over the route of - * the address it names; else over the parent link (waiting for it while it is down), unless it came on the parent - * link or this host never had one (`unavailable`). Throws as `reference` does. + * Where a hold arriving on `link` acting as `principal` goes, by the address it names: here when it is this host's + * (a relay never captures a hold for serving the same name); over the route of its kind; else over the parent link + * (waiting for it while it is down), unless it came on the parent link or this host never had one (`unavailable`). + * Throws as `reference` does. */ hold(request: HoldRequest, link: HostLink, principal: Principal | undefined, context: Context): Placed { - if (this.#types.has(request.type)) return HERE; const { address } = request; + if (this.#isHere(address)) return HERE; const routed = this.#pathTo(address, principal, context); if (routed !== undefined) return { kind: "path", path: routed, address }; if (link.role === "parent" || !this.#parent.linked) { @@ -67,15 +61,16 @@ export class Destination { } /** - * Throws `invalid/host_address` for a hold of `Type` with `params` (checked) served here but whose host address is - * another host (misrouted; never forwarded). The host address is computed here from the params: the address a hold - * names only routes it. + * The provider's own check of a request placed here for the remote `Type` it serves, with `params` (checked): + * `invalid/host_address` when its `hostAddress` puts the object on another host (code skew: the consumer computed + * this host), `undefined` when it lives here. A `hostAddress` that throws throws (the provider's own code). The + * address a request names only routes it. */ - assertHere(Type: Contract, params: ParamsOf): void { + misplaced(Type: Contract, params: ParamsOf): ChordError | undefined { const address = hostAddressOf(Type, params); - if (this.#isHere(address)) return; + if (this.#isHere(address)) return undefined; // A host serving a remote type has an address (`serve` refuses one without). - throw invalid("host_address", `${Type.name}: placed on ${address.kind}, this host is ${this.#address!.key}`); + return invalid("host_address", `${Type.name}: placed on ${address.kind}, this host is ${this.#address!.key}`); } /** diff --git a/packages/chord/src/next/host/host.ts b/packages/chord/src/next/host/host.ts index 5612e562a..d7d6ef47c 100644 --- a/packages/chord/src/next/host/host.ts +++ b/packages/chord/src/next/host/host.ts @@ -28,6 +28,7 @@ import { import { binaryCodec, type Codec, type Compression, type Link, type PartLimits } from "../link/index.ts"; import { linkLimits } from "../link/internal.ts"; import type { Registry, RegistryValue } from "../registries/index.ts"; +import type { RegistryValueSource } from "../registries/internal.ts"; import { type RegistryContribution, type RegistryEdge, @@ -41,6 +42,7 @@ import { Path, type ProviderSide, RemoteObject, + type Upstream, } from "../remote/internal.ts"; import { Batch, type BatchError } from "../state/internal.ts"; import type { ChordStore, Store } from "../store/store.ts"; @@ -58,6 +60,7 @@ import { Binding, type CheckedParams, type Held, type Reference, type Target } f import { RevocationKeys, type RevokedBy } from "./revocation.ts"; import { type RootOptions, RootScopes } from "./root.ts"; import { Route, type RouteOptions } from "./route.ts"; +import { UnboundIndex, type UnboundRequest } from "./unbound.ts"; import { uuidv7 } from "./uuid.ts"; /** What `createHost` takes; `R` is the app's revocation type (JSON). */ @@ -151,7 +154,12 @@ export type HostError = readonly restarts: number; } /** A blocked run was dropped without `finish`: aborted, or its object ended. */ - | { readonly kind: "job_dropped"; readonly object: string; readonly job: string; readonly id: string }; + | { readonly kind: "job_dropped"; readonly object: string; readonly job: string; readonly id: string } + /** + * An attempt parked `readyTimeout` (since `since`, `Date.now()` time) while `waitingFor`, a remote attempt, is only + * loading: once per parking episode, a diagnostic of a ring across hosts started at the same moment (nothing fails). + */ + | { readonly kind: "parked_long"; readonly object: string; readonly waitingFor: string; readonly since: number }; /** * What `accept` and `connect` take: the principal of the host at the other end, as this host stamps it; `trusted` skips @@ -184,22 +192,26 @@ export type AnyHost = Omit { /** - * Runs every facet's setup, then activates every served type with empty params and resolves when each has had its - * first outcome, live or failed (the start barrier; failures go to `onError`). A setup that throws rejects it and - * closes the host. Runs once. + * Runs every facet's setup and publishes what they declared in one batch (the types and registries served, the + * registries composed once, the contributions); requests that waited for the publication are placed after that + * batch. Then activates every served type with empty params and resolves when each has had its first outcome (the + * start barrier, decision (a)): live, failed by its own cause, parked with no blocker here still without its first + * outcome, or ended (failures go to `onError`). A setup that throws rejects it and closes the host. Runs once. */ load(facets: readonly { readonly name: string; readonly facet: Facet

    }[], context: Context): Promise; /** - * A reference once its object is live; waits for `load` first. It acts as the host's principal and goes where its - * type's host address puts it: here, over a route, or over the parent link (`Destination`). A broken failure - * rejects at once; through transient ones it waits until its deadline, then rejects with the last failure's error. + * A reference once its object is live (and hydrated). It acts as the host's principal and goes where its type's host + * address puts it: here, over a route, or over the parent link (`Destination`). It waits while the reference is + * `loading` (before the host's publication too), `failed { transient }` or `failed { waiting }` until its + * `readyTimeout`, then rejects with the last cause and releases the reference; a broken failure rejects at once. */ get(Type: C, params: ParamsOf, context: Context, options?: GetOptions): Promise>; /** * A reference at once, following its object until released: loading, live, failed (retried as its object is) and - * loading again while its link is down; it never rejects. Throws at once for invalid params (also before `load`) - * and a local type not served here; before `load` resolved, a type not served here fails the reference broken once - * it did. Its `variant` is `undefined` until its first binding. + * loading again while its link is down; it never rejects. Throws at once only for invalid params (also before + * `load`). A type not served where it goes is `loading` before the host's publication and `failed { waiting }` after + * it (bound when a publication serves it); a `hostAddress` or route that refuses fails it broken. Its `variant` is + * `undefined` until its first binding. */ hold(Type: C, params: ParamsOf, context: Context, options?: GetOptions): Held; /** @@ -244,11 +256,12 @@ export interface Host

    ; /** - * Registry `R`'s composed value as `R`'s version sees it, for code outside objects; waits for `load` first. Rejects - * when it has no value then (`unavailable/undeclared` without a provider, a first `compose` that threw, a version - * not served here: `contract_mismatch`); none can come before a reload. + * Registry `R`'s composed value as `R`'s version sees it, for code outside objects, once it is usable: waits for the + * host's publication, then while it has no value (`unavailable/undeclared` without a provider, a first `compose` + * that threw, a version not served here: `contract_mismatch`) until its `readyTimeout`, then rejects with the last + * error. */ - registry(R: Registry, context: Context): Promise>; + registry(R: Registry, context: Context, options?: GetOptions): Promise>; /** * Runs `fn` with a call context for code outside objects (a Durable tool, a CLI): the host's principal (or * `options.principal`), this host's incarnation, `context`'s telemetry and signal, `ctx.get`. When `fn` settles or @@ -263,6 +276,14 @@ export interface Host

    ; } +/** + * What a hold placed here binds to (`serving`): its object, or why it waits unbound (`failed { waiting }`: not served, + * another version, or placed elsewhere by the provider's code). + */ +export type Serving = + | { readonly kind: "object"; readonly object: ChordObject } + | { readonly kind: "waiting"; readonly cause: FailureCause }; + /** Creates a host. */ export function createHost

    ( options: HostOptions, @@ -305,7 +326,11 @@ export class ChordHost

    (); /** Teardowns still running, by identity: the next instance of an identity waits for its previous ones. */ readonly #endings = new Map>(); - readonly #loaded = Promise.withResolvers(); + /** The requests that cannot bind yet, by name: before the publication all of them, then those left unbound. */ + readonly #unbound = new UnboundIndex(); + /** Resolved at the host's first publication (`load`'s batch), or its close. */ + readonly #publication = Promise.withResolvers(); + #published = false; /** Every open link: children, routes and the parent. */ readonly #links = new Set(); /** Where holds go that are not served here and not routed: the parent link, across reconnects. */ @@ -320,7 +345,7 @@ export class ChordHost

    (); #nextOrigin = 1; - /** Aborted when the host closes: the context of every route's `resolve`. */ + /** Aborted when the host closes: the context of every route's `resolve`, and the end of waiting `host.registry`s. */ readonly #routing = new AbortController(); readonly #heartbeat: { readonly ping: number; readonly close: number }; /** Its registries by name, each made by its first use (a serve, a contribution or a reader). */ @@ -344,7 +369,7 @@ export class ChordHost

    | undefined); this.timeouts = { close: options.closeTimeout ?? 30_000, ready: options.readyTimeout ?? 60_000 }; this.#heartbeat = { ping: options.heartbeat?.ping ?? 15_000, close: options.heartbeat?.close ?? 30_000 }; - this.#destination = new Destination(this.address, this.#types, this.#routes, this.#parent); + this.#destination = new Destination(this.address, this.#routes, this.#parent); // Chord's own: `openStore` made it. this.store = options.store as ChordStore | undefined; this.store?.attach({ failed: (error) => this.#storageFailed(error), batch: this.batch }); @@ -357,8 +382,8 @@ export class ChordHost

    { for (const contribution of contributions) contribution.record.add(contribution); for (const record of this.#registries.values()) record.composeServed(); @@ -369,12 +394,15 @@ export class ChordHost

    {}); throw error; } + // After the batch: what waited for the publication is placed (the gate lasted only for the setups). + this.#published = true; + this.#publication.resolve(); + this.#unbound.wakeAll(); const objects = [...this.#types.values()] .filter((served) => served.kind === "local" && served.hostLifetime) .map((served) => this.#object(served, EMPTY_PARAMS, undefined, context)); - await Promise.allSettled(objects.map(({ settled }) => settled)); + await Promise.all(objects.map(({ firstOutcome }) => firstOutcome)); if (this.#state === "loading") this.#state = "loaded"; - this.#loaded.resolve(); } async get( @@ -435,9 +463,18 @@ export class ChordHost

    []>; } - async registry(R: Registry, context: Context): Promise> { + async registry(R: Registry, context: Context, options: GetOptions = {}): Promise> { this.batch.assertOutside("registry"); - const loading = this.pendingLoad; - if (loading !== undefined) await awaitWithContext(loading, context); this.#assertOpen(); - const value = this.registryRecord(R.name).value(R.version); - const start = value.start(); - if (start.kind === "failed" && start.last === undefined) throw start.error; - // Its values are the composition as `R`'s version has it, `V`. - return value as RegistryValue; + const timeout = options.readyTimeout ?? this.timeouts.ready; + const expired = new AbortController(); + const waiting = withLinkedAbortSignal(expired.signal, context); + let last: unknown; + const deadline = setTimeout(() => expired.abort(last ?? notLive(`registry ${R.name}`, timeout)), timeout); + // The host's close ends the wait at once, `released/closed` (its timer with it). + const closing = (): void => expired.abort(this.#closedError()); + this.#routing.signal.addEventListener("abort", closing, { once: true }); + try { + await awaitWithContext(this.#publication.promise, waiting.context); + this.#assertOpen(); + const value = this.registryRecord(R.name).value(R.version); + // Its values are the composition as `R`'s version has it, `V`. + const usable = (): boolean => { + const start = value.start(); + if (start.kind === "value" || start.last !== undefined) return true; + last = start.error; + return false; + }; + if (!usable()) await untilUsable(value, usable, waiting.context); + return value as RegistryValue; + } finally { + clearTimeout(deadline); + this.#routing.signal.removeEventListener("abort", closing); + waiting.release(); + } } async root(fn: (ctx: Ctx

    ) => Promise, context: Context, options: RootOptions

    = {}): Promise { @@ -499,11 +555,21 @@ export class ChordHost

    | undefined { - return this.#state === "new" || this.#state === "loading" ? this.#loaded.promise : undefined; + get published(): boolean { + return this.#published; + } + + /** `request` cannot bind yet: indexed under `name` (the type or registry it waits for) until woken or ended. */ + index(name: string, request: UnboundRequest): void { + this.#unbound.add(name, request); + } + + /** `request` left the unbound index's wait for `name` (it ended, or a retry checks it again). */ + unindex(name: string, request: UnboundRequest): void { + this.#unbound.delete(name, request); } /** Whether the host closes: its objects end through `close`, not through its links. */ @@ -513,8 +579,9 @@ export class ChordHost

    this.#place(binding, Type, params, context)); + return undefined; + } + // Checked above: served here at the caller's version. + const object = this.#object(served!, params, principal, context, NO_CHAIN, binding.dependent); + binding.bind(object); + return object; } - /** Where a hold arriving on `link` goes (`Destination.hold`); `released` once the host closed. */ + /** Where a hold arriving on `link` goes (`Destination.hold`: by the address it names); `released` once the host closed. */ place(request: HoldRequest, link: HostLink, caller: Caller, context: Context): Placed { this.#assertOpen(); return this.#destination.hold(request, link, caller.principal, context); } /** - * The object a hold placed here binds to, acting as `principal`: a remote type served here (otherwise - * `unavailable`) with the hold's version and fingerprint (`contract_mismatch`), valid params, and placed here, its - * host address computed from the params (`invalid/host_address`); not an activation the hold's chain waits for - * (`invalid/cycle`, Q5). A new object keeps the hold's chain. + * What a hold placed here binds to, acting as `principal`: the object, or why it waits unbound (`failed + * { waiting }`): a remote type not served here (`not_served`), not at the hold's version and fingerprint, or whose + * params put it on another host (code skew: `version`). Throws for a broken refusal: invalid params, an activation + * the hold's chain waits for (`invalid/cycle`, Q5). A new object keeps the hold's chain. */ - serving(request: HoldRequest, params: unknown, principal: Principal, context: Context): ChordObject { + serving(request: HoldRequest, params: unknown, principal: Principal, context: Context): Serving { const { type, version, fingerprint, chain = NO_CHAIN } = request; const served = this.#types.get(type); - if (served?.kind !== "remote") { - throw new ChordError({ code: "unavailable", message: `${type} is not served here` }); - } - this.#assertMatch(served, version, fingerprint); - const checked = checkParams(served.contract, params, context); - this.#destination.assertHere(served.contract, checked.params); - const object = this.#object(served, checked, principal, context, chain); + const unbound = unboundCause(served, "remote", type, version, fingerprint); + if (unbound !== undefined) return { kind: "waiting", cause: unbound }; + // Checked above: a remote type served here. + const remote = served as Extract; + const checked = checkParams(remote.contract, params, context); + const misplaced = this.#destination.misplaced(remote.contract, checked.params); + if (misplaced !== undefined) return { kind: "waiting", cause: { kind: "waiting", error: misplaced } }; + const object = this.#object(remote, checked, principal, context, chain); const activation = object.activationId; if (activation !== undefined && chain.includes(activation)) throw this.cycle(chain, activation, [object.label]); - return object; + return { kind: "object", object }; } /** * The path a remote reference of `Type` with `params` acting as `principal` goes over now: `ref.retry()` places a * reference failed for a part past the limits anew (a route may have forgotten its old path). Throws as - * `Destination.reference` does, and `unavailable` for a type served here now. + * `Destination.reference` does, and `unavailable` for a type placed here now. */ pathOf(Type: Contract, params: ParamsOf, principal: Principal | undefined): Path { // A retry has no context of its own: its admission checks count on none. const placed = this.#destination.reference(Type, params, principal, BACKGROUND_CONTEXT); if (placed.kind === "path") return placed.path; - throw new ChordError({ code: "unavailable", message: `${Type.name} is served here now` }); + throw new ChordError({ code: "unavailable", message: `${Type.name} is placed here now` }); } /** @@ -713,47 +788,55 @@ export class ChordHost

    { - // Released while `load` ran: nothing is made for it. - if (binding.refState.kind === "ended") return; - if (this.#state === "closed") return binding.end(this.#closedError()); - this.#bindOrFail(binding, Type, checked, context); - }); + const place = (): void => this.#place(binding, Type, checked, context); + if (this.#published) place(); + else this.#wait(binding, Type.name, undefined, place); return binding; } /** - * Binds a reference (`bind`); where it cannot go (`#assertRoute`, a `hostAddress` or `authorize` that throws or - * refuses) fails it broken, until `ref.retry()` binds it again. + * Places a reference (`bind`); where it cannot go (a `hostAddress` or `authorize` that throws or refuses) fails it + * broken, until `ref.retry()` places it again. One woken after the host closed ends as the close ended the others. */ - #bindOrFail(binding: Binding, Type: Contract, checked: CheckedParams, context: Context): void { + #place(binding: Binding, Type: Contract, checked: CheckedParams, context: Context): void { + if (this.#state === "closed") { + binding.end(this.#closedError()); + return; + } try { - this.#assertRoute(Type); this.bind(binding, Type, checked, context); } catch (error) { - // A retry after the host closed ends the reference as the close did the others. - const again = () => - this.#state === "closed" - ? binding.end(this.#closedError()) - : this.#bindOrFail(binding, Type, checked, context); + const again = (): void => this.#place(binding, Type, checked, context); binding.refuse({ kind: "broken", error: asChordError(error, Type.name) }, again); } } + /** + * The provider's own host-address check of a remote `Type` placed here by the consumer's contract, against the one + * served (`invalid/host_address`, `waiting`: code skew); none when both have the same `hostAddress` (the consumer + * computed it with the provider's function) or the type is local. A provider's `hostAddress` that throws throws. + */ + #misplaced(Type: Contract, served: Served, params: ParamsOf): FailureCause | undefined { + if (served.kind !== "remote" || contractData(Type).hostAddress === contractData(served.contract).hostAddress) { + return undefined; + } + const error = this.#destination.misplaced(served.contract, params); + return error === undefined ? undefined : { kind: "waiting", error }; + } + + /** Leaves `binding` unbound in the index under `name`: `cause` (`waiting`), or none before the publication. */ + #wait(binding: Binding, name: string, cause: FailureCause | undefined, place: () => void): void { + binding.wait(cause, place, () => this.#unbound.delete(name, binding)); + this.#unbound.add(name, binding); + } + /** A new reference's binding made by `caller`; its parked calls wait its `readyTimeout`. */ #binding(Type: Contract, dependent: ChordObject | undefined, caller: Caller, options: GetOptions): Binding { return new Binding(Type, this.batch, dependent, caller, options.readyTimeout ?? this.timeouts.ready); @@ -791,22 +874,6 @@ export class ChordHost

    route.close()), - ]; - for (const holder of holders) if (holder.kind === "remote") holder.lost(closed); + for (const route of this.#routes.values()) route.close(); + for (const holder of this.#pathHolders()) if (holder.kind === "remote") holder.lost(closed); + // At the fence too: requests still waiting for a publication or a provider end as the others did, and gets + // waiting for a publication that will not come fail now, not after the slow closes below (fix round 1). + this.#unbound.close(closed); + this.#publication.resolve(); this.backoff.clear(); // Every object is sealed before any teardown releases a reference, so none ends in the background. const teardowns = [...this.#objects.values()].reverse().map((object) => object.stop()); @@ -870,8 +937,6 @@ export class ChordHost

    0) { const message = `host.close: ${failures.length} native close(s) threw`; throw new AggregateError( @@ -881,6 +946,12 @@ export class ChordHost

    route.holders()); + return [...this.#parent.holders(), ...this.unplaced.holders(), ...routed]; + } + /** The store stopped after a backend error (it tells the host once): reported, and the host closes. */ #storageFailed(error: unknown): void { this.report({ kind: "storage_failed", error }); @@ -956,13 +1027,6 @@ export class ChordHost

    boolean, context: Context): Promise { + let changed = Promise.withResolvers(); + const release = value.wait({ + dependencyMoved: () => queueMicrotask(() => changed.resolve()), + // The value closed with its host: the wait ends with the close's error. + closed: (error) => changed.reject(error), + }); + try { + while (!usable()) { + await awaitWithContext(changed.promise, context); + changed = Promise.withResolvers(); + } + } finally { + release(); + } +} diff --git a/packages/chord/src/next/host/internal.ts b/packages/chord/src/next/host/internal.ts index a976a73a5..18c907631 100644 --- a/packages/chord/src/next/host/internal.ts +++ b/packages/chord/src/next/host/internal.ts @@ -3,9 +3,10 @@ export { nextDelay } from "./backoff.ts"; export { CallContext, type Caller } from "./ctx.ts"; -export type { AnyHost, ChordHost } from "./host.ts"; +export type { AnyHost, ChordHost, Serving } from "./host.ts"; export type { PinnedKey } from "./keyed-members.ts"; -export { type ChordObject, type Holder, settlement } from "./object.ts"; +export type { ChordObject, Holder } from "./object.ts"; export { LIVE, LOADING, type RefState } from "./ref-state.ts"; -export type { Binding, CheckedParams, Target } from "./reference.ts"; +export { type Binding, type CheckedParams, type RetryOperation, retryOperation, type Target } from "./reference.ts"; +export type { UnboundRequest } from "./unbound.ts"; export { uuidv7 } from "./uuid.ts"; diff --git a/packages/chord/src/next/host/object.ts b/packages/chord/src/next/host/object.ts index 91e0bd2b8..54c2400a9 100644 --- a/packages/chord/src/next/host/object.ts +++ b/packages/chord/src/next/host/object.ts @@ -1,15 +1,16 @@ // Objects (§6): one per identity, from its first attempt to its end. Its state across attempts (its documents, line -// and job runs, its member sources, slice 6a §1.2), its holders, retries after failures (backoff, dependency recovery -// R18, `ref.retry()`), a live object's failure (slice 6a §1.4), the waits and cycle checks of activations (Q5, its -// instance id and the activation chain it was created for), lingering after its last reference (§3, D8), the job runs -// that keep it alive (§15), its contributions to registries while live and its wake signals (`retryOn`, slice 8), and -// its ending, revocation included (§4, R11). +// and job runs, its member sources, slice 6a §1.2), its holders, retries after failures (backoff, `ref.retry()`, passed +// on while parked), what it shows while its attempt is parked on dependencies (dependency model §3), its first outcome +// (the start barrier, decision (a)), a live object's failure (slice 6a §1.4), the waits and cycle checks of activations +// (Q5, its instance id and the activation chain it was created for), lingering after its last reference (§3, D8), the +// job runs that keep it alive (§15), its contributions to registries while live and its wake signals (`retryOn`, slice +// 8), and its ending, revocation included (§4, R11). -import { awaitWithContext, type Context } from "../../context/index.ts"; +import type { Context } from "../../context/index.ts"; import type { Contract, ParamsOf } from "../contracts/index.ts"; import type { ContractShape } from "../contracts/internal.ts"; -import { ChordError, type FailureCause, isChordError, type Principal } from "../data/index.ts"; -import { invalid, type Provider } from "../data/internal.ts"; +import { ChordError, type FailureCause, type Principal } from "../data/index.ts"; +import { invalid, type Provider, sameCause } from "../data/internal.ts"; import { Documents, type KeyedSlot } from "../documents/internal.ts"; import type { JobDefinition } from "../jobs/index.ts"; import { JobRecords } from "../jobs/internal.ts"; @@ -17,13 +18,14 @@ import { CHAIN } from "../link/internal.ts"; import type { DeriveSource } from "../state/internal.ts"; import { Attempt, type Self } from "./attempt.ts"; import type { Caller } from "./ctx.ts"; +import type { Dependency } from "./dependencies.ts"; import type { AnyHost } from "./host.ts"; import { KeyedMembers, type PinnedKey } from "./keyed-members.ts"; import { MemberSource } from "./member-source.ts"; import type { KeyedBinding } from "./members.ts"; import type { CloseFailed, NativeCloseReason } from "./native.ts"; import { type Ended, type Failed, LIVE, LOADING, type RefState } from "./ref-state.ts"; -import type { Target } from "./reference.ts"; +import { type RetryOperation, retryOperation, type Target } from "./reference.ts"; import { type RetrySource, RetryWakes } from "./retry-on.ts"; import { uuidv7 } from "./uuid.ts"; @@ -60,7 +62,7 @@ type ServedType = { /** What holds an object: a reference's binding, or a hold arriving over a link. */ export interface Holder { - /** The object whose attempt declared the reference; retried when this object goes live (R18). */ + /** The object whose attempt declared the reference: it checks its dependencies again when this object changes. */ readonly dependent: ChordObject | undefined; /** Whom the reference acts for, with its origin's incarnation (`host.references`). */ readonly caller: Caller; @@ -71,34 +73,40 @@ export interface Holder { } /** - * Where an object stands. `waiting`: for its identity's backoff window before its first attempt. `failed` has the - * state its references show, and keeps the failed attempt in `holding` after a dependency failure (R18): its - * references keep the dependencies. + * Where an object stands. `waiting`: for its identity's backoff window before its first attempt. `activating` has what + * its references show: `loading`, or `failed { waiting }` while its attempt is parked on a failed dependency. `failed` + * has the state its references show. */ type ObjectState = | { readonly kind: "waiting" } - | { readonly kind: "activating"; readonly attempt: Attempt } + | { readonly kind: "activating"; readonly attempt: Attempt; readonly shown: RefState } | { readonly kind: "live"; readonly attempt: Attempt } - | { readonly kind: "failed"; readonly failure: Failed; readonly holding: Attempt | undefined } + | { readonly kind: "failed"; readonly failure: Failed } | { readonly kind: "ended"; readonly ended: Ended }; /** - * What an ended object still has to do once its previous teardowns finished (`previous`): drain and close its attempt, - * or release a failed attempt's references; then drop its documents and job runs (`end` alone). + * What an ended object still has to do once its previous teardowns finished (`previous`): drain and close its attempt; + * then drop its documents and job runs (`end` alone). */ type Teardown = | { readonly kind: "none" } | { readonly kind: "end"; readonly previous: Promise | undefined } - | { readonly kind: "close" | "release"; readonly attempt: Attempt; readonly previous: Promise | undefined }; + | { readonly kind: "close"; readonly attempt: Attempt; readonly previous: Promise | undefined }; const END: NativeCloseReason = { kind: "end" }; const CLOSE: NativeCloseReason = { kind: "close" }; +/** How many retry ids an object remembers: a diamond's shortcut only (a ring is stopped by a retry's `through`). */ +const RETRIES_KEPT = 8; + +/** No dependencies waited for: the shared empty list. */ +const NO_DEPENDENCIES: readonly Dependency[] = []; + /** - * One object. Each attempt is a fresh `Attempt`; the documents, the line and the job runs, the holders (and, after a - * dependency failure, the failed attempt's references) outlive it. A transient failure is retried after its backoff window, or at once when a - * dependency it failed on goes live or a reference asks (`retry`); a broken one stays failed until a reference asks. + * One object. Each attempt is a fresh `Attempt`; the documents, the line, the job runs and the holders outlive it. An + * attempt parked on its dependencies waits for them without a timer; a transient failure (its own) is retried after its + * backoff window, or at once when a reference asks (`retry`); a broken one stays failed until a reference asks. * The object ends with its last reference, a live one after its type's `linger` (never, for a host-lifetime object), * or when the host closes. */ @@ -127,13 +135,20 @@ export class ChordObject implements Target { #state: ObjectState = { kind: "waiting" }; /** References and holds of this object. */ readonly #holders = new Set(); - /** Objects this object's activation waits for; followed to find cycles (Q5). */ - readonly #waitingFor = new Set(); - /** The attempt a dependency went live during (R18): if it fails on a dependency, it is retried at once. */ - #woken: Attempt | undefined; - /** The current attempt's outcome: resolves when it goes live, rejects with its error; renewed by every retry. */ - #outcome = settlement(); - /** The error of the last failed attempt, until one goes live. */ + /** The dependencies its parked attempt waits for; their objects are followed to find cycles (Q5). */ + #waitingFor: readonly Dependency[] = NO_DEPENDENCIES; + /** + * Its first outcome (decision (a)): live, failed, parked with no blocker here still without one, or ended; once. + * The promise is made by the first who waits for it (the start barrier, for host-lifetime objects). + */ + #firstOutcome: PromiseWithResolvers | undefined; + #hasFirstOutcome = false; + /** + * The ids of the last retry operations it handled (at most `RETRIES_KEPT`, the newest last; made by the first): a + * retry reaching it twice through a diamond is handled once. + */ + #retried: string[] | undefined; + /** The error of the last failed attempt, or the last cause a parked attempt showed, until one goes live. */ #lastFailure: ChordError | undefined; /** Ends the object when its type's `linger` passed after its last reference; cleared by a new one. */ #lingering: ReturnType | undefined; @@ -189,7 +204,8 @@ export class ChordObject implements Target { const state = this.#state; if (state.kind === "failed") return state.failure; if (state.kind === "live") return LIVE; - return state.kind === "ended" ? state.ended : LOADING; + if (state.kind === "ended") return state.ended; + return state.kind === "activating" ? state.shown : LOADING; } /** Whether its state members have values: once live (a local object stays live until it ends). */ @@ -202,12 +218,27 @@ export class ChordObject implements Target { return this.#exposed !== undefined && this.#state.kind !== "ended"; } - /** The current attempt's outcome (the start barrier waits for the first). */ - get settled(): Promise { - return this.#outcome.promise; + /** Resolves at its first outcome (the start barrier waits for it); never rejects. */ + get firstOutcome(): Promise { + if (this.#hasFirstOutcome) return Promise.resolve(); + this.#firstOutcome ??= Promise.withResolvers(); + return this.#firstOutcome.promise; } - /** The error of the last failed attempt while none went live since. */ + /** Whether it holds up the start barrier: it has no first outcome yet. */ + get blocksFirstPass(): boolean { + return !this.#hasFirstOutcome; + } + + /** An object here: it is not held on a link. */ + get heldOnLink(): false { + return false; + } + + /** + * The error of its last failure while none went live since: a failed attempt's, or the cause its parked attempt + * showed last (kept when the dependency falls back to `loading`, so a getter's deadline names it). + */ get lastFailure(): ChordError | undefined { return this.#lastFailure; } @@ -251,21 +282,31 @@ export class ChordObject implements Target { const state = this.#state; if (state.kind !== "waiting" && state.kind !== "failed") return; this.#instance ??= uuidv7(); - const attempt = new Attempt(this, state.kind === "failed" ? state.holding : undefined, this.#context); - this.#state = { kind: "activating", attempt }; - if (state.kind === "failed") { - this.#outcome = settlement(); - this.#notify(); - } + const attempt = new Attempt(this, this.#context); + this.#state = { kind: "activating", attempt, shown: LOADING }; + if (state.kind === "failed") this.#notify(); void this.#run(attempt); } /** - * `ref.retry()` and a `retry` message: an attempt now when failed (a broken one too) or waiting, its backoff window - * skipped; ignored while an attempt runs and once live. + * `ref.retry()` and a `retry` message, as `operation` (handled once): an attempt now when failed (a broken one too) + * or waiting, its backoff window skipped; passed on to the failed dependencies while parked `failed { waiting }`, + * with this object added to the objects it passed through (at most `CHAIN`: a retry stops past 32 hops); ignored + * otherwise (an attempt runs, or it is live), and when it passed through this object already (a ring) or was handled + * here already (a diamond). */ - retry(): void { - if (this.#state.kind !== "waiting" && this.#state.kind !== "failed") return; + retry(operation: RetryOperation = retryOperation()): void { + const instance = this.#instance; + if (instance !== undefined && operation.through.includes(instance)) return; + if (this.#handled(operation.id)) return; + const state = this.#state; + if (state.kind === "activating") { + const { through } = operation; + if (state.shown.kind !== "failed" || instance === undefined || through.length >= CHAIN) return; + state.attempt.retryDependencies({ id: operation.id, through: [...through, instance] }); + return; + } + if (state.kind !== "waiting" && state.kind !== "failed") return; this.host.backoff.skip(this.identity); this.attempt(); } @@ -396,10 +437,8 @@ export class ChordObject implements Target { if (cause.kind === "transient") this.host.backoff.failed(this.identity); else this.host.backoff.delete(this.identity); const failure: Failed = { kind: "failed", cause, retryAt: this.host.backoff.until(this.identity) }; - this.#state = { kind: "failed", failure, holding: undefined }; + this.#state = { kind: "failed", failure }; this.#lastFailure = error; - this.#outcome = settlement(); - this.#outcome.reject(error); this.#withdraw(attempt); attempt.seal(error); // Members following another object's sources freeze now: no commit of this object changes them. @@ -476,37 +515,39 @@ export class ChordObject implements Target { throw error; } - /** - * Waits for `dependency`'s current attempt as part of this object's activation. Rejects at once when it is failed, - * and when its attempt fails, with `unavailable` naming it (R18), its error as the cause; a dependency that failed - * for a cycle rejects with that `invalid/cycle` itself. - */ - async waitFor(dependency: Target, context: Context): Promise { - this.#waitingFor.add(dependency); - try { - await awaitWithContext(dependency.settled, context); - } catch (error) { - if (isChordError(error, "invalid") && error.reason === "cycle") throw error; - const message = `${this.label}: dependency ${dependency.label} failed`; - throw new ChordError({ code: "unavailable", message }, { cause: error }); - } finally { - this.#waitingFor.delete(dependency); - } + /** Its parked attempt waits for `dependencies` (none once it leaves parking): the waits a cycle check follows. */ + waitsFor(dependencies: readonly Dependency[]): void { + this.#waitingFor = dependencies; } /** - * A dependency went live: retried at once (backoff window skipped) when it failed on a dependency (R18). A wake - * while it activates is remembered: that activation may still fail on the dependency's earlier failure. + * A dependency of its current attempt changed (its state, or its first outcome): a parked attempt checks again; an + * attempt past its dependency check ignores it (R18 after live), and so does any other state. */ - dependencyLive(): void { + dependencyMoved(): void { const state = this.#state; - if (state.kind === "activating") { - this.#woken = state.attempt; - return; + if (state.kind === "activating") state.attempt.dependencyMoved(); + } + + /** + * Its attempt `attempt` is parked showing `shown` (`loading` or `failed { waiting }`): holders are told when that + * changed (a cause compared by value). `firstPass`: no blocker here is still without its first outcome, so this + * object has its first outcome now. + */ + parked(attempt: Attempt, shown: RefState, firstPass: boolean): void { + const state = this.#state; + if (state.kind !== "activating" || state.attempt !== attempt) return; + if (!sameState(state.shown, shown)) { + this.#state = { kind: "activating", attempt, shown }; + if (shown.kind === "failed") this.#lastFailure = shown.cause.error; + this.#notify(); } - if (state.kind !== "failed" || state.holding === undefined) return; - this.host.backoff.skip(this.identity); - this.attempt(); + if (firstPass) this.#settleFirstOutcome(); + } + + /** Its attempt `attempt` left parking for its own steps: it shows `loading` until its outcome. */ + unparked(attempt: Attempt): void { + this.parked(attempt, LOADING, false); } /** @@ -540,14 +581,11 @@ export class ChordObject implements Target { case "live": return this.#wentLive(attempt); case "failed": - return this.#failed(attempt, outcome.cause, undefined); - case "dependency_failed": - return this.#failed(attempt, outcome.cause, attempt); + return this.#failed(attempt, outcome.cause); } } #wentLive(attempt: Attempt): void { - this.#woken = undefined; if (this.#wakes !== undefined) this.#wakes.during = undefined; this.#state = { kind: "live", attempt }; this.#exposed = attempt; @@ -562,21 +600,17 @@ export class ChordObject implements Target { } this.#lastFailure = undefined; this.host.wentLive(this); - this.#outcome.resolve(); this.#notify(); - // R18: objects that failed because this one had failed are retried now. - for (const holder of [...this.#holders]) holder.dependent?.dependencyLive(); + this.#settleFirstOutcome(); attempt.live(); } /** - * Records a failed attempt: reported, torn down in the background (reason `end`), its references released unless it - * is `holding` them (R18), its holders told. Transient: a backoff window, whose end is the failure's `retryAt`; - * broken: no automatic retry. + * Records a failed attempt: reported, torn down in the background (reason `end`), its references released once its + * natives closed, its holders told. Transient: a backoff window, whose end is the failure's `retryAt`; broken: no + * automatic retry. */ - #failed(attempt: Attempt, cause: FailureCause, holding: Attempt | undefined): void { - const woken = this.#woken === attempt; - this.#woken = undefined; + #failed(attempt: Attempt, cause: FailureCause): void { const wokenBy = this.#wakes?.during === attempt; if (this.#wakes !== undefined) this.#wakes.during = undefined; // What its activation loaded for the pinned keys goes; they stay frozen. @@ -584,25 +618,23 @@ export class ChordObject implements Target { if (cause.kind === "transient") this.host.backoff.failed(this.identity); else this.host.backoff.delete(this.identity); const failure: Failed = { kind: "failed", cause, retryAt: this.host.backoff.until(this.identity) }; - this.#state = { kind: "failed", failure, holding }; + this.#state = { kind: "failed", failure }; this.#lastFailure = cause.error; attempt.seal(this.ended()); const closing = attempt.close(END).then((failures) => { attempt.detach(); - // Released once its natives closed (a close may use them, §6, §13); R18's attempt keeps them for the retry. - if (holding === undefined) attempt.releaseReferences(); + // Released once its natives closed (a close may use them, §6, §13). + attempt.releaseReferences(); return failures; }); this.host.ending(this.identity, this.#reported(closing)); this.host.report({ kind: "activation_failed", object: this.label, failure: cause }); - this.#outcome.reject(cause.error); this.#notify(); + this.#settleFirstOutcome(); // Job runs kept it while it activated; failed, they keep it no longer. this.#idle(); - // A dependency went live while this attempt activated: the wake it missed retries it now (R18). - if (holding !== undefined && woken) this.dependencyLive(); // A wake signal arrived while it activated: it retries it now. - else if (wokenBy) this.retry(); + if (wokenBy) this.retry(); } /** @@ -614,7 +646,8 @@ export class ChordObject implements Target { // Called by its wakes only. const wakes = this.#wakes!; if (state.kind === "activating") { - wakes.during = state.attempt; + // While parked its dependencies decide (dependency model §3.8). + if (!state.attempt.isParked) wakes.during = state.attempt; return; } if (state.kind !== "failed" || wakes.leaving) return; @@ -661,7 +694,7 @@ export class ChordObject implements Target { for (const [member, source] of attempt.stateMembers) { if (which === "all" || !attempt.owns(source)) this.#memberSource(member, attempt).freeze(); } - // A pin still loading is given up with the failure: a consumer activating on it fails transient (R18). + // A pin still loading is given up with the failure: a consumer parked on it shows `failed { waiting }`. this.#keys.freeze(which, this.#lastFailure ?? this.ended()); } @@ -705,7 +738,8 @@ export class ChordObject implements Target { this.#stopLingering(); this.#wakes?.stop(); this.host.forget(this); - this.#outcome.reject(error); + // An object of the start barrier's cohort that ends before its first outcome settles it as ended. + this.#settleFirstOutcome(); // In a batch: the derives over its members are marked, so none keeps a cached value. this.host.batch.within(() => { for (const source of this.#members.values()) source.objectEnded(error); @@ -717,12 +751,8 @@ export class ChordObject implements Target { this.#holders.clear(); this.host.unreferenced(this, holders, this.#context); for (const holder of holders) holder.end(error); - if (state.kind === "waiting") return { kind: "end", previous }; - if (state.kind === "failed") { - // The failed attempt is closed (or closing) since it failed; only the references it holds (R18) are left. - const { holding } = state; - return holding === undefined ? { kind: "end", previous } : { kind: "release", attempt: holding, previous }; - } + // A failed attempt is closed (or closing) since it failed. + if (state.kind === "waiting" || state.kind === "failed") return { kind: "end", previous }; if (state.kind === "live") this.#withdraw(state.attempt); state.attempt.seal(error); return { kind: "close", attempt: state.attempt, previous }; @@ -767,8 +797,10 @@ export class ChordObject implements Target { if (teardown.kind === "none") return []; await teardown.previous; const failures = teardown.kind === "close" ? await teardown.attempt.close(reason) : []; - if (teardown.kind === "close") teardown.attempt.detach(); - if (teardown.kind !== "end") teardown.attempt.releaseReferences(); + if (teardown.kind === "close") { + teardown.attempt.detach(); + teardown.attempt.releaseReferences(); + } await this.documents.settled(); this.documents.end(); this.host.store?.claims.release(this); @@ -794,12 +826,35 @@ export class ChordObject implements Target { throw new ChordError({ code: "not_ready", message: `${this.label}: not live yet` }); } - /** The chain of waits from this activation to `target` (see `Target`). */ + /** + * Whether the retry operation `id` reached it before (among the last few: the diamond's shortcut; a ring is stopped + * by `through`); records it otherwise. + */ + #handled(id: string): boolean { + this.#retried ??= []; + const retried = this.#retried; + if (retried.includes(id)) return true; + if (retried.length === RETRIES_KEPT) retried.shift(); + retried.push(id); + return false; + } + + /** The settlement of its first outcome (once): dependents parked on it check their first pass again. */ + #settleFirstOutcome(): void { + if (this.#hasFirstOutcome) return; + this.#hasFirstOutcome = true; + this.#firstOutcome?.resolve(); + this.#firstOutcome = undefined; + for (const holder of this.#holders) holder.dependent?.dependencyMoved(); + } + + /** The chain of waits from this activation to `target` (see `Target`), through its parked attempt's dependencies. */ pathTo(target: ChordObject, visited: Set): ChordObject[] | undefined { if (this === target) return [this]; visited.add(this); - for (const next of this.#waitingFor) { - if (visited.has(next)) continue; + for (const dependency of this.#waitingFor) { + const next = dependency.kind === "reference" ? dependency.binding.target : undefined; + if (next === undefined || visited.has(next)) continue; const rest = next.pathTo(target, visited); if (rest !== undefined) return [this, ...rest]; } @@ -807,9 +862,8 @@ export class ChordObject implements Target { } } -/** A settlement nobody may wait for: its rejection is not unhandled. */ -export function settlement(): PromiseWithResolvers { - const settlement = Promise.withResolvers(); - settlement.promise.catch(() => {}); - return settlement; +/** Whether two states of a reference say the same: the same object, or failures with the same cause (by value). */ +function sameState(a: RefState, b: RefState): boolean { + if (a === b) return true; + return a.kind === "failed" && b.kind === "failed" && a.retryAt === b.retryAt && sameCause(a.cause, b.cause); } diff --git a/packages/chord/src/next/host/ref-state.ts b/packages/chord/src/next/host/ref-state.ts index 9e5720405..1c9234238 100644 --- a/packages/chord/src/next/host/ref-state.ts +++ b/packages/chord/src/next/host/ref-state.ts @@ -7,11 +7,13 @@ import type { ChordError, FailureCause } from "../data/index.ts"; import { type Batch, NativeSource } from "../state/internal.ts"; /** - * Where a reference stands. `loading`: its object activates or retries, it waits for `load` or a parent link, or its - * link is down (the reference's coarser view of its object's waiting and activating); `failed`: its object failed and - * no attempt runs (`retryAt` undefined: broken, no automatic retry); `live`: bound; `ended`: for good, with the error - * that says why (`released` without a reason: released by its holder; `released/ended`: its object ended; - * `released/closed`: its host closed; `forbidden/revoked`). + * Where a reference stands. `loading`: its object activates, retries or is parked on dependencies still under way, it + * waits for its host's publication or a parent link, or its link is down (the reference's coarser view of its object's + * waiting and activating); `failed`: its object failed and no attempt runs (`transient`: `retryAt` is when its backoff + * retries it; `broken`: no automatic retry), or nothing fits it or its object is parked on a failed dependency + * (`waiting`: an event wakes it, `retryAt` undefined); `live`: bound; `ended`: for good, with the error that says why + * (`released` without a reason: released by its holder; `released/ended`: its object ended; `released/closed`: its + * host closed; `forbidden/revoked`). */ export type RefState = | { readonly kind: "loading" } diff --git a/packages/chord/src/next/host/reference.ts b/packages/chord/src/next/host/reference.ts index 65da7ab33..07e3b2665 100644 --- a/packages/chord/src/next/host/reference.ts +++ b/packages/chord/src/next/host/reference.ts @@ -14,7 +14,7 @@ import { takesNoArgs, type VariantImplementation, } from "../contracts/internal.ts"; -import { ChordError, type FailureCause } from "../data/index.ts"; +import { ChordError, type FailureCause, type Principal } from "../data/index.ts"; import { Keyed, type KeyedSlot, type SealToken, Singleton } from "../documents/internal.ts"; import { type Batch, Constant, type DeriveSource, Source } from "../state/internal.ts"; import { type Caller, withLinkedAbortSignal } from "./ctx.ts"; @@ -23,6 +23,8 @@ import type { KeyedBinding } from "./members.ts"; import type { ChordObject, Holder } from "./object.ts"; import { ParkedCalls } from "./parked.ts"; import { type Ended, type Failed, LOADING, ReferenceState, type RefState } from "./ref-state.ts"; +import type { UnboundRequest } from "./unbound.ts"; +import { uuidv7 } from "./uuid.ts"; /** * A reference: a local contract's members, or for a remote contract the members of the variant its object chose, @@ -50,7 +52,10 @@ type InVariants = { type Lifecycle = { /** The reference's state: loading, live, failed or released; watches get every change. */ readonly state: NativeWatchable; - /** Asks a failed object for an attempt now (the backoff skipped, a broken one renewed); otherwise nothing. */ + /** + * Asks a failed object for an attempt now (the backoff skipped, a broken one renewed); an object parked on failed + * dependencies passes it on to them, once; otherwise nothing. + */ retry(): void; release(): void; }; @@ -68,12 +73,20 @@ export interface Target { readonly hydrated: boolean; /** The variant bound last; `undefined` for a local object and before the first binding. */ readonly variant: string | undefined; - /** The error of its last failed attempt while none went live since. */ + /** The error of its last failure (an attempt's, or the dependencies it is parked on) while none went live since. */ readonly lastFailure: ChordError | undefined; /** Its instance id while it activates on this host (Q5); `undefined` otherwise and over a link. */ readonly activationId: string | undefined; - /** The current attempt's outcome, which a dependent's activation waits for (R18). */ - readonly settled: Promise; + /** + * Whether it is held on a link now: a remote attempt answers for it there (`parked_long` counts it while it is + * loading); not while its path has no link, and never for an object here. + */ + readonly heldOnLink: boolean; + /** + * Whether it holds up this host's start barrier: an object here without its first outcome (decision (a)); a remote + * object never does. + */ + readonly blocksFirstPass: boolean; /** Counts `binding` as a reference; the binding reads `state` itself. */ hold(binding: Binding): void; /** `binding` was released. */ @@ -97,8 +110,11 @@ export interface Target { * attempts once loaded (`keyed-members.ts`); only a live local object takes it. */ pinKey(member: string, key: string, context: Context): Promise; - /** `ref.retry()`: an attempt now if failed; ignored otherwise. */ - retry(): void; + /** + * `ref.retry()` (`operation` once per retry, deduping diamonds): an attempt now if failed; an object parked on failed + * dependencies passes it on; ignored otherwise. + */ + retry(operation: RetryOperation): void; /** * The chain of activation waits from this object to `target`, both included (Q5); `undefined` when there is none. * `visited` keeps a search over shared dependencies linear in the waits. @@ -106,14 +122,29 @@ export interface Target { pathTo(target: ChordObject, visited: Set): ChordObject[] | undefined; } +/** + * One `ref.retry()` passing on through parked objects, across links too (the `retry` message's `operation` and + * `through`): its id (a uuidv7) and the instance ids of the objects it passed through, added hop by hop (at most + * `CHAIN`). An object in `through` drops it, so a ring of hosts stops after one lap however many retries run at once; + * an object remembers the last few ids it handled too, so a diamond retries it once. + */ +export type RetryOperation = { readonly id: string; readonly through: string[] }; + +/** A new retry operation, through nothing yet (`ref.retry()`, a `retry` message that names none). */ +export function retryOperation(): RetryOperation { + return { id: uuidv7(), through: [] }; +} + /** A reference's params as L1 checked them, with their identity key. */ export type CheckedParams = { readonly params: ParamsOf; readonly key: string }; /** - * Where a binding stands. `unbound`: waiting for `load` (a `host.hold`) or for its object's activation to bind it (a - * `self.get`); `refused`: a `host.hold` whose type turned out not to be gettable here once `load` resolved; a released - * binding keeps nothing but its name and its end (`ref.state`'s `ended`). `dependent` is the object whose attempt declared the reference with `self.get`, - * `undefined` for `host.get`, `host.hold` and `ctx.get`. + * Where a binding stands. `unbound`: waiting for its object's activation to bind it (a `self.get`); `waiting`: in the + * host's unbound index, `place` checking it again when woken: before the publication (no `failure`: `loading`), or after + * a check that found no fitting provider (`failed { waiting }`, `not_served` or `version`); `refused`: a broken refusal + * (a `hostAddress` or `authorize` that threw or refused) until `ref.retry()` places it again; a released binding keeps + * nothing but its name and its end (`ref.state`'s `ended`). `dependent` is the object whose attempt declared the + * reference with `self.get`, `undefined` for `host.get`, `host.hold` and `ctx.get`. */ type BindingState = | { readonly kind: "unbound"; readonly batch: Batch; readonly dependent: ChordObject | undefined } @@ -123,7 +154,22 @@ type BindingState = readonly dependent: ChordObject | undefined; readonly target: Target; } - | { readonly kind: "refused"; readonly batch: Batch; readonly failure: Failed; readonly bind: () => void } + | { + readonly kind: "waiting"; + readonly batch: Batch; + readonly dependent: ChordObject | undefined; + readonly failure: Failed | undefined; + readonly place: () => void; + /** Takes it out of the index (it ended). */ + readonly leave: () => void; + } + | { + readonly kind: "refused"; + readonly batch: Batch; + readonly dependent: ChordObject | undefined; + readonly failure: Failed; + readonly bind: () => void; + } | { readonly kind: "released"; readonly state: Ended }; /** What a wait for a state decides at each change. */ @@ -144,7 +190,7 @@ type Waiter = { readonly check: () => void; readonly end: (error: ChordError) => * change of its state. Calls run while live, park while loading (sent in order once live) and reject while failed. * Released, or ended with its object, it ends the watches taken through it and rejects the calls made through it. */ -export class Binding implements Holder { +export class Binding implements Holder, UnboundRequest { readonly reference: Reference; /** Who calls through this reference: the principal it was got as, and this host's incarnation. */ readonly caller: Caller; @@ -165,8 +211,6 @@ export class Binding implements Holder { readonly #waiters = new Set(); /** `ref.state`, made on first access. */ #refState: ReferenceState | undefined; - /** Resolved when the reference ends; made by the first `ended()`. */ - #ended: PromiseWithResolvers | undefined; /** The keys its object's factory pinned with `ref.member.at(key)`: pinned once bound, released with it. */ readonly #keys: KeyedMemberAt[] = []; @@ -206,26 +250,38 @@ export class Binding implements Holder { /** The object whose attempt declared this reference, until it is released. */ get dependent(): ChordObject | undefined { const state = this.#state; - return state.kind === "unbound" || state.kind === "bound" ? state.dependent : undefined; + return state.kind === "released" ? undefined : state.dependent; + } + + /** Whom it acts for (the unbound index's revocation). */ + get principal(): Principal | undefined { + return this.caller.principal; } /** The reference's state. */ get refState(): RefState { const state = this.#state; if (state.kind === "bound") return state.target.state; + if (state.kind === "waiting") return state.failure ?? LOADING; if (state.kind === "refused") return state.failure; return state.kind === "unbound" ? LOADING : state.state; } + /** The object it is bound to; `undefined` while it is not bound. */ + get target(): Target | undefined { + return this.#state.kind === "bound" ? this.#state.target : undefined; + } + /** The object in errors: its label once bound, else the contract's name. */ get label(): string { return this.#state.kind === "bound" ? this.#state.target.label : this.#name; } - /** The error of the object's last failed attempt while none went live since. */ + /** The error of the object's last failure while none went live since, or of the check that left it unbound. */ get lastFailure(): ChordError | undefined { const state = this.#state; if (state.kind === "refused") return state.failure.cause.error; + if (state.kind === "waiting") return state.failure?.cause.error; return state.kind === "bound" ? state.target.lastFailure : undefined; } @@ -233,26 +289,48 @@ export class Binding implements Holder { bind(target: Target): void { const state = this.#state; if (state.kind !== "unbound") return; - this.#state = { ...state, kind: "bound", target }; + this.#state = { kind: "bound", batch: state.batch, dependent: state.dependent, target }; target.hold(this); this.moved(); } /** - * A `host.hold` that cannot be bound (where it goes refused it): failed with `cause` until `ref.retry()` runs `bind` - * again (a route's `authorize` may admit it later). + * Leaves the reference unbound in the host's unbound index (`leave` takes it out): `loading` before the publication + * (no `cause`), else `failed` with `cause` (`waiting`); `place` checks it again when the index wakes it. + */ + wait(cause: FailureCause | undefined, place: () => void, leave: () => void): void { + const state = this.#state; + if (state.kind !== "unbound") return; + const failure: Failed | undefined = + cause === undefined ? undefined : { kind: "failed", cause, retryAt: undefined }; + this.#state = { kind: "waiting", batch: state.batch, dependent: state.dependent, failure, place, leave }; + // Unchanged while loading: nothing to tell. + if (failure !== undefined) this.moved(); + } + + /** Woken by the unbound index (out of it already): checked again where it goes. */ + wake(): void { + const state = this.#state; + if (state.kind !== "waiting") return; + this.#state = { kind: "unbound", batch: state.batch, dependent: state.dependent }; + state.place(); + } + + /** + * A reference that cannot be bound (where it goes refused it): failed broken with `cause` until `ref.retry()` runs + * `bind` again (a route's `authorize` may admit it later). */ refuse(cause: FailureCause, bind: () => void): void { const state = this.#state; if (state.kind !== "unbound") return; const failure: Failed = { kind: "failed", cause, retryAt: undefined }; - this.#state = { kind: "refused", batch: state.batch, failure, bind }; + this.#state = { kind: "refused", batch: state.batch, dependent: state.dependent, failure, bind }; this.moved(); } /** * The state may have changed: `ref.state`'s watches are told; live sends the parked calls in order, failed - * rejects them with the cause; waits for a state decide. + * rejects them with the cause; waits for a state decide; the object whose attempt declared it re-evaluates. */ moved(): void { this.#refState?.moved(); @@ -266,6 +344,7 @@ export class Binding implements Holder { this.#parked?.reject(state.cause.error); } for (const waiter of [...this.#waiters]) waiter.check(); + this.dependent?.dependencyMoved(); } /** Resolves when the reference is live; rejects with a broken failure's error, or its end. Getters wait for it. */ @@ -313,6 +392,11 @@ export class Binding implements Holder { } case "unbound": throw new ChordError({ code: "not_ready", message: `${this.#name}: not live yet` }); + case "waiting": + if (state.failure === undefined) { + throw new ChordError({ code: "not_ready", message: `${this.#name}: not live yet` }); + } + throw state.failure.cause.error; case "refused": throw state.failure.cause.error; case "released": @@ -369,6 +453,11 @@ export class Binding implements Holder { return at; } + /** Whether its object's factory pinned keys on it (`at(key)`). */ + get pinsKeys(): boolean { + return this.#keys.length > 0; + } + /** * Pins the keys of `at(key)` on the live object's keyed members and loads those not in memory, before the * dependent goes live (slice 6a §12); `undefined` when there are none. A failed load rejects. @@ -381,24 +470,18 @@ export class Binding implements Holder { } /** - * Resolves when the reference ends (released, or its object ended); never rejects. A wait that must not keep the - * reference races it (a `host.hold` waiting for `load`). + * `ref.retry()`, as `operation` (one per `ref.retry()`, passed on through parked objects): the object is asked for + * an attempt now (ignored unless failed; a parked one passes it on); a refused reference is placed again; one + * waiting in the index is left as it is (an event wakes it). */ - ended(): Promise { - if (this.#state.kind === "released") return Promise.resolve(); - this.#ended ??= Promise.withResolvers(); - return this.#ended.promise; - } - - /** `ref.retry()`: the object is asked for an attempt now (ignored unless failed); a refused hold is bound again. */ - retry(): void { + retry(operation: RetryOperation = retryOperation()): void { const state = this.#state; if (state.kind === "bound") { - state.target.retry(); + state.target.retry(operation); return; } if (state.kind !== "refused") return; - this.#state = { kind: "unbound", batch: state.batch, dependent: undefined }; + this.#state = { kind: "unbound", batch: state.batch, dependent: state.dependent }; this.moved(); state.bind(); } @@ -415,13 +498,15 @@ export class Binding implements Holder { /** * Ends the reference without releasing it from its object (it ended): watches end, calls (running and parked) - * reject with `error`, `ref.state` is `ended` with it. + * reject with `error`, `ref.state` is `ended` with it; the object whose attempt declared it checks again. */ end(error: ChordError): void { // Kept by `ref.state`: its stack is made text now, so the error holds none of the frames' objects. void error.stack; const ended: Ended = { kind: "ended", error }; + const previous = this.#state; this.#state = { kind: "released", state: ended }; + if (previous.kind === "waiting") previous.leave(); for (const at of this.#keys) at.release(); this.#calls?.abort(error); this.#calls = undefined; @@ -431,8 +516,8 @@ export class Binding implements Holder { this.#refState?.released(ended); // Each removes itself; promise reactions run later. for (const waiter of this.#waiters) waiter.end(error); - this.#ended?.resolve(); - this.#ended = undefined; + // An ended dependency breaks a parked dependent (it cannot come back). + if (previous.kind !== "released") previous.dependent?.dependencyMoved(); } #call(method: string, args: unknown, context: Context): Promise { diff --git a/packages/chord/src/next/host/route.ts b/packages/chord/src/next/host/route.ts index 7d4d53c07..78a26820c 100644 --- a/packages/chord/src/next/host/route.ts +++ b/packages/chord/src/next/host/route.ts @@ -98,6 +98,11 @@ export class Route { /** The host closes: no address resolves again. Returns every holder of every path, for the host to end. */ close(): Upstream[] { this.#backoff.clear(); + return this.holders(); + } + + /** Every holder of every path of this route, linked or not (revocation ends a revoked principal's). */ + holders(): Upstream[] { return [...this.#addresses.values()].flatMap(({ path }) => path.holders()); } diff --git a/packages/chord/src/next/host/unbound.ts b/packages/chord/src/next/host/unbound.ts new file mode 100644 index 000000000..d9be173f4 --- /dev/null +++ b/packages/chord/src/next/host/unbound.ts @@ -0,0 +1,62 @@ +// The unbound index (dependency model §1, §4): the requests a host decided cannot bind yet, by the name they wait for. +// Before the host's first publication every request made here, and every hold arriving on its links, waits here +// (`unpublished`); after it, those its check left unbound (`not_served`, `version`): a reference made here, a hold that +// arrived over a link and was refused for a name. A publication wakes the requests of the names it serves; a request +// leaves when it ends (released, settled, given up, its link or origin closed, revoked) or binds. + +import type { ChordError, Principal } from "../data/index.ts"; + +/** A request in the index: a reference made here or a hold that arrived over a link. */ +export interface UnboundRequest { + /** Whom it acts for: revocation ends the requests of a revoked principal. */ + readonly principal: Principal | undefined; + /** The name it waits for became usable (or the host published): it runs its check again, leaving the index first. */ + wake(): void; + /** It is revoked, or its host closes: it ends with `error` (a hold sends `gone`), leaving the index. */ + end(error: ChordError): void; +} + +/** A host's unbound requests by name. */ +export class UnboundIndex { + readonly #byName = new Map>(); + + /** Indexes `request` under `name` (once; again after a wake). */ + add(name: string, request: UnboundRequest): void { + let requests = this.#byName.get(name); + if (requests === undefined) { + requests = new Set(); + this.#byName.set(name, requests); + } + requests.add(request); + } + + /** `request` ended or binds: it leaves the index. */ + delete(name: string, request: UnboundRequest): void { + const requests = this.#byName.get(name); + if (requests === undefined || !requests.delete(request)) return; + if (requests.size === 0) this.#byName.delete(name); + } + + /** The host's publication: every request indexed (by any name) runs its check again, out of the index first. */ + wakeAll(): void { + const all = [...this.#byName.values()].flatMap((requests) => [...requests]); + this.#byName.clear(); + for (const request of all) request.wake(); + } + + /** Every indexed request acting for a principal `revoked` names ends with `error`. */ + revoke(revoked: (principal: Principal) => boolean, error: ChordError): void { + for (const requests of [...this.#byName.values()]) { + for (const request of [...requests]) { + if (request.principal !== undefined && revoked(request.principal)) request.end(error); + } + } + } + + /** The host closed: every indexed request ends with `error`. */ + close(error: ChordError): void { + const all = [...this.#byName.values()].flatMap((requests) => [...requests]); + this.#byName.clear(); + for (const request of all) request.end(error); + } +} diff --git a/packages/chord/src/next/link/binary.ts b/packages/chord/src/next/link/binary.ts index df9fbc80e..65be9335c 100644 --- a/packages/chord/src/next/link/binary.ts +++ b/packages/chord/src/next/link/binary.ts @@ -6,6 +6,7 @@ import { type BoundEntry, + CAUSE_KINDS, type Codec, type FailedMember, type Message, @@ -49,6 +50,11 @@ const PRESENT = 1; const SEGMENTS = 2; +/** A retry's optional fields, one bit each: its operation (16 raw bytes), then the ids it passed through (a list). */ +const OPERATION = 1; + +const THROUGH = 2; + /** A hold's optional fields, one bit each. */ const PRINCIPAL = 1; @@ -122,10 +128,20 @@ function encodeFields(message: Message, writer: Writer): void { return; } case "release": - case "retry": case "loading": writer.varint(message.hold); return; + case "retry": { + const { operation, through } = message; + writer.varint(message.hold); + writer.byte((operation === undefined ? 0 : OPERATION) | (through === undefined ? 0 : THROUGH)); + if (operation !== undefined) writer.raw(hex(operation.replaceAll("-", ""), 16)); + if (through !== undefined) { + writer.varint(through.length); + for (const instance of through) writer.raw(hex(instance.replaceAll("-", ""), 16)); + } + return; + } case "call": writer.varint(message.call); writer.varint(message.hold); @@ -162,7 +178,7 @@ function encodeFields(message: Message, writer: Writer): void { return; case "failed": writer.varint(message.hold); - writer.byte(message.cause.kind === "transient" ? 0 : 1); + writer.byte(CAUSE_KINDS.indexOf(message.cause.kind)); writer.json(message.cause.error); // 0 when absent, else one more than the delay. writer.varint(message.retryIn === undefined ? 0 : message.retryIn + 1); @@ -224,9 +240,16 @@ function decodeFields(kind: (typeof KINDS)[number], reader: Reader): Message { }; } case "release": - case "retry": case "loading": return { kind, hold: reader.varint() }; + case "retry": { + const hold = reader.varint(); + const flags = reader.byte(); + if (flags > (OPERATION | THROUGH)) throw new Error("binary: an unknown retry flag"); + const operation = flags & OPERATION ? { operation: uuid(reader.raw(16)) } : {}; + const through = flags & THROUGH ? { through: reader.list(() => uuid(reader.raw(16))) } : {}; + return { kind, hold, ...operation, ...through }; + } case "call": { const [call, hold, method] = [reader.varint(), reader.varint(), reader.varint()]; return { kind, call, hold, method, ...reader.payload("args") }; @@ -255,15 +278,15 @@ function decodeFields(kind: (typeof KINDS)[number], reader: Reader): Message { } case "failed": { const hold = reader.varint(); - const cause = reader.byte(); - if (cause > 1) throw new Error("binary: an unknown failure kind"); + const cause = CAUSE_KINDS[reader.byte()]; + if (cause === undefined) throw new Error("binary: an unknown failure kind"); // The schema check after decoding establishes the wire error. const error = reader.json() as Extract["error"]; const retryIn = reader.varint(); return { kind, hold, - cause: { kind: cause === 0 ? "transient" : "broken", error }, + cause: { kind: cause, error }, ...(retryIn === 0 ? {} : { retryIn: retryIn - 1 }), }; } diff --git a/packages/chord/src/next/link/message.ts b/packages/chord/src/next/link/message.ts index 84bab8629..49302273f 100644 --- a/packages/chord/src/next/link/message.ts +++ b/packages/chord/src/next/link/message.ts @@ -3,7 +3,7 @@ import Type from "typebox"; import type { Context } from "../../context/index.ts"; -import { arr, int, lit, obj, opt, type Static, str, union, unknown } from "../contracts/index.ts"; +import { arr, int, lit, obj, oneOf, opt, type Static, str, union, unknown } from "../contracts/index.ts"; import { validatorFor } from "../data/internal.ts"; import type { Part } from "./part.ts"; @@ -50,6 +50,9 @@ const incarnation = str({ pattern: "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f /** The longest activation chain a hold carries (Q5): a longer one keeps its last entries. */ export const CHAIN = 32; +/** The kinds of a `failed` message's cause, in their codes' order (the binary codec's byte, the text codec's letter). */ +export const CAUSE_KINDS: ["transient", "broken", "waiting"] = ["transient", "broken", "waiting"]; + /** The protocol's messages, one tagged union. Parts are opaque here (the owners parse them). */ const MESSAGE = union([ // `calls`: how many concurrent calls this side accepts (the call window). @@ -75,8 +78,16 @@ const MESSAGE = union([ obj({ kind: lit("release"), origin: opt(origin), hold: id }), obj({ kind: lit("call"), origin: opt(origin), call: id, hold: id, method: index, args: opt(part), segments }), obj({ kind: lit("abort"), origin: opt(origin), call: id }), - // Asks the provider for an attempt of a failed hold's object now (`ref.retry()`). - obj({ kind: lit("retry"), origin: opt(origin), hold: id }), + // Asks the provider for an attempt of a failed hold's object now (`ref.retry()`); `operation` names the one retry it + // belongs to (a uuidv7) and `through` the instance ids of the objects it passed through (with `operation` only), so + // an object it reaches twice (a diamond, a ring) handles it once. Relays forward both. + obj({ + kind: lit("retry"), + origin: opt(origin), + hold: id, + operation: opt(incarnation), + through: opt(Type.Array(incarnation, { maxItems: CHAIN })), + }), // Binds several holds at once (first bindings, re-holds), applied in one batch. obj({ kind: lit("bound"), @@ -103,8 +114,9 @@ const MESSAGE = union([ origin: opt(origin), hold: id, // A failure cause's error in the wire error format (a ChordError, checked by L1's envelope check, or a plugin's). - cause: obj({ kind: union([lit("transient"), lit("broken")]), error: wireError }), - // The rest of the object's backoff window in milliseconds: when it retries by itself (absent: broken). + // `waiting`: not the object's fault (no provider, no fitting version, a dependency failed); an event wakes it. + cause: obj({ kind: oneOf(...CAUSE_KINDS), error: wireError }), + // The rest of the object's backoff window in milliseconds: when it retries by itself (absent: broken, waiting). retryIn: opt(int({ min: 0 })), }), // The failed object of a hold started a new attempt. diff --git a/packages/chord/src/next/link/text.ts b/packages/chord/src/next/link/text.ts index e20c741bb..493657b17 100644 --- a/packages/chord/src/next/link/text.ts +++ b/packages/chord/src/next/link/text.ts @@ -8,6 +8,7 @@ import { type BoundEntry, + CAUSE_KINDS, type Codec, type FailedMember, type Message, @@ -91,7 +92,7 @@ function ownTuple(message: Message): string { case "state": return `["s",[${message.holds.map(stateEntry).join(",")}]]`; case "failed": { - const kind = message.cause.kind === "transient" ? "t" : "b"; + const kind = message.cause.kind[0]; const retryIn = message.retryIn === undefined ? [] : [message.retryIn]; return json(["f", message.hold, kind, message.cause.error, ...retryIn]); } @@ -104,7 +105,10 @@ function ownTuple(message: Message): string { case "revoke": return json(["V", message.revocation]); case "retry": - return json(["T", message.hold]); + // `through` travels only with an `operation` (Chord sends both). + if (message.operation === undefined) return json(["T", message.hold]); + if (message.through === undefined) return json(["T", message.hold, message.operation]); + return json(["T", message.hold, message.operation, message.through]); case "loading": return json(["l", message.hold]); case "ack": @@ -257,10 +261,11 @@ function fromFields(fields: unknown[], rest: Uint8Array): Message { }; case "f": { arity(3, 4); - if (f[1] !== "t" && f[1] !== "b") throw new Error("text: an unknown failure kind"); + const causeKind = CAUSE_KINDS.find((each) => each[0] === f[1]); + if (causeKind === undefined) throw new Error("text: an unknown failure kind"); // The schema check after decoding establishes the wire error. const error = f[2] as Extract["error"]; - const cause = { kind: f[1] === "t" ? "transient" : "broken", error } as const; + const cause = { kind: causeKind, error }; return { kind: "failed", hold: field(0), cause, ...(f.length === 4 ? { retryIn: field(3) } : {}) }; } case "r": @@ -276,8 +281,13 @@ function fromFields(fields: unknown[], rest: Uint8Array): Message { arity(1); return { kind: "revoke", revocation: f[0] }; case "T": - arity(1); - return { kind: "retry", hold: field(0) }; + arity(1, 2, 3); + return { + kind: "retry", + hold: field(0), + ...(f.length >= 2 ? { operation: field(1) } : {}), + ...(f.length === 3 ? { through: field(2) } : {}), + }; case "l": arity(1); return { kind: "loading", hold: field(0) }; diff --git a/packages/chord/src/next/registries/README.md b/packages/chord/src/next/registries/README.md index 070c98877..535973db0 100644 --- a/packages/chord/src/next/registries/README.md +++ b/packages/chord/src/next/registries/README.md @@ -39,8 +39,10 @@ host.serve(Server, (_, self) => { a change of the table marks it, and it composes when read or, while someone watches, observes or waits for it, in the batch that changed the table (after the commit's adoptions); an unchanged table (every entry the same) composes nothing. A `compose` that throws is the owner's bug: `registry_failed` once per episode, watches get one `failed` - call, `.value` keeps the last good composition (before any, it throws the error); the next good composition is a - value call, also when its value is the same. + call, `.value` keeps the last good composition while every contribution it was built from is still in the table + (before any, or once one of them left, it throws the error: a contributor that failed or ended may have closed the + native its entry held; decision 5, slice 7a); the next good composition is a value call, also when its value is the + same. - The table (`RegistryTable`): one `RegistryEntry { plugin, type?, version, value }` per live contributor, sorted by plugin name, then type (a facet's entry first): never load order. A contribution of an older version is adapted with `accepts[version].contribution` once per contribution value; a newer one, or an older one without an adapter, is left @@ -53,16 +55,18 @@ host.serve(Server, (_, self) => { `RegistryValue`: a native watchable (`.value`, `watch`) with `observe(observer, ctx)`, a synchronous observer of its compositions (Durable's shape, for `registryReader`). A reader whose token is older than the provider's gets the composition mapped back with `accepts[version].value`, once per composition, only while read or followed; a mapping - that throws fails that version only (`registry_failed`, its last good mapping kept). A reader without a mapping, or - newer than the provider, gets `contract_mismatch` (its object fails `broken`: only a reload helps). The mapping is - kept only while a reader or follower is there. Without a provider a registry has no value - (`unavailable/undeclared`); its contributions wait in the table. + that throws fails that version only (`registry_failed`, its last good mapping kept while every contribution the + composition it was mapped from was built from is still in the table: decision 5 holds for every version). A reader without a mapping, or + newer than the provider, gets `contract_mismatch`. The mapping is kept only while a reader or follower is there. + Without a provider a registry has no value (`unavailable/undeclared`); its contributions wait in the table. A + reader's attempt without a usable value parks `failed { waiting }` (`host/`), woken when the value turns usable; a + `host.registry` waits the same way up to its `readyTimeout`. - `keyedEntries(table, entriesOf, report)`: a map of every entry's keyed items, the first by plugin name keeping a key, a duplicate reported through `compose`'s `report`; `orderedEntries(table, itemsOf, orderOf?)`: every entry's items in table order, or by `orderOf`. Both keep each item's `plugin` (and `type`): a contribution is another plugin's code, so a use site catches its throws and rejections, reports the plugin, and goes on. -- Registries carry natives: an entry lives exactly as long as its contributor. A last good composition kept after a - throwing `compose` may still name a contributor that left: use sites look up at each use and keep nothing. +- Registries carry natives: an entry lives exactly as long as its contributor, and a last good composition no longer + than the contributions it was built from; use sites look up at each use and keep nothing. ## For Chord's layers (`internal.ts`) @@ -75,7 +79,11 @@ host.serve(Server, (_, self) => { (`value(version)`); `close()` at the host's close. - `RegistryContribution` (`contribution.ts`): one contributor's contribution, a `Dependent` of its source; `entry(version, provider)` is its table entry (the same object while its value is), or `undefined` when left out. -- `RegistryValueSource`: a `FollowedSource` per reader version; `read(waiter)` registers an activation's object - (`RegistryWaiter`), woken with `dependencyLive()` (on a microtask) when the value gets its first good value; readers - waiting for one count as followers, so a change composes for them. At the host's close it keeps its last start and - lets the record (and the host) go. +- `RegistryValueSource`: a `FollowedSource` per reader version; `read()` registers a reader (an attempt that declared + it, until its references are released: an older version's mapping is kept meanwhile); `wait(waiter)` registers a + `RegistryWaiter` (a parked attempt, a waiting `host.registry`), told with `dependencyMoved()` (inside the batch: it + only records) at every change of whether the value is usable or of its failure while it is not (errors compared by + value: a version mismatch is made per read), and with `closed(error)` when the host closes (`released/closed`: the + wait ends); waiters count as followers, so a change composes for them. At the host's close it keeps its last start + and lets the record (and the host) go. `RegistryRecord.composedFrom` and `inTable` give a mapping the provenance + decision 5 checks. diff --git a/packages/chord/src/next/registries/record.ts b/packages/chord/src/next/registries/record.ts index f879132f9..110ad7e51 100644 --- a/packages/chord/src/next/registries/record.ts +++ b/packages/chord/src/next/registries/record.ts @@ -44,9 +44,12 @@ export type RegistryEdge = { readonly from: RegistryRecord; readonly to: Registr /** The provider as served: the defining plugin's version, `compose` and adapters. */ type Provider = { readonly version: number; readonly provider: RegistryProvider }; -/** Where a composition stands: never composed, its last good value, and its failure while `compose` throws. */ +/** + * Where a composition stands: never composed, its last good value (with the contributions it was built from), and its + * failure while `compose` throws. + */ type Composition = { - readonly good: { readonly value: unknown } | undefined; + readonly good: { readonly value: unknown; readonly from: readonly RegistryContribution[] } | undefined; readonly failure: { readonly error: unknown } | undefined; }; @@ -59,7 +62,7 @@ export class RegistryRecord { readonly #report: (error: RegistryError) => void; #provider: Provider | undefined; /** The contributions of live contributors, sorted by plugin, then type (a facet's first). */ - readonly #contributions: RegistryContribution[] = []; + #contributions: readonly RegistryContribution[] = []; /** The last table composed, kept so an unchanged table composes nothing. */ #table: RegistryTable | undefined; #dirty = true; @@ -128,7 +131,8 @@ export class RegistryRecord { /** Adds a live contributor's contribution, inside a batch: the table changes. */ add(contribution: RegistryContribution): void { const at = this.#contributions.findIndex((other) => compareContributors(contribution, other) < 0); - this.#contributions.splice(at < 0 ? this.#contributions.length : at, 0, contribution); + // A new list per change (contributors come and go rarely): a composition keeps the one it was built from. + this.#contributions = this.#contributions.toSpliced(at < 0 ? this.#contributions.length : at, 0, contribution); const { reads } = contribution; if (reads !== undefined) this.#reads.set(reads, (this.#reads.get(reads) ?? 0) + 1); contribution.follow(); @@ -139,7 +143,7 @@ export class RegistryRecord { remove(contribution: RegistryContribution): void { const at = this.#contributions.indexOf(contribution); if (at < 0) return; - this.#contributions.splice(at, 1); + this.#contributions = this.#contributions.toSpliced(at, 1); const { reads } = contribution; if (reads !== undefined) { const left = this.#reads.get(reads)! - 1; @@ -176,6 +180,20 @@ export class RegistryRecord { return { kind: "value", value: good!.value }; } + /** The contributions the current good composition was built from; `undefined` while there is none. */ + get composedFrom(): readonly RegistryContribution[] | undefined { + return this.#composition.good?.from; + } + + /** + * Whether every contribution of `from` (what a composition, or a mapping of it, was built from) is still in the + * table (decision 5: one that left may have closed the native its entry held). + */ + inTable(from: readonly RegistryContribution[]): boolean { + const current = this.#contributions; + return from === current || from.every((each) => current.includes(each)); + } + /** The adapter mapping the composed value back for readers of `version`; `undefined` when there is none. */ valueAdapter(version: number): ((current: never) => unknown) | undefined { return this.#provider?.provider.accepts?.[version]?.value; @@ -203,7 +221,7 @@ export class RegistryRecord { */ close(): void { for (const contribution of this.#contributions) contribution.unfollow(); - this.#contributions.length = 0; + this.#contributions = []; this.#reads.clear(); this.#table = undefined; this.#dirty = false; @@ -226,8 +244,9 @@ export class RegistryRecord { /** * Builds the table and composes it, unless it is the last one composed, entry for entry (`compose` is pure). A - * throw is a failure episode: reported once, the last good composition kept; conflicts are reported once per - * episode, as their message and plugins. + * throw is a failure episode: reported once, the last good composition kept while every contribution it was built + * from is still in the table (decision 5: one whose contributor ended may hold a closed native); conflicts are + * reported once per episode, as their message and plugins. */ #recompose({ version, provider }: Provider): void { this.#dirty = false; @@ -247,10 +266,14 @@ export class RegistryRecord { value = this.#batch.compute(() => provider.compose(table, context)); } catch (error) { if (this.#composition.failure === undefined) this.failed(error); - this.#composition = { good: this.#composition.good, failure: { error } }; + const { good } = this.#composition; + this.#composition = { + good: good !== undefined && this.inTable(good.from) ? good : undefined, + failure: { error }, + }; return; } - this.#composition = { good: { value }, failure: undefined }; + this.#composition = { good: { value, from: this.#contributions }, failure: undefined }; for (const [key, conflict] of conflicts) { if (this.#conflicts.has(key)) continue; this.#report({ kind: "registry_conflict", registry: this.name, ...conflict }); diff --git a/packages/chord/src/next/registries/value.ts b/packages/chord/src/next/registries/value.ts index 7b3b8e1db..f08b25987 100644 --- a/packages/chord/src/next/registries/value.ts +++ b/packages/chord/src/next/registries/value.ts @@ -1,20 +1,40 @@ // Registry values (slice 8 §1 "Reading", S2, S3): a registry's composed value as the readers of one version see it, a // native source. The provider's version reads the composition itself; an older one reads it mapped back with // `accepts[old].value`, once per composition, kept only while someone reads or follows it; a version the provider -// cannot serve fails `contract_mismatch`. Readers declared by attempts are woken (R18) when the value gets its first -// good value. +// cannot serve fails `contract_mismatch`. Its readers (attempts that declared it) keep the mapping; its waiters +// (attempts parked on it, a waiting `host.registry`) are woken at every change of whether it is usable, and of its +// failure while it is not (dependency model §9). -import { ChordError } from "../data/index.ts"; +import { ChordError, isChordError } from "../data/index.ts"; import { type Batch, FollowedSource, type NativeStart } from "../state/internal.ts"; +import type { RegistryContribution } from "./contribution.ts"; import type { RegistryRecord } from "./record.ts"; -/** An object waiting for a registry's value (`self.registry`): retried at once when its first value arrives (R18). */ +/** Who waits for a registry's value (a parked attempt, a `host.registry`): told when it may have changed. */ export interface RegistryWaiter { - dependencyLive(): void; + /** The value may have changed (inside its batch: it only records). */ + dependencyMoved(): void; + /** The value closed with its host: the wait ends with `error` (`released/closed`). */ + closed(error: ChordError): void; } -/** The last good mapping for an older version, and the composition it was made from. */ -type Mapped = { readonly from: unknown; readonly value: unknown }; +const USABLE = { kind: "usable" } as const; + +/** Whether two failures say the same: the same error, or ChordErrors alike (a version mismatch is made per read). */ +function sameError(a: unknown, b: unknown): boolean { + if (a === b) return true; + return isChordError(a) && isChordError(b) && a.code === b.code && a.reason === b.reason && a.message === b.message; +} + +/** + * The last good mapping for an older version, the composition it was made from, and the contributions that composition + * was built from (kept only while all are in the table, decision 5). + */ +type Mapped = { + readonly from: unknown; + readonly value: unknown; + readonly contributions: readonly RegistryContribution[] | undefined; +}; /** A failed mapping of composition `from`, until another composition maps. */ type MapFailure = { readonly from: unknown; readonly error: unknown }; @@ -32,10 +52,15 @@ export class RegistryValueSource extends FollowedSource { readonly name: string; readonly version: number; #registered: Registered; - /** Readers declared by attempts, each a token of its own (one object may read twice), until released. */ - readonly #readers = new Set<{ readonly waiter: RegistryWaiter }>(); - /** Whether it had a good value: the first one wakes the readers. */ - #valued = false; + /** Its readers, each a token of its own (one object may read twice), until released. */ + readonly #readers = new Set(); + /** Its waiters, each a token of its own, until released. */ + readonly #waiters = new Set<{ readonly waiter: RegistryWaiter }>(); + /** + * What its waiters were last told: usable (a value, or a failure with a last good one), or the error it has none + * for; a change wakes them. + */ + #told: { readonly kind: "usable" } | { readonly kind: "failed"; readonly error: unknown } | undefined; #mapped: Mapped | undefined; #mapFailure: MapFailure | undefined; @@ -60,16 +85,15 @@ export class RegistryValueSource extends FollowedSource { return start.last.value; } - /** Its composition (mapped for an older version), composed now if its table changed; a first value wakes readers. */ + /** + * Its composition (mapped for an older version), composed now if its table changed; a change of whether it is + * usable, or of its failure while it is not, wakes its waiters. + */ override start(): NativeStart { const registered = this.#registered; if (registered.kind === "closed") return registered.last; const start = this.#start(registered.record); - if (!this.#valued && (start.kind === "value" || start.last !== undefined)) { - this.#valued = true; - // Not inside the batch that composed: a retry runs a factory, which may not run in a batch. - for (const { waiter } of this.#readers) queueMicrotask(() => waiter.dependencyLive()); - } + if (this.#waiters.size > 0) this.#tell(start); return start; } @@ -80,14 +104,14 @@ export class RegistryValueSource extends FollowedSource { return start.value; } - /** Followed while someone watches or observes it, and while readers wait for its first value. */ + /** Followed while someone watches or observes it, and while waiters wait on it. */ override get watched(): boolean { - return super.watched || (this.#readers.size > 0 && !this.#valued); + return super.watched || this.#waiters.size > 0; } - /** Registers `waiter` (an activation's declared dependency) until the returned release. */ - read(waiter: RegistryWaiter): () => void { - const token = { waiter }; + /** Registers a reader (an attempt that declared it) until the returned release: its mapping is kept meanwhile. */ + read(): () => void { + const token = {}; this.#readers.add(token); return () => { this.#readers.delete(token); @@ -95,6 +119,20 @@ export class RegistryValueSource extends FollowedSource { }; } + /** + * Registers `waiter` (a parked attempt's object, a waiting `host.registry`) until the returned release: it is told + * of every change of whether this value is usable (inside the batch that made it: it only records). + */ + wait(waiter: RegistryWaiter): () => void { + const token = { waiter }; + if (this.#waiters.size === 0) this.#told = undefined; + this.#waiters.add(token); + return () => { + this.#waiters.delete(token); + if (this.#waiters.size === 0) this.#told = undefined; + }; + } + /** The registry's table changed (inside a batch): its dependents are marked, its followers get the next frame. */ recomposed(): void { this.changed(); @@ -119,6 +157,11 @@ export class RegistryValueSource extends FollowedSource { } this.#registered = { kind: "closed", last }; this.#readers.clear(); + const waiters = [...this.#waiters]; + this.#waiters.clear(); + const error = new ChordError({ code: "released", reason: "closed", message: `registry ${this.name}: closed` }); + for (const { waiter } of waiters) waiter.closed(error); + this.#told = undefined; this.#dropMapping(); this.end(); } @@ -128,6 +171,17 @@ export class RegistryValueSource extends FollowedSource { if (this.#readers.size === 0) this.#dropMapping(); } + /** Wakes the waiters when `start` says something else than they were told (by usability, then by the error). */ + #tell(start: NativeStart): void { + const told = this.#told; + const usable = start.kind === "value" || start.last !== undefined; + if (usable ? told?.kind === "usable" : told?.kind === "failed" && sameError(told.error, start.error)) return; + this.#told = usable ? USABLE : { kind: "failed", error: (start as { readonly error: unknown }).error }; + // The first start of a wait wakes nobody: the waiter reads it itself. + if (told === undefined) return; + for (const { waiter } of this.#waiters) waiter.dependencyMoved(); + } + #dropMapping(): void { this.#mapped = undefined; this.#mapFailure = undefined; @@ -156,13 +210,18 @@ export class RegistryValueSource extends FollowedSource { if (current.kind === "failed" && current.last === undefined) return current; const from = current.kind === "value" ? current.value : current.last!.value; // `undefined` is a composition too: the records, not their fields, say whether there is one. + // A mapping built from a contribution that left is no last good value any more (decision 5). + if (this.#mapped?.contributions !== undefined && !record.inTable(this.#mapped.contributions)) { + this.#mapped = undefined; + } const mapped = this.#mapped; const failure = this.#mapFailure; if (mapped !== undefined && mapped.from === from) { this.#mapFailure = undefined; } else if (failure === undefined || failure.from !== from) { try { - this.#mapped = { from, value: record.adapt(() => map(from as never)) }; + const value = record.adapt(() => map(from as never)); + this.#mapped = { from, value, contributions: record.composedFrom }; this.#mapFailure = undefined; } catch (error) { if (failure === undefined) record.failed(error); diff --git a/packages/chord/src/next/remote/README.md b/packages/chord/src/next/remote/README.md index 16522f71b..19265eb12 100644 --- a/packages/chord/src/next/remote/README.md +++ b/packages/chord/src/next/remote/README.md @@ -55,8 +55,8 @@ within each link's limits, compression at the origin, activation chains and revo one, which a relayed hold carries upstream as `limit`. A call on a relayed hold whose arguments are past the link's wire limit, or on a served hold whose arguments are past its limits, is answered `too_large/part` and never admitted. Compressed params (on any link) and compressed arguments from a child are protocol errors: params are never - compressed, and clients never send compressed parts. Holds of a principal being revoked that still wait for `load` - end with `gone` (`revokeWaiting`). + compressed, and clients never send compressed parts. Holds of a principal being revoked that hold nothing (before the + publication, refused, relayed and waiting for their link) end with `gone` (`revokeUnbound`). - **The window** (K = 2, `ProviderSide`): at most two of this host's own `bound` and `state` messages unacknowledged per origin; every other message is control and never waits. The consumer sends `ack` with the origin after applying each. The side keeps the sender of every message in flight in order (`local`, or the upstream link a relayed one came @@ -81,13 +81,18 @@ within each link's limits, compression at the origin, activation chains and revo Trace (tab D through relay R, worker U, a derive over A and B on the tab): U's window for D's origin is full; a commit touching A and B joins U's pending queues. D acks one: R returns the `ack` to U, whose freed slot carries one `state` with A's and B's entries: the derive sees both at once, never A new with B old. -- `Hold` (one per `hold` served here, a `Holder` of its object): takes its object once `load` resolved; the hold's - `params` part is parsed then (malformed: a protocol error). A refused type, a mismatch, invalid params, a wrong - host address or a snapshot that throws is a broken `failed` that ends it (the provider forgets the id). While pending - it follows its object's `state`: each failure, transient or broken, is sent as `failed` with `retryIn`, the rest of - the object's backoff window (absent when broken), the current one at once when the object is failed already; the - next attempt is announced with `loading`; the hold stays, so a `retry` (from any holder) renews even a broken object - for every holder. Live, it binds through the window (`boundEntry`: the variant and each member's value as a part, +- `Hold` (one per `hold` served here, a `Holder` of its object, an `UnboundRequest` of the host's index): before the + host's publication it waits in the unbound index (`unplaced`, nothing sent: the consumer is `loading`) and the + publication starts it; then it takes its object in the turn it arrived; the hold's `params` part is parsed then + (malformed: a protocol error). A type not served here (or as a local type), a mismatch, or params the provider's + code places on another host leave it refused `failed { waiting }`, indexed under its type until a publication re-runs + its start (`loading`, then `bound` or `failed`); invalid params or a cycle leave it refused `failed { broken }`; both + are kept for a `retry`, which re-runs the start. While pending it follows its object's `state`: each failure is sent + as `failed` (with `retryIn`, the rest of the object's backoff window, for a transient one; absent for broken and + waiting), each new cause of an object parked on a failed dependency too, the current one at once when the object is + failed already; the next attempt (or the parked object's next `loading`) is announced with `loading`; the hold stays, + so a `retry` (from any holder) renews even a broken object for every holder (a parked object passes it on, one retry + operation of this host). Live, it binds through the window (`boundEntry`: the variant and each member's value as a part, names as indexes into the bound version). Its members' frames arrive as `Subscriber` calls inside the batch; at its end the hold's entry lists the members that changed with their ops, each encoded with that member's own Delta encoder (`entry`), or queues them (`queue`, later `pendingEntry`). Each entry counts one `chord.link.encodes`. A @@ -97,9 +102,10 @@ within each link's limits, compression at the origin, activation chains and revo dropping the member's pending ops; a value after it replaces it), and a retirement of a failed member waits for its next value (never listed beside its failure). Its object's end sends `gone` with its error (`forbidden/revoked` for a revocation; its running calls aborted with it). A - released pending hold stops its wait for `load` at once. Params past the link's wire limit, or its decoded one, fail - the hold broken `too_large/part`; a cycle its chain closes, broken `invalid/cycle` (`host.serving`). It counts as one - of the host's `references` (`host.activity`) while it keeps its object. + released hold before the publication leaves the index at once. Params past the link's wire limit, or its decoded + one, fail the hold broken `too_large/part`; a cycle its chain closes, broken `invalid/cycle` (`host.serving`). It + counts as one of the host's `references` (`host.activity`) from its arrival to its end, bound or not (relayed, it + counts at its destination instead). - `RelayedHold` (a hold this host neither serves nor parses, `host.place` decided it): id `h` of its downstream origin, held with the same id on its path's current link under that origin's number, as the arriving hold's caller, from its kept `hold` message (params copied once out of the received message, so no view keeps the buffer). `call`, @@ -134,9 +140,9 @@ within each link's limits, compression at the origin, activation chains and revo tells its owner (a route) when it needs a link or lost its last holder. - `RemoteObject` (one per remote reference, owned by its host across the links of its path): its `state` (`loading` until a link's hold is bound and again on every drop or `loading`, `failed` from `failed` with `retryAt = now + - retryIn`, `live` on `bound`, `released`), `settled` (pending while loading, resolved while live, rejected while - failed and by a drop or a relay's `loading` while loading: a dependent's activation waits for it and fails at once - then, R18, and `bound` retries the dependent), `lastFailure`, `variant` (set in `bound`'s batch, before derives recompute) and + retryIn` for a transient cause, `retryAt` undefined for a broken or waiting one, `live` on `bound`, `released`; + every change tells its binding, whose parked dependent checks again: no new hold, no backoff), `remote` and + `blocksFirstPass` (never: a wait on another host ends this host's first pass), `lastFailure`, `variant` (set in `bound`'s batch, before derives recompute) and `member` (the last binding's, kept while loading or failed; outside the variant `invalid/member`), `call` (only while live; outside the variant `invalid/member`), `retry` (sends `retry` while failed on an open link; after a part past the limits, placed anew with `host.pathOf` and held with a new id, now or on that path's next link). Its holds made diff --git a/packages/chord/src/next/remote/consumer.ts b/packages/chord/src/next/remote/consumer.ts index 8d654e291..e4b8516e6 100644 --- a/packages/chord/src/next/remote/consumer.ts +++ b/packages/chord/src/next/remote/consumer.ts @@ -8,6 +8,7 @@ import type { Context } from "../../context/index.ts"; import { type MethodData, takesNoArgs } from "../contracts/internal.ts"; import { ChordError, isChordError } from "../data/index.ts"; import { asChordError } from "../data/internal.ts"; +import type { RetryOperation } from "../host/internal.ts"; import { type BoundEntry, type ConsumerMessage, @@ -125,9 +126,10 @@ export class ConsumerSide { this.forget(hold, error); } - /** `ref.retry()` on a failed hold: the provider is asked for an attempt now. */ - retry(hold: number): void { - this.#link.send({ kind: "retry", hold }); + /** `ref.retry()` on a failed hold, as retry `operation`: the provider is asked for an attempt now. */ + retry(hold: number, { id, through }: RetryOperation): void { + const passed = through.length === 0 ? {} : { through }; + this.#link.send({ kind: "retry", hold, operation: id, ...passed }); } /** Drops hold `hold` from the tables; its calls, in flight and waiting, fail with `error` and free their slots. */ diff --git a/packages/chord/src/next/remote/hold.ts b/packages/chord/src/next/remote/hold.ts index b018c201b..21e5230c6 100644 --- a/packages/chord/src/next/remote/hold.ts +++ b/packages/chord/src/next/remote/hold.ts @@ -1,7 +1,8 @@ -// A hold arriving on a link and served here (§7, §22, W1): where it goes once `load` resolved, the object it keeps, -// the notices of its object's failures and new attempts, the snapshot and subscription taken in one synchronous step -// when a slot of the window is free, and the ops of its members per batch (sent at once, or pending while the window -// is full), each member with its own Delta encoder. +// A hold arriving on a link and served here (§7, §22, W1): where it goes once the host published (waiting in the +// unbound index until then), why it is unbound (refused: kept in the index while it waits for a name, dependency +// model §1), the object it keeps, the notices of its object's failures and new attempts, the snapshot and subscription +// taken in one synchronous step when a slot of the window is free, and the ops of its members per batch (sent at +// once, or pending while the window is full), each member with its own Delta encoder. import type { Context } from "../../context/index.ts"; import { type Encoder, encoder, type Op, PendingOps } from "../../delta/index.ts"; @@ -13,9 +14,16 @@ import { variantNames, variantOrder, } from "../contracts/internal.ts"; -import { type ChordError, isChordError } from "../data/index.ts"; +import { type ChordError, type FailureCause, isChordError, type Principal } from "../data/index.ts"; import { asChordError } from "../data/internal.ts"; -import type { Caller, ChordObject, Holder } from "../host/internal.ts"; +import { + type Caller, + type ChordObject, + type Holder, + retryOperation, + type Serving, + type UnboundRequest, +} from "../host/internal.ts"; import { type BoundEntry, type FailedMember, @@ -56,12 +64,13 @@ type Followed = { }; /** - * Where a hold stands: `pending` until its object is live (waiting for `load`, the activation and its retries; - * `toldFailed` once the consumer was told of a failure, so the next attempt is announced with `loading`); `binding` - * while its object is live and its `bound` waits for a slot; `bound` with the object's variant and the members it - * follows. + * Where a hold stands: `unplaced` in the unbound index until the host's publication places it; `pending` until its + * object is live (the activation, parking and retries; `toldFailed` once the consumer was told of a failure, so the + * next attempt is announced with `loading`); `binding` while its object is live and its `bound` waits for a slot; + * `bound` with the object's variant and the members it follows. */ type HoldState = + | { readonly kind: "unplaced"; readonly request: HoldRequest } | { readonly kind: "pending"; readonly toldFailed: boolean } | { readonly kind: "binding" } | { @@ -72,8 +81,12 @@ type HoldState = /** The members it follows, in the order they were subscribed. */ readonly followed: Followed[]; } - /** Refused where it arrived (host address, `authorize`, a mismatch, invalid params, a cycle): `retry` restarts it. */ - | { readonly kind: "refused"; readonly request: HoldRequest } + /** + * Refused where it arrived, with `cause`: `waiting` (not served here, another version, placed elsewhere by the + * provider's code) in the unbound index until a publication wakes it; `broken` (invalid params, a cycle, a route that + * refused, a part past the limits) until a `retry` restarts it. + */ + | { readonly kind: "refused"; readonly request: HoldRequest; readonly cause: FailureCause } | { readonly kind: "released" }; /** A bound hold's state. */ @@ -86,11 +99,13 @@ const BINDING: HoldState = { kind: "binding" }; /** * One hold served here: a reference to its object acting as its caller (the hold's principal). While pending, each * failure of its object is sent as `failed` (with the rest of its backoff window) and the next attempt as `loading`; a - * hold that cannot be taken (a refused type, a mismatch, invalid params or a wrong host address) is a broken `failed` - * that holds nothing until a `retry` starts it again. Once live it binds when its link's window has a slot, then sends a part per batch that changed its members - * (or keeps their ops pending while the window is full), and `gone` when its object ends. + * hold that cannot be taken holds nothing: `failed { waiting }` (not served, another version) until a publication + * re-runs its start, `failed { broken }` (invalid params, a cycle) until a `retry` does. Once live it binds when its + * link's window has a slot, then sends a part per batch that changed its members (or keeps their ops pending while the + * window is full), and `gone` when its object ends. It counts as one of the host's references from its arrival to its + * end, bound or not. */ -export class Hold implements Holder { +export class Hold implements Holder, UnboundRequest { readonly kind = "served"; readonly id: number; /** A hold is declared by no object on this host. */ @@ -103,11 +118,6 @@ export class Hold implements Holder { #state: HoldState = PENDING; /** The held object, from its binding until the hold is released. */ #object: ChordObject | undefined; - /** - * Resolved on release: a pending hold stops waiting for `load`. A plain resolution, not an abort: an aborted wait - * rejects with an error whose stack keeps this hold, and the waited promise keeps that rejection. - */ - readonly #released = Promise.withResolvers(); /** The members the current batch changed, with their ops, in order. */ #changes: (readonly [Followed, readonly Op[]])[] = []; /** The members that failed since the last message that carried this hold. */ @@ -115,14 +125,15 @@ export class Hold implements Holder { /** The members whose document was retired since the last message that carried this hold (ORed while it waits). */ #retired: readonly number[] = NO_RETIRED; #touched = false; - /** Whether the hold counts as one of the host's references (`host.activity`): while it keeps its object. */ - #counted = false; + /** Whether the hold counts as one of the host's references (`host.activity`): from its arrival until it ends. */ + #counted = true; constructor(id: number, provider: ProviderSide, caller: Caller, limits: Limits) { this.id = id; this.#provider = provider; this.caller = caller; this.limits = limits; + provider.link.host.count("references", 1); } /** Whether the hold is live: calls on it reach its object. */ @@ -130,33 +141,37 @@ export class Hold implements Holder { return this.#state.kind === "bound"; } - /** Whether the hold waits for `load`: admitted, nothing taken yet (revocation ends it, §4). */ - get waiting(): boolean { - return this.#state.kind === "pending" && this.#object === undefined; + /** Whom it acts for (the unbound index's revocation). */ + get principal(): Principal | undefined { + return this.caller.principal; + } + + /** Whether the hold holds no object: before the publication, or refused (revocation ends it, §4). */ + get unbound(): boolean { + return this.#object === undefined && this.#state.kind !== "released"; } /** - * Admits the hold once `load` resolved: its params within the link's part limit (`too_large/part`), then relayed - * (it is replaced by a relayed hold), or served here, its params opened and parsed (a malformed part is a protocol - * error), its object taken and followed (`moved`). A hold that cannot go anywhere, a type that does not match, - * invalid params, a wrong host address or a cycle (Q5) is a broken `failed`; the hold is kept, refused, for a - * `retry`. + * Admits the hold: before the host's publication it waits in the unbound index, placed by the publication. + * Then its params within the link's part limit (`too_large/part`), then relayed (it is replaced by a relayed hold), + * or served here, its params opened and parsed (a malformed part is a protocol error), its object taken and followed + * (`moved`) in the turn it arrived. A hold that cannot be served here waits refused (`#refuse`). */ - async start(request: HoldRequest): Promise { + start(request: HoldRequest): void { const { link } = this.#provider; - // Waits only for a `load` still running (it never rejects): once loaded, the reference is taken in the turn the - // hold arrived. - const pendingLoad = link.host.pendingLoad; - if (pendingLoad !== undefined) await Promise.race([pendingLoad, this.#released.promise]); - // Released meanwhile (the wait may have settled first): nothing is taken. - if (this.#state.kind !== "pending") return; - let object: ChordObject; + if (!link.host.published) { + this.#state = { kind: "unplaced", request: owned(request) }; + link.host.index(request.type, this); + return; + } + let served: Serving; try { const oversize = oversized(request.params, link.limits.part); if (oversize !== undefined) throw oversize; const placed = link.host.place(request, link, this.caller, link.context); if (placed.kind === "path") { - this.#state = { kind: "released" }; + // Counted where it is served: a relayed hold is not this host's reference. + this.drop(); new RelayedHold(this.id, this.#provider, request, this.caller, placed.path).start(); return; } @@ -170,18 +185,35 @@ export class Hold implements Holder { link.close(); return; } - object = link.host.serving(request, params, this.caller.principal!, link.context); + served = link.host.serving(request, params, this.caller.principal!, link.context); } catch (error) { - return this.#refuse(request, asChordError(error, request.type)); + this.#refuse(request, { kind: "broken", error: asChordError(error, request.type) }); + return; } - // Counted before the object lists it in its presence: a derive over that may close this link synchronously. + if (served.kind === "waiting") { + this.#refuse(request, served.cause); + return; + } + const { object } = served; this.#object = object; - this.#counted = true; - link.host.count("references", 1); object.hold(this); this.moved(); } + /** + * Woken by the unbound index (out of it already): before the publication, placed now; refused `waiting`, its start + * runs again, announced with `loading` (then `bound` or `failed`). + */ + wake(): void { + const state = this.#state; + if (state.kind === "unplaced") { + this.#state = PENDING; + this.start(state.request); + } else if (state.kind === "refused") { + this.#restart(state.request); + } + } + /** * The object's state changed, or the hold took it: a pending hold binds once it is live (when a slot is free), * sends each failure, and announces the attempt after a failure with `loading`. A live object stays live until it @@ -195,6 +227,7 @@ export class Hold implements Holder { this.#state = BINDING; this.#provider.bind(this, this.#provider.link.context); } else if (state.kind === "failed") { + // Each failure and each new cause (a parked object's blocker changed) is a message of its own. const { cause, retryAt } = state; const retryIn = retryAt === undefined ? {} : { retryIn: Math.max(0, retryAt - Date.now()) }; const failed = { kind: cause.kind, error: wireError(cause.error) }; @@ -207,18 +240,26 @@ export class Hold implements Holder { } /** - * A `retry` message: the object is asked for an attempt now (it ignores this unless failed); a refused hold is - * started again (a route's `authorize` may admit it now, a type may be served here now). + * A `retry` message, of its retry operation (a new one when it names none) through the objects it names: the object is + * asked for an attempt now (it ignores this unless failed, or when the operation reached it before; parked, it + * passes it on); a refused hold is + * started again (a route's `authorize` may admit it now). */ - retry(): void { + retry({ operation, through = [] }: Extract): void { const state = this.#state; if (state.kind !== "refused") { - this.#object?.retry(); + this.#object?.retry(operation === undefined ? retryOperation() : { id: operation, through }); return; } + if (state.cause.kind === "waiting") this.#provider.link.host.unindex(state.request.type, this); + this.#restart(state.request); + } + + /** Starts a refused hold again, announced with `loading`. */ + #restart(request: HoldRequest): void { this.#state = PENDING; this.#provider.send({ kind: "loading", hold: this.id }); - void this.start(state.request); + this.start(request); } /** Calls a method of the held object as the hold's caller. */ @@ -324,18 +365,21 @@ export class Hold implements Holder { /** Ends the hold without releasing its object (the host closes and ends its objects itself). */ drop(): void { - if (this.#state.kind === "bound") { - for (const { source, subscriber } of this.#state.followed) source.unsubscribe(subscriber); + const state = this.#state; + const { host } = this.#provider.link; + if (state.kind === "bound") { + for (const { source, subscriber } of state.followed) source.unsubscribe(subscriber); + } else if (state.kind === "unplaced" || (state.kind === "refused" && state.cause.kind === "waiting")) { + host.unindex(state.request.type, this); } this.#state = { kind: "released" }; - this.#released.resolve(); - if (this.#counted) this.#provider.link.host.count("references", -1); + if (this.#counted) host.count("references", -1); this.#counted = false; } /** - * The object ended while held (revocation; a reload, slice 7), or the hold was revoked while it waited for `load`: - * the hold sends `gone` with `error`. + * The object ended while held (revocation; a reload, slice 7b), or the hold was revoked (or its host closed) while it + * held no object: the hold sends `gone` with `error`. */ end(error: ChordError): void { if (this.#state.kind === "released") return; @@ -426,16 +470,27 @@ export class Hold implements Holder { this.#provider.touched(this); } - /** Refuses the hold: a broken `failed`; it keeps `request` for a `retry`, holding nothing until then. */ - #refuse(request: HoldRequest, error: ChordError): void { - const { params } = request; - // Its params owned, not a view of the message they came in (which the link's buffers may hold). - const owned = typeof params === "string" ? params : (params as Uint8Array).slice(); - this.#state = { kind: "refused", request: { ...request, params: owned } }; - this.#provider.send({ kind: "failed", hold: this.id, cause: { kind: "broken", error: wireError(error) } }); + /** + * Refuses the hold with `cause`, holding nothing: it keeps `request`, indexed under its type while `waiting` (a + * publication starts it again), for a `retry` otherwise. + */ + #refuse(request: HoldRequest, cause: FailureCause): void { + this.#state = { kind: "refused", request: owned(request), cause }; + if (cause.kind === "waiting") this.#provider.link.host.index(request.type, this); + this.#provider.send({ + kind: "failed", + hold: this.id, + cause: { kind: cause.kind, error: wireError(cause.error) }, + }); } } +/** `request` with its params owned, not a view of the message they came in (which the link's buffers may hold). */ +function owned(request: HoldRequest): HoldRequest { + const { params } = request; + return typeof params === "string" ? request : { ...request, params: (params as Uint8Array).slice() }; +} + /** A `hold` message's fields naming whom it acts for. */ type ActingFor = Pick; diff --git a/packages/chord/src/next/remote/host-link.ts b/packages/chord/src/next/remote/host-link.ts index f0da9e79f..0afe07e20 100644 --- a/packages/chord/src/next/remote/host-link.ts +++ b/packages/chord/src/next/remote/host-link.ts @@ -193,9 +193,9 @@ export class HostLink implements ChannelHandler { } } - /** Ends the holds on this link still waiting for `load` that act for a principal `revoked` names (§4). */ - revokeWaiting(revoked: (principal: Principal) => boolean, error: ChordError): void { - for (const provider of this.#providers.values()) provider.revokeWaiting(revoked, error); + /** Ends the holds on this link that hold nothing and act for a principal `revoked` names (§4). */ + revokeUnbound(revoked: (principal: Principal) => boolean, error: ChordError): void { + for (const provider of this.#providers.values()) provider.revokeUnbound(revoked, error); } closed(): void { diff --git a/packages/chord/src/next/remote/provider.ts b/packages/chord/src/next/remote/provider.ts index 4325f7c12..ca49e18e8 100644 --- a/packages/chord/src/next/remote/provider.ts +++ b/packages/chord/src/next/remote/provider.ts @@ -90,7 +90,7 @@ export class ProviderSide { this.#abort(message.call); break; case "retry": - this.#entries.get(message.hold)?.retry(); + this.#entries.get(message.hold)?.retry(message); break; case "ack": this.#ack(); @@ -128,15 +128,15 @@ export class ProviderSide { } /** - * Ends with `error` every hold of this origin that acts for a principal `revoked` names and still waits: for `load` - * (served here), or for its route's link (relayed). + * Ends with `error` every hold of this origin that acts for a principal `revoked` names and holds nothing: served + * here but unbound (before the publication, refused), or relayed and waiting for its route's link. */ - revokeWaiting(revoked: (principal: Principal) => boolean, error: ChordError): void { + revokeUnbound(revoked: (principal: Principal) => boolean, error: ChordError): void { for (const hold of [...this.#entries.values()]) { // A hold arriving on a link acts for a principal: the link's, or one its parent named. - if (!hold.waiting || !revoked(hold.caller.principal!)) continue; - if (hold.kind === "served") hold.end(error); - else hold.revoked(error); + if (!revoked(hold.caller.principal!)) continue; + if (hold.kind === "served" && hold.unbound) hold.end(error); + else if (hold.kind === "relayed" && hold.waiting) hold.revoked(error); } } @@ -283,7 +283,7 @@ export class ProviderSide { /** * A hold message: a protocol error for a live id or compressed params (params are never compressed), otherwise - * admitted once `load` resolved (`Hold.start`). Its activation chain is kept only from a trusted link (Q5); its + * admitted (`Hold.start`, placed once the host published). Its activation chain is kept only from a trusted link (Q5); its * decoded limit is the link's, or the smaller one a parent stamped (Q2), and travels with it when it is relayed. */ #hold(message: Extract): void { @@ -310,7 +310,7 @@ export class ProviderSide { const limits = limit === link.limits.decoded ? link.limits : { part: link.limits.part, decoded: limit }; const hold = new Hold(id, this, this.#caller(message), limits); this.#entries.set(id, hold); - void hold.start(request); + hold.start(request); } /** diff --git a/packages/chord/src/next/remote/relay.ts b/packages/chord/src/next/remote/relay.ts index b8857992c..982207a42 100644 --- a/packages/chord/src/next/remote/relay.ts +++ b/packages/chord/src/next/remote/relay.ts @@ -109,10 +109,10 @@ export class RelayedHold { upstream?.link.send({ kind: "abort", origin: upstream.origin, call: id }); } - /** A downstream `retry`, forwarded. */ - retry(): void { + /** A downstream `retry`, forwarded with its operation and the objects it passed through. */ + retry(message: Extract): void { const upstream = this.#upstream; - upstream?.link.send({ kind: "retry", origin: upstream.origin, hold: this.id }); + upstream?.link.send({ ...message, origin: upstream.origin }); } /** Released downstream: released upstream (its calls with it) and no later link holds it. */ diff --git a/packages/chord/src/next/remote/remote-object.ts b/packages/chord/src/next/remote/remote-object.ts index 05ea9cea9..855f04867 100644 --- a/packages/chord/src/next/remote/remote-object.ts +++ b/packages/chord/src/next/remote/remote-object.ts @@ -24,7 +24,7 @@ import { LOADING, type PinnedKey, type RefState, - settlement, + type RetryOperation, type Target, } from "../host/internal.ts"; import { @@ -78,8 +78,6 @@ export class RemoteObject implements Target { /** The hold on the path's link while one is open: that link's consumer side and this hold's id on it. */ #hold: { readonly side: ConsumerSide; readonly id: number } | undefined; #binding: Binding | undefined; - /** Pending while loading, resolved while live, rejected while failed: a dependent's activation waits for it (R18). */ - #settled = settlement(); #lastFailure: ChordError | undefined; constructor( @@ -112,9 +110,14 @@ export class RemoteObject implements Target { return this.#state; } - /** Pending while loading; resolves when bound, rejects with a failure or the reference's end. */ - get settled(): Promise { - return this.#settled.promise; + /** Whether its hold is on a link now (a remote attempt answers for it), not while its path has no link. */ + get heldOnLink(): boolean { + return this.#hold !== undefined; + } + + /** An object over a link is no blocker of this host's first pass. */ + get blocksFirstPass(): false { + return false; } /** The error of the last `failed`, until `bound`. */ @@ -151,12 +154,15 @@ export class RemoteObject implements Target { attach(link: HostLink): void { const chain = this.#dependent?.activationChain(); this.#hold = { side: link.consumer, id: link.consumer.attach(this, chain) }; + // Held on a link now: a parked dependent checks again (`parked_long` counts a hold on a link only). + this.#dependent?.dependencyMoved(); } /** The link dropped: loading until a new link's hold is bound (`#startOver`). */ dropped(_error: ChordError): void { this.#hold = undefined; this.#startOver(); + this.#dependent?.dependencyMoved(); } /** @@ -214,17 +220,18 @@ export class RemoteObject implements Target { } /** - * `ref.retry()`: a failed object's provider is asked for an attempt (a hold it refused is started there again). One - * that failed for a part past the link's limits is placed anew and held with a new id, now or on its path's next - * link (a `hostAddress` that throws fails it broken again). + * `ref.retry()`: a failed object's provider is asked for an attempt (a hold it refused is started there again; a + * parked object there passes it on as an operation of its own host). One that failed for a part past the link's + * limits is placed anew and held with a new id, now or on its path's next link (a `hostAddress` that throws fails it + * broken again). */ - retry(): void { + retry(operation: RetryOperation): void { if (this.#state.kind !== "failed") return; if (this.#path === this.#host.unplaced) { this.#place(); return; } - this.#hold?.side.retry(this.#hold.id); + this.#hold?.side.retry(this.#hold.id, operation); } /** Places a reference whose part was past the limits anew, on its path now (a throwing `hostAddress` fails it). */ @@ -313,17 +320,16 @@ export class RemoteObject implements Target { }; } - /** After a `bound` message's batch: the reference goes live (its parked calls are sent) and R18 retries follow. */ + /** After a `bound` message's batch: the reference goes live (its parked calls are sent, a parked dependent goes on). */ wentLive(): void { this.#lastFailure = undefined; this.#move(LIVE); - // R18: a dependent that failed while waiting for this object is retried now. - this.#binding?.dependent?.dependencyLive(); } /** - * `failed`: the provider's object failed and no attempt runs; `retryIn` is the rest of its backoff window. Ignored - * once live (only a pending hold is told). + * `failed`: the provider's object failed and no attempt runs (`retryIn`: the rest of its backoff window), or the hold + * is unbound there or its object parked on a failed dependency (`waiting`), each new cause a new message. Ignored once + * live (only a pending hold is told). */ failed(message: Extract): void { if (this.#state.kind === "live") return; @@ -383,6 +389,17 @@ export class RemoteObject implements Target { }; } + /** + * Its principal was revoked on this host (`host.revoke`): released at its provider if it is held there, then ended + * with `error` as `lost` does; no later link holds it. + */ + revoke(error: ChordError): void { + const hold = this.#hold; + this.#hold = undefined; + hold?.side.release(hold.id, error); + this.lost(error); + } + /** `gone`, or the host closes: the reference ends, its calls reject with `error`. */ lost(error: ChordError): void { const binding = this.#binding; @@ -399,28 +416,23 @@ export class RemoteObject implements Target { this.#dependent = undefined; this.#hold = undefined; this.#bound = undefined; - this.#settled.reject(error); this.#path.detach(this); for (const replica of bound?.replicas.values() ?? []) replica.end(error); } /** * Loading again until the next binding; the members keep their last values. A drop while it is loading already - * changes nothing: a dependent's activation waiting for it goes on waiting for the next link's binding, bounded by - * its `readyTimeout` (§6, R18; a drop is not a dependency failure). + * changes nothing: a dependent parked on it goes on waiting for the next link's binding (a drop is not a dependency + * failure). */ #startOver(): void { const { kind } = this.#state; if (kind !== "ended" && kind !== "loading") this.#move(LOADING); } - /** Moves to `state` and tells the reference; `settled` follows (a new wait each time it is loading again). */ + /** Moves to `state` and tells the reference. */ #move(state: RefState): void { - const previous = this.#state; this.#state = state; - if (state.kind === "loading" && previous.kind !== "loading") this.#settled = settlement(); - if (state.kind === "live") this.#settled.resolve(); - if (state.kind === "failed") this.#settled.reject(state.cause.error); this.#binding?.moved(); } } diff --git a/packages/chord/test/next/data/failure.test.ts b/packages/chord/test/next/data/failure.test.ts index b833f4d8a..c7e88861e 100644 --- a/packages/chord/test/next/data/failure.test.ts +++ b/packages/chord/test/next/data/failure.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from "vitest"; -import { broken } from "../../../src/next/data/index.ts"; -import { invalid, openFailure } from "../../../src/next/data/internal.ts"; +import { broken, ChordError } from "../../../src/next/data/index.ts"; +import { invalid, openFailure, sameCause, waitingFor } from "../../../src/next/data/internal.ts"; describe("failure causes", () => { it("a value thrown by an open is transient unless marked broken", () => { @@ -25,3 +25,65 @@ describe("failure causes", () => { expect(openFailure(broken(chord), "open")).toEqual({ kind: "broken", error: chord }); }); }); + +describe("a dependent's waiting cause (dependency model §3)", () => { + const root = new ChordError({ code: "unavailable", message: "app.catalog is not served here" }); + + it("names the nearest blocker and the root cause, its blocker's error as the cause", () => { + const cause = waitingFor("app.chat(S1)", root); + expect(cause.kind).toBe("waiting"); + expect(cause.error.toJSON()).toEqual({ + code: "unavailable", + message: "waiting for app.chat(S1): app.catalog is not served here", + }); + expect(cause.error.cause).toBe(root); + // A chain names its nearest blocker and its root only (fix round 1, Opus F1). + const chain = waitingFor("app.panel", cause.error); + expect(chain.error.message).toBe("waiting for app.panel: app.catalog is not served here"); + }); + + it("reaches a fixed point when a ring of waiting attempts passes its causes round (Opus F1)", () => { + let b = waitingFor("X", new ChordError({ code: "unavailable", message: "X is not served here" })).error; + let rounds = 0; + for (; rounds < 10; rounds++) { + const next = waitingFor("A", waitingFor("B", b).error).error; + if (next.message === b.message) break; + b = next; + } + expect(rounds).toBeLessThanOrEqual(1); + expect(b.message).toBe("waiting for A: X is not served here"); + }); + + it("keeps the root cause within 500 characters: a long label is clipped first (Sol 8)", () => { + const original = new ChordError({ code: "unavailable", message: "ORIGINAL: missing provider" }); + const message = waitingFor(`sol.item(${"x".repeat(600)})`, original).error.message; + expect(message.length).toBeLessThanOrEqual(500); + expect(message).toContain(original.message); + expect(message.startsWith("waiting for sol.item(xxx")).toBe(true); + }); + + it("keeps a long chain's root cause, never repeated ellipses (Sol 8)", () => { + const original = new ChordError({ code: "unavailable", message: "ORIGINAL: missing provider" }); + let error = original; + for (let i = 0; i < 2000; i++) error = waitingFor(`level-${i}`, error).error; + expect(error.message).toBe("waiting for level-1999: ORIGINAL: missing provider"); + }); + + it("clips a long root past its share, and both when both are long", () => { + const long = new ChordError({ code: "unavailable", message: "y".repeat(600) }); + const short = waitingFor("app.a", long).error.message; + expect(short).toHaveLength(500); + expect(short.startsWith("waiting for app.a: yyy")).toBe(true); + const both = waitingFor("z".repeat(600), long).error.message; + expect(both).toHaveLength(500); + expect(both.endsWith("y…")).toBe(true); + }); + + it("is compared by value: kind, code, reason and message", () => { + expect(sameCause(waitingFor("a", root), waitingFor("a", root))).toBe(true); + expect(sameCause(waitingFor("a", root), waitingFor("b", root))).toBe(false); + expect(sameCause({ kind: "broken", error: root }, { kind: "transient", error: root })).toBe(false); + const other = new ChordError({ code: "unavailable", reason: "undeclared", message: root.message }); + expect(sameCause({ kind: "waiting", error: root }, { kind: "waiting", error: other })).toBe(false); + }); +}); diff --git a/packages/chord/test/next/examples/05-errors.ts b/packages/chord/test/next/examples/05-errors.ts index 7ea09797d..fabdaf3b9 100644 --- a/packages/chord/test/next/examples/05-errors.ts +++ b/packages/chord/test/next/examples/05-errors.ts @@ -78,9 +78,10 @@ await attempt("bad params", () => host.get(Echo, { id: "one" }, context)); // An implementation with a member its contract does not declare fails activation. await attempt("unknown member", () => host.get(Sloppy, { id: 1 }, context)); -// A get needs the provider's version and fingerprint; another declaration of the type does not fit. +// A get needs the provider's version and fingerprint; another declaration of the type does not fit. That is not the +// caller's fault (a reload may bring the version): the get waits up to its readyTimeout, then rejects with the cause. const EchoV2 = local("example.echo", { version: 2, params: { id: int() }, members: { id: int() } }); -await attempt("contract mismatch", () => host.get(EchoV2, { id: 1 }, context)); +await attempt("contract mismatch", () => host.get(EchoV2, { id: 1 }, context, { readyTimeout: 100 })); // A factory that throws fails broken: the get rejects at once with its error, and no object is left behind. await attempt("throwing factory", () => host.get(Broken, { id: 1 }, context)); diff --git a/packages/chord/test/next/examples/07-failures.ts b/packages/chord/test/next/examples/07-failures.ts index 2d8bb9697..31dc81a42 100644 --- a/packages/chord/test/next/examples/07-failures.ts +++ b/packages/chord/test/next/examples/07-failures.ts @@ -36,7 +36,8 @@ const facet = defineFacet((host) => { return { status: constant("connected") }; }); host.serve(Inbox, ({ user }, self) => { - // A dependency that fails fails this object transient too; it is retried the moment the dependency is live. + // A dependency that fails parks this object (`failed { waiting }`, no timer of its own, the factory not run + // again); it goes on the moment the dependency is live (31-parked-dependencies.ts). const remote = self.get(Remote, {}); console.log(`${at()}: inbox ${user}, attempt`); return { status: self.derive([remote.status], ([status]) => `${user}: ${status}`) }; diff --git a/packages/chord/test/next/examples/31-parked-dependencies.ts b/packages/chord/test/next/examples/31-parked-dependencies.ts new file mode 100644 index 000000000..938bd0b63 --- /dev/null +++ b/packages/chord/test/next/examples/31-parked-dependencies.ts @@ -0,0 +1,148 @@ +// Parked dependencies (dependency model, slice 7a): a reference is a request that outlives every provider it meets. An +// object whose dependency is not there yet is parked, not failed: it shows `failed { waiting }` naming what it waits +// for (the nearest blocker and the root cause), no timer runs for it, its factory is not run again, and it goes on the +// moment its dependency is live. Here a tab's panel needs `Notes` from a server that does not serve it yet; the server +// restarts with the notes plugin, the tab reconnects, and the panel goes live without anyone asking. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/31-parked-dependencies.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { arr, hostKind, int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { isChordError } from "../../../src/next/data/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet, type RefState } from "../../../src/next/host/index.ts"; +import { memoryLinkPair } from "../../../src/next/link/index.ts"; +import type { NativeWatchable } from "../../../src/watch.ts"; + +const context = BACKGROUND_CONTEXT; +const started = Date.now(); +const at = (): string => `${((Date.now() - started) / 1000).toFixed(1)} s`; + +type Principal = { readonly kind: "server" } | { readonly kind: "owner" }; + +const Server = hostKind("server"); + +/** The server's notes (the notes plugin), placed on the server. */ +const Notes = remote("example.notes", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { editor: { state: { list: arr(str()) }, methods: { add: method(obj({ text: str() })) } } }, +}); + +/** A panel in the tab: a local object over the server's notes. */ +const Panel = local("example.panel", { version: 1, params: { id: int() }, members: { count: int() } }); + +/** A local object the panel's sibling needs, which no plugin of the tab serves. */ +const Theme = local("example.theme", { version: 1, params: {}, members: { name: str() } }); +const Header = local("example.header", { version: 1, params: { id: int() }, members: { title: str() } }); + +const List = defineState("example.list", { + schema: obj({ list: arr(str()) }), + version: 1, + initial: () => ({ list: [] }), +}); + +/** The notes plugin's server facet. */ +const notesPlugin = defineFacet((host) => { + host.serve(Notes, (_params, _principal, self) => { + const list = self.state(List); + return { + variant: "editor", + members: { + list: self.derive([list], ([l]) => l.list), + add: async ({ text }, ctx) => { + await list.change((draft) => void draft.list.push(text), ctx); + return null; + }, + }, + }; + }); +}); + +/** A server with the given plugins, its link to the tab accepted. */ +async function startServer(plugins: { readonly name: string; readonly facet: typeof notesPlugin }[]) { + const server = createHost({ address: Server(), principal: { kind: "server" }, onError: () => {} }); + await server.load(plugins, context); + const [up, down] = memoryLinkPair(); + server.accept(up, { principal: { kind: "owner" } }, context); + return { server, tabLink: down, close: () => up.close() }; +} + +// The tab: its panel declares the notes (a dependency); its header a theme nobody serves. +let panels = 0; +const tab = createHost({ onError: (error) => console.log(`${at()}: reported ${error.kind}`) }); +await tab.load( + [ + { + name: "@example/panel", + facet: defineFacet((host) => { + host.serve(Panel, (_, self) => { + panels++; + const notes = self.get(Notes, {}); + return { count: self.derive([notes.list], ([list]) => list.length) }; + }); + host.serve(Header, (_, self) => ({ title: self.get(Theme, {}).name })); + }), + }, + ], + context, +); + +/** Prints every state of `state` as it changes. */ +async function follow(label: string, state: NativeWatchable): Promise { + const watch = await state.watch(context); + watch.start(async (call) => { + if (call.kind === "value") console.log(`${at()}: ${label} ${describe(call.value)}`); + }); +} + +function describe(state: RefState): string { + if (state.kind !== "failed") return state.kind; + return `failed { ${state.cause.kind} }: ${state.cause.error.message}`; +} + +// 1. The first server has no notes plugin. Its answer is `failed { waiting }`: not the panel's fault. +const first = await startServer([]); +tab.connect(first.tabLink, { principal: { kind: "server" } }, context); +const panel = tab.hold(Panel, { id: 1 }, context); +await follow("panel:", panel.state); +await new Promise((resolve) => setTimeout(resolve, 10)); +console.log(`${at()}: panel now ${describe(panel.state.value)}`); + +// 2. A local type nobody serves here is the same answer: the request waits unbound, indexed under the type's name. +const header = tab.hold(Header, { id: 1 }, context); +await new Promise((resolve) => setTimeout(resolve, 10)); +console.log(`${at()}: header ${describe(header.state.value)}`); + +// 3. A getter waits through `waiting` up to its readyTimeout, then rejects with the cause (a hold just shows it). +try { + await tab.get(Panel, { id: 1 }, context, { readyTimeout: 200 }); +} catch (error) { + if (isChordError(error)) console.log(`${at()}: get gave up after 200 ms: ${error.message}`); +} + +// 4. Time passes: no timer runs for the parked panel, its factory is not run again. +await new Promise((resolve) => setTimeout(resolve, 1500)); +console.log(`${at()}: still ${panel.state.value.kind}, factory runs: ${panels}`); + +// 5. The server restarts with the notes plugin (the link drop shows as loading); the tab reconnects: its request is +// held again and binds, and the parked panel goes on. +first.close(); +await new Promise((resolve) => setTimeout(resolve, 10)); +const second = await startServer([{ name: "@example/notes", facet: notesPlugin }]); +tab.connect(second.tabLink, { principal: { kind: "server" } }, context); +const live = await tab.get(Panel, { id: 1 }, context); +console.log(`${at()}: panel live, count ${live.count.value}, factory runs: ${panels}`); +// The panel's notes are the owner's object on the server (a remote object per principal): the tab adds through its own. +const notes = await tab.get(Notes, {}, context); +await notes.add({ text: "first" }, context); +await new Promise((resolve) => setTimeout(resolve, 10)); +console.log(`${at()}: panel count ${live.count.value}`); + +header.release(); +panel.release(); +live.release(); +notes.release(); +await tab.close(context); +await second.server.close(context); +await first.server.close(context); diff --git a/packages/chord/test/next/examples/32-startup.ts b/packages/chord/test/next/examples/32-startup.ts new file mode 100644 index 000000000..2c2169e10 --- /dev/null +++ b/packages/chord/test/next/examples/32-startup.ts @@ -0,0 +1,104 @@ +// Startup (dependency model §6, slice 7a): `load` publishes every facet's declarations in one batch first, then starts +// the host-lifetime objects; requests made here and holds arriving on links wait only for that publication, never for +// the start barrier. Two hosts whose objects need each other's types (an acyclic graph) used to deadlock: the server's +// eager `Bootstrap` needs the worker's `Agent`, the `Agent` needs the server's `Catalog`, and the worker's hold for +// `Catalog` waited for the server's whole `load`, which waited for `Bootstrap`. Now it starts at once, no timeout. +// `load` resolves when each host-lifetime object had its first outcome: live, failed by its own cause, or parked with +// no blocker here still without one (a wait on another host ends the first pass). +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/32-startup.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { hostKind, int, local, remote, str } from "../../../src/next/contracts/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { memoryLinkPair } from "../../../src/next/link/index.ts"; +import { constant } from "../../../src/next/state/index.ts"; + +const context = BACKGROUND_CONTEXT; +const started = Date.now(); +const at = (): string => `${Date.now() - started} ms`; + +type Principal = { readonly kind: "server" } | { readonly kind: "worker"; readonly session: string }; + +const Server = hostKind("server"); +const Session = hostKind("session", { id: str() }); + +/** On the server, a host-lifetime object (local, empty params): it starts at `load`. */ +const Bootstrap = local("example.bootstrap", { version: 1, params: {}, members: { agents: int() } }); + +/** On the session's worker. */ +const Agent = remote("example.agent", { + version: 1, + params: { session: str() }, + hostAddress: ({ session }) => Session({ id: session }), + variants: { v: { state: { models: int() } } }, +}); + +/** On the server. */ +const Catalog = remote("example.catalog", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: { models: int() } } }, +}); + +/** The worker of a session, started by the server's route; it connects first, then loads (W6). */ +function startWorker(id: string) { + const [up, down] = memoryLinkPair(); + const worker = createHost({ + address: Session({ id }), + principal: { kind: "worker", session: id }, + onError: (error) => console.log(`${at()}: worker reported ${error.kind}`), + }); + worker.connect(down, { principal: { kind: "server" }, trusted: true }, context); + const agents = defineFacet((host) => + host.serve(Agent, (_params, _principal, self) => { + console.log(`${at()}: worker: Agent declares the server's Catalog`); + return { variant: "v", members: { models: self.get(Catalog, {}).models } }; + }), + ); + void worker + .load([{ name: "@example/agents", facet: agents }], context) + .then(() => console.log(`${at()}: worker loaded`)); + return { worker, link: up }; +} + +const workers: ReturnType[] = []; +const server = createHost({ + address: Server(), + principal: { kind: "server" }, + // No readyTimeout is needed to break anything: a long one shows nothing waits for it. + readyTimeout: 600_000, + onError: (error) => console.log(`${at()}: server reported ${error.kind}`), +}); +server.route( + Session, + { + authorize: () => true, + resolve: async (address) => { + const started = startWorker(address.params.id); + workers.push(started); + return { link: started.link, principal: { kind: "worker", session: address.params.id }, trusted: true }; + }, + }, + context, +); +const core = defineFacet((host) => { + host.serve(Catalog, () => ({ variant: "v", members: { models: constant(3) } })); + host.serve(Bootstrap, (_, self) => { + console.log(`${at()}: server: Bootstrap declares the worker's Agent`); + return { agents: self.get(Agent, { session: "S1" }).models }; + }); +}); + +// 1. The server's load: publication (Catalog served), then Bootstrap starts and parks on the worker's Agent. A wait on +// another host is a first outcome, so `load` resolves without waiting for the worker. +await server.load([{ name: "@example/core", facet: core }], context); +console.log(`${at()}: server loaded; Bootstrap is ${server.hold(Bootstrap, {}, context).state.value.kind}`); + +// 2. The worker's hold for Catalog bound on the published server; Agent went live, then Bootstrap. +const bootstrap = await server.get(Bootstrap, {}, context); +console.log(`${at()}: Bootstrap live, it sees ${bootstrap.agents.value} models through the worker`); + +bootstrap.release(); +await server.close(context); +for (const { worker } of workers) await worker.close(context); diff --git a/packages/chord/test/next/host/activity.test.ts b/packages/chord/test/next/host/activity.test.ts index cdfa929b7..63ac47af6 100644 --- a/packages/chord/test/next/host/activity.test.ts +++ b/packages/chord/test/next/host/activity.test.ts @@ -27,6 +27,19 @@ describe("host.activity", () => { expect(server.activity.value.references).toBe(0); }); + it("counts a hold placed here from its arrival, bound or not: a missing provider keeps the worker alive (M§10)", async () => { + const { server, client } = await linked(() => {}); + const missing = client.hold(Notes, { session: "S1" }, context); + await transported(); + expect(missing.state.value).toMatchObject({ kind: "failed", cause: { kind: "waiting" } }); + expect(server.activity.value.references).toBe(1); + // Presence lists bound references only. + expect(server.references(Notes).value).toEqual([]); + missing.release(); + await transported(); + expect(server.activity.value.references).toBe(0); + }); + it("counts method calls running on this host, local and remote, until they settle", async () => { const { server, client } = await linked((h) => void serveNotes(h)); const remoteRef = await client.get(Notes, { session: "S1" }, context); diff --git a/packages/chord/test/next/host/attempt.test.ts b/packages/chord/test/next/host/attempt.test.ts index d1227bc4a..088ee54ec 100644 --- a/packages/chord/test/next/host/attempt.test.ts +++ b/packages/chord/test/next/host/attempt.test.ts @@ -2,7 +2,7 @@ import { createHook } from "node:async_hooks"; import { afterEach, beforeEach, describe, expect, expectTypeOf, it, vi } from "vitest"; import { type Context, BACKGROUND_CONTEXT as context, withCancel } from "../../../src/context/index.ts"; import { arr, int, local, method, obj, opt, remote, str } from "../../../src/next/contracts/index.ts"; -import { broken, isChordError } from "../../../src/next/data/index.ts"; +import { broken, ChordError, isChordError } from "../../../src/next/data/index.ts"; import { defineState, type SingletonHandle } from "../../../src/next/documents/index.ts"; import { createHost, @@ -12,6 +12,7 @@ import { type Self, type StoreSelf, } from "../../../src/next/host/index.ts"; +import { ChordObject } from "../../../src/next/host/object.ts"; import { defineRegistry } from "../../../src/next/registries/index.ts"; import { constant } from "../../../src/next/state/index.ts"; import type { StagedWrite } from "../../../src/next/store/step.ts"; @@ -22,9 +23,11 @@ import { evolveTable } from "../../../src/next/store/table-evolve.ts"; import { TableHandle } from "../../../src/next/store/table-handle.ts"; import { prepareTables } from "../../../src/next/store/table-migration.ts"; import { + failLive, failures, fakeTimers, flushed, + instancesOf, loaded, memoryStore, metered, @@ -33,6 +36,8 @@ import { tableOwner, } from "../support.ts"; +const objects = instancesOf(ChordObject, "attempt"); + const List = defineState("list", { schema: obj({ items: arr(str()) }), version: 1, initial: () => ({ items: [] }) }); const Page = local("app.page", { version: 1, params: { id: int() }, members: { label: str(), slow: method() } }); @@ -892,3 +897,93 @@ describe("eager migrations at activation (slice 6a §9)", () => { await store.close(context); }); }); + +describe("parked attempts (dependency model §3)", () => { + fakeTimers(); + const View = local("app.view", { version: 1, params: {}, members: { n: int() } }); + const Missing = local("app.missing", { version: 1, params: {}, members: { n: int() } }); + /** A view with params: no host lifetime, so `load` does not wait for it. */ + const Viewer = local("app.viewer", { version: 1, params: { id: int() }, members: { n: int() } }); + + it("run no timer while parked: no dependency deadline, no backoff, the factory run once", async () => { + const factories = vi.fn(); + const { host, errors } = await loaded((h) => + h.serve(View, (_, self) => { + factories(); + return { n: self.get(Missing, {}).n }; + }), + ); + const view = host.hold(View, {}, context); + await flushed(); + await vi.advanceTimersByTimeAsync(10 * 60_000); + expect(view.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "waiting", error: { message: "waiting for app.missing: app.missing is not served here" } }, + retryAt: undefined, + }); + expect(vi.getTimerCount()).toBe(0); + expect([factories.mock.calls.length, errors]).toEqual([1, []]); + view.release(); + }); + + it("pause the readyTimeout deadline while parked: the own steps get the budget left", async () => { + const Slow = local("app.slow", { version: 1, params: { id: int() }, members: { n: int() } }); + const { host, errors } = await loaded( + (h) => { + h.serve(Slow, (_, self) => { + self.open( + () => new Promise((resolve) => setTimeout(resolve, 5_000)), + () => {}, + ); + return { n: constant(1) }; + }); + h.serve(Viewer, (_, self) => { + const slow = self.get(Slow, { id: 1 }); + self.open( + () => new Promise(() => {}), + () => {}, + ); + return { n: slow.n }; + }); + }, + { readyTimeout: 10_000 }, + ); + const view = host.hold(Viewer, { id: 1 }, context); + // Parked 5 s on its dependency; then its own open hangs: it fails 10 s after the boundary, not 10 s after start. + await vi.advanceTimersByTimeAsync(5_000 + 9_999); + expect(failures(errors)).toEqual([]); + await vi.advanceTimersByTimeAsync(1); + expect(failures(errors)).toEqual(["app.viewer(1): transient"]); + view.release(); + }); + + it("go on past the dependency check: a dependency leaving live during the own steps changes nothing (R18)", async () => { + const Dependency = local("app.dependency", { version: 1, params: {}, members: { n: int() } }); + const gate = Promise.withResolvers(); + let dependencySelf: Self | undefined; + const { host, errors } = await loaded((h) => { + h.serve(Dependency, (_, self) => { + dependencySelf = self; + return { n: constant(1) }; + }); + h.serve(Viewer, (_, self) => { + const dependency = self.get(Dependency, {}); + self.open( + () => gate.promise, + () => {}, + ); + return { n: dependency.n }; + }); + }); + const view = host.hold(Viewer, { id: 1 }, context); + await flushed(); + // The view opens its own native; its dependency fails live meanwhile. + failLive(objects, dependencySelf!, new ChordError({ code: "unavailable", message: "offline" })); + await flushed(); + gate.resolve(); + await flushed(); + expect(view.state.value).toEqual({ kind: "live" }); + expect(failures(errors)).toEqual([]); + view.release(); + }); +}); diff --git a/packages/chord/test/next/host/dependencies.test.ts b/packages/chord/test/next/host/dependencies.test.ts new file mode 100644 index 000000000..ca413718e --- /dev/null +++ b/packages/chord/test/next/host/dependencies.test.ts @@ -0,0 +1,294 @@ +import { describe, expect, it, vi } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { int, local, remote } from "../../../src/next/contracts/index.ts"; +import { createHost, defineFacet, type HostError } from "../../../src/next/host/index.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import { + bindHold, + failures, + fakeTimers, + flushed, + loaded, + Notes, + rawParent, + rejection, + relayed, + Server, + Session, + statesOf, + type TestPrincipal, + transported, +} from "../support.ts"; + +const View = local("app.view", { version: 1, params: { id: int() }, members: { n: int() } }); + +describe("the dependency check", () => { + it("breaks a dependent off a cycle with the cycle's error: it would wait on a ring that never heals", async () => { + const Ping = local("app.ping", { version: 1, params: {}, members: { n: int() } }); + const Pong = local("app.pong", { version: 1, params: {}, members: { n: int() } }); + const { host, errors } = await loaded((h) => { + h.serve(Ping, (_, self) => ({ n: self.get(Pong, {}).n })); + h.serve(Pong, (_, self) => ({ n: self.get(Ping, {}).n })); + h.serve(View, (_, self) => ({ n: self.get(Ping, {}).n })); + }); + const view = host.hold(View, { id: 1 }, context); + await flushed(); + expect(view.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "broken", error: { code: "invalid", reason: "cycle" } }, + }); + expect(failures(errors)).toContain("app.view(1): broken"); + view.release(); + }); + + it("breaks a dependent whose dependency ended (gone): the request cannot come back", async () => { + const errors: HostError[] = []; + const client = createHost({ onError: (error) => errors.push(error) }); + const peer = await rawParent(client); + await client.load( + [ + { + name: "@test/view", + facet: defineFacet((h) => + h.serve(View, (_, self) => ({ n: self.get(Notes, { session: "S1" }).count })), + ), + }, + ], + context, + ); + const view = client.hold(View, { id: 1 }, context); + const { hold } = await peer.next("hold"); + const revoked = { code: "forbidden", reason: "revoked", message: "revoked" } as const; + peer.send({ kind: "gone", hold, error: { kind: "chord", error: revoked } }); + await transported(); + expect(view.state.value).toMatchObject({ kind: "failed", cause: { kind: "broken", error: revoked } }); + expect(failures(errors)).toEqual(["app.view(1): broken"]); + view.release(); + }); + + it("goes live once every dependency is: one parked on two, the second binding last", async () => { + const client = createHost({ onError: () => {} }); + const peer = await rawParent(client); + const facet = defineFacet((h) => + h.serve(View, (_, self) => { + const a = self.get(Notes, { session: "S1" }).count; + const b = self.get(Notes, { session: "S2" }).count; + return { n: self.derive([a, b], ([x, y]) => x + y) }; + }), + ); + await client.load([{ name: "@test/view", facet }], context); + const view = client.hold(View, { id: 1 }, context); + const first = await peer.next("hold"); + const second = await peer.next("hold"); + bindHold(peer, first.hold, [{ text: "a" }]); + await transported(); + expect(view.state.value).toEqual({ kind: "loading" }); + bindHold(peer, second.hold, [{ text: "b" }, { text: "c" }]); + await transported(); + expect([view.state.value, view.n.value]).toEqual([{ kind: "live" }, 3]); + view.release(); + }); + + it("breaks a dependent whose dependency a relay's authorize refused: forbidden heals only by retry() (fix round 1, Fable)", async () => { + const errors: HostError[] = []; + const client = createHost({ onError: (error) => errors.push(error) }); + const peer = await rawParent(client); + const facet = defineFacet((h) => + h.serve(View, (_, self) => ({ n: self.get(Notes, { session: "S1" }).count })), + ); + await client.load([{ name: "@test/view", facet }], context); + const view = client.hold(View, { id: 1 }, context); + const { hold } = await peer.next("hold"); + const forbidden = { code: "forbidden", message: "session|S1: not authorized" } as const; + peer.send({ kind: "failed", hold, cause: { kind: "broken", error: { kind: "chord", error: forbidden } } }); + await transported(); + expect(view.state.value).toMatchObject({ kind: "failed", cause: { kind: "broken", error: forbidden } }); + expect(failures(errors)).toEqual(["app.view(1): broken"]); + view.release(); + }); + + it("settles a waiting ring across hosts after one round: the root cause kept (fix round 1, Opus F1)", async () => { + const A = remote("app.a", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: { n: int() } } }, + }); + const B = remote("app.b", { + version: 1, + params: {}, + hostAddress: () => Session({ id: "S1" }), + variants: { v: { state: { n: int() } } }, + }); + const X = local("app.x", { version: 1, params: {}, members: { n: int() } }); + const Kick = local("app.kick", { version: 1, params: {}, members: { n: int() } }); + const Front = local("app.front", { version: 1, params: {}, members: { n: int() } }); + const Back = local("app.back", { version: 1, params: {}, members: { n: int() } }); + const { workers } = await relayed({ + trusted: true, + server: (h) => { + // The server's host object starts the worker; its A and the worker's B then start in the same turns, so + // the activation chains miss the ring (M§7's gap). + h.serve(Kick, (_p, self) => ({ n: self.get(B, {}).n })); + h.serve(A, (_p, _principal, self) => ({ variant: "v", members: { n: self.get(B, {}).n } })); + }, + worker: (h) => { + h.serve(Front, (_p, self) => ({ n: self.get(A, {}).n })); + h.serve(Back, (_p, self) => ({ n: self.get(B, {}).n })); + h.serve(B, (_p, _principal, self) => { + const a = self.get(A, {}); + self.get(X, {}); + return { variant: "v", members: { n: a.n } }; + }); + }, + }); + await transported(); + const worker = workers.get("session|S1")!; + const b = worker.hold(B, {}, context); + for (let round = 0; round < 50; round++) await transported(); + const states = statesOf(await b.state.watch(context)); + for (let round = 0; round < 50; round++) await transported(); + expect(states).toEqual([]); + expect(b.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "waiting", error: { message: expect.stringMatching(/: app\.x is not served here$/) } }, + }); + b.release(); + }); +}); + +describe("parked_long (dependency model §7)", () => { + fakeTimers(); + + it("starts its clock when the hold gets on a link: a late link is not reported at once (fix round 2, Opus F2)", async () => { + const errors: HostError[] = []; + const client = createHost({ readyTimeout: 1_000, onError: (error) => errors.push(error) }); + const facet = defineFacet((h) => + h.serve(View, (_, self) => ({ n: self.get(Notes, { session: "S1" }).count })), + ); + await client.load([{ name: "@test/view", facet }], context); + const view = client.hold(View, { id: 1 }, context); + await vi.advanceTimersByTimeAsync(5_000); + const peer = await rawParent(client); + await peer.next("hold"); + const linked = Date.now(); + await vi.advanceTimersByTimeAsync(10); + expect(errors).toEqual([]); + await vi.advanceTimersByTimeAsync(990); + expect(errors).toEqual([ + { kind: "parked_long", object: "app.view(1)", waitingFor: "app.notes(S1)", since: linked }, + ]); + view.release(); + }); + + it("is not reported while the path has no link: no remote attempt answers for it (fix round 1, Opus F2)", async () => { + const errors: HostError[] = []; + const client = createHost({ readyTimeout: 1_000, onError: (error) => errors.push(error) }); + const facet = defineFacet((h) => + h.serve(View, (_, self) => ({ n: self.get(Notes, { session: "S1" }).count })), + ); + await client.load([{ name: "@test/view", facet }], context); + const view = client.hold(View, { id: 1 }, context); + await vi.advanceTimersByTimeAsync(5_000); + expect(errors).toEqual([]); + expect(view.state.value).toEqual({ kind: "loading" }); + // Once a link holds it and the remote attempt is only loading, it is. + const peer = await rawParent(client); + await peer.next("hold"); + await vi.advanceTimersByTimeAsync(5_000); + expect(errors.map(({ kind }) => kind)).toEqual(["parked_long"]); + view.release(); + }); + + it("is reported once per parking episode, after readyTimeout on a remote attempt only loading; nothing fails", async () => { + const errors: HostError[] = []; + const client = createHost({ readyTimeout: 1_000, onError: (error) => errors.push(error) }); + const peer = await rawParent(client); + const facet = defineFacet((h) => + h.serve(View, (_, self) => ({ n: self.get(Notes, { session: "S1" }).count })), + ); + await client.load([{ name: "@test/view", facet }], context); + const view = client.hold(View, { id: 1 }, context); + await peer.next("hold"); + await vi.advanceTimersByTimeAsync(999); + expect(errors).toEqual([]); + await vi.advanceTimersByTimeAsync(1); + expect(errors).toEqual([{ kind: "parked_long", object: "app.view(1)", waitingFor: "app.notes(S1)", since: 0 }]); + await vi.advanceTimersByTimeAsync(60_000); + expect(errors).toHaveLength(1); + expect(view.state.value).toEqual({ kind: "loading" }); + view.release(); + }); + + it("is not reported for a remote blocker that failed: the parked attempt shows its cause instead", async () => { + const errors: HostError[] = []; + const client = createHost({ readyTimeout: 1_000, onError: (error) => errors.push(error) }); + const peer = await rawParent(client); + const facet = defineFacet((h) => + h.serve(View, (_, self) => ({ n: self.get(Notes, { session: "S1" }).count })), + ); + await client.load([{ name: "@test/view", facet }], context); + const view = client.hold(View, { id: 1 }, context); + const { hold } = await peer.next("hold"); + const error = { kind: "chord", error: { code: "unavailable", message: "app.notes is not served here" } } as const; + peer.send({ kind: "failed", hold, cause: { kind: "waiting", error } }); + await vi.advanceTimersByTimeAsync(60_000); + expect(errors).toEqual([]); + expect(view.state.value).toMatchObject({ kind: "failed", cause: { kind: "waiting" } }); + expect(vi.getTimerCount()).toBe(0); + view.release(); + }); + + it("is not reported for a local blocker: its own deadline bounds it", async () => { + const Slow = local("app.slow", { version: 1, params: { id: int() }, members: { n: int() } }); + const { host, errors } = await loaded( + (h) => { + h.serve(Slow, (_, self) => { + self.open( + () => new Promise(() => {}), + () => {}, + ); + return { n: constant(1) }; + }); + h.serve(View, (_, self) => ({ n: self.get(Slow, { id: 1 }).n })); + }, + { readyTimeout: 1_000 }, + ); + const view = host.hold(View, { id: 1 }, context); + await vi.advanceTimersByTimeAsync(5_000); + expect(errors.map(({ kind }) => kind)).not.toContain("parked_long"); + view.release(); + }); +}); + +describe("a getter of a parked object (fix round 1, Astra 6)", () => { + it("rejects with the last parked cause after its dependency fell back to loading, the newest of several", async () => { + const host = createHost({ onError: () => {} }); + const peer = await rawParent(host); + await host.load( + [ + { + name: "@test/view", + facet: defineFacet((h) => + h.serve(View, (_, self) => ({ n: self.get(Notes, { session: "S1" }).count })), + ), + }, + ], + context, + ); + const pending = rejection(host.get(View, { id: 1 }, context, { readyTimeout: 80 })); + const { hold } = await peer.next("hold"); + for (const message of ["first absence", "specific provider absence"]) { + const error = { kind: "chord", error: { code: "unavailable", message } } as const; + peer.send({ kind: "failed", hold, cause: { kind: "waiting", error } }); + await transported(); + } + peer.send({ kind: "loading", hold }); + await transported(); + expect(await pending).toMatchObject({ + code: "unavailable", + message: "waiting for app.notes(S1): specific provider absence", + }); + await host.close(context); + }); +}); diff --git a/packages/chord/test/next/host/host.test.ts b/packages/chord/test/next/host/host.test.ts index 0074d240e..8c9e0528f 100644 --- a/packages/chord/test/next/host/host.test.ts +++ b/packages/chord/test/next/host/host.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it, vi } from "vitest"; import { BACKGROUND_CONTEXT as context, withCancel } from "../../../src/context/index.ts"; -import { int, local, method, remote, str } from "../../../src/next/contracts/index.ts"; +import { hostKind, int, local, method, remote, str } from "../../../src/next/contracts/index.ts"; import { broken } from "../../../src/next/data/index.ts"; import { type Ctx, @@ -13,6 +13,7 @@ import { import type { ChordHost } from "../../../src/next/host/internal.ts"; import { memoryLinkPair } from "../../../src/next/link/index.ts"; import { jsonPart } from "../../../src/next/link/internal.ts"; +import { defineRegistry } from "../../../src/next/registries/index.ts"; import { constant } from "../../../src/next/state/index.ts"; import { sqliteBackend } from "../../../src/next/store/backend.ts"; import type { StagedWrite } from "../../../src/next/store/step.ts"; @@ -31,6 +32,7 @@ import { partValue, RawPeer, rejection, + relayed, SERVER_ADDRESS, Server, Session, @@ -153,19 +155,296 @@ describe("the start barrier", () => { await vi.advanceTimersByTimeAsync(1); expect(done).toBe(true); }); + + it("waits for a local blocker's first outcome: a host object parked on another one here (CodeMode → Harness)", async () => { + const Harness = local("app.harness", { version: 1, params: {}, members: { n: int() } }); + const CodeMode = local("app.codemode", { version: 1, params: {}, members: { n: int() } }); + const opened = Promise.withResolvers(); + const host = createHost(noErrors); + const facet = defineFacet((h) => { + h.serve(Harness, (_, self) => { + self.open( + () => opened.promise, + () => {}, + ); + return { n: constant(1) }; + }); + h.serve(CodeMode, (_, self) => ({ n: self.get(Harness, {}).n })); + }); + let done = false; + const loading = host.load([{ name: "@app/core", facet }], context).then(() => { + done = true; + }); + await flushed(); + expect(done).toBe(false); + opened.resolve(); + await loading; + expect(host.hold(CodeMode, {}, context).state.value).toEqual({ kind: "live" }); + }); + + it("ends its pass at a local blocker's own failure: the dependent parked failed { waiting } has its first outcome", async () => { + const Harness = local("app.harness", { version: 1, params: {}, members: { n: int() } }); + const CodeMode = local("app.codemode", { version: 1, params: {}, members: { n: int() } }); + const { host, errors } = await loaded((h) => { + h.serve(Harness, (_, self) => { + self.open( + async () => { + throw new Error("offline"); + }, + () => {}, + ); + return { n: constant(1) }; + }); + h.serve(CodeMode, (_, self) => ({ n: self.get(Harness, {}).n })); + }); + expect(failures(errors)).toEqual(["app.harness: transient"]); + expect(host.hold(CodeMode, {}, context).state.value).toMatchObject({ + kind: "failed", + cause: { kind: "waiting", error: { message: "waiting for app.harness: app.harness: offline" } }, + }); + }); + + it("ends its pass at a remote blocker: a wait on another host is a first outcome", async () => { + const CodeMode = local("app.codemode", { version: 1, params: {}, members: { n: int() } }); + const host = createHost(noErrors); + const [a, b] = memoryLinkPair(); + host.connect(a, { principal: { kind: "server" }, codec: testCodec() }, context); + const parent = new RawPeer(b); + const facet = defineFacet((h) => + h.serve(CodeMode, (_, self) => ({ n: self.get(Notes, { session: "S1" }).count })), + ); + await host.load([{ name: "@app/codemode", facet }], context); + expect(await parent.next("hold")).toMatchObject({ type: Notes.name }); + expect(host.hold(CodeMode, {}, context).state.value).toEqual({ kind: "loading" }); + }); + + it("is settled by host.close: an object of its cohort that ends before its first outcome settles it as ended", async () => { + const host = createHost(noErrors); + const facet = defineFacet((h) => + h.serve(Info, (_, self) => { + self.open( + () => new Promise(() => {}), + () => {}, + ); + return { title: constant("t") }; + }), + ); + let done = false; + const loading = host.load([{ name: "a", facet }], context).then(() => { + done = true; + }); + await flushed(); + expect(done).toBe(false); + await host.close(context); + await loading; + expect(done).toBe(true); + }); + + it("waits for a local blocker parked failed { waiting } on one still in its first attempt (fix round 1, Sol 4)", async () => { + const Core = local("app.core", { version: 1, params: {}, members: { n: int() } }); + const Middle = local("app.middle", { version: 1, params: { id: int() }, members: { n: int() } }); + const Slow = local("app.slow", { version: 1, params: { id: int() }, members: { n: int() } }); + const Missing = local("app.missing", { version: 1, params: {}, members: { n: int() } }); + const gate = Promise.withResolvers(); + const host = createHost(noErrors); + const facet = defineFacet((h) => { + h.serve(Slow, (_, self) => { + self.open( + () => gate.promise, + () => {}, + ); + return { n: constant(1) }; + }); + h.serve(Middle, ({ id }, self) => { + const slow = self.get(Slow, { id }); + self.get(Missing, {}); + return { n: slow.n }; + }); + h.serve(Core, (_, self) => ({ n: self.get(Middle, { id: 1 }).n })); + }); + let done = false; + const loading = host.load([{ name: "@app/core", facet }], context).then(() => { + done = true; + }); + await flushed(); + expect(host.hold(Middle, { id: 1 }, context).state.value).toMatchObject({ + kind: "failed", + cause: { kind: "waiting" }, + }); + expect(done).toBe(false); + gate.resolve(); + await loading; + await host.close(context); + }); + + it("waits for it whichever it declares first (fix round 1, Astra 3)", async () => { + const A = local("app.a", { version: 1, params: {}, members: {} }); + const B = local("app.b", { version: 1, params: { id: int() }, members: {} }); + const Slow = local("app.slow", { version: 1, params: { id: int() }, members: {} }); + const Missing = local("app.missing", { version: 1, params: {}, members: {} }); + const gate = Promise.withResolvers(); + const host = createHost(noErrors); + const facet = defineFacet((h) => { + h.serve(A, (_, self) => { + self.get(B, { id: 1 }); + return {}; + }); + h.serve(B, (_, self) => { + self.get(Missing, {}); + self.get(Slow, { id: 1 }); + return {}; + }); + h.serve(Slow, (_, self) => { + self.open( + () => gate.promise, + () => {}, + ); + return {}; + }); + }); + let done = false; + const loading = host.load([{ name: "@app/a", facet }], context).then(() => { + done = true; + }); + await flushed(); + const premature = done; + gate.resolve(); + await loading; + await host.close(context); + expect(premature).toBe(false); + }); +}); + +describe("startup across hosts (dependency model §6)", () => { + // Timers never advance: nothing here may wait for a deadline. + fakeTimers(); + + /** On the server, waiting for a worker's `Agent`. */ + const Bootstrap = local("app.bootstrap", { version: 1, params: {}, members: { n: int() } }); + /** On worker S1, waiting for the server's `Catalog`. */ + const Agent = remote("app.agent", { + version: 1, + params: { session: str() }, + hostAddress: ({ session }) => Session({ id: session }), + variants: { v: { state: { n: int() } } }, + }); + const Catalog = remote("app.catalog", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { v: { state: { n: int() } } }, + }); + + it("starts two hosts that would have deadlocked (Sol's probe): an acyclic graph needs no timeout", async () => { + const { server, errors } = await relayed({ + server: (h) => { + h.serve(Bootstrap, (_, self) => ({ n: self.get(Agent, { session: "S1" }).n })); + h.serve(Catalog, () => ({ variant: "v", members: { n: constant(7) } })); + }, + worker: (h) => { + // The worker's own host object waits for the server's catalog during the worker's load. + h.serve(Agent, (_params, _principal, self) => ({ variant: "v", members: { n: self.get(Catalog, {}).n } })); + }, + }); + await transported(); + await transported(); + const bootstrap = server.hold(Bootstrap, {}, context); + expect(bootstrap.state.value).toEqual({ kind: "live" }); + expect(bootstrap.n.value).toBe(7); + expect([errors.server, errors.worker]).toEqual([[], []]); + }); + + it("places a hold arriving before the publication here, not over the parent", async () => { + const worker = createHost({ + address: Session({ id: "S1" }), + principal: { kind: "worker", session: "S1" }, + ...noErrors, + }); + const [a, b] = memoryLinkPair(); + worker.connect(a, { principal: { kind: "server" }, codec: testCodec() }, context); + const parent = new RawPeer(b); + const [c, d] = memoryLinkPair(); + worker.accept(c, { principal: { kind: "owner" }, codec: testCodec() }, context); + const child = new RawPeer(d); + child.send({ + kind: "hold", + hold: 0, + type: PlacedNotes.name, + version: PlacedNotes.version, + fingerprint: PlacedNotes.fingerprint, + params: jsonPart({ session: "S1" }), + address: { kind: "session", params: { id: "S1" } }, + }); + const mine = worker.hold(PlacedNotes, { session: "S1" }, context); + await transported(); + expect(mine.state.value).toEqual({ kind: "loading" }); + await worker.load( + [{ name: "@test/w", facet: defineFacet((h) => serveNotes(h, PlacedNotes)) }], + context, + ); + expect((await child.next("bound")).holds[0]).toMatchObject({ hold: 0 }); + await transported(); + expect(mine.state.value).toEqual({ kind: "live" }); + expect(parent.kinds()).toEqual(["hello"]); + }); + + it("wakes a parked chain over three hosts in order, its causes passed on (tab → worker → server)", async () => { + const Chat = remote("app.chat", { + version: 1, + params: { session: str() }, + hostAddress: ({ session }) => Session({ id: session }), + variants: { v: { state: { n: int() } } }, + }); + const order: string[] = []; + let catalogs = 0; + const { tab } = await relayed({ + server: (h) => + h.serve(Catalog, (_params, _principal, self) => { + const attempt = ++catalogs; + self.open( + async () => { + if (attempt === 1) throw new Error("offline"); + }, + () => {}, + ); + self.onLive(async () => void order.push("catalog")); + return { variant: "v", members: { n: constant(attempt) } }; + }), + worker: (h) => + h.serve(Chat, (_params, _principal, self) => { + const catalog = self.get(Catalog, {}); + self.onLive(async () => void order.push("chat")); + return { variant: "v", members: { n: catalog.n } }; + }), + }); + const view = tab.hold(Chat, { session: "S1" }, context); + for (let round = 0; round < 5; round++) await transported(); + expect(view.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "waiting", error: { message: "waiting for app.catalog: app.catalog: offline" } }, + retryAt: undefined, + }); + // The catalog's own backoff (1 s) is the only timer: it retries, and the chain goes on in order. + await vi.advanceTimersByTimeAsync(1_000); + for (let round = 0; round < 5; round++) await transported(); + expect(view.state.value).toEqual({ kind: "live" }); + expect(order).toEqual(["catalog", "chat"]); + expect(view.n.value).toBe(2); + }); }); describe("get", () => { - it("refuses an unserved type and a contract mismatch", async () => { + it("waits for an unserved type and a contract mismatch until its readyTimeout, then rejects with the cause", async () => { const { host } = await loaded((h) => h.serve(Info, () => ({ title: constant("t") }))); - expect(await rejection(host.get(Page, { id: 1 }, context))).toEqual({ + const quick = { readyTimeout: 5 }; + expect(await rejection(host.get(Page, { id: 1 }, context, quick))).toEqual({ code: "unavailable", message: "app.page is not served here", }); const InfoV2 = local("app.info", { version: 2, params: {}, members: { title: str() } }); const Changed = local("app.info", { version: 1, params: {}, members: { title: int() } }); - expect(await rejection(host.get(InfoV2, {}, context))).toMatchObject({ code: "contract_mismatch" }); - expect(await rejection(host.get(Changed, {}, context))).toMatchObject({ code: "contract_mismatch" }); + expect(await rejection(host.get(InfoV2, {}, context, quick))).toMatchObject({ code: "contract_mismatch" }); + expect(await rejection(host.get(Changed, {}, context, quick))).toMatchObject({ code: "contract_mismatch" }); const Same = local("app.info", { version: 1, params: {}, members: { title: str() } }); expect((await host.get(Same, {}, context)).title.value).toBe("t"); }); @@ -612,7 +891,11 @@ forEachCodec(() => { it("a local type never crosses a link; a remote type without a parent link waits for one", async () => { const { client } = await linked(() => {}); - expect(await rejection(client.get(Inner, { id: 1 }, context))).toEqual({ + expect(client.hold(Inner, { id: 1 }, context).state.value).toMatchObject({ + kind: "failed", + cause: { kind: "waiting" }, + }); + expect(await rejection(client.get(Inner, { id: 1 }, context, { readyTimeout: 5 }))).toEqual({ code: "unavailable", message: "app.inner is not served here", }); @@ -645,6 +928,81 @@ forEachCodec(() => { }); }); +describe("host.registry and the host's close (fix round 1)", () => { + const Absent = defineRegistry("app.absent", { version: 1 }); + + it("ends a waiting host.registry at the close with released/closed, its deadline timer with it (Sol 5, Astra 5)", async () => { + const host = createHost(noErrors); + await host.load([], context); + let outcome: unknown; + void rejection(host.registry(Absent, context, { readyTimeout: 60_000 })).then((error) => { + outcome = error; + }); + await flushed(); + await host.close(context); + await flushed(); + expect(outcome).toMatchObject({ code: "released", reason: "closed" }); + }); + + it("ends one still waiting for the publication at the close", async () => { + const host = createHost(noErrors); + const waiting = rejection(host.registry(Absent, context, { readyTimeout: 60_000 })); + await host.close(context); + expect(await waiting).toMatchObject({ code: "released", reason: "closed" }); + }); +}); + +describe("the selected provider's own checks for a request made here (fix round 1)", () => { + const Here = hostKind("app-here"); + const Elsewhere = hostKind("app-elsewhere"); + const shape = { version: 1, params: {}, variants: { view: { state: { n: int() } } } } as const; + + it("waits a same-host remote request the provider's code places elsewhere: invalid/host_address (Astra 4)", async () => { + const Consumer = remote("app.placed", { ...shape, hostAddress: () => Here() }); + const Provider = remote("app.placed", { ...shape, hostAddress: () => Elsewhere() }); + const host = createHost({ address: Here(), principal: { kind: "owner" }, onError: () => {} }); + const facet = defineFacet((h) => h.serve(Provider, () => ({ variant: "view", members: { n: constant(1) } }))); + await host.load([{ name: "@app/provider", facet }], context); + const ref = host.hold(Consumer, {}, context); + await flushed(); + expect(ref.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "waiting", error: { code: "invalid", reason: "host_address" } }, + }); + ref.release(); + await host.close(context); + }); + + it("fails broken when the provider's hostAddress throws: its own code (Sol 7)", async () => { + const Consumer = remote("app.remote", { ...shape, hostAddress: () => Server() }); + const Provider = remote("app.remote", { + ...shape, + hostAddress: () => { + throw new Error("provider address bug"); + }, + }); + const server = createHost({ address: Server(), principal: { kind: "server" }, onError: () => {} }); + const client = createHost({ onError: () => {} }); + const facet = defineFacet((h) => h.serve(Provider, () => ({ variant: "view", members: { n: constant(1) } }))); + await server.load([{ name: "@app/provider", facet }], context); + await client.load([], context); + const [down, up] = memoryLinkPair(); + server.accept(down, { principal: { kind: "server" } }, context); + client.connect(up, { principal: { kind: "server" } }, context); + const held = client.hold(Consumer, {}, context); + await transported(); + expect(held.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "broken", error: { message: "app.remote: provider address bug" } }, + }); + // The same throw for a request made on the provider's host fails it broken too. + const here = server.hold(Consumer, {}, context); + expect(here.state.value).toMatchObject({ kind: "failed", cause: { kind: "broken" } }); + await client.close(context); + await server.close(context); + }); +}); + describe("a load whose setup throws", () => { it("closes the host as close does, links included, and rejects with the setup's error", async () => { const host = createHost({ principal: { kind: "server" }, onError: () => {} }); @@ -687,23 +1045,29 @@ describe("acquiring: hold and get", () => { const offline = { code: "unavailable", message: "app.svc(1): offline" }; - it("invalid params and a local type not served here: at once", async () => { + it("invalid params at once; a local type not served here: hold waits failed, get rejects at its readyTimeout", async () => { const host = createHost({ address: Server(), ...noErrors, principal: { kind: "server" } }); await host.load([{ name: "@test/app", facet: defineFacet((h) => void serveNotes(h)) }], context); const params = { code: "invalid", reason: "params" }; expect(() => host.hold(Notes, { session: 1 } as never, context)).toThrow("params"); expect(await rejection(host.get(Notes, { session: 1 } as never, context))).toMatchObject(params); const unserved = { code: "unavailable", message: "app.info is not served here" }; - expect(() => host.hold(Info, {}, context)).toThrow(unserved.message); - expect(await rejection(host.get(Info, {}, context))).toEqual(unserved); + expect(host.hold(Info, {}, context).state.value).toMatchObject({ + kind: "failed", + cause: { kind: "waiting", error: unserved }, + retryAt: undefined, + }); + const got = rejection(host.get(Info, {}, context, { readyTimeout: 5 })); + await vi.advanceTimersByTimeAsync(5); + expect(await got).toEqual(unserved); }); - it("before load resolved: hold is loading, get waits; once it resolved both bind, or an unserved type fails", async () => { + it("before load resolved: hold is loading, get waits; once it resolved both bind, or an unserved type waits", async () => { const host = createHost(noErrors); const held = host.hold(Info, {}, context); const unserved = host.hold(Page, { id: 1 }, context); const got = host.get(Info, {}, context); - const refused = rejection(host.get(Page, { id: 1 }, context)); + const refused = rejection(host.get(Page, { id: 1 }, context, { readyTimeout: 50 })); expect([held.state.value, unserved.state.value]).toEqual([{ kind: "loading" }, { kind: "loading" }]); await host.load( [{ name: "@test/app", facet: defineFacet((h) => h.serve(Info, () => ({ title: constant("t") }))) }], @@ -715,9 +1079,10 @@ describe("acquiring: hold and get", () => { const error = { code: "unavailable", message: "app.page is not served here" }; expect(unserved.state.value).toMatchObject({ kind: "failed", - cause: { kind: "broken", error }, + cause: { kind: "waiting", error }, retryAt: undefined, }); + await vi.advanceTimersByTimeAsync(50); expect(await refused).toEqual(error); }); @@ -866,25 +1231,25 @@ forEachCodec(() => { expect((await child.next("bound")).holds[0]).toMatchObject({ hold: 0 }); }); - it("refuses a hold served here whose params place it elsewhere, whatever address it names: invalid/host_address (race 5)", async () => { + it("keeps a hold naming this host whose params the code here places elsewhere waiting: invalid/host_address (code skew, M§1)", async () => { const { child } = await worker(); holdOf(child, "S2", "S1"); expect(await child.next("failed")).toMatchObject({ hold: 0, - cause: { kind: "broken", error: { kind: "chord", error: misplaced } }, - }); - // A refused hold is kept (a retry starts it again): the next hold takes the next id. - holdOf(child, "S2", "S2", PlacedNotes.name, 1); - expect(await child.next("failed")).toMatchObject({ - hold: 1, - cause: { kind: "broken", error: { kind: "chord", error: misplaced } }, + cause: { kind: "waiting", error: { kind: "chord", error: misplaced } }, }); + // A refused hold is kept (a publication or a retry starts it again): the next hold takes the next id. + holdOf(child, "S2", "S1", PlacedNotes.name, 1); + expect(await child.next("failed")).toMatchObject({ hold: 1, cause: { kind: "waiting" } }); }); - it("places a hold served here by its params, not by the address it names (that only routes it)", async () => { - const { child } = await worker(); - holdOf(child, "S1", "S2"); - expect((await child.next("bound")).holds[0]).toMatchObject({ hold: 0 }); + it("places a hold by the address it names, never by the name it serves (a relay never captures it)", async () => { + const { child, parent } = await worker({ parent: true }); + holdOf(child, "S2", "S2"); + const relayed = await parent!.next("hold"); + expect(relayed).toMatchObject({ type: PlacedNotes.name, address: { kind: "session", params: { id: "S2" } } }); + holdOf(child, "S1", "S2", PlacedNotes.name, 1); + expect(await parent!.next("hold")).toMatchObject({ hold: 1 }); }); it("sends a reference made here whose host address is another host over the parent link, though it serves the type", async () => { diff --git a/packages/chord/test/next/host/keyed-members.test.ts b/packages/chord/test/next/host/keyed-members.test.ts index d2570dd56..1ae93211c 100644 --- a/packages/chord/test/next/host/keyed-members.test.ts +++ b/packages/chord/test/next/host/keyed-members.test.ts @@ -1,18 +1,19 @@ // Tests of keyed members across attempts (slice 6a §1.2, §1.4, §12): the keys references hold on an object's keyed // members, frozen while it is failed, re-bound by name (loaded first) when its next attempt goes live, released once. -import { describe, expect, it } from "vitest"; +import { describe, expect, it, vi } from "vitest"; import { BACKGROUND_CONTEXT as context, withAbortSignal } from "../../../src/context/index.ts"; import { int, keyed, local, obj, str } from "../../../src/next/contracts/index.ts"; import { ChordError } from "../../../src/next/data/index.ts"; import { defineState, type KeyedHandle } from "../../../src/next/documents/index.ts"; -import type { Self } from "../../../src/next/host/index.ts"; +import { createHost, defineFacet, type Self } from "../../../src/next/host/index.ts"; import { ChordObject } from "../../../src/next/host/object.ts"; import { changesOf, delivered, expectCollected, failLive, + flushed, instancesOf, loaded, memoryStore, @@ -291,7 +292,8 @@ describe("a key whose first pin still loads", () => { await delivered(); expect([viewer.value.value, viewer.double.value]).toEqual([{ n: 2 }, 4]); expect([pinsOf(providers.a!), pinsOf(providers.b!)]).toEqual([undefined, 1]); - expect(errors.filter((error) => error.kind === "activation_failed").length).toBe(1); + // The pin's failed load parked the viewer (a dependency failure is not its own): nothing failed it. + expect(errors.filter((error) => error.kind === "activation_failed").length).toBe(0); const values = changesOf(await viewer.value.watch(context)); await providers.a!.change( "k", @@ -312,10 +314,11 @@ describe("a key whose first pin still loads", () => { await close(); }); - it("the object fails and stays failed: the consumer's activation fails transient, never live on it", async () => { - const { host, errors, exposer, providers, hold, gate, entered, fail, close } = await cold(() => "a"); + it("the object fails and stays failed: the consumer parks failed { waiting } on it, never live on it", async () => { + const { host, exposer, providers, hold, gate, entered, fail, close } = await cold(() => "a"); hold(); const abort = new AbortController(); + const viewer = host.hold(Viewer, { id: 1 }, context); const loading = host.get(Viewer, { id: 1 }, withAbortSignal(abort.signal, context)); await entered.promise; fail(); @@ -323,8 +326,8 @@ describe("a key whose first pin still loads", () => { await delivered(); await new Promise((resolve) => setTimeout(resolve, 10)); expect(exposer.state.value.kind).toBe("failed"); - const failure = errors.find((error) => error.kind === "activation_failed"); - expect(failure).toMatchObject({ failure: { kind: "transient" } }); + expect(viewer.state.value).toMatchObject({ kind: "failed", cause: { kind: "waiting" } }); + viewer.release(); abort.abort(); await expect(loading).rejects.toThrow(); expect(pinsOf(providers.a!)).toBeUndefined(); @@ -450,3 +453,54 @@ describe("chains and releases", () => { await expectCollected(...refs); }); }); + +describe("a key whose first pin still loads at startup (scenario 20, Astra)", () => { + it("keeps the first pass pending during an otherwise-ready dependency's delayed keyed pin", async () => { + const Data = defineState("app.pin-data", { + schema: Schema, + version: 1, + keyed: true, + initial: () => ({ n: 4 }), + }); + const Store = local("app.pin-store", { version: 1, params: {}, members: { notes: keyed(Schema) } }); + const Reader = local("app.pin-reader", { version: 1, params: {}, members: { value: Schema } }); + const gate = Promise.withResolvers(); + const original = ChordObject.prototype.pinKey; + const spy = vi.spyOn(ChordObject.prototype, "pinKey").mockImplementation(async function ( + this: ChordObject, + member, + key, + ctx, + ) { + await gate.promise; + return original.call(this, member, key, ctx); + }); + const host = createHost({ onError: () => {} }); + let done = false; + const loading = host + .load( + [ + { + name: "@test/pin", + facet: defineFacet((h) => { + h.serve(Store, (_, self) => ({ notes: self.state(Data) })); + h.serve(Reader, (_, self) => ({ value: self.get(Store, {}).notes.at("k") })); + }), + }, + ], + context, + ) + .then(() => { + done = true; + }); + await flushed(); + const premature = done; + gate.resolve(); + await loading; + spy.mockRestore(); + const ref = await host.get(Reader, {}, context); + expect(ref.value.value).toEqual({ n: 4 }); + await host.close(context); + expect(premature).toBe(false); + }); +}); diff --git a/packages/chord/test/next/host/object.test.ts b/packages/chord/test/next/host/object.test.ts index ab1675fb2..a6959394a 100644 --- a/packages/chord/test/next/host/object.test.ts +++ b/packages/chord/test/next/host/object.test.ts @@ -30,6 +30,7 @@ import { Server, Session, StoreRestartable, + statesOf, storeRestartable, testCodec, transported, @@ -240,7 +241,7 @@ describe("retries", () => { }); }); -describe("dependency recovery (R18)", () => { +describe("parked on dependencies (dependency model §3)", () => { fakeTimers(); const View = local("app.view", { version: 1, params: { id: int() }, members: { n: int() } }); @@ -254,7 +255,7 @@ describe("dependency recovery (R18)", () => { return { times, factory }; } - it("fails a dependent transient with the dependency's cause and retries it at once when it goes live", async () => { + it("parks a dependent of a failed dependency failed { waiting } naming it, with no timer; it goes on when the dependency is live", async () => { const core = flaky((attempt) => attempt === 1); const viewer = view(Core); const { host, errors } = await loaded((h) => { @@ -262,26 +263,28 @@ describe("dependency recovery (R18)", () => { h.serve(View, viewer.factory); }); await vi.advanceTimersByTimeAsync(500); + const held = host.hold(View, { id: 1 }, context); const got = host.get(View, { id: 1 }, context); await flushed(); - const failed = errors.find((error) => error.kind === "activation_failed" && error.object === "app.view(1)"); - expect(failed).toMatchObject({ - failure: { - kind: "transient", - error: { code: "unavailable", message: "app.view(1): dependency app.core failed" }, + expect(held.state.value).toEqual({ + kind: "failed", + cause: { + kind: "waiting", + error: expect.objectContaining({ message: "waiting for app.core: app.core: offline" }), }, + retryAt: undefined, }); - expect(failed?.kind === "activation_failed" && failed.failure.error.cause).toMatchObject({ - message: "app.core: offline", - }); - // The core's retry at 1 s succeeds; the view is retried then, not at the end of its own window (1.5 s). + // A parked attempt is no failure of its own: nothing is reported for the view. + expect(failures(errors)).toEqual(["app.core: transient"]); + // The core's own backoff retries it at 1 s; the view goes on then, its factory not run again. await vi.advanceTimersByTimeAsync(500); expect((await got).n.value).toBe(2); - await vi.advanceTimersByTimeAsync(5000); - expect(viewer.times).toEqual([500, 1000]); + await vi.advanceTimersByTimeAsync(120_000); + expect(viewer.times).toEqual([500]); + held.release(); }); - it("keeps the dependency while failed; releasing the dependent releases it", async () => { + it("keeps its references while parked; a getter rejects with the waiting cause at its deadline; releasing ends the wait", async () => { const svc = flaky(() => true); const viewer = view(Svc); const { host } = await loaded((h) => { @@ -290,81 +293,77 @@ describe("dependency recovery (R18)", () => { }); const got = rejection(host.get(View, { id: 1 }, context, { readyTimeout: 1500 })); await vi.advanceTimersByTimeAsync(1500); - expect(await got).toEqual({ code: "unavailable", message: "app.view(1): dependency app.svc(1) failed" }); + expect(await got).toEqual({ + code: "unavailable", + message: "waiting for app.svc(1): app.svc(1): offline", + }); await vi.advanceTimersByTimeAsync(10_000); - // Held by the failed view, the dependency was retried at 1 s; released with it at 1.5 s, it is not any more. + // Held by the parked view, the dependency was retried at 1 s; released with it at 1.5 s, it is not any more. expect(svc.times).toEqual([0, 1000]); + expect(viewer.times).toEqual([0]); }); - it("binds a retry's references before releasing the previous attempt's", async () => { - const svc = flaky((attempt) => attempt === 1); - const viewer = view(Svc); + it("shows every change of a dependency: loading, failed, a new cause, loading again, live", async () => { + let cores = 0; + const viewer = view(Core); const { host } = await loaded((h) => { - h.serve(Svc, svc.factory); + h.serve(Core, (_, self) => { + const attempt = ++cores; + self.open( + async () => { + if (attempt < 3) throw new Error(`offline ${attempt}`); + }, + () => {}, + ); + return { n: constant(attempt) }; + }); h.serve(View, viewer.factory); }); - const got = host.get(View, { id: 1 }, context); - await vi.advanceTimersByTimeAsync(1000); - // Held only by the view: released first, the live dependency would have ended and started anew. - expect((await got).n.value).toBe(2); - expect([svc.times, viewer.times]).toEqual([ - [0, 1000], - [0, 1000], - ]); + const held = host.hold(View, { id: 1 }, context); + await flushed(); + expect(held.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "waiting", error: { message: "waiting for app.core: app.core: offline 1" } }, + }); + const states = statesOf(await held.state.watch(context)); + await vi.advanceTimersByTimeAsync(10_000); + // The core's retries at 1 s and 3 s: each attempt shows as loading, each failure as a new cause. + expect(states).toEqual(["loading", "failed: waiting for app.core: app.core: offline 2", "loading", "live"]); + expect(viewer.times).toEqual([0]); + held.release(); }); - it("remembers a dependency going live while the dependent activates: the dependent is retried at once (audit)", async () => { - const viewer = view(Core); + it("passes a retry on to its broken dependency, once through a diamond", async () => { + const Top = local("app.top", { version: 1, params: {}, members: { n: int() } }); + const Left = local("app.left", { version: 1, params: { id: int() }, members: { n: int() } }); let cores = 0; const { host } = await loaded((h) => { - // Its first factory throws (broken); the second goes live at once, without natives. h.serve(Core, () => { - if (++cores === 1) throw new Error("first factory"); - return { n: constant(2) }; + if (++cores === 1) throw new Error("bug"); + return { n: constant(cores) }; }); - h.serve(View, viewer.factory); - }); - await flushed(); - // The view activates on the core's failure while the core is retried and goes live at once. - const dependency = host.hold(Core, {}, context); - const dependent = host.hold(View, { id: 1 }, context); - dependency.retry(); - await flushed(); - expect(dependency.state.value).toEqual({ kind: "live" }); - expect(dependent.state.value).toEqual({ kind: "live" }); - expect(viewer.times).toEqual([0, 0]); - dependent.release(); - dependency.release(); - }); - - it("ignores a trigger while an attempt runs: simultaneous triggers give one attempt", async () => { - const gate = Promise.withResolvers(); - const svc = flaky((attempt) => attempt === 1, gate.promise); - const viewer = view(Svc); - const { host } = await loaded((h) => { - h.serve(Svc, svc.factory); - h.serve(View, viewer.factory); - }); - const got = host.get(View, { id: 1 }, context); - // At 1 s both windows end: the dependency's retry waits at its gate, the view's waits for it. - await vi.advanceTimersByTimeAsync(1000); - gate.resolve(); - expect((await got).n.value).toBe(2); - expect(viewer.times).toEqual([0, 1000]); - }); - - it("retries a dependent of a broken dependency with its backoff, for as long as it is held", async () => { - const viewer = view(Core); - const { host } = await loaded((h) => { - h.serve(Core, () => { - throw new Error("bug"); + h.serve(Left, (_, self) => ({ n: self.get(Core, {}).n })); + h.serve(Top, (_, self) => { + const a = self.get(Left, { id: 1 }).n; + const b = self.get(Left, { id: 2 }).n; + return { n: self.derive([a, b], ([x, y]) => x + y) }; }); - h.serve(View, viewer.factory); }); - const got = rejection(host.get(View, { id: 1 }, context, { readyTimeout: 16_000 })); - await vi.advanceTimersByTimeAsync(16_000); - expect(await got).toEqual({ code: "unavailable", message: "app.view(1): dependency app.core failed" }); - expect(viewer.times).toEqual([0, 1000, 3000, 7000, 15000]); + const top = host.hold(Top, {}, context); + await flushed(); + expect(top.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "waiting", error: { message: "waiting for app.left(1): app.core: bug" } }, + }); + await vi.advanceTimersByTimeAsync(60_000); + // A broken dependency is not retried by any timer. + expect(cores).toBe(1); + top.retry(); + await flushed(); + expect(cores).toBe(2); + expect(top.state.value).toEqual({ kind: "live" }); + expect(top.n.value).toBe(4); + top.release(); }); it("fails every object of a cycle broken; none is retried", async () => { @@ -517,6 +516,39 @@ describe("GC of failed objects", () => { }); }); +describe("GC of a parked chain", () => { + it("collects a chain parked on a type not served, released from its top: objects, attempts and index entries", async () => { + const Top = local("app.top", { version: 1, params: { id: int() }, members: { n: int() } }); + const Mid = local("app.mid", { version: 1, params: { id: int() }, members: { n: int() } }); + const Missing = local("app.missing", { version: 1, params: {}, members: { n: int() } }); + const refs: WeakRef[] = []; + const { host } = await loaded((h) => { + h.serve(Mid, (_, self) => { + refs.push(new WeakRef(self)); + return { n: self.get(Missing, {}).n }; + }); + h.serve(Top, ({ id }, self) => { + refs.push(new WeakRef(self)); + return { n: self.get(Mid, { id }).n }; + }); + }); + const top = host.hold(Top, { id: 1 }, context); + await flushed(); + expect(top.state.value).toMatchObject({ + kind: "failed", + cause: { + kind: "waiting", + error: { message: "waiting for app.mid(1): app.missing is not served here" }, + }, + }); + top.release(); + await flushed(); + expect(refs).toHaveLength(2); + await expectCollected(...refs); + expect(host).toBeDefined(); + }); +}); + describe("ref.retry()", () => { fakeTimers(); diff --git a/packages/chord/test/next/host/reference.test.ts b/packages/chord/test/next/host/reference.test.ts index 2f2375b1c..04abbfe61 100644 --- a/packages/chord/test/next/host/reference.test.ts +++ b/packages/chord/test/next/host/reference.test.ts @@ -613,7 +613,7 @@ describe("keyed state members (slice 6a §12)", () => { await store.close(context); }); - it("a pin load that fails its provider fails the consumer as a dependency failure (transient)", async () => { + it("a pin load that fails its provider parks the consumer failed { waiting } on it (a dependency failure)", async () => { const store = await seeded(); const { host, errors } = await persistedNotes(store, persistedMemo(2)); const provider = await host.get(NoteStore, {}, context); @@ -621,7 +621,7 @@ describe("keyed state members (slice 6a §12)", () => { await vi.waitFor(() => expect(view.state.value.kind).toBe("failed")); expect(view.state.value).toMatchObject({ kind: "failed", - cause: { kind: "transient", error: { message: expect.stringContaining("cannot migrate") } }, + cause: { kind: "waiting", error: { message: expect.stringContaining("cannot migrate") } }, }); expect(provider.state.value).toMatchObject({ kind: "failed", cause: { kind: "broken" } }); expect(errors.map(({ kind }) => kind)).toContain("object_failed"); diff --git a/packages/chord/test/next/host/retry-on.test.ts b/packages/chord/test/next/host/retry-on.test.ts index 5677b6c28..31710c226 100644 --- a/packages/chord/test/next/host/retry-on.test.ts +++ b/packages/chord/test/next/host/retry-on.test.ts @@ -76,6 +76,27 @@ describe("retryOn", () => { await host.close(context); }); + it("does nothing while the object is parked failed { waiting }: its dependencies decide (dependency model §3.8)", async () => { + const Missing = local("app.missing", { version: 1, params: {}, members: {} }); + let factories = 0; + const { host } = await loaded((h) => { + serveModels(h); + h.serve(User, (_, self) => { + factories++; + self.retryOn(self.registry(Models)); + self.get(Missing, {}); + return {}; + }); + }); + const user = host.hold(User, { id: 1 }, context); + await flushed(); + expect(user.state.value).toMatchObject({ kind: "failed", cause: { kind: "waiting" } }); + await (await host.get(Runtime, {}, context)).set(1, context); + await flushed(); + expect([user.state.value.kind, factories]).toEqual(["failed", 1]); + await host.close(context); + }); + it("wakes on value calls only: a throwing compose wakes nothing, its next good composition does", async () => { const opens = { n: 0 }; const { host } = await loaded((h) => { diff --git a/packages/chord/test/next/host/revocation.test.ts b/packages/chord/test/next/host/revocation.test.ts index ed5bab268..22bc7a9de 100644 --- a/packages/chord/test/next/host/revocation.test.ts +++ b/packages/chord/test/next/host/revocation.test.ts @@ -1,12 +1,12 @@ // Tests of revocation (§4, R10, R11): revocation keys, \`host.revoke\` on one host (objects, running calls, activations, -// holds still waiting for \`load\`), child links and \`revoke\` messages, GC, and revocation across a relay. +// holds bound or not), child links and \`revoke\` messages, GC, and revocation across a relay. import { describe, expect, it } from "vitest"; import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; -import { int, local, remote } from "../../../src/next/contracts/index.ts"; +import { hostKind, int, local, remote } from "../../../src/next/contracts/index.ts"; import { createHost, defineFacet, type FacetHost, type Host } from "../../../src/next/host/index.ts"; import { ChordObject } from "../../../src/next/host/object.ts"; -import { memoryLinkPair } from "../../../src/next/link/index.ts"; +import { type Link, memoryLinkPair } from "../../../src/next/link/index.ts"; import { jsonPart } from "../../../src/next/link/internal.ts"; import { constant } from "../../../src/next/state/index.ts"; import { @@ -18,6 +18,7 @@ import { RawPeer, rawParent, rawWorkers, + rejection, relayed, SERVER_ADDRESS, Server, @@ -312,6 +313,22 @@ forEachCodec(() => { await host.close(context); }); + it("ends an unbound hold of the principal (refused, waiting for a provider): gone, nothing left of it", async () => { + const { host, parent } = await worker(); + const Absent = { ...Notes, name: "app.absent" } as unknown as typeof Notes; + holdAs(parent, 0, S1, Absent); + expect(await parent.next("failed")).toMatchObject({ hold: 0, cause: { kind: "waiting" } }); + expect(host.activity.value.references).toBe(1); + parent.send({ kind: "revoke", revocation: { session: "S1" } }); + expect(await parent.next("gone")).toMatchObject({ + hold: 0, + error: { kind: "chord", error: { code: "forbidden", reason: "revoked" } }, + }); + await transported(); + expect(host.activity.value.references).toBe(0); + await host.close(context); + }); + it("binds a hold made for a revoked principal, then the parent's revoke after it ends it (race 3, R10's window)", async () => { const { host, parent } = await worker(); holdAs(parent, 0, S1); @@ -443,3 +460,120 @@ describe("revocation across a relay", () => { expect(again.state.value.kind).toBe("live"); }); }); + +describe("revocation of this host's own requests over its paths (fix round 1)", () => { + type Guest = { kind: "owner" } | { kind: "guest"; grant: string }; + const There = hostKind("app-there"); + const Remote = remote("app.remote", { + version: 1, + params: {}, + hostAddress: () => There(), + variants: { view: { state: {}, methods: {} } }, + }); + const guestRevokedBy = (p: Guest) => (p.kind === "guest" ? [p.grant] : []); + + it("ends a guest's ctx.get waiting for its route before any link exists, at once (Astra 1, Sol 1)", async () => { + const host = createHost({ + principal: { kind: "owner" }, + revokedBy: guestRevokedBy, + onError: () => {}, + }); + host.route(There, { authorize: () => true, resolve: () => new Promise(() => {}) }, context); + await host.load([], context); + let outcome: unknown; + void rejection( + host.root((ctx) => ctx.get(Remote, {}, { readyTimeout: 60_000 }), context, { + principal: { kind: "guest", grant: "g" }, + }), + ).then((error) => { + outcome = error; + }); + await transported(); + host.revoke("g", context); + await transported(); + expect(outcome).toMatchObject({ code: "forbidden", reason: "revoked" }); + await host.close(context); + }); + + it("never lets the revoked request bind on the route's later link (Astra 1)", async () => { + const host = createHost({ + principal: { kind: "owner" }, + revokedBy: guestRevokedBy, + onError: () => {}, + }); + const gate = Promise.withResolvers<{ link: Link; principal: Guest; trusted: boolean }>(); + host.route(There, { authorize: () => true, resolve: () => gate.promise }, context); + await host.load([], context); + const pending = rejection( + host.root( + async (ctx) => { + await ctx.get(Remote, {}, { readyTimeout: 500 }); + throw new Error("revoked request became live"); + }, + context, + { principal: { kind: "guest", grant: "g" } }, + ), + ); + await transported(); + host.revoke("g", context); + const worker = createHost({ + address: There(), + principal: { kind: "owner" }, + revokedBy: guestRevokedBy, + onError: () => {}, + }); + const [up, down] = memoryLinkPair(); + worker.connect(up, { principal: { kind: "owner" }, trusted: true }, context); + const facet = defineFacet((h) => h.serve(Remote, () => ({ variant: "view", members: {} }))); + await worker.load([{ name: "@test/late", facet }], context); + gate.resolve({ link: down, principal: { kind: "owner" }, trusted: true }); + expect(await pending).toMatchObject({ code: "forbidden", reason: "revoked" }); + await transported(); + expect(worker.activity.value.references).toBe(0); + await host.close(context); + await worker.close(context); + }); + + it("ends a guest's reference over a route link that dropped, and nothing holds it on the next one", async () => { + const routed: RawPeer[] = []; + const host = createHost({ + principal: { kind: "owner" }, + revokedBy: guestRevokedBy, + onError: () => {}, + }); + host.route( + There, + { + authorize: () => true, + resolve: async () => { + const [a, b] = memoryLinkPair(); + routed.push(new RawPeer(b)); + return { link: a, principal: { kind: "owner" }, codec: testCodec() }; + }, + }, + context, + ); + await host.load([], context); + const states: string[] = []; + void host.root( + async (ctx) => { + const held = await ctx.get(Remote, {}, { readyTimeout: 60_000 }).catch((error: Error) => error); + states.push(held instanceof Error ? held.message : "live"); + }, + context, + { principal: { kind: "guest", grant: "g" } }, + ); + await transported(); + const first = routed[0]!; + await first.next("hold"); + first.close(); + await transported(); + host.revoke("g", context); + await transported(); + expect(states).toEqual(["app.remote: revoked"]); + expect(routed.flatMap((peer, i) => (i === 0 ? [] : peer.received.filter(({ kind }) => kind === "hold")))).toEqual( + [], + ); + await host.close(context); + }); +}); diff --git a/packages/chord/test/next/host/routines.test.ts b/packages/chord/test/next/host/routines.test.ts index 17dfe5eb1..7dfa40565 100644 --- a/packages/chord/test/next/host/routines.test.ts +++ b/packages/chord/test/next/host/routines.test.ts @@ -727,8 +727,9 @@ describe("on", () => { }, ctx); }); }); + // 0 → 1 → 2 → 3, then the write of 3 changes nothing and nothing calls again (waited for, not timed: Opus F3). + await vi.waitFor(() => expect(calls).toBe(4), { timeout: 2_000 }); await new Promise((resolve) => setTimeout(resolve, 20)); - // 0 → 1 → 2 → 3, then the write of 3 changes nothing and nothing calls again. expect(calls).toBe(4); }); diff --git a/packages/chord/test/next/host/unbound.test.ts b/packages/chord/test/next/host/unbound.test.ts new file mode 100644 index 000000000..8da3929e4 --- /dev/null +++ b/packages/chord/test/next/host/unbound.test.ts @@ -0,0 +1,171 @@ +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { int, local, str } from "../../../src/next/contracts/index.ts"; +import { ChordError, type Principal } from "../../../src/next/data/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { Binding } from "../../../src/next/host/reference.ts"; +import { UnboundIndex, type UnboundRequest } from "../../../src/next/host/unbound.ts"; +import { memoryLinkPair } from "../../../src/next/link/index.ts"; +import { jsonPart } from "../../../src/next/link/internal.ts"; +import { Hold } from "../../../src/next/remote/hold.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import { + delivered, + expectCollected, + instancesOf, + RawPeer, + rejection, + SERVER_ADDRESS, + Server, + type TestPrincipal, + testCodec, + transported, +} from "../support.ts"; + +const bindings = instancesOf(Binding, "release"); +const holds = instancesOf(Hold, "release"); + +const Info = local("app.info", { version: 1, params: {}, members: { title: str() } }); +const Missing = local("app.missing", { version: 1, params: { id: int() }, members: { n: int() } }); + +/** A request recording its wakes and ends. */ +function request(principal?: Principal): UnboundRequest & { readonly log: string[] } { + const log: string[] = []; + return { principal, log, wake: () => log.push("wake"), end: (error) => log.push(`end ${error.message}`) }; +} + +describe("the index", () => { + it("wakes every request at the publication, out of the index first; a deleted one is not woken", () => { + const index = new UnboundIndex(); + const [a, b, c] = [request(), request(), request()]; + index.add("app.a", a); + index.add("app.a", b); + index.add("app.b", c); + index.delete("app.a", b); + index.wakeAll(); + expect([a.log, b.log, c.log]).toEqual([["wake"], [], ["wake"]]); + // Out of it: a second publication wakes nobody. + index.wakeAll(); + expect(a.log).toEqual(["wake"]); + }); + + it("ends the requests of a revoked principal only, and every request at the host's close", () => { + const index = new UnboundIndex(); + const guest = request({ kind: "guest" }); + const owner = request({ kind: "owner" }); + const own = request(); + for (const each of [guest, owner, own]) index.add("app.a", each); + const revoked = new ChordError({ code: "forbidden", reason: "revoked", message: "revoked" }); + index.revoke((principal) => principal.kind === "guest", revoked); + expect([guest.log, owner.log, own.log]).toEqual([["end revoked"], [], []]); + index.close(new ChordError({ code: "released", reason: "closed", message: "closed" })); + expect([owner.log, own.log]).toEqual([["end closed"], ["end closed"]]); + }); +}); + +describe("references waiting in the index", () => { + it("before the publication: loading, placed by it; a type not served then waits failed { waiting }", async () => { + const host = createHost({ onError: () => {} }); + const info = host.hold(Info, {}, context); + const missing = host.hold(Missing, { id: 1 }, context); + expect([info.state.value, missing.state.value]).toEqual([{ kind: "loading" }, { kind: "loading" }]); + await host.load( + [{ name: "@test/a", facet: defineFacet((h) => h.serve(Info, () => ({ title: constant("t") }))) }], + context, + ); + await delivered(); + expect(info.state.value).toEqual({ kind: "live" }); + expect(missing.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "waiting", error: { code: "unavailable", message: "app.missing is not served here" } }, + retryAt: undefined, + }); + // A retry leaves it waiting: an event wakes it, nothing else. + missing.retry(); + expect(missing.state.value).toMatchObject({ kind: "failed", cause: { kind: "waiting" } }); + await host.close(context); + expect(missing.state.value).toMatchObject({ kind: "ended", error: { reason: "closed" } }); + }); + + it("leave nothing behind once released, before and after the publication (GC)", async () => { + const host = createHost({ onError: () => {} }); + host.hold(Missing, { id: 1 }, context).release(); + await host.load([], context); + host.hold(Missing, { id: 2 }, context).release(); + const got = host.get(Missing, { id: 3 }, context, { readyTimeout: 1 }).catch(() => {}); + await got; + await expectCollected(...bindings); + expect(host).toBeDefined(); + }); + it("end at the start of the host's close, not after its slow native closes (fix round 1, Sol 6)", async () => { + const Core = local("app.core", { version: 1, params: {}, members: { n: int() } }); + const closeGate = Promise.withResolvers(); + const host = createHost({ onError: () => {} }); + const facet = defineFacet((h) => + h.serve(Core, (_, self) => { + self.open( + async () => 1, + () => closeGate.promise, + ); + return { n: constant(1) }; + }), + ); + await host.load([{ name: "@test/a", facet }], context); + let outcome: unknown; + void rejection(host.get(Missing, { id: 1 }, context)).then((error) => { + outcome = error; + }); + const held = host.hold(Missing, { id: 2 }, context); + const closing = host.close(context); + await delivered(); + expect(outcome).toMatchObject({ code: "released", reason: "closed" }); + expect(held.state.value).toMatchObject({ kind: "ended", error: { reason: "closed" } }); + closeGate.resolve(); + await closing; + }); +}); + +describe("holds waiting in the index", () => { + /** A server not loaded yet, a raw consumer peer on a child link, and a hold of `app.missing` it sent. */ + async function early() { + const host = createHost({ address: Server(), principal: { kind: "server" }, onError: () => {} }); + const [a, b] = memoryLinkPair(); + host.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + const peer = new RawPeer(b); + peer.send({ + kind: "hold", + hold: 0, + type: "app.missing", + version: 1, + fingerprint: "0123456789abcdef", + params: jsonPart({}), + address: SERVER_ADDRESS, + }); + await transported(); + return { host, peer, link: a }; + } + + it("wait for the publication without a word, then are refused failed { waiting } and kept", async () => { + const { host, peer } = await early(); + expect(peer.kinds()).toEqual(["hello"]); + await host.load([], context); + expect(await peer.next("failed")).toMatchObject({ hold: 0, cause: { kind: "waiting" } }); + expect(host.activity.value.references).toBe(1); + await host.close(context); + }); + + it("leave nothing behind once their link closed, before and after the publication (GC)", async () => { + const before = await early(); + before.link.close(); + await transported(); + await before.host.load([], context); + const after = await early(); + await after.host.load([], context); + await after.peer.next("failed"); + after.link.close(); + await transported(); + expect([before.host.activity.value.references, after.host.activity.value.references]).toEqual([0, 0]); + await expectCollected(...holds); + expect([before.host, after.host]).toBeDefined(); + }); +}); diff --git a/packages/chord/test/next/link/message.test.ts b/packages/chord/test/next/link/message.test.ts index 968612a2a..ceeb6c599 100644 --- a/packages/chord/test/next/link/message.test.ts +++ b/packages/chord/test/next/link/message.test.ts @@ -100,6 +100,12 @@ const EVERY_KIND: Message[] = [ cause: { kind: "transient", error: { kind: "chord", error: { code: "unavailable", message: "m" } } }, retryIn: 0, }, + // Not the object's fault (dependency model §2): no backoff window, an event wakes it. + { + kind: "failed", + hold: 0, + cause: { kind: "waiting", error: { kind: "chord", error: { code: "contract_mismatch", message: "m" } } }, + }, // The wire error format in `failed` and `gone`: a plugin's error travels too. { kind: "failed", hold: 0, cause: { kind: "broken", error: { kind: "plugin", code: "config", message: "m" } } }, { kind: "gone", hold: 1, error: { kind: "plugin", code: "config", message: "m", details: { at: 1 } } }, @@ -110,6 +116,10 @@ const EVERY_KIND: Message[] = [ retryIn: 60_000, }, { kind: "retry", hold: 300 }, + // The retry operation it belongs to (fix round 1): an object reached twice handles it once. + { kind: "retry", hold: 300, operation: incarnation }, + // And the instance ids of the objects it passed through (fix round 2): a ring stops after one lap. + { kind: "retry", hold: 300, operation: incarnation, through: [incarnation, incarnation] }, { kind: "loading", hold: 300 }, { kind: "result", call: 3, value: '{"n":1}' }, { kind: "result", call: 3 }, @@ -333,6 +343,21 @@ describe("the binary codec", () => { expect([...binaryCodec.encode({ kind: "closed", origin: 1 })]).toEqual([14 | 0x80, 1]); }); + it("writes a failure's cause as one byte, 0 transient, 1 broken, 2 waiting (dependency model §2); refuses another", () => { + const error = { kind: "chord", error: { code: "unavailable", message: "m" } } as const; + const failed = (kind: "transient" | "broken" | "waiting"): Message => ({ + kind: "failed", + hold: 0, + cause: { kind, error }, + }); + expect( + ["transient", "broken", "waiting"].map((kind) => binaryCodec.encode(failed(kind as "waiting"))[2]), + ).toEqual([0, 1, 2]); + const unknown = binaryCodec.encode(failed("waiting")); + unknown[2] = 3; + expect(() => binaryCodec.decode(unknown)).toThrow("an unknown failure kind"); + }); + it("costs a state entry without retirements no byte: the failure count is doubled, plus one when they follow", () => { const state = (retired: number[]): Message => ({ kind: "state", @@ -413,8 +438,16 @@ describe("the binary codec", () => { expect(() => binaryCodec.decode(new Uint8Array([1, 0, 1, 0xff]))).toThrow(); }); - it("writes retry and loading as their kind and hold, and failed's retryIn as one more than it (0: absent)", () => { - expect([...binaryCodec.encode({ kind: "retry", hold: 300 })]).toEqual([11, 0xac, 0x02]); + it("writes retry as its kind, hold, flags, operation (16 raw bytes) and objects passed through, loading as its kind and hold, and failed's retryIn as one more than it (0: absent)", () => { + expect([...binaryCodec.encode({ kind: "retry", hold: 300 })]).toEqual([11, 0xac, 0x02, 0]); + expect(binaryCodec.encode({ kind: "retry", hold: 300, operation: incarnation })).toHaveLength(4 + 16); + const through = [incarnation, incarnation]; + expect(binaryCodec.encode({ kind: "retry", hold: 300, operation: incarnation, through })).toHaveLength( + 4 + 16 + 1 + 32, + ); + const flagged = binaryCodec.encode({ kind: "retry", hold: 300 }); + flagged[3] = 4; + expect(() => binaryCodec.decode(flagged)).toThrow("an unknown retry flag"); expect([...binaryCodec.encode({ kind: "loading", hold: 1 })]).toEqual([12, 1]); const cause = { kind: "broken", error: { kind: "chord", error: { code: "released", message: "" } } } as const; const absent = binaryCodec.encode({ kind: "failed", hold: 0, cause }); @@ -446,6 +479,10 @@ describe("the text codec", () => { expect(text({ kind: "failed", hold: 2, cause, retryIn: 750 })).toBe( '["f",2,"t",{"kind":"chord","error":{"code":"unavailable","message":"m"}},750]', ); + expect(text({ kind: "failed", hold: 2, cause: { ...cause, kind: "waiting" } })).toBe( + '["f",2,"w",{"kind":"chord","error":{"code":"unavailable","message":"m"}}]', + ); + expect(() => textCodec.decode(new TextEncoder().encode('["f",2,"x",{}]'))).toThrow("an unknown failure kind"); expect(text({ kind: "ack" })).toBe('["a"]'); // An origin wraps the message's own tuple. expect(text({ kind: "release", origin: 4, hold: 3 })).toBe('["@",4,["R",3]]'); diff --git a/packages/chord/test/next/registries/record.test.ts b/packages/chord/test/next/registries/record.test.ts index c25b27fcf..5066df8ea 100644 --- a/packages/chord/test/next/registries/record.test.ts +++ b/packages/chord/test/next/registries/record.test.ts @@ -320,7 +320,7 @@ describe("compose", () => { await host.close(context); }); - it("before any good composition, `.value` and `host.registry` throw the error", async () => { + it("before any good composition, `.value` throws the error and `host.registry` rejects with it at its readyTimeout", async () => { const Broken = defineRegistry("app.broken", { version: 1 }); const { host, errors } = await loadedPlugins({ "@app/broken": (host) => @@ -330,7 +330,7 @@ describe("compose", () => { }, }), }); - await expect(host.registry(Broken, context)).rejects.toThrow("bug"); + await expect(host.registry(Broken, context, { readyTimeout: 5 })).rejects.toThrow("bug"); expect(kinds(errors)).toEqual(["registry_failed"]); await host.close(context); }); @@ -623,6 +623,31 @@ describe("contributions", () => { }); }); +describe("the last good composition (decision 5)", () => { + it("is dropped when a contribution it was built from leaves: `.value` throws until `compose` succeeds", async () => { + const Count = defineRegistry("app.count", { version: 1 }); + const compose = (table: RegistryTable): number => { + if (table.length === 0) throw new Error("no contributor"); + return table[0]!.value; + }; + const { host, fail, retry } = await storeRestartable( + objects, + (attempt, self) => self.contribute(Count, attempt), + { + serve: (host) => host.serve(Count, { compose }), + }, + ); + const count = await host.registry(Count, context); + expect(count.value).toBe(1); + // Its contributor failed: the last good value was built from its contribution (a native may have closed with it). + fail(); + expect(() => count.value).toThrow("no contributor"); + await retry(); + expect(count.value).toBe(2); + await host.close(context); + }); +}); + describe("the table's memory", () => { it("keeps nothing of a closed host: records, tables and followed contributions go with it", async () => { const Doc = defineState("app.kept", { schema: obj({ on: bool() }), version: 1, initial: () => ({ on: true }) }); diff --git a/packages/chord/test/next/registries/value.test.ts b/packages/chord/test/next/registries/value.test.ts index e23722948..b2e6c6579 100644 --- a/packages/chord/test/next/registries/value.test.ts +++ b/packages/chord/test/next/registries/value.test.ts @@ -2,7 +2,8 @@ import { getEventListeners } from "node:events"; import { describe, expect, it } from "vitest"; import { BACKGROUND_CONTEXT as context, withAbortSignal, withCancel } from "../../../src/context/index.ts"; import { int, local, method, str } from "../../../src/next/contracts/index.ts"; -import type { defineFacet } from "../../../src/next/host/index.ts"; +import type { FollowObserver } from "../../../src/next/host/follow.ts"; +import { createHost, defineFacet, type Native } from "../../../src/next/host/index.ts"; import { ChordObject } from "../../../src/next/host/object.ts"; import { defineRegistry, type Registry } from "../../../src/next/registries/index.ts"; import type { NativeWatchCall, WatchEnd } from "../../../src/watch.ts"; @@ -83,7 +84,7 @@ describe("older readers (S2)", () => { await host.close(context); }); - it("without a value mapping, and for a reader newer than the provider, fail `contract_mismatch`: broken, reported", async () => { + it("without a value mapping, and for a reader newer than the provider, park readers failed { waiting } (contract_mismatch)", async () => { const Newer = local("app.newer", { version: 1, params: {}, members: { total: int() } }); const Older = local("app.older", { version: 1, params: {}, members: { total: int() } }); const { host, errors } = await loadedPlugins({ @@ -93,11 +94,103 @@ describe("older readers (S2)", () => { host.serve(Older, (_, self) => ({ total: self.derive([self.registry(SumV1)], ([v]) => v) })); }, }); - expect(await rejection(host.registry(SumV1, context))).toMatchObject({ code: "contract_mismatch" }); - expect(await rejection(host.registry(SumV3, context))).toMatchObject({ code: "contract_mismatch" }); - // Only a reload helps: no backoff, the mismatch itself the cause. - expect(failures(errors)).toEqual(["app.newer: broken", "app.older: broken"]); - expect(errors[0]).toMatchObject({ failure: { error: { code: "contract_mismatch" } } }); + const quick = { readyTimeout: 5 }; + expect(await rejection(host.registry(SumV1, context, quick))).toMatchObject({ code: "contract_mismatch" }); + expect(await rejection(host.registry(SumV3, context, quick))).toMatchObject({ code: "contract_mismatch" }); + // Not their fault: waiting (a reload serving the version wakes them), no backoff, nothing failed. + expect(failures(errors)).toEqual([]); + expect(host.hold(Newer, {}, context).state.value).toMatchObject({ + kind: "failed", + cause: { + kind: "waiting", + error: { + message: "waiting for registry app.sum: registry app.sum: version 3 is not served here (version 2 is)", + }, + }, + retryAt: undefined, + }); + await host.close(context); + }); + + it("drops an older reader's last good mapping after a contributor it was built from failed live (fix round 1, Astra 2)", async () => { + const R = defineRegistry("app.mapped", { version: 2 }); + const Old = defineRegistry("app.mapped", { version: 1 }); + const Contributor = local("app.contributor", { version: 1, params: {}, members: { n: int() } }); + let observer: FollowObserver | undefined; + const { host } = await loadedPlugins({ + "@app/mapping": (h) => { + h.serve(R, { + compose: (table) => table.reduce((n, e) => n + e.value, 0), + accepts: { + 1: { + value: (n) => { + if (n === 0) throw new Error("old version cannot map the empty table"); + return n; + }, + }, + }, + }); + h.serve(Contributor, (_, self) => { + self.contribute(R, 42); + return { + n: self.follow(async (followed) => { + observer = followed; + return { value: 42, detach() {} }; + }), + }; + }); + }, + }); + const old = await host.registry(Old, context); + expect(old.value).toBe(42); + observer!.end({ kind: "failed", error: new Error("runtime ended") }); + await flushed(); + expect(() => old.value).toThrow("old version cannot map the empty table"); + await host.close(context); + }); + + it("never hands out a mapped native whose contributor failed and closed it (fix round 1, Sol 2)", async () => { + type Resource = { closed: boolean }; + const Current = defineRegistry, readonly Native[]>("app.resources", { version: 2 }); + const Old = defineRegistry, Native>("app.resources", { version: 1 }); + const Owner = local("app.owner", { version: 1, params: {}, members: {} }); + const resource = { closed: false }; + let failRuntime: () => void = () => {}; + const { host } = await loadedPlugins({ + "@app/resources": (h) => { + h.serve(Current, { + compose: (table) => table.map(({ value }) => value), + accepts: { + 1: { + value: (current) => { + if (current.length === 0) throw new Error("no current resource"); + return current[0]!; + }, + }, + }, + }); + h.serve(Owner, (_, self) => { + const native = self.open( + async () => resource, + () => { + resource.closed = true; + }, + ); + self.contribute(Current, native); + self.follow(async (observer) => { + failRuntime = () => observer.end({ kind: "failed", error: new Error("runtime closed") }); + return { value: 1, detach: () => {} }; + }); + return {}; + }); + }, + }); + const value = await host.registry(Old, context); + expect(value.value.value).toBe(resource); + failRuntime(); + await flushed(); + expect(resource.closed).toBe(true); + expect(() => value.value).toThrow("no current resource"); await host.close(context); }); @@ -162,29 +255,30 @@ describe("readers (S3)", () => { R: Registry = Sum, ) => host.serve(Reader, (_, self) => ({ total: self.derive([self.registry(R)], ([v]) => v.total) })); - it("activate once the registry has a value; a reader of a registry nobody serves fails at once, `load` waits for nothing", async () => { + it("activate once the registry has a value; a reader of a registry nobody serves parks failed { waiting }, `load` waits for nothing", async () => { const started = Date.now(); const { host, errors } = await loadedPlugins({ "@app/reader": (host) => serveReader(host) }); expect(Date.now() - started).toBeLessThan(1_000); - expect(errors).toMatchObject([ - { - kind: "activation_failed", - object: "app.reader", - failure: { - kind: "transient", - error: { - code: "unavailable", - message: "app.reader: registry app.sum has no value", - cause: { code: "unavailable", reason: "undeclared" }, - }, + expect(errors).toEqual([]); + expect(host.hold(Reader, {}, context).state.value).toMatchObject({ + kind: "failed", + cause: { + kind: "waiting", + error: { + code: "unavailable", + message: "waiting for registry app.sum: registry app.sum: no plugin serves it here", + cause: { code: "unavailable", reason: "undeclared" }, }, }, - ]); - expect(await rejection(host.registry(Sum, context))).toMatchObject({ code: "unavailable", reason: "undeclared" }); + }); + expect(await rejection(host.registry(Sum, context, { readyTimeout: 5 }))).toMatchObject({ + code: "unavailable", + reason: "undeclared", + }); await host.close(context); }); - it("a reader failed on a first `compose` that threw is retried at once by the first good composition (R18)", async () => { + it("a reader parked on a first `compose` that threw goes on at the first good composition", async () => { const { host, errors } = await loadedPlugins({ "@app/sum": (host) => { host.serve(Sum, { @@ -198,15 +292,46 @@ describe("readers (S3)", () => { }, "@app/setter": (host) => setter(host, 0), }); - expect(failures(errors)).toEqual(["app.reader: transient"]); + expect(failures(errors)).toEqual([]); + const held = host.hold(Reader, {}, context); + expect(held.state.value).toMatchObject({ kind: "failed", cause: { kind: "waiting" } }); const setters = await host.get(Setter, {}, context); await setters.set(3, context); - // The reader's backoff is a second at least: the composition retried it. + // Woken by the composition: no timer of its own. const reader = await host.get(Reader, {}, context, { readyTimeout: 500 }); + held.release(); expect(reader.total.value).toBe(3); await host.close(context); }); + it("host.registry before the publication waits for it, then resolves with the provider it served", async () => { + const host = createHost({ onError: () => {} }); + const waiting = host.registry(Sum, context); + await host.load([{ name: "@app/sum", facet: defineFacet((h) => h.serve(Sum, { compose: total })) }], context); + expect((await waiting).value).toEqual({ total: 0 }); + await host.close(context); + }); + + it("host.registry waits through no value until a first good composition, woken by it, its wait then gone", async () => { + const { host } = await loadedPlugins({ + "@app/sum": (host) => { + host.serve(Sum, { + compose: (table) => { + const sum = total(table); + if (sum.total === 0) throw new Error("empty"); + return sum; + }, + }); + setter(host, 0); + }, + }); + const waiting = host.registry(Sum, context, { readyTimeout: 500 }); + await flushed(); + await (await host.get(Setter, {}, context)).set(5, context); + expect((await waiting).value).toEqual({ total: 5 }); + await host.close(context); + }); + it("once live, a later throwing `compose` changes nothing for the reader: `.value` keeps the last good one", async () => { const { host } = await loadedPlugins({ "@app/sum": (host) => { diff --git a/packages/chord/test/next/remote/hold.test.ts b/packages/chord/test/next/remote/hold.test.ts index 03d27f74c..ffbe77fcc 100644 --- a/packages/chord/test/next/remote/hold.test.ts +++ b/packages/chord/test/next/remote/hold.test.ts @@ -2,13 +2,13 @@ import { describe, expect, it, vi } from "vitest"; import { type Context, BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; import type * as Delta from "../../../src/delta/index.ts"; import { decoder } from "../../../src/delta/index.ts"; -import { bool, int, local, method, obj, remote } from "../../../src/next/contracts/index.ts"; +import { bool, hostKind, int, local, method, obj, remote } from "../../../src/next/contracts/index.ts"; import { broken } from "../../../src/next/data/index.ts"; import { defineState } from "../../../src/next/documents/index.ts"; import { createHost, defineFacet, type Host, type Self } from "../../../src/next/host/index.ts"; import type { ChordHost } from "../../../src/next/host/internal.ts"; import { type Link, memoryLinkPair } from "../../../src/next/link/index.ts"; -import { jsonPart } from "../../../src/next/link/internal.ts"; +import { binaryCodec, jsonPart } from "../../../src/next/link/internal.ts"; import { Hold } from "../../../src/next/remote/hold.ts"; import { constant } from "../../../src/next/state/index.ts"; import { @@ -102,7 +102,9 @@ function providerObject(server: unknown) { const host = server as ChordHost; const { name: type, version, fingerprint } = Halves; const request = { type, version, fingerprint, params: "{}", address: SERVER_ADDRESS }; - return host.serving(request, {}, { kind: "owner" }, context); + const served = host.serving(request, {}, { kind: "owner" }, context); + if (served.kind !== "object") throw new Error("Halves is served here"); + return served.object; } /** A remote type whose activation never completes: its open waits for its abort. */ @@ -457,6 +459,65 @@ forEachCodec(() => { }); }); + describe("holds that bind to nothing yet (dependency model §1, §3)", () => { + const Absent = { name: "app.absent", version: 1, fingerprint: "0123456789abcdef" }; + + it("a refused hold waits failed { waiting }, counted; a retry re-runs its start (loading, failed); its link's close drops it", async () => { + const { server, peer } = await rawLinked(() => {}); + hold(peer, 0, Absent); + const failed = await peer.next("failed"); + expect(failed).toMatchObject({ hold: 0, cause: { kind: "waiting" } }); + expect(failed.retryIn).toBeUndefined(); + expect(server.activity.value.references).toBe(1); + expect(server.references(Halves).value).toEqual([]); + peer.send({ kind: "retry", hold: 0 }); + expect(await peer.next("loading")).toEqual({ kind: "loading", hold: 0 }); + expect(await peer.next("failed")).toMatchObject({ hold: 0, cause: { kind: "waiting" } }); + peer.close(); + await transported(); + expect(server.activity.value.references).toBe(0); + }); + + it("a hold on an object parked on a failed dependency is told each new cause, then loading, then bound", async () => { + const Core = local("app.core", { version: 1, params: {}, members: { n: int() } }); + let cores = 0; + const { peer } = await rawLinked((h) => { + h.serve(Core, () => { + if (++cores < 3) throw new Error(`bug ${cores}`); + return { n: constant(cores) }; + }); + h.serve(Halves, (_, _principal, self) => { + const core = self.get(Core, {}); + const n = self.state(N); + return { + variant: "v", + members: { + n, + half: self.derive([core.n], ([c]) => c), + set: async () => null, + wait: async () => null, + }, + }; + }); + }); + hold(peer, 0, Halves); + const waiting = (message: string) => ({ + kind: "failed", + hold: 0, + cause: { kind: "waiting", error: { kind: "chord", error: { code: "unavailable", message } } }, + }); + // The core failed at load (its first outcome): the parked object names it at once. + expect(await peer.next("failed")).toEqual(waiting("waiting for app.core: app.core: bug 1")); + // A retry passes on through the parked object to its broken dependency (one host's retry operation). + // (Its new attempt failed within the batch of changes the parked object checks once: no loading between.) + peer.send({ kind: "retry", hold: 0 }); + expect(await peer.next("failed")).toEqual(waiting("waiting for app.core: app.core: bug 2")); + peer.send({ kind: "retry", hold: 0 }); + await peer.next("loading"); + expect((await peer.next("bound")).holds[0]).toMatchObject({ hold: 0 }); + }); + }); + describe("a reconnecting consumer", () => { const Tally = remote("app.tally", { version: 1, @@ -715,3 +776,107 @@ forEachCodec(() => { }); }); }); + +/** Lets every pending reaction run (a memory link delivers by microtask). */ +async function flush(): Promise { + for (let i = 0; i < 100; i++) await Promise.resolve(); +} + +/** A local type nobody serves. */ +const Missing = local("app.missing", { version: 1, params: {}, members: { n: int() } }); + +describe("retry operations across links (fix rounds 1 and 2, Sol 3)", () => { + it.each([1, 9, 50])( + "quiesce after %i concurrent Retries round a waiting ring across hosts: each stops after one lap", + async (operations) => { + type P = { kind: "server" } | { kind: "worker" }; + const Server = hostKind("app.retry-server"); + const Worker = hostKind("app.retry-worker"); + const LocalA = local("app.a", { version: 1, params: {}, members: { n: int() } }); + const LocalB = local("app.b", { version: 1, params: {}, members: { n: int() } }); + const RemoteA = remote("app.remote-a", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { reader: { state: { n: int() } } }, + }); + const RemoteB = remote("app.remote-b", { + version: 1, + params: {}, + hostAddress: () => Worker(), + variants: { reader: { state: { n: int() } } }, + }); + let retries = 0; + const count = (link: Link): Link => ({ + send: (bytes) => { + if (binaryCodec.decode(bytes).kind === "retry") retries++; + link.send(bytes); + }, + onMessage: (listener) => link.onMessage(listener), + onClose: (listener) => link.onClose(listener), + close: () => link.close(), + }); + const [down, up] = memoryLinkPair(); + const server = createHost

    ({ address: Server(), principal: { kind: "server" }, onError: () => {} }); + const worker = createHost

    ({ address: Worker(), principal: { kind: "worker" }, onError: () => {} }); + server.route( + Worker, + { + authorize: () => true, + resolve: async () => ({ link: count(down), principal: { kind: "worker" }, trusted: true }), + }, + context, + ); + worker.connect(count(up), { principal: { kind: "server" }, trusted: true }, context); + try { + const left = server.load( + [ + { + name: "@test/a", + facet: defineFacet

    ((h) => { + h.serve(LocalA, (_, self) => ({ n: self.get(RemoteB, {}).n })); + h.serve(RemoteA, (_, _principal, self) => ({ + variant: "reader", + members: { n: self.get(LocalA, {}).n }, + })); + }), + }, + ], + context, + ); + const right = worker.load( + [ + { + name: "@test/b", + facet: defineFacet

    ((h) => { + h.serve(LocalB, (_, self) => { + self.get(Missing, {}); + return { n: self.get(RemoteA, {}).n }; + }); + h.serve(RemoteB, (_, _principal, self) => ({ + variant: "reader", + members: { n: self.get(LocalB, {}).n }, + })); + }), + }, + ], + context, + ); + await Promise.all([left, right]); + await flush(); + const a = server.hold(LocalA, {}, context); + expect(a.state.value).toMatchObject({ kind: "failed", cause: { kind: "waiting" } }); + // More at once than an object remembers ids: `through` stops each at the object it passed through already. + for (let i = 0; i < operations; i++) a.retry(); + await flush(); + // Each went round once: server → worker → server, then stopped at the object it started from. + expect(retries).toBe(2 * operations); + await flush(); + expect(retries).toBe(2 * operations); + } finally { + await server.close(context); + await worker.close(context); + } + }, + ); +}); diff --git a/packages/chord/test/next/remote/provider.test.ts b/packages/chord/test/next/remote/provider.test.ts index a93e54c62..dc9f30aed 100644 --- a/packages/chord/test/next/remote/provider.test.ts +++ b/packages/chord/test/next/remote/provider.test.ts @@ -117,14 +117,14 @@ forEachCodec(() => { { fingerprint: "0000000000000000" }, { code: "contract_mismatch", message: "app.notes v1 does not match the served v1" }, ], - ])("refuse %s with a broken failed", async (_, override, error) => { + ])("keep %s refused, failed { waiting }", async (_, override, error) => { const { peer } = await notesServer(); const { type, ...rest } = override as { type?: string }; hold(peer, 0, { ...Notes, name: type ?? Notes.name } as Contract, { session: "S1" }, rest); - expect(await peer.next("failed")).toMatchObject({ - hold: 0, - cause: { kind: "broken", error: { kind: "chord", error } }, - }); + const failed = await peer.next("failed"); + expect(failed).toMatchObject({ hold: 0, cause: { kind: "waiting", error: { kind: "chord", error } } }); + // No backoff window: an event (a publication) wakes it. + expect(failed.retryIn).toBeUndefined(); }); it("refuse invalid params with a broken failed", async () => { @@ -135,7 +135,7 @@ forEachCodec(() => { }); }); - it("arriving before load resolved wait for it", async () => { + it("arriving before the publication wait for it, and bind before the start barrier is over", async () => { const server = createHost({ address: Server(), principal: { kind: "server" }, @@ -505,12 +505,12 @@ forEachCodec(() => { }); describe("local types over a link", () => { - it("never cross one", async () => { + it("never cross one: a hold naming one is not served (failed { waiting })", async () => { const Store = local("app.store", { version: 1, params: {}, members: { n: int() } }); const { peer } = await rawLinked((h) => h.serve(Store, () => ({ n: constant(1) }))); hold(peer, 0, Store, {}); expect(await peer.next("failed")).toMatchObject({ - cause: { kind: "broken", error: { kind: "chord", error: { code: "unavailable" } } }, + cause: { kind: "waiting", error: { kind: "chord", error: { code: "unavailable" } } }, }); }); }); diff --git a/packages/chord/test/next/remote/remote-object.test.ts b/packages/chord/test/next/remote/remote-object.test.ts index eeb3fc7a9..c9a281f44 100644 --- a/packages/chord/test/next/remote/remote-object.test.ts +++ b/packages/chord/test/next/remote/remote-object.test.ts @@ -302,16 +302,20 @@ forEachCodec(() => { }); }); - describe("self.get of a type over the link (R18)", () => { - it("a failed hold fails the dependent transiently; bound retries it at once with a new hold", async () => { + describe("self.get of a type over the link (dependency model §3)", () => { + it("a failed hold parks the dependent failed { waiting }: no new attempt or hold; bound wakes it", async () => { const View = local("app.view", { version: 1, params: {}, members: { count: int() } }); const errors: HostError[] = []; const client = createHost({ onError: (error) => errors.push(error) }); const [a, b] = memoryLinkPair(); client.connect(a, { principal: { kind: "server" }, codec: testCodec() }, context); const peer = new RawPeer(b); + let factories = 0; const facet = defineFacet((h) => - h.serve(View, (_, self) => ({ count: self.get(Notes, { session: "S1" }).count })), + h.serve(View, (_, self) => { + factories++; + return { count: self.get(Notes, { session: "S1" }).count }; + }), ); const loading = client.load([{ name: "@test/view", facet }], context); const { hold } = await peer.next("hold"); @@ -320,25 +324,20 @@ forEachCodec(() => { hold, cause: { kind: "transient", error: { kind: "chord", error: { code: "unavailable", message: "warming" } } }, }); + // Parked on a remote blocker: its first outcome, so the start barrier is over. await loading; - expect(errors).toMatchObject([ - { - kind: "activation_failed", - object: "app.view", - failure: { kind: "transient", error: { code: "unavailable" } }, - }, - ]); - const members = [ - { member: EDITOR.state.notes!, value: "[]" }, - { member: EDITOR.state.count!, value: "0" }, - ]; - peer.send({ kind: "bound", holds: [{ hold, variant: EDITOR.variant, members, failed: [] }] }); - // The retry binds its own hold before the failed attempt's is released. - const retried = await bindNext(peer, [{ text: "a" }]); - expect(retried).toBe(1); - expect(await peer.next("release")).toMatchObject({ hold }); - const view = await client.get(View, {}, context); + await transported(); + expect(errors).toEqual([]); + const view = client.hold(View, {}, context); + expect(view.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "waiting", error: { message: "waiting for app.notes(S1): warming" } }, + }); + bindHold(peer, hold, [{ text: "a" }]); + await transported(); + expect(view.state.value).toEqual({ kind: "live" }); expect(view.count.value).toBe(1); + expect([factories, peer.kinds().filter((kind) => kind === "hold" || kind === "release")]).toEqual([1, []]); }); }); @@ -370,7 +369,7 @@ forEachCodec(() => { // Timers stand still: neither the backoff window nor a readyTimeout ends while the test runs. fakeTimers(); - it("fails at once with the real cause when its load holds it (no wait for a timeout)", async () => { + it("parks at once on the real cause when its load holds it (no wait for a timeout)", async () => { const Flaky = remote("app.flaky", { version: 1, params: {}, @@ -399,16 +398,12 @@ forEachCodec(() => { second.connect(b, { principal: { kind: "server" }, codec: testCodec() }, context); const facet = defineFacet((h) => h.serve(View, (_, self) => ({ n: self.get(Flaky, {}).n }))); await second.load([{ name: "@test/view", facet }], context); - expect(errors).toMatchObject([ - { - kind: "activation_failed", - object: "app.view", - failure: { - kind: "transient", - error: { message: "app.view: dependency app.flaky failed", cause: { message: "app.flaky: offline" } }, - }, - }, - ]); + await transported(); + expect(errors).toEqual([]); + expect(second.hold(View, {}, context).state.value).toMatchObject({ + kind: "failed", + cause: { kind: "waiting", error: { message: "waiting for app.flaky: app.flaky: offline" } }, + }); await first.close(context); await failing; }); @@ -633,7 +628,7 @@ forEachCodec(() => { const toUpgraded = await relink(upgraded, links[0]); expect(notes.state.value).toMatchObject({ kind: "failed", - cause: { kind: "broken", error: { code: "contract_mismatch" } }, + cause: { kind: "waiting", error: { code: "contract_mismatch" } }, retryAt: undefined, }); notes.retry(); @@ -652,7 +647,7 @@ forEachCodec(() => { await transported(); expect(notes.state.value).toMatchObject({ kind: "failed", cause: { kind: "broken" } }); notes.retry(); - expect(await peer.next("retry")).toEqual({ kind: "retry", hold }); + expect(await peer.next("retry")).toEqual({ kind: "retry", hold, operation: expect.any(String) }); bindHold(peer, hold); await transported(); expect(notes.state.value).toEqual({ kind: "live" }); @@ -771,15 +766,6 @@ forEachCodec(() => { h.serve(View, (_, self) => ({ count: self.get(Notes, { session: "S1" }).count })), ); - /** Binds the kept hold (0) on `next`; the retry's own hold (1) is bound and the kept one released (R18). */ - async function retried(client: Host, next: RawPeer): Promise { - expect(await bindNext(next, [{ text: "a" }])).toBe(0); - // Retried at once, long before its backoff window (1 s) ends: its new attempt holds anew. - expect(await bindNext(next, [{ text: "a" }])).toBe(1); - expect(await next.next("release")).toEqual({ kind: "release", hold: 0 }); - expect((await client.get(View, {}, context, { readyTimeout: 1 })).count.value).toBe(1); - } - it("stays loading through the drop, no failure reported, and goes live when the next link binds it", async () => { const errors: HostError[] = []; const client = createHost({ onError: (error) => errors.push(error) }); @@ -798,28 +784,32 @@ forEachCodec(() => { expect(errors).toEqual([]); }); - it("dropped and never bound again, fails transient at its readyTimeout (the bound of the wait)", async () => { + it("dropped and never bound again, stays parked loading: nothing fails, no parked_long without a link", async () => { const errors: HostError[] = []; const client = createHost({ readyTimeout: 1_000, onError: (error) => errors.push(error) }); const peer = await rawParent(client); const loading = client.load([{ name: "@test/view", facet }], context); await peer.next("hold"); + // A remote blocker ends the start barrier's first pass. + await loading; peer.close(); await transported(); + await vi.advanceTimersByTimeAsync(10_000); + // No remote attempt answers for it while its path has no link (fix round 1, Opus F2). expect(errors).toEqual([]); - await vi.advanceTimersByTimeAsync(1_000); - await loading; - expect(failures(errors)).toEqual(["app.view: transient"]); + expect(client.hold(View, {}, context).state.value).toEqual({ kind: "loading" }); }); - it("without any link fails at its readyTimeout, keeps the reference, and is retried when the first link binds it", async () => { + it("without any link stays loading past its readyTimeout and goes live when the first link binds its one hold", async () => { const errors: HostError[] = []; const client = createHost({ readyTimeout: 1_000, onError: (error) => errors.push(error) }); - const loading = client.load([{ name: "@test/view", facet }], context); - await vi.advanceTimersByTimeAsync(1_000); - await loading; - expect(failures(errors)).toEqual(["app.view: transient"]); - await retried(client, await rawParent(client)); + await client.load([{ name: "@test/view", facet }], context); + await vi.advanceTimersByTimeAsync(500); + expect(failures(errors)).toEqual([]); + const next = await rawParent(client); + expect(await bindNext(next, [{ text: "a" }])).toBe(0); + expect((await client.get(View, {}, context, { readyTimeout: 1 })).count.value).toBe(1); + expect(next.kinds().filter((kind) => kind === "hold" || kind === "release")).toEqual([]); }); }); diff --git a/packages/chord/test/next/store/table-migration.test.ts b/packages/chord/test/next/store/table-migration.test.ts index d58bef1b7..e5938015b 100644 --- a/packages/chord/test/next/store/table-migration.test.ts +++ b/packages/chord/test/next/store/table-migration.test.ts @@ -218,18 +218,33 @@ describe("eager migrations", () => { }); }); +/** + * Records the `setTimeout` calls the migration's own code makes (its stack passes through `table-migration.ts`); timers + * of the test runner or other work under load pass through unrecorded (fix round 2: the test watched every timer). + */ +function migrationTimers(): { readonly calls: string[]; restore(): void } { + const calls: string[] = []; + const original = globalThis.setTimeout; + const spy = vi.spyOn(globalThis, "setTimeout").mockImplementation(((...args: Parameters) => { + const stack = new Error().stack ?? ""; + if (stack.includes("src/next/store/table-migration.ts")) calls.push(stack); + return original(...args); + }) as typeof setTimeout); + return { calls, restore: () => spy.mockRestore() }; +} + describe("the chunk yield", () => { it("yields between chunks without Node's setImmediate (Workers, browsers), through a message, not a timer", async () => { await filled(3 * MIGRATION_CHUNK); const saved = globalThis.setImmediate; // Runtimes without it: Cloudflare Workers, browsers. Reflect.deleteProperty(globalThis, "setImmediate"); - const timers = vi.spyOn(globalThis, "setTimeout"); + const timers = migrationTimers(); try { await prepared(EditsV2); - expect(timers).not.toHaveBeenCalled(); + expect(timers.calls).toEqual([]); } finally { - timers.mockRestore(); + timers.restore(); globalThis.setImmediate = saved; } expect(store.table(TABLE)).toMatchObject({ version: 2, state: "ready" }); @@ -242,12 +257,12 @@ describe("the chunk yield", () => { setTimeout(() => { fired = true; }, 0); - const timers = vi.spyOn(globalThis, "setTimeout"); + const timers = migrationTimers(); try { await prepared(EditsV2); - expect(timers).not.toHaveBeenCalled(); + expect(timers.calls).toEqual([]); } finally { - timers.mockRestore(); + timers.restore(); } expect(fired).toBe(true); expect(store.table(TABLE)).toMatchObject({ version: 2, state: "ready" }); From 972d7017af55ce15eb05059c56a647c9530c55f5 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 11:54:38 +0000 Subject: [PATCH 025/175] docs(chord): slice 7b brief: reload, decided and open points --- packages/chord/docs/next/slice-7b/book.md | 304 +++++++++++ packages/chord/docs/next/slice-7b/index.html | 500 +++++++++++++++++++ 2 files changed, 804 insertions(+) create mode 100644 packages/chord/docs/next/slice-7b/book.md create mode 100644 packages/chord/docs/next/slice-7b/index.html diff --git a/packages/chord/docs/next/slice-7b/book.md b/packages/chord/docs/next/slice-7b/book.md new file mode 100644 index 000000000..bda87641d --- /dev/null +++ b/packages/chord/docs/next/slice-7b/book.md @@ -0,0 +1,304 @@ +--- +title: Slice 7b: reloading plugins in a running Chord app +cover: Slice 7b / Reload +kicker: A design brief · Chord v2 · slice 7b · 2026-10-29 +subtitle: What happens to objects, references, jobs and registries when a plugin's code changes on one host or across many, what is decided and what the owner still decides +stats: + - 4 | steps of a reload on one host + - 3 | fates of a live object + - 26 | carry-over items from slices 3a to 7a + - 5 | decisions open +note: | + A brief before the slice 7b spec, written by the redesign session from `design.md` §21, the dependency model + (`dependencies/model.md`, decided and built for 7a in pi `339847eb1`), the carry-over in `slices.md` and the code at + `339847eb1`. Marks: **Decided** points cite the decision; **Open** points are the owner's, with a recommendation. +colophon: Slice 7b brief · 2026-10-29 · Chord v2 +--- + +# Before we begin {.preface} + +Mario edits the notes plugin while a session is open: a new method, a renamed field in a document, a second state +member. He presses Reload. The server, the session's worker and two browser tabs run the old notes code; within a +second they should all run the new one, with the notes, the open views and the running export job intact, and without +restarting a single process. + +Chord v2 can do everything around that today: objects, references across hosts, the store, jobs, registries, and since +7a a dependency model in which a missing provider is a wait, not an error. What it cannot do yet is change code. `load` +runs once per host (`host.ts`: "load runs once per host"). Slice 7b adds the reload, on one host and across hosts. + +## Six things to know + +:::findings +1. **A reload is a transaction per host.** Load the bundles, stage the new declarations, commit atomically, then let + each object follow; any code error before the commit leaves the host entirely on its old code. +2. **A live object has three fates.** Pure objects swap in place; objects with natives drain and restart; objects of a + removed type end, and their holders wait. +3. **Old calls finish on old code.** A pure swap keeps the running calls' old handles valid until they settle; only new + calls see the new code. +4. **Versions are the provider's business.** A provider accepts older consumers through `accepts`; a consumer that does + not fit waits (7a), it is not ended. +5. **7a already decided most of the hard part.** Removal, dropped versions and installs are waits woken by a + publication; a reload's commit is such a publication. +6. **Five points are open:** the reload API, the rollout across hosts, loader D, a `writable` across a swap, and the + install watchdog. +::: + +## Words used throughout + +A **reload** changes the code of one or more plugins on one host. A **facet** is a plugin's code for one host kind; a +**bundle** is the built JavaScript of a facet. **Staging** runs the new facets and factories against a recording +`self` without activating anything. The **commit** switches the host to the new code at once and is a **publication** +in 7a's sense. A **pure** object has no natives (`self.open`), in its old or its new code. A **rollout** is a reload +of the same code set on every host, server first. + +# One Host Reloads {#host lede="Four steps, all or nothing up to the commit."} + +Design §21 fixes the shape. **Decided.** + +```text +1. load the new bundles; an import or syntax error rejects +2. staging facet setups and every live object's new factory run with a recording `self` + (declarations only, nothing activated); any throw rejects +3. commit the host switches to the new code atomically: a publication (7a) +4. after each object follows its fate (chapter II) +``` + +Rollback is all or nothing per host, and only for code errors: an import error, a throwing setup or factory. A failed +`open`, a registry conflict or a failing `migrate` after the commit are runtime states of single objects, never a +reason to roll back. A host is at revision R or R−1, never in between. + +Staging runs factories against a recording `self` because a factory only declares: it returns members and calls +`self.state`, `self.get`, `self.open` and so on, and the recording tells Chord what changed (a new dependency, a native +added or removed, a member gone) before anything happens. + +:::sequence animate | A reload of the notes plugin on the worker. Nothing is visible to callers until the commit. +actor app "base code" "reload control" +actor host "worker W" "" +actor objs "notes objects" "live" +app -> host: reload([notes v2], ctx) +host -> host: load bundles +host -> host: staging: setups + new factories (recording self) +host => host: commit: types, registries, contributions swapped; publication +host => objs: each follows its fate +objs --> app: resolved: { reloaded, restarted, ended, failed } +::: + +:::review | Open 1: the API +Today a host has `load(facets, ctx)`, once. Proposal: `host.reload(facets, ctx)` with the plugins' new facets (by +plugin name: added, changed; a plugin missing from the list is removed), resolving with a report of what each object +did, rejecting with the code error and nothing changed. `load` stays the first call. Alternative: one +`host.load` that may be called again. Recommended: the separate `reload`, because its result differs (a report, not a +barrier) and its rollback rule does not apply to the first load (a throwing first setup closes the host). +::: + +# Three Fates {#fates lede="What happens to each live object at the commit."} + +The commit compares each live object's old and new declarations. **Decided** (design §21, owner 2026-10-26, the +dependency model). + +| The object | Fate | Calls | Documents, jobs | +| --- | --- | --- | --- | +| pure, old and new (no `self.open`) | **swapped in place**: new methods, derives and routines; references kept, new ones made | running calls finish on old handles; new calls see new code | stay | +| has natives, old or new | **restarted**: new calls park, running calls drain (`closeTimeout`), natives close (`reload`), the new factory runs, natives reopen | park during the restart | stay; running job runs end `aborted { reload }`, `finish` runs on new code | +| its type is gone | **ended**: natives close (`end`), callers rejected, handlers drain | rejected | in-memory gone; persisted stay until retired | +| a new dependency is missing (any of the above) | **parks** (decision 4): gate closed, `failed { waiting }` | reject | stay | + +```ts +// notes v1 // notes v2: pure, swapped in place +host.serve(NoteStore, (_, self) => { host.serve(NoteStore, (_, self) => { + const notes = self.state(Notes); const notes = self.state(Notes); // same document, v2 migrates it + return { const count = self.derive([notes], ([n]) => n.list.length); + notes, return { + add: (a, ctx) => self.tx(…, ctx), notes, count, // a new exposed member + }; add: (a, ctx) => self.tx(…, ctx), +}); rename: (a, ctx) => self.tx(…, ctx), // a new method + }; + }); +``` + +Rules that come with the fates, all **decided**: + +- **R15, old code and migrated documents:** an old call that touches a document whose version rose throws once; the + document stays intact; the call is not resent. +- **Derives** have no names: exposed derives are matched by their member name, internal ones are rebuilt. +- **Routines** hand over per object: the old ones are aborted, the new ones start once the old calls finished, bounded + by `closeTimeout`; calls do not wait for it. +- **Instance ids:** a restarted object gets a new instance id (an old activation chain must not match it); a swapped one + keeps it. +- **Broken objects** of a reloaded plugin are retried, as `ref.retry()` does. +- **A parked object** of a reloaded plugin is replaced by the new factory's attempt in one handover: the new requests + bind before the old ones are released (the mechanism 7a removed comes back here). +- **In-memory documents** migrate on reload through their definition's `migrate` (6a built the version check). + +:::review | Open 2: a `writable` across a pure swap +A pure swap runs the new factory, so `self.writable(initial)` makes a new writable at its initial value. An object that +contributes MCP's tool list through a writable then contributes an empty list until its restarted routine sets it again: +the tools vanish for a moment. Options: (a) accept it (the routine refills it within one call); (b) keep a writable's +value across a swap when the new factory declares a writable at the same position (as members are matched by name). +Recommended: (a); writables have no names, and position matching would be the first positional rule in Chord. +::: + +# Consumers Across a Reload {#consumers lede="Holds, versions and old calls; 7a decided that nothing is ended for a code change."} + +A reference on another host points at an object whose code may change under it. **Decided** (design §21, §22 and the +dependency model): + +| The provider's reload | The consumer's hold | It sees | +| --- | --- | --- | +| still accepts the hold's version, object swapped | kept; re-subscribed through the version's mappings | one `[["r", value]]` per changed member | +| still accepts it, object restarted | kept | `loading`, then a fresh `bound` | +| stops accepting the version | unbound `version` | `failed { waiting }`; it binds again if a later reload accepts it | +| removes the type | unbound `not_served` | `failed { waiting }`; it binds again when the type comes back | + +A provider accepts older consumers through `accepts`, written once with the provider, never in the shared contract +(Q10): + +```ts +export const NotesV2 = evolve(NotesV1, (e) => e.state("editor", "notes", arr(NoteV2)).args("add", NoteV2)); + +host.serve(NotesV2, factory, { + accepts: { + 1: { + state: { editor: { notes: (current) => current.map(toV1) } }, // what a v1 tab sees, as a diff per commit + methods: { add: { args: (old) => toV2(old) } }, // a v1 tab's call, mapped forward + }, + }, +}); +``` + +- **Old-version calls:** `checkArgs` returns `{ kind: "method", args, result? }` or `{ kind: "call", args, call }`; Chord + runs the handler or the provider's replacement and maps the result back (L1 built the check; 7b invokes it). +- **Adapted bindings get diffs:** Chord keeps the last adapted value per binding and sends `diffRevisions`, so an old + tab never receives a whole state per commit. +- **On the wire** (7b's half of 7a's model): a `failed` on a bound hold unbinds it; the provider drops its unsent + pending work and keeps the ack slots of messages already sent; a consumer whose own code moves to a new version + releases its old hold and makes a new one. +- **A consumer reloaded itself** gets new requests from its new factory; the handover binds them before it releases the + old. + +Today `host.serve` binds every provider with an empty accepts table (`host.ts`: `bindProvider(contract, {})`); 7b passes +the provider's table. Small and **decided** in shape (Q10, `bindProvider`); the option's place is `host.serve(Type, +factory, { accepts, linger })`. + +# Jobs Across a Reload {#jobs lede="Runs belong to the object; their versions are mapped forward."} + +**Decided** (design §15, owner 2026-10-24 and 2026-10-26): + +```ts +const ExportJob = defineJob("notes.export", { + version: 2, + args: obj({ format: oneOf("md", "html") }), + result: obj({ path: str() }), + accepts: { 1: { args: (old) => ({ format: "md" }), result: (old) => ({ path: old.file }) } }, +}); +``` + +- A run started on an older version finishes on its old code across a pure swap; `finish` and `running` see its args + and result mapped forwards, once per run. +- Without an adapter, or for a run of a newer version (a rollback), the run is `blocked { version }` until a reload + brings matching code; blocked runs are retried on every reload. Nothing is aborted or discarded for a version change. +- A native restart aborts running runs (`aborted { reload }`) and `finish` runs on the new code. +- A job the new code no longer declares: its runs are `blocked { version }` (the same family), kept until a reload + declares it again or they are aborted. +- Persisted runs of another version or fingerprint stay blocked with their record as stored, resolved by the reload + that brings their version. + +# Registries Across a Reload {#registries lede="A provider and its contributions are swapped in one batch."} + +**Decided** (slice 8 §6, design §17): the commit replaces a plugin's contributions and a registry's provider in one +batch; staging composes nothing (D1). The record keeps the provider, facet contributions by plugin and object +contributions by object apart, so a reload of one plugin replaces exactly its parts (today `RegistryRecord.serve` +refuses a second serve). Older readers across a reload of the provider follow its new `accepts[old].value`. A reader +waits (7a) when the new provider no longer serves its version. `retryOn` watches are re-declared by the new factory. + +# Across Hosts {#rollout lede="The same reload on every host, in an order that cannot deadlock."} + +**Decided** (design §21 and the dependency model): a revision is the complete code set; the server installs first, +then workers, then clients; installs are serialized per host; a host that starts or reconnects installs the current +revision first; a revision carries a code-set digest so an unchanged server restart reinstalls nothing; an install +never waits for activations (a reverse-order rollout would deadlock); each host knows only its own install: while it +installs, holds that do not fit its running code wait `installing`, re-evaluated when the install ends; a host whose +own install was rejected fails its own requests with the reason. + +:::steps | The notes contract goes from v1 to v2; the server installs first ++++ Before +S, W and T run revision R. T's notes view holds W's `Notes` v1. All live. ++++ S installs R+1 +S has no notes objects of its own; its install commits. T, still on R, holds nothing on S that changed. ++++ W installs R+1 with `accepts[1]` +W's `Notes` v2 swaps in place and still accepts v1: T's hold is re-subscribed through the mapping and gets one diff. ++++ W installs R+1 without `accepts[1]` +T's hold becomes unbound `version`: `failed { waiting }`, last notes readable, calls reject with "update the page". ++++ T reloads its page +T's new code holds v2 with a new hold id; it binds; the old hold is released. +::: + +:::review | Open 3: who runs the rollout +Chord's part is per host: `host.reload`, the `installing` state, and what a hold sees. The rest is distribution: +building bundles, sending them to workers, telling tabs to reload, ordering, the digest. Options: (a) Chord ships a +rollout protocol over its links (an `install` message from the server down, `installed`/`rejected` up, the digest in +`hello`); (b) distribution is the app's (p3's base code calls `host.reload` on each host and orders them), as p2 does +today (`apps/p2/src/hosts/server.ts` `performReload`: prepare every host, commit, publish bundles). Recommended: (b) for +7b, with Chord's per-host guarantees; a protocol only if a second app needs the same code. +::: + +:::review | Open 4: the install watchdog (design §26 #24) +An install that hangs (a bundle that never loads, a staging factory that awaits forever, which it may not) leaves its +holds `installing` forever. Options: (a) a watchdog: `host.reload` rejects after `installTimeout` (default 60 s) and the +host stays on its old code; (b) no Chord timer: the app's context bounds the reload. Recommended: (b): `reload` takes a +context like everything else, and the app passes its deadline; a host option would be the only timer left in reload. +::: + +# Loader D {#loader lede="How a plugin's new code gets into a running process."} + +p2 designed it (`apps/p2/docs/plugin-loading.md` §9, §14); Chord has not built it. **Decided in p2's design:** + +- **One base bundle per surface** holds every host package once: Chord, Durable, pi-ai, pi-tui, TypeBox, React and + `react-dom`, plus the app's runtime. A change to it restarts the hosts. +- **One script per facet**, an esbuild bundle wrapped as a function of its host packages, which are external and + passed in, so there is one Chord and one React per process: + + ```js + // notes/worker.facet.js, built + (require, chord, durable, react) => { + const { defineFacet, local, str } = chord; // the host's instance, not a copy + /* the plugin's own files, contracts included (bundled: their versions are the plugin's) */ + return defineFacet((host, self) => { … }); + } + ``` + +- **Evaluation:** `vm.Script.runInThisContext` on Node (server, workers, TUI worker), a classic ` + From adfa8f3827e92947420d7c85458ff80566759d5e Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Thu, 29 Oct 2026 12:15:57 +0000 Subject: [PATCH 026/175] docs(chord): slice 7b brief: plugin layout, building and loading, assets, HTTP, the app kit, measurements --- packages/chord/docs/next/slice-7b/book.md | 379 ++++++++++++++-- packages/chord/docs/next/slice-7b/index.html | 443 ++++++++++++++++--- 2 files changed, 728 insertions(+), 94 deletions(-) diff --git a/packages/chord/docs/next/slice-7b/book.md b/packages/chord/docs/next/slice-7b/book.md index bda87641d..8e2566b9c 100644 --- a/packages/chord/docs/next/slice-7b/book.md +++ b/packages/chord/docs/next/slice-7b/book.md @@ -2,12 +2,13 @@ title: Slice 7b: reloading plugins in a running Chord app cover: Slice 7b / Reload kicker: A design brief · Chord v2 · slice 7b · 2026-10-29 -subtitle: What happens to objects, references, jobs and registries when a plugin's code changes on one host or across many, what is decided and what the owner still decides +subtitle: What a plugin is, how it is built and loaded, how a mini p2 is assembled from Chord, and what happens when a plugin's code changes on one host or across many stats: - 4 | steps of a reload on one host - 3 | fates of a live object - 26 | carry-over items from slices 3a to 7a - - 5 | decisions open + - 11 ms | to rebuild one facet after an edit + - 6 | decisions open note: | A brief before the slice 7b spec, written by the redesign session from `design.md` §21, the dependency model (`dependencies/model.md`, decided and built for 7a in pi `339847eb1`), the carry-over in `slices.md` and the code at @@ -26,7 +27,7 @@ Chord v2 can do everything around that today: objects, references across hosts, 7a a dependency model in which a missing provider is a wait, not an error. What it cannot do yet is change code. `load` runs once per host (`host.ts`: "load runs once per host"). Slice 7b adds the reload, on one host and across hosts. -## Six things to know +## Eight things to know :::findings 1. **A reload is a transaction per host.** Load the bundles, stage the new declarations, commit atomically, then let @@ -39,8 +40,12 @@ runs once per host (`host.ts`: "load runs once per host"). Slice 7b adds the rel not fit waits (7a), it is not ended. 5. **7a already decided most of the hard part.** Removal, dropped versions and installs are waits woken by a publication; a reload's commit is such a publication. -6. **Five points are open:** the reload API, the rollout across hosts, loader D, a `writable` across a swap, and the - install watchdog. +6. **A plugin is a directory with a `package.json` and one file per surface.** Each facet is bundled alone; host + packages are passed in, never copied; a whole plugin set builds in a few hundred milliseconds. +7. **Chord should ship the app kit** (proposed): loader, HTTP, workers, web, TUI and dev reload as optional + components, so a mini p2 is a page of code. +8. **Six points are open:** the reload API, the rollout across hosts, a `writable` across a swap, the install + watchdog, where the loader lives, and the app kit. ::: ## Words used throughout @@ -51,6 +56,332 @@ A **reload** changes the code of one or more plugins on one host. A **facet** is in 7a's sense. A **pure** object has no natives (`self.open`), in its old or its new code. A **rollout** is a reload of the same code set on every host, server first. +# A Plugin, File by File {#plugin lede="What a plugin looks like on disk, and what each file is for."} + +A plugin is a directory. Its name is its `package.json` name; its code is one file per host kind it runs on, a +**facet**; everything else is ordinary TypeScript the facets import. This is p2's layout today (`apps/p2/plugins/*`), +plus the `package.json` that loader D needs (p2's `plugin-loading.md` §8; p2's built-in plugins have none yet and are +named by their directory). + +```text +plugins/notes/ + package.json name = the plugin's name; peerDependencies = host packages; dependencies = bundled npm deps + contract.ts the plugin's object types and registries (shared by its facets and by other plugins) + server.ts facet for the server (default export: defineFacet(...)) + worker.ts facet for session workers + web.tsx facet for browser tabs + tui.ts facet for the TUI + store.ts plain modules the facets import + NoteList.tsx + notes.css + assets/ + template.md files the plugin ships (read at runtime, or offered for download) + icon.svg + skills/notes/SKILL.md optional resources (p2 §4.4) +``` + +```json +{ + "name": "@pi/notes", + "version": "1.4.0", + "type": "module", + "peerDependencies": { + "@earendil-works/chord": "^2", + "@earendil-works/pi-durable": "^1", + "react": "^19" + }, + "dependencies": { + "marked": "16.4.1" + } +} +``` + +- **`name`** identifies the plugin everywhere: its facets, its registry contributions, diagnostics, reloads. +- **`peerDependencies`** are **host packages**: the running app supplies them, one instance per process (one Chord, + one React; `instanceof ChordError` and React hooks only work that way). The bundler never copies them. +- **`dependencies`** are the plugin's own npm packages (`marked`): installed into the plugin's `node_modules` with + `--ignore-scripts` and bundled into every facet that imports them. +- **No `tsconfig.json` is needed.** The bundler strips types and does not type-check; JSX is compiled with the + automatic React runtime, a fixed setting of the loader. A plugin may keep a `tsconfig.json` for its editor and for + `tsc --noEmit` (the SDK ships one to extend); the loader ignores it. + +A facet only declares (chapter III of the model): it serves object types, serves or contributes to registries, and +returns nothing. + +```ts +// plugins/notes/contract.ts +import { defineRegistry, local, remote, str, obj, arr, method } from "@earendil-works/chord/next"; +import { Session } from "@app/core"; // the app's host kinds, a host package +export const NoteStore = local("notes.store", { version: 1, params: {}, members: { /* … */ } }); +export const Notes = remote("notes", { + version: 2, + params: { session: str() }, + hostAddress: ({ session }) => Session({ id: session }), + variants: { editor: { state: { list: arr(obj({ id: str(), text: str() })) }, methods: { add: method(obj({ text: str() })) } } }, +}); + +// plugins/notes/worker.ts +import { defineFacet } from "@app/core"; // bound to the app's principal type +import { NoteStore, Notes } from "./contract.ts"; +import { serveStore } from "./store.ts"; +export default defineFacet((host) => { + host.serve(NoteStore, serveStore); + host.serve(Notes, ({ session }, principal, self) => { /* … */ }); +}); + +// plugins/notes/web.tsx +import { defineFacet } from "@app/core"; +import { NoteList } from "./NoteList.tsx"; // a React component, bundled +import "./notes.css"; // bundled into the facet's stylesheet +export default defineFacet((host, self) => { + self.contribute(Panels, { notes: NoteList }); // the app's UI registry +}); +``` + +# Building and Loading {#loading lede="How the bundler knows what to pull in, and how the code gets into a running process."} + +Each facet is built on its own, as one script. The bundler starts at the facet file and follows its imports: + +| An import of | Becomes | +| --- | --- | +| the plugin's own files (`./store.ts`, `./NoteList.tsx`) | inlined into the facet's script | +| another plugin's contract (`../sessions/contract.ts`, or `@pi/sessions/contract` for an installed package) | inlined too: a copy, so a plugin carries the contract version it was built against (`accepts` needs that); only contracts and side-effect-free modules may be imported across plugins | +| an npm package in `dependencies` (`marked`) | inlined, resolved from the plugin's own `node_modules` | +| a host package (`@earendil-works/chord/next`, `react`, `@app/core`, …) | **external**: a parameter of the script, the host's instance at load time | +| a Node builtin (`node:fs`) | external, through the `require` the loader passes (server and worker facets only; a web facet that imports one fails to build) | +| CSS (`import "./notes.css"`) | collected into one stylesheet per web facet | +| an asset by URL (`import iconUrl from "./assets/icon.svg?url"`) | a content-hashed file served by the app; the import is its URL | + +The built script is a function of its host packages. That is the whole trick of loader D: there is no module map, no +second copy of Chord, and an old version is garbage-collected once nothing holds its closures. + +```js +// what the bundler emits for plugins/notes/worker.ts (simplified) +(function (require, __files) { + const { defineFacet } = require("@app/core"); // the host's instance + const { local, remote, str } = require("@earendil-works/chord/next"); + const { marked } = /* marked's code, inlined */; + /* contract.ts, store.ts, inlined */ + return defineFacet((host) => { /* … */ }); +}) +``` + +```ts +// the loader (proposed @earendil-works/chord/next/loader), Node side +import { buildFacet, evaluateFacet } from "@earendil-works/chord/next/loader"; + +const HOST_PACKAGES = { // the app's base bundle exports these + "@earendil-works/chord/next": chordNext, "@earendil-works/pi-durable": durable, + "@app/core": core, react, "react/jsx-runtime": jsxRuntime, +}; +const script = await buildFacet({ plugin: "/data/plugins/notes", facet: "worker.ts", hostPackages: Object.keys(HOST_PACKAGES), platform: "node" }); +const facet = evaluateFacet(script, { hostPackages: HOST_PACKAGES, require: createRequire(import.meta.url), dir: "/data/plugins/notes" }); +await host.reload([{ name: "@pi/notes", facet }], ctx); // or host.load the first time +``` + +- `buildFacet` is esbuild with: the facet as entry, the host packages and `node:*` external, `format: "cjs"` wrapped + as the function above, minified, an `onResolve` that refuses anything outside the plugin, its `node_modules`, a + contract of another plugin or a host package. It returns the script, its content hash and esbuild's metafile (the + dev loop knows which files a facet used). +- `evaluateFacet` runs the script with `vm.Script` and a stable filename per (facet, hash) on Node, or as a classic + ` - diff --git a/packages/chord/docs/next/explainer/src/00-front.md b/packages/chord/docs/next/explainer/src/00-front.md deleted file mode 100644 index 36d62e8d0..000000000 --- a/packages/chord/docs/next/explainer/src/00-front.md +++ /dev/null @@ -1,111 +0,0 @@ ---- -title: Chord v2: a guide to building apps that reload -cover: Chord v2 / A Guide -kicker: A guide · Chord v2 target design · 2026-10-20 -subtitle: How to build an app like p2 from nothing on Chord v2 and pi-durable, one object at a time, so that new plugin code takes over while the agent keeps running -stats: - - 11 | layers in the rebuild, L1 to L11 - - ~1,300 | lines built so far (L1 contracts) - - 6–7k | target size of Chord v2, without tests - - 0.004 ms | Delta's apply per small change -note: | - This guide teaches the **designed** API of Chord v2 (`packages/chord/src/next`, export `@earendil-works/chord/next`) - from `packages/chord/docs/next/design.md` and the newer decision log `packages/chord/docs/next/decisions.md`, which wins - where the two differ. Most of it is **not built yet**: a first kernel of about 10k lines is frozen as a reference and - Chord v2 is being rebuilt layer by layer; only the L1 contract layer exists (`src/next/contracts/`, commit - `7d2bb8ea8`). Code in this guide is therefore the target, not something you can run today. Every chapter says what is - built. API names are working names. Notes marked *review* are points that are undecided, or where the sources - disagree; they name the files. The reader is assumed to know pi-durable; Chord is explained from the start. -colophon: Chord v2 · a guide · first edition · 2026-10-20 · based on design.md (target design v2), redesign-decisions.md (review rounds 2 and 3), chord-next-walkthrough.md, layer-1.md, contracts/README.md, the p3 functionality map, questions.md and the pi-durable README ---- - -# Before we begin {.preface} - -It is late in the evening. Mario is on his phone, connected to his laptop at home through the Radius relay. In session -S1 the agent is twenty minutes into a refactoring; right now it is running a test suite through `bash`. A colleague, -Ann, follows the session read-only from her own browser. Mario notices that the notes panel sorts notes the wrong way -round. He fixes one line in the notes plugin on the laptop and presses **Reload**. - -What should happen is easy to say. The notes panel on both screens shows the new order. The test run keeps running. The -agent's turn continues as if nothing happened. Ann's view does not flicker into an error. Nothing that was written is -lost, and no process restarts. - -The obvious ways to get there fail. **Restarting the processes** loses the turn in flight, the open terminal and every -connection; pi-durable would resume the turn, but the test run and the connections are gone. **Hot-swapping modules** -in place, as front-end tooling does, leaves the old code alive in closures, timers, event listeners and sockets that -nobody knows about: the old sort function keeps running inside a callback someone stored, a file watcher opened by the -old code is never closed, and two versions of the plugin now write the same data without any order. - -Chord v2 is the runtime that makes the first outcome normal. It does so by being strict about one thing: **every piece -of state, every open OS handle and every piece of running work has exactly one owner, an object, and objects only -declare what they need.** Chord itself activates, connects, reloads and ends them. Because Chord knows every timer, -watcher, transaction and reference an object has, it can swap the object's code while its data stays, restart only -what touches the operating system, and route calls between browser tabs, the server and per-session worker processes -without the plugin author writing any of that. - -This guide teaches Chord v2 by building an app called **Studio**: a small version of p2 (Pi Mobile), with a server, -one worker process per session, browser tabs as clients, and an agent from pi-durable on each worker. The first -chapters grow one plugin, `@studio/notes`, until it uses every concept; the last three chapters assemble the rest of a -p2-like app from the same parts. - -:::review | Review: what exists today -Nothing in this guide runs yet except the contract layer. `packages/chord/src/next/contracts/` (L1 part 1: `remote`, -`local`, `method`, `evolve`, `checkAccepts`, the schema helpers) is built and reviewed. The rest of L1 (inputs, -argument checks, principals, errors) is a draft spec (`packages/chord/docs/next/layer-1.md`). The frozen kernel in -`src/next/*.ts` implements an older version of the design (host values, a development freeze, JSONL storage, -descriptions on the wire); do not read it as the target. Build on the design, not on the frozen code. -::: - -## Ten things to know - -:::findings -1. **An app is a tree of hosts.** One server, any number of workers it starts on demand, any number of clients (tabs, - TUIs) that connect to it. Clients reach workers only through the server, which relays. -2. **Objects are the unit of everything.** An object type is a contract (name, input, placement, members, version); an - object is one instance per input and caller. The first `get` creates it, later gets share it, the last release ends - it. -3. **Factories declare, Chord activates.** A factory is synchronous: it declares documents, references, derived - values, OS handles, routines and jobs through `self`, and returns the members. Chord opens, binds and starts them, - and undoes them in reverse. -4. **Links stamp who is calling.** A caller's data never says who it is; the link it arrived on does. Each caller gets - its own object and the variant of members it may use. -5. **Shared data lives once, behind views.** A `local` shared store holds the data; per-caller `remote` views expose - what each caller may see. Only stable identities persist documents. -6. **State changes only in transactions.** Documents are JSON values written in `self.tx`; commits are atomic for - every viewer and travel as small operations, never as whole values. -7. **Failures look like disconnects.** A reference survives link drops, restarts and failures: calls park or fail, - values stay readable, and Chord retries by itself. There are no request ids. -8. **Everything long-lived has a declared form.** Standing behaviour is a routine, work that outlives a call is a job, - input from a person is a flow, behaviour from other plugins comes through a registry. Nothing else may hold a timer - or a callback. -9. **Reload swaps pure objects in place and restarts native ones.** Objects without OS handles get the new code with - their data and running work intact; objects with OS handles close and reopen them. A broken change leaves the whole - host on its old code. -10. **pi-durable runs inside a core object.** The agent harness is a host-lifetime object on each worker; models, - environments and tools reach it through registries, and tools reach Chord through `chord(fn, context)`. -::: - -## Words used throughout - -A **host** is one participant of the app: a process, a browser tab or a Durable Object, identified by a logical -identity and an **incarnation** that changes on every start. A **link** connects two hosts. The **server** is the one -addressable root host; a **worker** is a host the server starts for one session; a **client** is a tab or TUI, not -addressable. A **plugin** is a directory with a `package.json`; its code for one kind of host is a **facet**. **Base -code** is the app's own code (Chord calls its facet the core facet); it restarts the hosts when it changes, while -plugins reload. An **object type** (or **contract**) describes objects; an **object** is one instance; its **input** is -its key among objects of that type, and its **placement** computes from the input on which host it lives. A **factory** builds an -object; **`self`** is what a factory declares through. A **principal** is who calls, stamped by the link; a -**variant** is the set of members a principal gets. A **reference** is what `get` returns. A **readable** is a value -plus its ordered changes. A **document** is a named JSON value an object owns, written in a **transaction**. A -**derive** is a value computed from readables. A **native** is an OS-level value (a socket, a watcher, a process) -opened with `self.open`; an object with natives is a **native object**, one without is a **pure object**. A -**routine** is standing behaviour, a **job** is owned work that outlives its call, a **flow** is input into running -work. A **registry** is an owner's extension point, filled with **contributions**. **`ctx`** is what every call -receives: cancellation, caller, call-scoped references. A **reload** swaps a plugin's code inside running hosts; a -**restart** ends a process; a **revision** names the complete code set the hosts run. - -:::note | Reading the code in this guide -Imports are shown once per file and then omitted. `obj`, `str`, `arr` and friends are Chord's schema helpers (they -return ordinary TypeBox schemas). Helper functions that belong to the app (`authenticate`, `watchDir`, `uuidv7`) are -named and not shown. Where the design gives no name, the guide says it uses a working name. -::: diff --git a/packages/chord/docs/next/explainer/src/01-shape.md b/packages/chord/docs/next/explainer/src/01-shape.md deleted file mode 100644 index db18b5cfe..000000000 --- a/packages/chord/docs/next/explainer/src/01-shape.md +++ /dev/null @@ -1,246 +0,0 @@ - -# The Shape of an App {#shape lede="Which processes a Chord app has, how they find each other, and the start code that connects them."} - -Before any plugin exists, Studio needs processes and connections between them. In Chord each participant is a host, -and the app's own start code creates every host, connects them and decides how workers start. Chord deliberately does -none of that itself: a laptop app forks child processes, a test runs everything in one process, a Cloudflare -deployment routes through the edge. This chapter writes Studio's start code: `server.ts`, `worker.ts`, the browser -entry, and a test setup. - -## Three roles - -A Node app on Chord has one shape: - -:::flow wide | The three roles. Clients connect to the one server; the server starts workers and relays between them. -row "Clients: not addressable" muted -box tab1 "browser tab" "owner, local" -box tab2 "browser tab" "guest Ann" -box tui "TUI" "owner" -row "Server: exactly one" accent -gap -box server "server" "`Server()`" accent -gap -row "Workers: one per session, on demand" blue -box w1 "worker" "`Session({ id: \"S1\" })`" blue -box w2 "worker" "`Session({ id: \"S2\" })`" blue -box w3 "worker" "`Session({ id: \"S3\" })`" dashed -arrow tab1 -> server "WebSocket" -arrow tab2 -> server "WebSocket" -arrow tui -> server "WebSocket" -arrow server -> w1 "IPC" -arrow server -> w2 "IPC" -note "A worker for S3 does not exist until something asks for an object that lives there." -::: - -| Role | Addressable | Who it is comes from | Lifecycle | -| --- | --- | --- | --- | -| client | no | the authentication of its link | connects by itself | -| server | yes, exactly one | its own declaration | runs | -| worker | yes, many, by a typed address | stamped by the server, its parent | started on demand by the app | - -The **server** holds what all sessions share (the session list, access, settings) and starts workers. A **worker** -runs one session: its agent, its files, its terminal connections. **Clients** are browser tabs and TUIs. Nobody can -send a request *to* a client, so a client can only serve objects that nobody outside it needs; chapter IV names them -`local` types. - -Things that look like hosts but are not: a code mode sandbox (a worker thread inside a worker, owned by an object), a -remote environment's daemon (it speaks its own protocol, owned by an object), the TUI's sandbox behind its launcher. -A detached process such as a terminal host could become another kind of child of the server. - -## Host kinds and addresses - -Every addressable host has a typed address. The app declares its **host kinds** once, in base code: - -```ts -// studio/src/hosts.ts -import { hostKind, obj, str } from "@earendil-works/chord/next"; - -export const Server = hostKind("server"); -export const Session = hostKind("session", obj({ id: str() })); - -Server(); // the address of the server -Session({ id: "S1" }); // the address of the worker of session S1 -``` - -There are no host-name strings anywhere in app code. Chord serializes addresses internally; your code only calls the -kinds. An object type names its host with a placement function from its input to an address (chapter II), so the -type checker knows that `Session({ id: session })` needs a `session` field. - -## Links - -A **link** is an ordered, two-way channel of messages between two hosts; it only delivers. Chord ships three: a memory -pair for tests, IPC for a Node child process forked with `serialization: "advanced"`, and WebSocket. All three carry -the same binary message format and the same flow control (chapter XVII). Recovery policy belongs to the link adapter: -IPC needs no heartbeat, a WebSocket through a relay needs one, because TCP does not notice that a laptop went to sleep. - -Each link has a **direction**. A host has at most one link to its parent (`connect`) and any number of links from -below (`accept` for clients, `route` for workers). Requests arriving on a link act as the principal the link was given; -chapter III explains why that matters. - -## The server - -```ts -// studio/src/server.ts -import { createHost, ipcLink, webSocketLink } from "@earendil-works/chord/next"; -import { fork } from "node:child_process"; -import { WebSocketServer } from "ws"; -import { Server, Session } from "./hosts.ts"; -import { coreServerFacet } from "./core/server.ts"; -import type { P } from "./principal.ts"; - -const server = createHost

    ({ - address: Server(), - principal: { kind: "server" }, // who this host's own code acts as - storage: await openServerStorage(dataDir), // persisted documents (chapter V); memory if omitted - onError: (e) => { if (e.kind === "storage_failed") process.exit(75); else log(e); }, -}); - -await server.load([ - { name: "@studio/core", facet: coreServerFacet(config) }, // base code, under a fixed name - ...(await loadPlugins("server")), // [{ name, facet }] from the plugin loader -]); - -const wss = new WebSocketServer({ server: http, path: "/chord", maxPayload: 64 * 1024 * 1024 }); -wss.on("connection", (ws, req) => { - const principal = authenticate(req); // app code: cookie, loopback, device token - if (!principal) return ws.close(4401); - server.accept(webSocketLink(ws), { principal }); -}); - -server.route(Session, { - authorize: (p, { id }) => - p.kind === "owner" || p.kind === "server" || (p.kind === "guest" && p.session === id), - resolve: async ({ id }, { signal }) => { - const principal: P = { kind: "worker", session: id }; - const child = fork("worker.ts", [], { serialization: "advanced", signal }); - child.send({ id, principal, init: await initFor(id, signal) }); // the app's own start message - return { link: ipcLink(child), principal }; // Chord stamps `principal` on it - }, -}); -``` - -Each call has one job: - -| Call | Meaning | -| --- | --- | -| `createHost

    ({ address?, principal?, ...options })` | with an address: an addressable host whose own code acts as `principal`; without: a client. `P` is the app's principal type | -| `load([{ name, facet }])` | the facets this host runs, each with its plugin's `package.json` name; duplicates and invalid names are refused. Base code passes its core facet the same way | -| `accept(link, { principal })` | a link from a client; every request on it acts as `principal` | -| `route(Kind, { authorize, resolve })` | how to reach hosts of a kind. `authorize(principal, address)` runs before anything starts; `resolve(address, { signal })` returns `{ link, principal }` | -| `connect(link, { principal })` | the link to the parent, at most one, replaced after a reconnect; `principal` says who the parent is | -| `root(fn, { principal?, signal? })` | a call scope for code outside objects (a CLI command): gives `fn` a `ctx` (chapter VI) | - -Chord has no `listen`. Entry points differ too much: a loopback WebSocket server, a tunnel, the Radius relay -delivering channels, a Durable Object edge, a memory link in a test. Each entry point authenticates and calls -`accept`. Radius, for example, is one more line: - -```ts -radius.onChannel((ws) => - server.accept(webSocketLink(ws), { principal: { kind: "owner", access: "relay" } })); -``` - -## The worker - -```ts -// studio/src/worker.ts -const { id, principal, init } = await startMessage(process); // app code: the first IPC message -const worker = createHost

    ({ - address: Session({ id }), - principal, // { kind: "worker", session: id } - storage: await openSessionStorage(init.dataDir, id), // one storage per session -}); -await worker.load([ - { name: "@studio/core", facet: coreWorkerFacet(init) }, // serves the start data (chapter IX) - ...(await loadPlugins("worker")), -]); -worker.connect(ipcLink(process), { principal: { kind: "server" } }); -``` - -## A browser tab - -```ts -// studio/web/main.tsx -const client = createHost

    (); // no address: a client -await client.load(await loadPlugins("web")); -client.connect(webSocketLink(new WebSocket("/chord")), { principal: { kind: "server" } }); -const notes = await client.get(Notes, { session: "S1" }); // tab → server → worker S1 -``` - -## The first `get`, step by step - -The tab asks for the notes of S1. No worker for S1 runs yet. - -:::sequence animate | The first get for an object on a worker that is not running. The server authorizes, the app starts the worker, Chord stamps the link. -actor tab "browser tab" "owner" -actor server "server" accent -actor app "route(Session)" "app code" muted -actor worker "worker S1" blue -tab -> tab: placement: Session({ id: "S1" }) -tab -> server: hold Notes { session: "S1" } -server -> server: not mine: route(Session) -server -> app: authorize(owner, { id: "S1" }) -app --> server: true -server => app: resolve({ id: "S1" }) -app -> worker: fork + start message -worker -> worker: createHost, load, connect -app --> server: { link, principal: worker S1 } -server => worker: hold (stamped: owner) -worker -> worker: placement check, run factory -worker --> server: bound: snapshot -server --> tab: bound: snapshot -::: - -`resolve` runs once per address while its link lives. When the link drops (the worker crashed or the supervisor -stopped it) and references to that worker still exist, Chord calls `resolve` again by itself, with a per-address -backoff, so a tab that only watches recovers without a click. A worker start that already produced a link finishes -even if the request that triggered it went away. - -**Incarnations** make restarts visible. Chord mints a fresh id (a uuidv7) on every host start and sends it in the link -handshake; the other side sees a restart as a new incarnation of the same identity, and references to it re-hold -(chapter VI). - -## Tests: everything in one process - -The same code runs in a test with memory links. `resolve` builds the worker host in memory and returns one end of the -pair: - -```ts -server.route(Session, { - authorize: () => true, - resolve: async ({ id }) => { - const [up, down] = memoryLinkPair(); - const principal: P = { kind: "worker", session: id }; - const worker = createHost

    ({ address: Session({ id }), principal }); - await worker.load([{ name: "@studio/core", facet: coreWorkerFacet({ id, dataDir }) }, notesWorker]); - worker.connect(up, { principal: { kind: "server" } }); - return { link: down, principal }; - }, -}); -``` - -Memory links deliver the same encoded messages asynchronously, so a test runs the production protocol path. - -## What stays app code - -Starting processes, the start message, authentication, supervising and stopping idle workers are app code. Chord -provides one readable for the last one: `host.activity`, with `{ references, calls, jobs, lingering }` (viewers, calls -in flight, unsettled jobs, objects kept warm after their last reference). Studio's core reports it to the server -together with the agent's running tasks, and the server's supervisor stops a worker when everything has been zero for -five minutes, as p2 does. Chord counts; it has no idle policy. - -## Host options and timeouts - -`createHost` takes the options introduced later in this guide: `storage` (V), `principalKey` and `revokedBy` (III, -VII), `telemetry` (XVII) and `onError`. `onError` receives background errors and `{ kind: "storage_failed" }`; every -app must handle the latter, because Chord never ends the process. Studio exits with its own code so its supervisor -restarts it. - -Timeouts are host defaults with per-thing overrides; the full table is in the appendix. The two you meet first: -`readyTimeout` (60 s: how long an activation and an awaited `get` wait) and the park deadline (60 s: how long a call -waits through a link drop or a restart before failing `unavailable`). - -:::note | Built so far -Nothing in this chapter is built in Chord v2 yet: hosts, links and routes are layers L6 and L7. The frozen kernel has -working versions with the same shape (`createHost`, `connect`, `accept`, `route`, `load`), which the rebuild -simplifies. -::: diff --git a/packages/chord/docs/next/explainer/src/02-first.md b/packages/chord/docs/next/explainer/src/02-first.md deleted file mode 100644 index 5c01e5213..000000000 --- a/packages/chord/docs/next/explainer/src/02-first.md +++ /dev/null @@ -1,187 +0,0 @@ - -# A First Object {#first-object lede="A plugin, a contract, a factory, and a browser tab that calls it on another process."} - -Studio's first feature is session notes: a list of short notes per session, shown in a panel, that anyone allowed can -add to. Several places need "the notes of S1": the owner's tab, a guest's tab, later a download route and an agent -tool. If each built its own copy, the copies would drift apart and nobody would know when to clean them up. In Chord -there is **one object** for "the notes of S1", everyone asks for it by its type and input, and Chord routes them all to it. - -## The plugin's files - -A plugin is a directory with a `package.json`. Its `name` (npm style, globally unique and stable) is the plugin's -identity: Chord finds its stored documents, reports its errors and orders its contributions by it, never by a -directory name or load order. - -| File | Runs on | Holds | -| --- | --- | --- | -| `package.json` | read by the loader | `"name": "@studio/notes"` | -| `contract.ts` | every host that imports it | object types: names, shapes, method signatures; no running code | -| `worker.ts` | every session worker | the worker facet | -| `server.ts` | the server | the server facet (later: the download route) | -| `web.tsx` | browser tabs | the web facet: the panel | - -## The contract - -An object type is a contract: a name, an **input** (the object's key among objects of this type), a **placement** -(on which host it lives, computed from the input), the **members** (state you can read and methods you can call) and a **version**. - -```ts -// notes/contract.ts -import { arr, int, method, obj, remote, str, type Static } from "@earendil-works/chord/next"; -import { Session } from "@studio/core/hosts"; - -export const Note = obj({ id: str(), text: str({ max: 2000 }), by: str(), at: int() }); -export type Note = Static; - -export const Notes = remote("notes", { - version: 1, - input: obj({ session: str() }), // the key: { session: "S1" } - placement: ({ session }) => Session({ id: session }), // input → the host's address - variants: { - editor: { - state: { notes: obj({ items: arr(Note) }) }, - methods: { - add: method(obj({ text: str({ min: 1, max: 2000 }) }), str()), // returns the new id - remove: method(obj({ id: str() })), // no result - }, - }, - }, -}); -``` - -`remote` means objects of this type can be reached from any host, across links (chapter IV introduces `local`). The -schema helpers (`obj`, `str`, `int`, `arr`, `map`, `opt`, `lit`, `oneOf`, `union`, `bool`, `num`, `nul`, `bytes`, -`unknown`) return ordinary TypeBox schemas; short options such as `{ min, max }` bound lengths or values. Schemas must -describe data: no callbacks, no class instances, no custom codecs. - -`method()` has four forms: `method()` (no arguments, no result), `method(args)`, `method(args, result)` and -`method(undefined, result)`. - -A contract contains no code that runs, so every host can import it. When you declare it, Chord materializes its -description and computes a **fingerprint**, a hash of its input, variants and members. Two hosts compare contracts by -version and fingerprint (chapter XVI). Placement is not part of the fingerprint. - -## Placement - -The input says *which* object; `placement` says *where* it lives. The caller runs the placement function on the input and sends the resulting address with its request; the server -routes by it. The provider runs the function too and refuses an object that is not its own with `invalid/placement`. -So a worker for S2 never builds "Notes for S1", even if a caller has an outdated rule. Because placement is a typed -function, a `Session` placement on a type whose input has no `session` field does not compile. - -Inputs are **normalized** before anything else: defaults are applied, unknown fields removed, and a canonical key -computed. `{ session: "S1" }` and `{ session: "S1", extra: 1 }` are the same object. - -## The facet and the factory - -A **facet** is a plugin's setup for one kind of host. It runs once when the plugin loads on that host and tells the -host which object types it serves: - -```ts -// notes/worker.ts -import { defineState } from "@earendil-works/chord/next"; -import { defineFacet } from "@studio/core"; // Studio's defineFacet: principals typed as Studio's P (chapter III) -import { Note, Notes } from "./contract.ts"; - -const NoteList = defineState("notes.list", { // a document (chapter V); in memory for now - schema: obj({ items: arr(Note) }), - version: 1, - initial: () => ({ items: [] }), - checkpointWhen: () => true, -}); - -export default defineFacet((host, self) => { - host.serve(Notes, ({ session }, principal, self) => { - const list = self.state(NoteList); // THE note list of this object - return { - variant: "editor", - notes: list, // a document handle is a readable: expose it as state - add: ({ text }, ctx) => - self.tx(async (tx) => { - const id = uuidv7(); - (await tx.draft(list)).items.push({ id, text, by: "someone", at: Date.now() }); - return id; - }, ctx), - remove: ({ id }, ctx) => - list.change((d) => { d.items = d.items.filter((n) => n.id !== id); }, ctx), - }; - }); -}); -``` - -Read `host.serve(Notes, factory)` as "this host builds `Notes` objects like this". The factory receives the -normalized input, the caller's principal (chapter III) and `self`, through which it declares what the object needs. -It returns the members of one variant, tagged with `variant`. - -**Factories declare; Chord activates.** The factory is synchronous and has no side effects. It does not await, open -files or start timers. `self.state(NoteList)` does not load anything: it declares that the object has this document -and returns a **handle** that stays valid for the object's whole life. Chord then performs the activation: it loads the -document, binds references, opens OS handles, starts routines, and only then lets calls in. The rule exists for -reload: a reload runs the factory again, and anything the factory did itself would happen twice. - -## Calling it - -```tsx -// notes/web.tsx -export function NotesPanel({ session }: { session: string }) { - const notes = useRef(Notes, { session }); // app-side hook over host.get / release - const items = useReadable(notes?.notes)?.items ?? []; - return ( -

    - {items.map((n) =>

    {n.text}

    )} - notes?.add({ text })} /> -
    - ); -} -``` - -Underneath the hooks is the reference API: - -```ts -const notes = await client.get(Notes, { session: "S1" }); // live and hydrated once awaited -notes.notes.value.items; // the current list -notes.notes.subscribe((v) => render(v.items)); // every committed change -const id = await notes.add({ text: "check the flaky test" }); -notes.release(); // the object may end now -``` - -An awaited reference is **live and hydrated**: every state member has a value. State members are **readables**: -`.value` is the current value, `subscribe` is called after every commit. The first `get` for `{ session: "S1" }` runs -the factory; every later `get` for the same input and caller shares that object. The object ends when the last -reference is released (a type may keep it a little longer with `linger`). - -:::sequence animate | A call to add. Arguments are checked once at the provider; the commit reaches the viewer as one small change. -actor tab "browser tab" -actor server "server" muted -actor worker "worker S1" blue -actor obj "Notes { S1 }" accent -tab -> server: call add { text } -server -> worker: relay (header only) -worker -> obj: strip + validate args -obj => obj: self.tx: push note, commit -obj --> worker: result: id -worker --> server: state: ops of the commit -worker --> server: result -server --> tab: state, then result -::: - -## Two rules for members - -**Methods return data, never objects.** A method that wants to point you to another object returns its address -(`{ session: "S2" }`), and you `get` it yourself. That keeps every reference owned by someone who releases it. - -**No callbacks as arguments.** A callback carries code, and everything it closes over, from one object into another. -Suppose an inbox object passed `onFile` to another object, which stored it. After a reload of the inbox's plugin, the -other object still calls the *old* `onFile`; after the inbox ends, it calls into an object whose file watcher is -already closed. Chord forbids this by construction: members are data (JSON plus bytes). Behaviour from elsewhere goes -through registries (chapter XIII), multi-step work through jobs (XI), input from a person through flows (XII). - -## What is wrong with this version - -This first version works for one person. It has two problems, which the next two chapters fix. Everyone gets the -`editor` variant, including a read-only guest. And, less obvious, the owner and a guest do not see the same notes: as -chapter III shows, a remote object exists per caller, so each caller's `Notes` object has its own `NoteList`. - -:::note | Built so far -`remote`, `method`, the schema helpers, `Static` and fingerprints are built (`src/next/contracts/`). Serving, `get`, -references and documents are layers L3 to L5. -::: diff --git a/packages/chord/docs/next/explainer/src/03-principals.md b/packages/chord/docs/next/explainer/src/03-principals.md deleted file mode 100644 index 52a4c1064..000000000 --- a/packages/chord/docs/next/explainer/src/03-principals.md +++ /dev/null @@ -1,190 +0,0 @@ - -# Who Is Calling {#principals lede="Principals: how Chord knows who calls, why a caller cannot lie about it, and how the answer picks the object and its members."} - -The owner may add and remove notes. Ann, a read-only guest, may only read them and should not even see who wrote -what. The object must therefore know who is calling, and the caller must not be able to claim to be someone else. In -Chord the answer is a **principal**, a small JSON value the app defines and the links stamp. - -## The app's principal type - -Chord defines no principal kinds; the app does. Every variant has a `kind`. Studio uses p2's set: - -```ts -// studio/src/principal.ts -export type P = - | { kind: "owner"; access: "local" | "relay" | "device"; device?: string } - | { kind: "guest"; session: string; mode: "read" | "write"; grant: string; revision: number } - | { kind: "server" } - | { kind: "worker"; session: string }; -``` - -| `kind` | Who | -| --- | --- | -| `owner` | the person running the app, on loopback, through the relay, or on a paired device | -| `guest` | someone invited into one session, with a mode and the grant (and its revision) that let them in | -| `server` | the server's own code calling a worker | -| `worker` | a session worker calling the server | - -## Links stamp principals - -A principal never comes from a request's data. The server authenticates a client once, when its WebSocket connects, -and stamps the result on the whole link (`server.accept(link, { principal })`). The server tells Chord who each worker -is when it resolves it (`return { link, principal }`). When the server relays a client's request to a worker, Chord -writes the client's principal into the forwarded request; a principal field coming from a client is always -overwritten, and only a trusted parent link may carry one. - -Where principals come from is app code. Studio's sharing feature creates a share link per session and mode; a guest -who opens it gets a guest record (grant `g1`) and a cookie; `authenticate(req)` turns the cookie into -`{ kind: "guest", session: "S1", mode: "read", grant: "g1", revision: 1 }` on every new connection. Chord never sees -cookies or grants, only the finished principal. - -## The caller is part of the identity - -A `remote` object exists once per **(type, input, principal key)**. The **principal key** is a string computed from -the principal; by default its canonical JSON (sorted keys). The app may pass its own `principalKey` to `createHost`, -and must pass the same function on every host, or identities split silently. - -```ts -const principalKey = (p: P) => - p.kind === "owner" ? `owner:${p.access}:${p.device ?? ""}` - : p.kind === "guest" ? `guest:${p.grant}:r${p.revision}` - : p.kind === "worker" ? `worker:${p.session}` - : p.kind; -``` - -The design wants the key to include how the client is connected and the grant's revision, so that two access paths -never share objects and a changed grant never reuses the old grant's object: - -| Caller | Principal key | `Notes { S1 }` object | -| --- | --- | --- | -| owner, tab A, local | `owner:local:` | 1 | -| owner, tab B, local | `owner:local:` | 1 (shared with tab A) | -| owner through the relay | `owner:relay:` | 2 | -| guest g1, revision 1 | `guest:g1:r1` | 3 | -| guest g1 after a change to write mode, revision 2 | `guest:g1:r2` | 4 (new) | - -## Variants - -The contract lists named sets of members; the factory receives the principal and returns the variant that principal -may use: - -```ts -// notes/contract.ts (version 1, final form for now) -variants: { - editor: { - state: { notes: obj({ items: arr(Note) }) }, - methods: { add: method(obj({ text: str({ min: 1, max: 2000 }) }), str()), remove: method(obj({ id: str() })) }, - }, - reader: { - state: { notes: obj({ items: arr(PublicNote) }) }, // no `by`: authors are hidden - }, -}, -``` - -```ts -// notes/worker.ts -host.serve(Notes, ({ session }, principal, self) => { - if (canWrite(principal, session)) return { variant: "editor", notes: …, add: …, remove: … }; - return { variant: "reader", notes: … }; -}); -``` - -A reader has no `add` at all. Chord refuses a call to a member that is not in the caller's variant, so `add` needs no -check of its own. Three rules keep variants compatible: a member name has the same kind (state or method) in every -variant that has it; a method has the same arguments and result in every variant; only state schemas may differ per -variant (the reader's notes have no authors). `remote(...)` enforces them at the type level, and at runtime for -contracts that arrive as data. - -## How a facet knows `P` - -`P` is Studio's type, not Chord's, and every host is created with it (`createHost

    `). A plugin's facets are written -and compiled separately, then handed to `host.load` as values, so the type checker cannot infer `P` from the host into -the factory's `principal`. Studio's base code therefore exports its own `defineFacet`, already bound to `P`, and every -facet of every plugin imports that one instead of Chord's: - -```ts -// studio/src/facet.ts (base code, exported as @studio/core) -import { bindFacet } from "@earendil-works/chord/next"; -import type { P } from "./principal.ts"; - -export const defineFacet = bindFacet

    (); // a facet's principals are Studio's P -export type { P, Revocation } from "./principal.ts"; -``` - -```ts -// notes/worker.ts (server.ts, web.tsx and tui.ts import the same defineFacet) -import { defineFacet } from "@studio/core"; - -export default defineFacet((host, self) => { - host.serve(Notes, ({ session }, principal, self) => { // principal: P - if (principal.kind === "guest" && principal.mode === "read") return { variant: "reader", notes: … }; - … - }); -}); -``` - -The import is cheap: `@studio/core` is a host package plugins already import at runtime (the contract imports `Session` -from it), `P` is a type-only import, and `bindFacet` is a one-line function. The host checks the pairing: `load` -accepts only facets typed for its own `P`, so a facet compiled against another app's principal type, or one built with -Chord's untyped `defineFacet`, does not compile into Studio's start code. Client facets (`web.tsx`, `tui.ts`) use the -same `defineFacet`; they rarely see a principal, because clients serve only `local` types. - -None of this is checked at runtime, and none of it needs to be: the principal a factory receives is the one the link -stamped. - -:::review | Review: binding facets to the app's principal type (open) -The design shows `defineFacet((host, self) => …)` without saying how `P` reaches the factory. The frozen kernel takes -it as an explicit type parameter, `defineFacet

    (…)`, defaulting to `{ kind: string }` -(`packages/chord/src/next/facet.ts`), and the playground writes `defineFacet(…)`. The pre-bound -`defineFacet` above (`bindFacet` is a working name) is the owner's preferred direction, recorded as an open point in -`packages/chord/docs/next/layer-1.md` §4. -::: - -## Narrowing on the caller's side - -Owner and guest run the same `web.tsx`, and which variant they get is known only at runtime. The types force every -caller to narrow before using variant members: - -```ts -const notes = await client.get(Notes, { session: "S1" }); -notes.add({ text: "hi" }); // type error: `add` exists only in "editor" -if (inVariant(notes, "editor")) await notes.add({ text: "hi" }); -expectVariant(notes, "editor"); // or assert (throws) -function Editor({ notes }: { notes: RefIn }) { … } -``` - -The protection is not the type checker; it is Chord refusing the call at the provider. A client that ignores the -types gets the same error. - -## Admission before a worker starts - -Principals also decide whether a caller may reach a host at all. `route(Session, { authorize })` runs before Chord -resolves an address, so a guest of S1 asking for anything in S2 is refused at the server, and no worker starts for -them. Finer authority lives in the per-principal view's methods; there is no asynchronous "may this caller activate -this object" guard. An object whose address carries authority (say, a login flow by id) may exist; what a caller can -see and do is decided by the variant its factory picks. - -## Inside methods - -`ctx.principal` is the caller of the current call. In Studio's notes the editor's `add` uses it to record the -author: - -```ts -add: ({ text }, ctx) => store.add({ text, by: authorOf(ctx.principal) }, ctx), -``` - -Where principals act, in summary: - -| Purpose | Where | -| --- | --- | -| identity: which object a caller gets | principal key | -| variant: which members | the factory | -| admission: whether a host is reachable | `route.authorize` | -| attribution and fine checks | `ctx.principal` in methods | -| revocation | `revokedBy` (chapter VII) | - -## The bug from chapter II - -Now the bug is visible. The owner's `Notes { S1 }` and Ann's `Notes { S1 }` are two objects (rows 1 and 3 above), and -each declared its own `NoteList`. The owner's notes are not in Ann's list. Per-caller objects must not hold the data; -they must be views over something that exists once. That is the next chapter. diff --git a/packages/chord/docs/next/explainer/src/04-store.md b/packages/chord/docs/next/explainer/src/04-store.md deleted file mode 100644 index 44d3739ff..000000000 --- a/packages/chord/docs/next/explainer/src/04-store.md +++ /dev/null @@ -1,132 +0,0 @@ - -# One Store, Many Views {#shared-store lede="Local types, and the pattern every Chord plugin uses: shared data once in a store, per-caller views in front of it."} - -Every caller has its own `Notes` object, which is what makes per-caller variants possible. But data must exist once. -The fix is the most common pattern in Chord: a **shared store**, a `local` object that holds the data and does the -heavy work once, behind thin per-principal **views**, the `remote` objects callers get. - -## `local` types - -```ts -// notes/contract.ts -export const NoteStore = local("notes.store", { - version: 1, - input: obj({ session: str() }), - members: { - notes: obj({ items: arr(Note) }), // state: a schema - public: obj({ items: arr(PublicNote) }), - add: method(obj({ text: str(), by: str() }), str()), // methods: method(...) - remove: method(obj({ id: str() })), - }, -}); -``` - -A `local` contract has flat `members` and no variants. The differences: - -| | `remote` | `local` | -| --- | --- | --- | -| reachable from | any host, across links | its own host only | -| identity | (type, input, principal key) | (type, input) | -| principals, variants | yes | no | -| factory | `(input, principal, self)` | `(input, self)` | -| arguments | stripped to the provider's schema and validated, for every caller | see the review note below | -| values on the same host | passed by reference, never copied | passed by reference, never copied | -| `ctx.principal` | the caller | none | -| may persist documents | no (chapter V) | yes | - -A local object has no principal: it is trusted code on its own host, acting for the host. So attribution ("who wrote -this note") is passed as data (`by`), by the view that knows who called. Clients serve only local types, because -nobody can address a client. - -## The pattern - -```ts -// notes/worker.ts -export default defineFacet((host, self) => { - host.serve(NoteStore, ({ session }, self) => { - const list = self.state(NoteList); - const pub = self.derive("public", [list], ([l]) => ({ items: l.items.map(stripAuthor) })); - return { - notes: list, - public: pub, // computed once, for every reader - add: ({ text, by }, ctx) => - self.tx(async (tx) => { - const id = uuidv7(); - (await tx.draft(list)).items.push({ id, text, by, at: Date.now() }); - return id; - }, ctx), - remove: ({ id }, ctx) => list.change((d) => { d.items = d.items.filter((n) => n.id !== id); }, ctx), - }; - }); - - host.serve(Notes, ({ session }, principal, self) => { - const store = self.get(NoteStore, { session }); // a reference, declared: held for the view's life - if (canWrite(principal, session)) return { - variant: "editor", - notes: store.notes, // the store's readable, exposed as-is - add: ({ text }, ctx) => store.add({ text, by: authorOf(principal) }, ctx), - remove: (args, ctx) => store.remove(args, ctx), - }; - return { variant: "reader", notes: store.public }; - }); -}); -``` - -:::flow wide | One store, three views. The redaction runs once in the store; both readers read the same value. -row "Per-principal views (remote)" blue -box ve "Notes { S1 }" "owner · editor" blue -box vr1 "Notes { S1 }" "guest Ann · reader" blue -box vr2 "Notes { S1 }" "guest Bob · reader" blue -row "Shared store (local)" accent -gap -box store "NoteStore { S1 }" "notes, public (a derive)" accent -gap -arrow ve -> store "self.get" -arrow vr1 -> store "self.get" -arrow vr2 -> store "self.get" -note "Authority is the view's: the reader variant has no add. The store does not check; it is trusted and acts for the host." -::: - -`self.get(NoteStore, { session })` in a factory declares a **reference** that the view holds for its whole life; Chord -binds it during activation and waits until the store is live (chapter VI). The view exposes the store's readables -directly. Owner, Ann and Bob now give three `Notes` objects, one `NoteStore`, one note list and one redacted list. - -Any heavy or shared part belongs in a store like this: a pi-durable watch, a file watcher, a sandbox, a search index. -Base code uses the same pattern with the same primitives; nothing about it is special to core. - -## Derived values, briefly - -`self.derive(name, sources, fn)` computes a value from readables and recomputes it when a source changes; viewers get -only the difference. The reader's list above is one. Chapter VIII covers derives in depth, including the one footgun -that matters: `stripAuthor` must return the *same* object for an unchanged note, or every commit costs the size of the -whole list. The tool for that is `mapShared`. - -## Arguments are checked at the provider - -Trusted code is not type-checked code: plugins are bundled with esbuild without type checks, and agents write -JavaScript plugins. So the provider checks arguments itself (decision R7): every method's arguments are stripped to -the **provider's own** schema (unknown fields removed) and validated with it, whoever the caller is, and never with a -schema a peer sent. Schemas are compiled lazily on first use and cached per schema object. Bytes are checked as leaves -(`instanceof Uint8Array` plus an optional length bound), never walked. Stripping copies only when a field was removed; -the caller's value is never mutated (proposal Q12, in the L1 draft). Results and state are not validated: they come -from the provider. - -On the same host nothing is copied: arguments and results are passed by reference. Values are **immutable by -convention** (Delta's rule, as in Durable): nobody mutates a value it received, and Chord neither copies nor freezes -values to enforce it. There is no development mode that would freeze them; one behaviour everywhere (R9). - -:::review | Review: are local arguments validated? (Q15, open) -Design §5's table says `local` arguments are "neither validated nor copied"; §3 and R7 say arguments are validated -"for every caller". Review round 3 raised this as Q15 (apply R7 to `local` too, or keep `local` as the one exception), -and `layer-1.md` lists it as open. Until it is decided, write local methods so that they do not depend on either -answer: trust the shape, but never trust content (lengths, ids) that came from a remote caller without the view -checking it. `chord-next-walkthrough.md` concept 6 still describes the older rule ("validated only when the caller is -not a host principal"), which R7 replaced. -::: - -## Choosing between the two - -Use a `remote` type for what callers address: views, things with per-caller authority. Use a `local` type for what -exists once per input on one host: stores, caches, owners of OS handles, and the core objects a host provides -(chapter IX). If a type is reached across a link, it is remote; if a type persists documents, it is local (or a -host-lifetime object, chapter V). diff --git a/packages/chord/docs/next/explainer/src/05-documents.md b/packages/chord/docs/next/explainer/src/05-documents.md deleted file mode 100644 index 2a25a1caa..000000000 --- a/packages/chord/docs/next/explainer/src/05-documents.md +++ /dev/null @@ -1,258 +0,0 @@ - -# Writing It Down {#documents lede="Documents and transactions: how an object stores data, changes it atomically, survives restarts and migrates it."} - -Notes must be stored, changed safely while others read them, sent to viewers as small changes, and survive a worker -restart. An object's data lives in **documents**: named JSON values the object owns, changed only in -**transactions**. The design is a port of pi-durable's store, so most rules will look familiar; this chapter states -them for Chord. - -## Defining a document - -```ts -// notes/worker.ts -const NoteList = defineState("notes.list", { - schema: obj({ items: arr(Note) }), - version: 1, - initial: () => ({ items: [] }), - persist: true, // stored; default: memory only - checkpointWhen: (_value, _ops, info) => info.deltasSinceBase > 64, -}); - -const Draft = defineState("notes.draft", { // one unfinished note per author - schema: obj({ text: str() }), - version: 1, - initial: () => ({ text: "" }), - keyed: true, // many per object, by key - persist: true, - checkpointWhen: () => true, // small: a full base per write -}); -``` - -`checkpointWhen` is required on every definition, with no default (R14, as in Durable since pi `5c1736f82`): it says -when a commit stores a complete base instead of a delta, which bounds how much is replayed on load. Small documents -checkpoint on every write; growing ones after a number of deltas. - -A document's root is always a JSON object (lists live in a field). Documents hold JSON only; bytes are a value and wire -type, not a document type. - -## Singleton and keyed - -```ts -host.serve(NoteStore, ({ session }, self) => { - const list = self.state(NoteList); // THE note list of this object - const drafts = self.state(Draft); // all drafts of this object - … -}); -``` - -| | singleton (no `keyed`) | keyed | -| --- | --- | --- | -| how many | one per object | many per object, by a short key | -| loaded | at activation, kept in memory | on demand | -| read | `list.value` (synchronous) | `await drafts.get(key)`: the value or `undefined` | -| list | (it is one value) | `for await (const key of drafts.keys())` pages keys | -| expose as state | yes: the handle is a readable | no: expose a method or a singleton index | - -There is no live key list. A collection shown live is a singleton (an object holding a list or a map); a large -collection is a method returning pages. - -## Transactions - -All writes go through `self.tx(fn, ctx)`. Inside, `await tx.draft(handle, key?, { tags? })` returns a draft of the -current document, or a draft over `initial()` if it is absent. Drafts record operations (assignments, `push`, -`Object.assign`); asking for the same document twice in one transaction returns the same draft. `tx.retire(handle, -key?)` removes a document. There is no `tx.get`: reading is drafting, and a draft is an overlay over the current -value, so nothing is copied. A draft you do not change commits nothing, so reading never stores an absent document. - -```ts -saveDraft: ({ author, text }, ctx) => - self.tx(async (tx) => { (await tx.draft(drafts, author)).text = text; }, ctx), -``` - -Shorthands for one document: `handle.change(fn, ctx)` (keyed: `handle.change(key, fn, ctx)`) (loads the document, then runs the synchronous `fn` on -its draft) and `handle.retire(ctx)` (keyed: `handle.retire(key, ctx)`). - -### Four patterns - -Two decisions (R12 and R13, both as in Durable) shape how transactions are written. **Drafts are revoked when the -transaction settles**, so a transaction returns at most plain values, never something taken from a draft. And -**there is no existence check inside a transaction.** - -```ts -// 1. Return data: write in the transaction, read after the commit. -await self.tx(async (tx) => { (await tx.draft(list)).items.push(note); }, ctx); -return list.value.items.length; // the committed revision, or a newer one - -// 2. Remove if present: retire of an absent document is a no-op. -await drafts.retire(author, ctx); - -// 3. Create with tags if missing: the draft creates it, only if it changes. -await self.tx(async (tx) => { - (await tx.draft(drafts, author, { tags: [SessionTag(session)] })).text = text; -}, ctx); - -// 4. Branch on existence: read first, outside the transaction. -if ((await drafts.get(author)) === undefined) { … } -``` - -### Durable's rule - -Inside the callback, only `await tx.*`. No other objects, no network, no models, no processes, no second `self.tx`, no -reads past `tx`. Breaking it is not an error but a hang: the object's line is held while the callback waits. Fetch -first; decide and write inside. - -```ts -// Wrong: a fetch inside the transaction holds the object's line. -await self.tx(async (tx) => { (await tx.draft(list)).items.push({ …, title: await fetchTitle(url) }); }, ctx); -// Right: fetch, then write. -const title = await fetchTitle(url, ctx.signal); -await self.tx(async (tx) => { (await tx.draft(list)).items.push({ …, title }); }, ctx); -``` - -## What a commit does - -:::flow wide animate | The commit sequence, Durable's. Nothing is copied, frozen or revalidated; viewers get one batch per commit. -row "Inside self.tx" accent -box line "wait for the line" "per object" -box run "run the callback" "drafts record ops" accent -box prep "prepare" "Delta checks placed values" -box write "storage batch" "deltas or a base" accent -row "After the write" blue -box adopt "adopt" "pointer swap" -box local "notify locally" "once, with the ops" blue -box wire "send" "one batch per link" -box revoke "revoke drafts" "touching one throws" -arrow line -> run -arrow run -> prep -arrow prep -> write -arrow write -> adopt "stored" -arrow adopt -> local -arrow local -> wire -arrow wire -> revoke -note "An error or an abort before the storage write discards everything. Storage commits are serialized per host." -::: - -Lines are per object, never across objects; reads outside a transaction do not take the line. Subscribers on the same -host get the new value and the commit's operations by reference. Remote viewers get the commit's operations, never the -document again; chapter XVII shows how one commit touching several holds still arrives as one message. - -## Tags and deleting by tag - -Tags are typed tokens set when a document is created: - -```ts -export const SessionTag = defineTag("studio.session"); // base code, shared by every plugin -await self.tx(async (tx) => { (await tx.draft(drafts, author, { tags: [SessionTag(session)] })).text = text; }, ctx); -// when the session is deleted -await store.retireByTag(SessionTag, "S1"); // every document tagged S1 -``` - -The design names the operation `store.retireByTag(tag, value)` without saying who holds the store; session delete is -one of the app decisions still open (#18, chapter XIX). - -## Who may persist - -`persist: true` is allowed only in `local` objects and `lifetime: "host"` objects: identities without a principal. A -`remote` object's identity contains the principal key, which changes with the access path and the grant revision. Its -stored documents would scatter across access paths and be orphaned after a reconnect. The rule is enforced at the type -level (`self.state` in a remote factory accepts only non-persisted definitions) and at declaration ("persist needs a -stable identity: put the documents into a local shared store"). This is the second reason for the shared store. - -What survives a host teardown (a crash, a restart) is exactly what is persisted. In-memory documents live as long as -their object: they survive reloads and native restarts of a live object, and go when it ends. Everything else -(natives, derives, routines, references) is rebuilt on activation. - -## Facet documents - -A facet has its own identity (its plugin's name) and may hold documents too, for plugin-wide data such as -preferences. Objects the facet serves may read and write their facet's documents, and one transaction may span the -object's own documents and its facet's (lock order: own line, then the facet's). The facet never touches a served -object's documents. - -## Ownership across plugins - -No plugin reads or writes another plugin's documents, Chord or Durable. Data crosses plugins only through state and -methods. Data that core enforces lives in a format core owns and is handed to core through its API when the data is -created (as with Durable's `createConversation({ init })`), never written into core's documents. - -## Versions and migration - -A change of shape or meaning raises `version` and adds a pure `migrate(old, from)`: - -```ts -const NoteList = defineState("notes.list", { - schema: obj({ items: arr(NoteV2) }), // NoteV2 adds `pinned` - version: 2, - migrate: (old, from) => ({ items: old.items.map((n) => ({ ...n, pinned: false })) }), - initial: () => ({ items: [] }), - persist: true, - checkpointWhen: (_v, _o, info) => info.deltasSinceBase > 64, -}); -``` - -Documents migrate on access by the new definition: singletons during activation (hydration counts as access), keyed -ones on their next access. The migrated base is persisted with the next commit; right after a bump, a read-only -transaction writes it. - -Chord does not walk values against schemas. Instead, each definition computes a **schema fingerprint** once -(descriptions and titles ignored), and every stored record keeps the fingerprint it was written with. On access Chord -compares them: the same version with a different fingerprint fails the object `broken` ("schema changed without a -version bump"). So every structural change needs a bump, an additive one with a trivial `migrate`. The one exception -to "no value checks" (R8): the result of `migrate` and of `initial()` is validated against the new schema once, when -produced. A failed check, a throwing `migrate`, or old code meeting a newer document refuses the access, keeps the -stored data and fails the owning object. A changed meaning without a schema change cannot be detected; that is on -you. - -## Storage - -Behind the store is a `DocumentStorage`, passed as `createHost({ storage })`; memory by default. The server has one -shared storage, each worker one for its session. - -```ts -interface DocumentStorage { - commit(writes: readonly DocumentWrite[]): Promise; // create/change/retire, atomic - find(address: DocumentAddress): Promise; - load(id: string): Promise; // version, value, deltasSinceBase - scan(query: { owner: string; definition: string }, limit: number, cursor?: string): - Promise<{ keys: string[]; next?: string }>; - byTag(tag: string, value: string, limit: number, cursor?: string): Promise<{ ids: string[]; next?: string }>; - mintId(): Promise; -} -``` - -Chord ships a memory storage and a SQLite storage (over `@earendil-works/chord/sqlite`, the driver layer that moved -from Durable; Node and Durable Object drivers). Storages check only metadata (incarnation, version, sequence, address -occupancy, fingerprint) and write the supplied delta or base; they never rebuild or validate content on a commit. -Chord documents never commit together with Durable transcripts: Chord has its own storage, not an adapter over -Durable's. Your own storage can prove itself with `registerDocumentStorageConformance(runner, name, withStorage)` from -`@earendil-works/chord/testing`. - -Keyed persisted documents that nobody reads, writes or watches for 10 minutes leave memory and reload on the next -access. A storage error ends the store and closes the host; `onError` receives `{ kind: "storage_failed" }` and the -app decides (chapter I). - -## Kinds of state - -| Kind | For | API | -| --- | --- | --- | -| document | data the object owns | `self.state`, `self.tx` | -| derive | computed values | `self.derive` (chapter VIII) | -| source readable | external data: Durable views, files, logs | value plus ordered changes; feeds derives | -| paged method | large or queryable collections | a method returning pages (chat history) | -| plugin database | full-text search, large blobs | a native opened by its owner (chapter IX) | - -App state lives in Chord documents. Only transcript-bound data, which must fork and rewind with a conversation, lives -in Durable. - -:::review | Review: does an absent singleton read `undefined` or `initial()`? (Q11, open) -Design §9 says "whether a document exists is read outside the transaction (`handle.value` / `handle.get` give -`undefined` for an absent one)" and also "a singleton that does not exist yet reads `initial()` through its handle; -retiring resets it to `initial()`". Review round 3 proposes: singletons always read `initial()`, only absent keyed -documents read `undefined`, and singleton existence needs no API. This guide follows the proposal (`list.value` is -never `undefined`); it is not decided. -::: - -:::note | Built so far -Documents, transactions and storages are layer L3, a port of Durable's session store; not started. The frozen kernel -has a working version with a JSONL storage, which the design rejected. -::: diff --git a/packages/chord/docs/next/explainer/src/06-references.md b/packages/chord/docs/next/explainer/src/06-references.md deleted file mode 100644 index b852916b5..000000000 --- a/packages/chord/docs/next/explainer/src/06-references.md +++ /dev/null @@ -1,185 +0,0 @@ - -# Holding On {#references lede="References, the object lifecycle and ctx: what a caller sees while objects start, fail, restart and move, and why nobody writes retry code."} - -A tab shows the notes of S1 for an hour. During that hour the laptop sleeps, the worker is stopped for idleness and -started again, the notes plugin reloads twice and the store's storage hiccups once. The panel should show the notes the -whole time, say "reconnecting" when it cannot reach them, and recover by itself. In Chord this is what a -**reference** does; plugin code contains no retry loops and no request ids. - -## Three ways to hold - -| Where | API | Released | -| --- | --- | --- | -| a factory | `self.get(Type, input)` | when the object ends | -| a call | `await ctx.get(Type, input, { state? })` | when the call ends | -| outside objects | `await host.get(Type, input, { signal, state?, parkTimeout? })` | `ref.release()` | - -Use `self.get` for what an object needs its whole life (the view's store), `ctx.get` for what one call needs, and -`host.get` only in UI code, tests and entry points. `{ state: false }` binds for methods only, without receiving state. -A reference keeps its object alive while held. - -## The object lifecycle - -```text -(get) → activating → live → ending → ended - └→ failed (backoff) → activating … -``` - -**Activating:** Chord binds the declared references and waits until they are live and hydrated, loads singleton -documents, opens natives (one after another, in declaration order), computes derives and starts routines. The object -goes live when all of that succeeded. Calls wait behind its **gate** until then. - -**Ending:** routines stop, the object's calls and jobs are aborted as their rules say, natives close in reverse order, -references are released. An object with `lifetime: "referenced"` (the default) ends when its last reference goes, -after the type's `linger` if it has one. A `lifetime: "host"` object (only for types without a required input) lives -as long as the host. A running, finishing or blocked job counts as a reference (chapter XI). - -## `ref.state` - -Every reference reports what is going on as a tagged value; listeners get each new one: - -```ts -type RefState = - | { kind: "loading" } // first hold, reconnecting, restarting - | { kind: "live" } - | { kind: "failed"; cause: FailureCause; retryAt: number | undefined } // undefined: no automatic retry - | { kind: "released" }; -``` - -| Situation | Calls | Values | -| --- | --- | --- | -| link drop | park until reconnected | readable, `loading` | -| object loading or restarting | park until live | readable, `loading` | -| rollout: provider ahead or installing | park until installed | readable | -| object `failed` | reject at once with the cause | `failed`, with cause and `retryAt` | - -Values stay readable through all of this: the panel keeps showing the last notes. Parked calls share one deadline (60 -s by default, `parkTimeout` per `get`), then fail `unavailable`. Otherwise a call ends only through its caller's -signal; there is no per-call deadline. A call that was running on an instance that ends fails with an error and is -not resent: Chord cannot know whether it took effect. - -**Errors look like disconnects.** A consumer handles a failed call exactly as a dropped link: show the error, let the -user retry. Work that must not run twice carries its own idempotency key as data (Durable's `requestId`, a job's -`key`). - -:::steps | The worker restarts while a tab watches the notes -+++ Watching -The tab holds `Notes { S1 }`. `ref.state` is `live`; `notes.notes.value` has 40 notes. -+++ The worker goes away -The supervisor stops worker S1 (or it crashes). The server's IPC link to it drops. The server tells the tab that its -hold is gone; the tab's reference goes `loading`. The 40 notes stay readable. -+++ A click meanwhile -Ann's colleague clicks **Add**. The call **parks** on the reference; nothing is sent yet. The deadline is 60 s. -+++ Chord resolves again -Holds on S1 still exist, so Chord calls `route(Session).resolve({ id: "S1" })` again, with the per-address backoff. -The app forks a new worker: a new **incarnation** of `Session({ id: "S1" })`. -+++ Re-hold -The server re-holds every hold on S1 over the new link. The new worker activates `NoteStore { S1 }` (its notes load -from storage) and the views; it sends fresh snapshots. The tab's reference goes `live`. -+++ The parked call runs -The parked `add` is sent now and succeeds. The panel shows 41 notes. Nobody wrote recovery code. -::: - -## Failure and retry - -Any broken part fails the **whole** object: a half-working object could break rules that rely on its documents -changing together. Chord tags the cause by where it failed: - -```ts -type FailureCause = - | { kind: "transient"; error: ChordError } // an open threw, a dependency was unreachable: automatic backoff - | { kind: "broken"; error: ChordError }; // factory, derive, hydration or migrate threw: retried only by a trigger -``` - -**Transient** causes get a backoff per address: 1 s, doubling to 60 s; concurrent gets share one attempt; success -resets it. **Broken** causes would fail the same way again (a code bug, invalid data), so they are retried only by a -**trigger**: a plugin reload, `ref.retry()` (a Retry button), or a readable the factory declared with -`self.retryOn(...readables)`. Only an opener may declare its own failure broken, with `throw broken(error)`, for -example for invalid configuration. A failed reference stays held; on success it goes `loading`, then `live`. - -```ts -host.serve(EnvConnection, ({ env }, self) => { - const config = self.get(EnvConfig, {}); // a facet or registry value, outlives the attempt - self.retryOn(config.envs); // a config change retries a broken open - const ssh = self.open(async ({ signal }) => { - const c = config.envs.value[env]; - if (!c) throw broken(new Error(`unknown environment ${env}`)); - return connect(c, signal); - }, (s) => s.close()); - … -}); -``` - -`retryOn` sources must live independently of the attempt (facet documents, registry values); a member of one of the -object's own references is a type error, since it goes away with the attempt. - -## Waiting for dependencies - -An object whose references are not live yet stays `loading` and goes live the moment they are, bounded by -`readyTimeout` (60 s). The decision for the failure case (R18) is what makes recovery automatic: - -:::sequence animate | Dependency recovery. The view fails transient with the store's cause, keeps its reference, and is retried the moment the store goes live. -actor tab "tab" -actor view "Notes { S1 }" blue -actor store "NoteStore { S1 }" accent -tab -> view: hold -view -> store: self.get (activation) -store -> store: open fails: disk not mounted -store --> view: failed (transient) -view --> tab: failed: transient, cause inside -note view..store: the view keeps its reference; its state is the retry trigger -store -> store: backoff retry: open succeeds -store --> view: live -view -> view: retried at once (backoff cancelled) -view --> tab: live -::: - -A dependency that fails, or does not become live in time, fails the dependent **transient** with the dependency's -cause inside, never broken: the cause is the dependency's. The dependent keeps its references while failed, and Chord -uses their state as a retry trigger. Factory authors write `self.get` and know none of this. After an object is live, -a dependency leaving `live` does not change it: its calls that use the dependency get the dependency's error, or park -while it loads. - -**Cycles are rejected** (Q5). If A's activation gets B and B's gets A, neither would ever lose its last reference. -Every object instance has a uuidv7 instance id; on one host Chord follows the waiting edges of activating objects, and -across hosts a hold made during an activation carries the chain of instance ids. A cycle fails the activation -`broken`, naming it ("A → B → A"), and keeps no references. - -## No optional references - -`get` has no `optional` flag. When a dependency's plugin may be missing, turn the dependency around: the dependent -owns a registry and the other plugin contributes to it. Radius uses forwards from the ports plugin, but ports is -optional, so radius owns `radius.forwards` and ports contributes its forwards; without ports the table is empty and -radius works (chapter XIII). - -## `ctx` - -Every call gets a `ctx` as its last argument: - -| Field | Meaning | -| --- | --- | -| `signal` | fires when the caller cancelled or the object ended; pass it to everything you wait on | -| `principal` | the caller (remote types only) | -| `incarnation` | the calling host's incarnation, for calls that arrived over a link (chapter X) | -| `get` | references released when the call ends | -| `telemetry`, `trace(name, attributes, fn)` | the current span; a child span with a child `ctx` (chapter XVII) | -| `jobId`, `progress` | in a job's `run` (chapter XI) | - -```ts -exportText: async ({}, ctx) => { - const catalog = await ctx.get(Catalog, {}); // released when the call ends - const title = catalog.sessions.value[session]?.title ?? session; - const res = await fetch(renderUrl, { method: "POST", body: render(title, list.value), signal: ctx.signal }); - return await res.text(); -}, -``` - -There are no code versions and no request ids in `ctx`. Durable's `Context` is the same module (`chord/context`): -`ctx` extends it, so passing `ctx` into pi-durable needs no conversion. Code outside objects gets a `ctx` from -`host.root(fn, { principal?, signal? })`; inside Durable tools it uses `chord(fn, context)` (chapter XIV). - -:::note | Built so far -References, `ctx` and the lifecycle are layers L4 and L5. `ChordError` and `FailureCause` are specified in the L1 -draft (`layer-1.md`), with codes `invalid`, `contract_mismatch`, `forbidden`, `gone`, `not_ready`, `released`, -`too_large`, `unavailable` and `aborted`, and a finer `reason` such as `invalid/args`. -::: diff --git a/packages/chord/docs/next/explainer/src/07-revocation.md b/packages/chord/docs/next/explainer/src/07-revocation.md deleted file mode 100644 index 1cacf921b..000000000 --- a/packages/chord/docs/next/explainer/src/07-revocation.md +++ /dev/null @@ -1,113 +0,0 @@ - -# Taking Access Away {#revocation lede="Revocation: how removing a guest ends their objects on every host at once, and why reconnecting does not bring access back."} - -The owner removes Ann from S1. Her tab still has a WebSocket open and a `Notes { S1 }` view on worker S1. Both must end -now, on the server and on the worker, and her tab's automatic reconnect must not restore anything. Chord provides the -mechanism; the app decides who is blocked and remembers it. - -## Labels, not predicates - -The app declares, once, which **revocation labels** each principal carries. Revoking means revoking one label; -everything carrying it is hit. - -```ts -// studio/src/principal.ts -export type Revocation = { grant: string } | { device: string } | { session: string }; - -export const revokedBy = (p: P): Revocation[] => - p.kind === "guest" ? [{ grant: p.grant }, { session: p.session }] - : p.kind === "owner" && p.access === "device" && p.device ? [{ device: p.device }] - : []; - -const server = createHost({ address: Server(), principal: { kind: "server" }, revokedBy }); -// every worker is created with the same revokedBy -``` - -| Principal | Labels | -| --- | --- | -| guest Ann (grant g1) in S1 | `{ grant: "g1" }`, `{ session: "S1" }` | -| guest Bob (grant g2) in S1 | `{ grant: "g2" }`, `{ session: "S1" }` | -| owner on the paired phone | `{ device: "phone" }` | -| owner, local | none | - -| Call | Hits | -| --- | --- | -| `server.revoke({ grant: "g1" })`: remove Ann | Ann | -| `server.revoke({ session: "S1" })`: archive S1 | Ann and Bob | -| `server.revoke({ device: "phone" })`: sign out the phone | the phone | - -A label is plain data because revocation is an event that must travel from the server to the workers; a predicate -function could not be sent. - -## What `revoke` does - -`host.revoke(r)` ends every object and closes every link whose principal `p` has `r` in `revokedBy(p)` (compared by -canonical JSON; never the host's own principal), and passes `r` to its children, which apply their own `revokedBy`. - -:::sequence animate | Removing guest Ann. The label travels down; her objects end on every host; her reconnect is refused by the app's own record. -actor owner "owner tab" -actor server "server" accent -actor worker "worker S1" blue -actor ann "Ann's tab" "guest g1" muted -owner -> server: Sharing.remove({ grant: "g1" }) -server -> server: app: mark g1 blocked (document) -server => server: revoke({ grant: "g1" }) -server -> ann: close link -server => worker: revoke { grant: "g1" } -worker -> worker: end Notes { S1 } as g1 -ann -> server: reconnect -server -> server: authenticate: g1 blocked -server --> ann: refused (4401) -::: - -On worker S1, for `revoke({ grant: "g1" })`: - -| Object | Principal | Hit | -| --- | --- | --- | -| `Notes { S1 }` | owner, local | no | -| `Notes { S1 }` | guest g1 | **yes**: ends | -| `Notes { S1 }` | guest g2 | no | -| `NoteStore { S1 }` | none (local) | no | - -Ending Ann's view releases its reference to the store, which keeps serving the others. - -## Chord persists nothing here - -`revoke` cuts live access only. Lasting denial lives in the app's data (the sharing plugin's guest records), which -`authenticate` checks on every new connection. After a server restart nothing live exists to revoke; every client -authenticates again and blocked grants are refused there. - -**No re-stamping.** A changed grant (Ann moves from read to write) is a revocation of the old grant. Her link closes, -her tab reconnects and authenticates with the new grant revision, which gives a new principal key, and her references -re-hold by themselves, now as an editor. Reconnecting never restores revoked authority. - -**Revocation beats jobs** (R11). Normally a running job keeps its object alive after the last reference is released -(a closed tab does not abort an import). Revoking a principal ends its objects at once regardless; their jobs get the -abort signal and the normal close timeout. - -## Work in shared stores - -Revocation ends objects whose principal carries the label. A shared store has no principal, so work it runs *on -behalf of* someone (a question waiting for Ann's answer, a shell command a guest started) is not ended by `revoke`. -Such a store keeps the grant as data with the work and reconciles from the grant owner's live state with a routine -(chapter X): - -```ts -// in a shared store that runs work for guests -const sharing = self.get(Grants, {}); // the sharing plugin's live grants -self.routines.on(sharing.live, async (live, ctx) => { - for (const run of shell.running.value) - if (run.args.grant && !live[run.args.grant]) shell.abort(run.id); // idempotent -}); -``` - -There is no revocation hook: a hook can be missed (a crash between revoke and hook), a routine over current state -cannot. The window between the revocation and the routine's run is accepted (R10); a step started in it still runs. - -## Who does what - -| Chord | The app | -| --- | --- | -| stamps principals on links | what a grant is; share links, cookies, `authenticate` | -| keeps one object per principal key | defines `Revocation` and `revokedBy` | -| on `revoke(r)`: ends matching objects and links, passes `r` down | decides when to revoke; stores who is blocked | diff --git a/packages/chord/docs/next/explainer/src/08-derives.md b/packages/chord/docs/next/explainer/src/08-derives.md deleted file mode 100644 index 720542ef5..000000000 --- a/packages/chord/docs/next/explainer/src/08-derives.md +++ /dev/null @@ -1,114 +0,0 @@ - -# Computed Values {#derives lede="Derives, the one footgun that makes them expensive, mapShared, and presence: who is looking."} - -The reader's notes without authors, a note count, the number of people viewing: values computed from other values. -Storing copies would go stale; recomputing them on every read would be wasteful and could not be watched. A -**derive** is computed once per change and sent to viewers as a difference. - -## `self.derive` - -```ts -const count = self.derive("count", [list], ([l]) => l.items.length); -const summary = self.derive("summary", [list, store.pinned], ([l, pinned], previous) => ({ - total: l.items.length, - pinned: pinned.length, -})); -``` - -`self.derive(name, sources, (values, previous) => value)`. Sources can be documents, readables, other derives and -state members of references. The value is recomputed when a source changes and diffed into operations for the wire. -Derived values are adopted before ordinary observers run and share their source commit's wire batch, so a viewer -never sees a note list and a count from different commits. - -`previous` is the derive's last value, kept across factory re-runs (reloads) by name. It is a cache, empty after a -host teardown, so a derive must be correct without it. A derive that throws fails its whole owner `broken`. - -## How the difference is found - -Chord compares the new value with the previous one using Delta's `diffRevisions`. Parts that are the **same objects** -as last time cost one `===` each. Parts that are new objects must be compared by content. - -**The footgun.** A derive that builds new objects for unchanged data makes every commit cost the size of the whole -value, per view. The reader list from chapter IV, written naively: - -```ts -self.derive("public", [list], ([l]) => ({ items: l.items.map((n) => ({ id: n.id, text: n.text, at: n.at })) })); -// one note added to 2,000: 2,001 new objects, every one compared by content -``` - -**`mapShared(f)`** keeps one result per source element: `f` runs only for new or changed elements, unchanged ones -return their previous result object. - -```ts -import { mapShared } from "@earendil-works/chord/delta"; - -const stripAuthor = mapShared((n: Note): PublicNote => ({ id: n.id, text: n.text, at: n.at })); -self.derive("public", [list], ([l]) => ({ items: l.items.map(stripAuthor) })); -// one note added: one new object, 2,000 identical references; the diff emits one insert -``` - -:::review | Review: mapShared is planned, not available -Design §12 introduces `mapShared` as coming "from `@earendil-works/chord/delta`", but nothing in `packages/chord/src` -defines it yet, and Delta does not yet treat bytes as atomic leaves in `diffRevisions`. Design §28 lists both as the -step "Delta in pi", before the layered rebuild. Until then, keep a `Map` from source element to result yourself, which -is what `mapShared` will do. -::: - -## Cookbook - -| Case | How | -| --- | --- | -| select a field, look up by key, a small aggregate | compute it; the value is small | -| filter or map a list | `mapShared` over the elements | -| redaction | preferably separate documents per visibility; otherwise `mapShared`, computed once in the shared store, never per view | -| join two sources | `mapShared` plus the looked-up source's revision | -| a window over a large list | take a section of the source elements without rebuilding them; cache per-element measurements by identity; keep the window's start stable so changes arrive as appends | -| a large view combined from several sources | re-path the sources' operations, as Durable's `ConversationView` does; application code never interprets ops | - -**Container width** (R24): structural sharing is O(change) along a path, not across a container's width. Changing one -item of a flat array copies the array's slots, and `mapShared` visits every element once (one identity comparison -each). Large collections are keyed documents or bounded windows. A benchmark of long arrays is to set the documented -limit. - -## Readables that are not documents - -Three small readables round out the toolkit: - -- `constant(value)`: never changes. Start data uses it (chapter IX). -- `self.writable(initial)`: a host-local readable of any in-process value (functions included), compared by identity, - with `set`. It never goes on the wire; it exists for registry contributions that change (chapter XIII). -- Source readables from elsewhere: a Durable view or document watch is already a Chord readable, because Durable and - Chord v2 share the readable type (`chord/src/state.ts`), so it can feed a derive or be exposed as state directly. - -:::review | Review: exposing a writable (Q14, open) -A `writable` may hold functions; nothing yet stops a factory from returning one as a state member, which would reach -the wire encoder. Review round 3 proposes that only members with a JSON schema in the contract are exposable, checked -when the object is served. Undecided. -::: - -## Presence: who is looking - -`host.references(Type)` is a readable of who holds references to objects of a type on this host: input, principal and -client. Client holds carry an opaque origin id, so two tabs are two clients. - -```ts -host.serve(NoteStore, ({ session }, self) => { - const viewers = self.derive("viewers", [host.references(Notes)], ([refs]) => - refs.filter((r) => r.input.session === session).length); - return { …, viewers }; -}); -``` - -A link drop is a departure: accurate and simple. Smoothing (a phone switching networks makes the count flicker) is the -UI's job, for example a debounced count. Counts per person are derived by the app from principals and origins. There -is no consumer-side linger: a closed tab closes its link and keeps nothing. The provider's `linger` keeps a -just-closed object warm, and that also keeps the worker alive through `host.activity`. - -**"Leaving" is never derived from an unmount.** A UI reload unmounts and remounts components; a component unmounted by -a reload has not left. Leaving is explicit navigation, or presence (a dropped reference). This replaces p2's presence, -viewer and login-abandon leases with one mechanism. - -:::note | Built so far -Readables and derives are layer L2 (a readable is today's `replicatedState`, derives reuse today's derived cells over -it). Not started in the rebuild. -::: diff --git a/packages/chord/docs/next/explainer/src/09-natives.md b/packages/chord/docs/next/explainer/src/09-natives.md deleted file mode 100644 index ed7dacbbf..000000000 --- a/packages/chord/docs/next/explainer/src/09-natives.md +++ /dev/null @@ -1,160 +0,0 @@ - -# Things the OS Gives You {#natives lede="Start data, host-lifetime objects and natives: how a plugin opens a file watcher or a socket so that it is closed exactly once, also across reloads."} - -Studio's notes get an inbox: a text file dropped into the session's `inbox/` directory becomes a note. That needs a -file watcher, a **native**: an OS-level value (a file handle, a socket, a process, a database connection) that cannot -be JSON, cannot be copied, must be opened, and must be closed exactly once. It also needs the session's data -directory, which the plugin does not know. This chapter covers both. - -## Start data is an object - -A worker knows things before any plugin loads: its session id, the data directory, init data from the server's start -message. There is no special mechanism for them. Start data is an ordinary `local` object with `lifetime: "host"` and -constant state, served by base code's core facet: - -```ts -// studio/src/core/contract.ts -export const Init = obj({ id: str(), dataDir: str(), cwd: str(), env: opt(str()) }); -export const SessionInfo = local("studio.session-info", { - version: 1, - input: obj({}), - members: { init: Init }, -}); - -// studio/src/core/worker.ts -export const coreWorkerFacet = (init: Static) => defineFacet((host) => { - host.serve(SessionInfo, () => ({ init: constant(init) }), { lifetime: "host" }); - // … the Harness and other core objects (chapter XIV) -}); -``` - -Any plugin on the worker declares a reference and reads it **after activation**: in an opener, a derive, a routine or -a method, never synchronously in the factory. - -```ts -const info = self.get(SessionInfo, {}); -// later: info.init.value.id, info.init.value.dataDir -``` - -Paths are derived by the plugin from generic start data (`join(dataDir, id, "notes-inbox")`), never provided per -plugin. Platform bindings work the same way: on a Durable Object, a core object's `self.open` returns the binding -(`state`, `env`) and its close does nothing. - -## `self.open` - -```ts -const watcher = self.open( - async ({ signal }) => watchDir(path, { signal }), // open: during activation - (w, reason) => w.close(), // close: when the object ends or restarts - { closeTimeout: 10_000 }, // optional; host default 30 s -); -watcher.value; // the watcher; reading it before activation throws not_ready -``` - -`self.open` returns a handle with `.value` (type `Native`), not a proxy. The rules: - -- **Natives live only in objects.** Not at module level, not in the facet. A module-level watcher would survive a - reload with nobody to close it. -- **Order:** natives open one after another in declaration order (a later opener may read an earlier handle's - `.value`) and close in reverse. The object goes live after all opens succeeded. -- **The value never changes during an attempt.** No replacement, no versions, no stale-handle logic. To change a live - native, the object restarts. -- **A failed open** makes the object `failed`: transient with backoff by default, broken if the opener says so - (`throw broken(error)`). A partly failed activation closes the natives already open, with reason `end`; the next - attempt opens all of them anew. -- **`close` gets a reason:** `{ kind: "reload" }` (the plugin reloads and this native object restarts), `{ kind: - "stop" }` (the host shuts down) or `{ kind: "end" }` (the last reference ended, or the type was removed). Each - `close` is bounded by `closeTimeout`, then reported as a possible leak while the rest continues. A throwing `close` - does not stop the others; `host.close()` collects the errors, background ends report to `onError`. -- **Behaviour around a native goes into routines** (chapter X). Listeners on natives outside routines are forbidden. - -An object with at least one `self.open` is a **native object**; one without is a **pure object**. The difference -decides what a reload does (chapter XV): pure objects are swapped in place, native objects restart. - -## The inbox - -The inbox must run as long as the worker, whether anyone looks at the notes or not. So it is a `lifetime: "host"` -object. The facet cannot `self.get` a type it serves, and the inbox has no meaningful input, so it is an object -without input that reads its session from the start data: - -```ts -// notes/contract.ts -export const NoteInbox = local("notes.inbox", { version: 1, input: obj({}), members: {} }); - -// notes/worker.ts -host.serve(NoteInbox, (_input, self) => { - const info = self.get(SessionInfo, {}); - const watcher = self.open(async ({ signal }) => { - const { id, dataDir } = info.init.value; // read in the opener: after activation - return watchDir(join(dataDir, id, "notes-inbox"), { signal }); - }, (w) => w.close()); - - self.routines.loop("import", async (ctx) => { // chapter X - const store = await ctx.get(NoteStore, { session: info.init.value.id }); - for await (const file of watcher.value.files(ctx.signal)) { - await store.add({ text: await readFile(file, "utf8"), by: "inbox" }, ctx); - await rm(file); - } - }); - return {}; -}, { lifetime: "host" }); -``` - -Note `ctx.get(NoteStore, …)` in the routine rather than `self.get` in the factory. The factory runs before the -session id is known (start data is read after activation), so a host-lifetime object without input gets -session-scoped objects with `ctx.get` in a routine or method (R22). A `get` whose input is computed after activation is -to be added only if a real case needs it. - -## Natives with work in flight - -A restart is sequential (R16): close the old native, then open the new one. When the old one still has work running, -use one of two patterns: - -- **Drain in the background.** `close` detaches the old resource and returns at once; it disposes of it when its work - finishes, or after `closeTimeout` at most. Chord opens the new one immediately. Code mode's sandbox does this: a - script running during a reload finishes on the old sandbox (for up to 30 s, Q7; a longer one is aborted and its turn - fails), new scripts start on the new one. -- **Swap through a registry.** The resource is a registry contribution (chapter XIII). Readers always look up the - current one at use; a reader that must wait for one uses a cancellable "wait until present" with the park deadline. - -Chord aborts only the object's own calls and jobs on close. Other work that uses the native (Durable tool calls on a -sandbox) is the owner's business: `close` may drain instead of tearing down. What an old native does after its owner -restarts is also the owner's `close`: a model runtime keeps working until collected, a closed SSH connection fails its -in-flight `exec`. - -## Natives that follow a relocation - -A session can move to another directory. The inbox watcher must follow. Restarting the object on every move would be -heavy-handed; instead the native is a stable manager opened once, and a routine on the session location switches its -internal resource and bumps a generation, so late events from the old location are dropped: - -```ts -const watcher = self.open(async ({ signal }) => new SwitchableWatcher({ signal }), (w) => w.close()); -self.routines.on(location.cwd, (cwd) => watcher.value.switchTo(join(cwd, ".notes-inbox"))); -``` - -## Call-scoped natives - -A native that lives exactly as long as one call (a temporary SSH session for a directory listing, a file handle for a -download) is opened and closed inside the method with `try/finally`. There is no `ctx.open`: - -```ts -list: async ({ path }, ctx) => { - const session = await ssh.open(config, ctx.signal); - try { return await session.readDir(path, ctx.signal); } - finally { session.close(); } -}, -``` - -## Core vs. plugin - -`SessionInfo` comes from base code, `NoteInbox` from a plugin; both use the same primitives. Where code lives decides -which it is. **Base code** (the app's `src/`, imported statically by the start code under a fixed name such as -`@studio/core`) is always there, not switchable, and a change restarts the hosts. **A plugin** (its own directory, -found by listing, named by its `package.json`) is switchable and reloads. Core is what the app always needs and -everything builds on: the harness, the session list, access, the session location. Optional features are plugins; -code mode is a plugin. - -:::note | Built so far -Natives and host-lifetime objects are layer L4. `constant` is decided (R23), not built. -::: diff --git a/packages/chord/docs/next/explainer/src/10-routines.md b/packages/chord/docs/next/explainer/src/10-routines.md deleted file mode 100644 index 4191f21d0..000000000 --- a/packages/chord/docs/next/explainer/src/10-routines.md +++ /dev/null @@ -1,145 +0,0 @@ - -# Standing Behaviour {#routines lede="Routines, reconciling instead of notifying, and how two owners keep a rule between them without a shared transaction."} - -Notes older than 30 days should go every hour. Inbox files should be imported as they arrive. Notes of an archived -session should become read-only. Each of these is behaviour that stands, not a call someone makes. Written as bare -`setInterval`s and event listeners, it would keep running old code after a reload and touch state from nowhere. Chord -has one declared form for it: the **routine**. - -## Four kinds - -```ts -self.routines.every("cleanup", HOUR, (ctx) => dropOlderThan(30 * DAY, ctx)); // due time survives reloads -self.routines.on(settings, async (value, ctx) => applyLimit(value, ctx)); // a readable or event source -self.routines.loop("import", async (ctx) => { // runs until the object ends - while (!ctx.signal.aborted) await importNext(ctx); -}); -self.routines.custom("ticker", ({ signal, run }) => { // anything else - const timer = setInterval(() => run(tick), 1000); - return () => clearInterval(timer); // cleanup undoes the setup -}); -``` - -Routines start when the object goes live, stop when it ends, and restart with the new code on every reload of their -plugin. Each run gets its own `ctx`; errors are caught and reported, never uncaught. Routines are named (`on` -optionally by its source) so Chord recognizes them across reloads; that is how `every` keeps its due time. - -## Serialized per routine - -The new instance of a routine starts only after the old handler in flight has settled; only that routine waits, for at -most the routine handover timeout (10 s), after which it starts anyway and the stuck handler is reported. - -The reason, as a trace: a supervisor routine reconciles pending worker restarts. The old handler is midway through -`restart(W1)` when the plugin reloads. Without serialization the new routine starts at once, sees W1 still pending, and -restarts it a second time. - -## Reconcile, don't track - -A routine **reconciles from the current value**: it makes the world match what it sees now. It writes no "in -progress" flags, and it starts a job for long work (chapter XI). A handler that only cares about the latest value -compares with the current value itself; there is no `overlap` option. Subscribing to the object's own documents is -gapless: every commit is seen. - -No bare timers or callbacks that touch state outside routines, jobs or calls. That is what lets Chord stop all of an -object's behaviour when it ends or reloads. - -## Rules between two owners - -Every thing has exactly one owner, the smallest object it belongs to. Some rules span two: "notes of an archived -session are read-only" involves the catalog (which knows archiving) and `NoteStore` (which holds notes). Two tempting -designs are wrong: - -- **A cross-owner transaction** (archive and lock the notes atomically) would make the catalog know every plugin and - couple their storage. -- **A notification** (`catalog` calls `notes.archive(session)`) can be lost: the worker is not running, the link drops, - the process crashes between the archive and the call. - -The design's rule: **the owner of the dependent data enforces the rule** by reconciling from the other owner's state -with an idempotent routine. - -```ts -// NoteStore: an archived session's notes are read-only -const flags = self.state(NoteFlags); // { archived: boolean }, NoteStore's own document -const catalog = self.get(Catalog, {}); // the server's session list: another owner -self.routines.on(catalog.sessions, async (sessions, ctx) => { - const archived = sessions[session]?.archived ?? false; - if (flags.value.archived !== archived) await flags.change((d) => { d.archived = archived; }, ctx); -}); -// add and remove refuse while flags.value.archived -``` - -After a crash the routine runs again with the current value. The catalog does not know the notes plugin exists. The -same pattern closes a forward of an archived session in the ports plugin, aborts a job whose grant was revoked -(chapter VII), and keeps any invariant between plugins. - -## Reading without starting a worker - -The session list in the sidebar shows each session's note count. Getting `NoteStore { S1 }` for every session would -start every worker. The answer is plain publishing between two objects of one plugin, no Chord mechanism: the worker -side reports a summary to a server-side object by a method call from a routine, and clients read the server object. - -```ts -// notes/contract.ts -export const NoteCounts = remote("notes.counts", { - version: 1, input: obj({}), placement: () => Server(), - variants: { - owner: { state: { counts: map(int()) } }, - worker: { methods: { report: method(obj({ session: str(), count: int() })) } }, - }, -}); - -// notes/server.ts -const Counts = defineState("notes.counts", { - schema: obj({ bySession: map(obj({ count: int(), incarnation: str() })) }), - version: 1, initial: () => ({ bySession: {} }), persist: true, // survives a server restart - checkpointWhen: (_v, _o, info) => info.deltasSinceBase > 64, -}); - -host.serve(NoteCountStore, (_input, self) => { // local, lifetime "host" - const counts = self.state(Counts); - return { - counts: self.derive("counts", [counts], ([c]) => mapValues(c.bySession, (e) => e.count)), - report: ({ session, count, incarnation }, ctx) => - counts.change((d) => { - const last = d.bySession[session]; - if (last && last.incarnation > incarnation) return; // uuidv7: an older worker, ignore - d.bySession[session] = { count, incarnation }; - }, ctx), - }; -}, { lifetime: "host" }); - -host.serve(NoteCounts, (_input, principal, self) => { - const store = self.get(NoteCountStore, {}); - if (principal.kind === "worker") return { - variant: "worker", - report: ({ session, count }, ctx) => { - if (session !== principal.session) throw forbidden(); // a worker reports only itself - return store.report({ session, count, incarnation: ctx.incarnation }, ctx); - }, - }; - return { variant: "owner", counts: store.counts }; -}); - -// notes/worker.ts, inside NoteStore -const counts = self.get(NoteCounts, {}); // a reference to the server -self.routines.on(count, (n, ctx) => inVariant(counts, "worker") && counts.report({ session, count: n }, ctx)); -``` - -The server object keeps the latest value per session (in a document if it must survive a server restart) and -**ignores reports from an older worker incarnation** (incarnations are uuidv7, so they order by start time). A worker that was replaced may still deliver a late report; the -fence is the caller's incarnation, which Chord stamps on every call from the link handshake (R20): `ctx.incarnation`, -next to `ctx.principal`, not part of the principal or its key, never sent as data, so it cannot be forged. Session -status, subagent lists, unread marks and artifact lists in p2 all follow this pattern. The store is local and has no -caller incarnation of its own, so the view passes it on as data. - -:::review | Review: `ctx.incarnation` across a relay (Q13, open) -A worker's call to the server arrives directly, so `ctx.incarnation` is the worker's. But a call relayed through the -server (a tab's call to a worker) carries the server's incarnation on the worker's side, because the worker's -handshake is with the server. Review round 3 proposes that relays stamp the origin host's incarnation, as they stamp -the principal. Undecided; the pattern above only needs the direct case. -::: - -:::note | Built so far -Routines are layer L8. The design treats `every` as one of four forms over one runner; review round 3 proposed cutting -to `on` and `loop` only (C4), not decided. -::: diff --git a/packages/chord/docs/next/explainer/src/11-jobs.md b/packages/chord/docs/next/explainer/src/11-jobs.md deleted file mode 100644 index e506c05df..000000000 --- a/packages/chord/docs/next/explainer/src/11-jobs.md +++ /dev/null @@ -1,171 +0,0 @@ - -# Work That Outlives the Call {#jobs lede="Jobs: owned work started by a method that keeps running after its caller leaves, with cancel, dedupe and a recorded outcome."} - -Importing a 50 MB text file into notes takes a minute. The tab that started it may close; the import must go on, be -cancellable, not run twice when the tab retries after a link blip, and its result must be recorded. A method cannot do -this (it dies with its caller), a routine is standing behaviour rather than a request, and a Durable task is for work -that must resume after a crash. Chord's form for it is the **job**, built in pi-durable's sandwich: **intent** (a -transaction), **effect** (anything), **outcome** (a transaction in the current code). - -## Defining and running a job - -```ts -// notes/worker.ts -const ImportJob = defineJob("notes.import", { - version: 1, - args: obj({ path: str() }), - progress: obj({ lines: int() }), - result: obj({ added: int() }), - concurrency: 1, // one import at a time per NoteStore -}); - -const ImportResults = defineState("notes.import-result", { - schema: obj({ outcome: union([obj({ added: int() }), obj({ error: str() })]) }), - version: 1, initial: () => ({ outcome: { added: 0 } }), keyed: true, - checkpointWhen: () => true, -}); - -host.serve(NoteStore, ({ session }, self) => { - const list = self.state(NoteList); - const results = self.state(ImportResults); - const imports = self.job(ImportJob, { - run: async ({ path }, ctx) => { // 2. effect - let lines = 0; - for await (const line of readLines(path, ctx.signal)) { - await addNote({ id: `${ctx.jobId}:${lines}`, text: line, by: "import" }, ctx); // keyed by job id - ctx.progress({ lines: ++lines }); - } - return { added: lines }; - }, - finish: async (outcome, tx) => { // 3. outcome, current code - (await tx.draft(results, outcome.id)).outcome = - outcome.kind === "done" ? outcome.result : { error: outcome.error }; - }, - }); - return { - …, - imports: imports.running, // live list of runs - startImport: ({ path, requestId }, ctx) => - imports.start({ path }, ctx, { key: requestId }), // 1. intent (refined below) - abortImport: ({ id }) => imports.abort(id), - }; -}); -``` - -`defineJob(name, { version, args, progress, result, concurrency?, persist?, accepts? })`; `progress` and the outcome -are typed by the schemas. - -:::sequence animate | The job sandwich. The intent commits before anything runs; the effect runs outside any transaction; the outcome commits in the current code. -actor tab "tab" -actor store "NoteStore { S1 }" accent -actor run "run" "effect" blue -tab -> store: startImport({ path, requestId }) -store => store: 1. intent: busy check, dedupe by key, commit -store --> tab: { id } -store -> run: start after the commit -run -> run: read lines, add notes (keyed by jobId) -run --> store: progress { lines } (throttled) -note tab..store: the tab may close here: the job counts as a reference -run --> store: settles: { kind: "done", result } -store => store: 3. record outcome, finish(outcome, tx), ack -::: - -## The three steps - -**1. Intent.** `job.start(args, ctx, { key? })` is shorthand for `tx.start(job, args, { key? })` inside a -transaction, which may contain other writes. The busy check (`concurrency`) and deduplication (the same key returns the -existing run) are atomic and in the same commit; `run` starts only after the commit. Run ids are uuidv7. - -**2. Effect.** `run(args, ctx)` may do anything: processes, network, other objects. It writes no document itself; it -returns the outcome. External writes carry a dedupe key derived from `ctx.jobId` (above: note ids), so an interrupted -attempt cannot apply them twice. On a reload `run` keeps its old code. - -**3. Outcome.** When `run` settles, Chord records the outcome in its own record, then calls the **current** code's -`finish(outcome, tx)` and acknowledges the run in the same commit. The outcome is tagged: - -```ts -type Outcome = { id: string } & ({ kind: "done"; result: R } | { kind: "failed"; error: "threw" | "aborted" | "lost" }); -``` - -A throwing `finish` changes nothing and is retried with a backoff (1 s to 60 s), and after the next reload. `finish` -is optional and follows Durable's transaction rule. - -## Dedupe ends with the run - -Deduplication covers **unsettled runs only** (Q9). Once `finish` has committed, Chord drops the run's record and key -and releases its waiters' references; it keeps nothing beyond a run's lifetime. Dedupe after completion is the -owner's. Studio records which run a request started, in the same commit as the intent, and `finish` records the -result by run id (`ImportResults` above): - -```ts -const Requests = defineState("notes.import-request", { - schema: obj({ run: str() }), version: 1, initial: () => ({ run: "" }), keyed: true, persist: true, - checkpointWhen: () => true, -}); - -startImport: async ({ path, requestId }, ctx) => { - const prior = await requests.get(requestId); // read outside the transaction (R13) - if (prior) return { id: prior.run }; // started before, maybe finished long ago - return self.tx(async (tx) => { - const run = await tx.start(imports, { path }, { key: requestId }); // dedupes a concurrent retry - (await tx.draft(requests, requestId, { tags: [SessionTag(session)] })).run = run.id; - return { id: run.id }; - }, ctx); -}, -``` - -Two concurrent retries both miss `prior`, but the key makes `tx.start` return the same unsettled run. The client -watches `imports` for progress and reads the result by run id once it is gone from `running`. - -Why Chord does not keep completed runs itself: either 100,000 keyed imports would stay in memory forever, or a -retention limit would let a retried key rerun its effect. The owner knows which results matter. - -## Watching, waiting, aborting - -- **`running`** is a readable of `{ id, version, args, progress, phase }`. `phase` is `running`, `finishing` (outcome - recorded, `finish` not yet committed) or `blocked` (with the reason). `concurrency` counts only active runs. The - panel shows a progress bar from it. -- **Progress** is in memory, throttled and sent as deltas. For output, use `OutputBuffer` and `Progress` from - `@earendil-works/chord/output`: bounded windows, adaptive throttling, no write amplification. -- **Waiting:** `start` returns a run with `outcome(ctx)`. The caller's abort ends only its wait, never the run. -- **Abort is a request:** `job.abort(id)` fires the run's `ctx.signal`; the run decides what it returns (a result - that already arrived stays a success). A run that ignores the signal keeps its slot until it ends. `finish` runs in - Chord's own context. - -## Lifetime, persistence, revocation - -**Jobs keep their owner alive.** A running, finishing or blocked job counts as a reference to its object, which ends -only once all its jobs are settled. Starting an import and closing the tab does not abort it. Unsettled runs also -count as activity for the worker's keep-alive. - -**Revocation takes precedence** (R11). Revoking a principal ends its objects at once, whatever jobs remain; the jobs -get the abort signal and the normal close timeout. A job runs as its object's principal. - -**Persistence:** jobs are in memory by default and gone with the process. `persist: true` (only for stable -identities) keeps the record: a run live at a crash ends `lost` at the next activation and `finish` records it. -Nothing is rerun automatically. - -**Versions:** a run started under an older job version keeps its version; `finish` and `running` see it through pure -`accepts` adapters, and stored outcomes are never rewritten. A version bump without an adapter does not refuse the -reload: the old outcome is `blocked`, kept and reported until a later reload brings the adapter. - -**Native objects:** a job in a native object ends `aborted` when the object restarts on a reload (chapter XV). In a -pure object it keeps running on its old `run` code. - -## What is not a job - -| Work | Use | -| --- | --- | -| request and response that may die with its caller | a method | -| input into running work (a pasted code, an approval) | a flow (chapter XII) | -| work that must resume after a crash | a Durable task | -| standing behaviour | a routine | - -:::review | Review: jobs are the largest open simplification (#9, C3) -Decision #9 (deferred until login is ported): a login or a long relocation is a job whose state consumers want to -address and watch (`LoginFlow { id }`). Options: (a) jobs stay records inside their owner, watched through `running`; -(b) a job is an object of its own, addressed by its id, living until done. Separately, review round 3 proposed cut C3: -shrink jobs to a runner over documents (no job versions, `accepts`, `blocked`, `persist`) or move them out of the -kernel. Not decided. `chord-next-walkthrough.md` concept 17 still shows the older outcome shape `{ ok: true, result }`; -the design tags outcomes by `kind`. -::: diff --git a/packages/chord/docs/next/explainer/src/12-flows.md b/packages/chord/docs/next/explainer/src/12-flows.md deleted file mode 100644 index cf6a986c1..000000000 --- a/packages/chord/docs/next/explainer/src/12-flows.md +++ /dev/null @@ -1,141 +0,0 @@ - -# Asking a Person {#flows lede="Flows and the Questions service: how running work waits for an answer from someone who may be on any tab, may reconnect, and may answer twice."} - -A provider login asks for a pasted code, then an account choice. A dangerous tool call waits for the owner's -approval. The agent asks everyone present a question. In each case running work waits for a person, who may be on any -tab, may lose the connection halfway, may answer late or twice. Two patterns cover this: a **flow** kept as its -owner's state, and the **Questions** service, a shared store for questions. - -## A flow is state, not a call - -The tempting design is a long-held server call, `step()`, that blocks until the next step. A link drop fails it, and -the consumer cannot tell whether it missed a step. Chord's answer is that the owner keeps the flow as **state**, which -re-holds by itself after any drop: - -```ts -// models/contract.ts (sketch from the design) -start: method(obj({ provider: str() }), obj({ id: str() })) // uuidv7 -answer: method(obj({ id: str(), step: int(), value: str() }), Answered) // ok | "step no longer active" -cancel: method(obj({ id: str() })) - -// LoginFlow { id }: state -type FlowState = - | { kind: "running"; step?: { id: number; prompt: LoginPrompt }; notices: Notice[] } - | { kind: "done" | "failed" | "cancelled"; notices: Notice[] }; -``` - -The UI watches the state and renders the current step. A late or second answer to a step that is gone is refused. A -promise-based callback inside the work (pi-ai's `prompt` during a login) stays **inside** the owner: its job keeps the -resolver, and nothing crosses an object boundary. - -Imperative consumers (tests, scripts, a CLI, an agent tool) use a generic client helper over the same state, not a -second protocol: - -```ts -async function* steps(flow: LoginFlowRef, ctx: Ctx) { - let last = 0; - for (;;) { - const s = await until(flow.state, (v) => v.kind !== "running" || (v.step !== undefined && v.step.id > last), ctx); - if (s.kind !== "running") return s.kind; - last = s.step!.id; - yield s.step!; - } -} - -for await (const step of steps(await ctx.get(LoginFlow, { id }), ctx)) - await login.answer({ id, step: step.id, value: answerFor(step) }, ctx); -``` - -`last` keeps steps from being lost or repeated across re-holds. - -:::review | Review: where a flow lives (#11, deferred) -Decision #11, deferred until login is ported: (a) a keyed document of its owner, or (b) an object `LoginFlow { id }` -living until done; tied to #9 (jobs as objects). The `LoginFlow { id }` address above assumes (b). Also undecided is -how "leaving" ends a login (p2 cancels when the starting owner leaves everywhere); with presence it is a dropped -reference, never an unmount (chapter VIII). -::: - -## The Questions service - -The Questions service (`apps/p2/docs/questions.md`) is a p2 design, not a Chord feature, built from Chord's -primitives: a shared store with per-principal views. The asker gets `ask`; viewers get the open questions and -`answer`. - -```ts -export type Question = - | { kind: "text"; message: string; secret?: boolean; help?: string } - | { kind: "select"; message: string; options: readonly { id: string; label: string }[] } - | { kind: "confirm"; message: string; allow: string; deny: string }; - -type Policy = { mode: "first" } | { mode: "quorum"; count: number } | { mode: "all"; join?: "live" | "snapshot" }; - -ask(question, { - audience, // a readable or a list of principals - policy, - key?, // stable per asking flow: a re-ask reattaches to the pending question - deadline?, // then resolve with what is there; `missing` names who did not answer -}, ctx): Promise; -``` - -Its rules: - -- **The store is the truth.** Pending questions and their resolvers live in the store; `open` is a readable over it, - filtered per viewer. Nothing is a document, so a reload or migration cannot leave a ghost question. -- **One settlement point.** Register the waiter and the abort listener before publishing the question; an answer, an - abort or the deadline atomically takes the entry out before resolving. A late or second answer gets "no longer - active". Answers are validated against the stored question (a select id must be one of the options). Never await an - answer inside a transaction. -- **Policies over a live audience.** `first` resolves with the first answer; `quorum(n)` after n answers; `all` when - every member of the audience has answered. With `join: "live"` (the default), someone who joins becomes required and - someone who leaves drops out. -- **Authorization.** The asker is stamped by the host (a worker asks only in its own session's scope; secret questions - only from server askers). Each question stores who may answer it; a guest's send access does not imply approval - rights. Secret answers never appear in `open`, documents or logs. -- **Link drops and crashes.** A rejected `ask` is not a denial. Askers retry with the same `key` until their own signal - aborts; a pending `ask` counts as an active call for keep-alive. -- **Where it lives.** On the host where the asking work runs: session questions on the session's worker, login - questions on the server. UIs reach both through the usual relay. - -The store in Chord v2 terms: - -```ts -host.serve(QuestionStore, ({ scope }, self) => { - const pending = self.open(async () => new PendingQuestions(), (p) => p.rejectAll("store closed")); - const open = self.derive("open", [pending.value.readable], ([all]) => all); // data only, no resolvers - return { open, ask: (args, ctx) => pending.value.ask(args, ctx), answer: (args, ctx) => pending.value.answer(args, ctx) }; -}); - -host.serve(Questions, ({ scope }, principal, self) => { - const store = self.get(QuestionStore, { scope }); - if (isHost(principal)) return { variant: "asker", ask: (args, ctx) => store.ask({ ...args, asker: principal }, ctx) }; - if (!mayView(principal, scope)) throw forbidden(); - return { - variant: "viewer", - open: self.derive("mine", [store.open], ([all]) => openFor(all, principal)), - answer: ({ id, value }, ctx) => store.answer({ id, value, by: principal }, ctx), - }; -}); -``` - -The pending map with its resolvers is in-process state with promises, so it is a native of the store (opened and -closed with it). Its `close` rejects every waiter; askers retry with their key against the new store after a reload. - -:::review | Review: questions.md predates the v2 rules -`apps/p2/docs/questions.md` writes the store as `self.resource(...)` with a `replace` hand-over, which the design -rejected ("resources replaced in place"), and opens Chord call scopes in Durable hooks with `host.root`, which Q6 -replaced with `chord(fn, context)` (chapter XIV). Design §20 says the Questions store "must live in an object (it -predates natives only in objects)". The sketch above is this guide's translation, with the derive per viewer; filtering -per viewer in the store, once per variant, may be cheaper for large audiences. The `ask` signature passes typed -objects as arguments, which is fine (data), but the `audience` readable form cannot cross a link: across links, -audience must be data (principals) or computed by the store. -::: - -## Choosing - -| Case | Use | -| --- | --- | -| a multi-step login in one owner | a flow: the owner's job keeps the resolver, state shows the steps | -| a tool needs the owner's approval | Questions, `first`, audience the owner | -| the agent asks everyone present | Questions, `all`, audience from presence | -| deploy needs 2 of 3 approvals | Questions, `quorum(2)` | -| a notice (auth URL, device code, progress) | not a question: the asking work's own progress state | diff --git a/packages/chord/docs/next/explainer/src/13-registries.md b/packages/chord/docs/next/explainer/src/13-registries.md deleted file mode 100644 index 22efc7019..000000000 --- a/packages/chord/docs/next/explainer/src/13-registries.md +++ /dev/null @@ -1,185 +0,0 @@ - -# Extension Points {#registries lede="The facet, registries and the HTTP component: how plugins add behaviour to each other without callbacks, and how the owner stays in charge."} - -The notes plugin wants a download route, `GET /s/:session/notes.txt`. A logs plugin wants to time every request. -Several plugins add model providers, tools for the agent, commands for the TUI. Someone must combine these -contributions, decide conflicts, and replace one plugin's part as a whole when it reloads. Between plugins exactly two -things pass: object types and **registries**. - -## The facet, completed - -The facet has appeared since chapter II as the place that serves types. It is also itself a **pure object** with the -host's lifetime: - -```ts -// notes/server.ts -export default defineFacet((host, self) => { - const prefs = self.state(NotesPrefs); // plugin-wide documents - self.contribute(Http, { routes: { "GET /s/:session/notes.txt": exportNotes } }); - host.serve(NoteCountStore, …, { lifetime: "host" }); - host.serve(NoteCounts, …); -}); -``` - -- It may hold documents, references, routines and contributions; **no natives** (those need an object with its own - lifetime). -- Its identity is the plugin's name, so its persisted documents survive restarts and reloads. -- Served objects go live after the facet and may read and write its documents (chapter V). -- `host.serve` and `self.contribute` are valid only while the setup runs. The facet never `self.get`s a type it - serves. - -## Registries - -A registry is an extension point owned by someone (a Chord component, the app, a plugin). Each contributor -contributes **at most one value** per registry; the table is an immutable, ordered list of `(contributor, value)`; -the registry's `compose` turns it into the value consumers use. - -```ts -// owned by the HTTP component -export const Http = defineRegistry("chord.http", { - version: 2, - accepts: { 1: (old: HttpContributionV1): HttpContribution => upgrade(old) }, - compose: (table, { report }) => buildRouter(table, report), // pure; always delivers a value -}); - -self.contribute(Http, { routes: { … }, wraps: [ … ] }); // a contributor -const router = host.registry(Http).value; // a consumer: live -``` - -**`compose` runs at runtime, whenever the table changes:** after a reload's commit, when a contributor goes live or -ends, when a contribution changes. It is not a staging check: contributions can be dynamic (known only once a native -is open), so no check before a reload could be complete. `compose` always delivers a value with its fallback built in -(e.g. the first by plugin name wins) and reports conflicts through `report`, which become owner notices and telemetry -events. Readers get a plain value and never handle conflicts. Only if `compose` throws is the registry in an error -state, and readers get an error when they use it. Work that started earlier keeps the value it read. - -**Rules:** - -- **Who contributes** (R21): only facets and `lifetime: "host"` objects; only their `self` has `contribute` (a type - error elsewhere). Every object may read registries. -- **A contribution is in the table exactly while its contributor is live:** a facet's while its plugin is loaded; an - object's from `live` until it ends, fails or restarts. -- **Order** is sorted by plugin name: deterministic on every host and across reloads, and deliberately meaningless. - Meaningful order comes from the owner's composition (HTTP phases, below). Helpers: `keyed(table, entriesOf)`, - `ordered(table, itemsOf, orderOf?)`. -- **One authoritative declaration** (Q8): only the defining plugin's facet declares the registry to the host - (`compose`, version, `accepts`). Other plugins' tokens only identify it by name and types. A reload of the defining - plugin replaces the declaration atomically and the table recomposes. If the defining plugin is not loaded, the - registry has no value (readers get an error like a failed reference) and contributions wait in the table. -- **Versions** (Q8): a contribution carries the registry version its plugin was built against; older ones are adapted - by the defining facet's pure `accepts`. One without a matching version or adapter is left out and reported naming - plugin, registry and versions; its plugin keeps working. Contributions are in-process values (often functions) and - are not schema-checked. -- **Use sites guard contributions.** A contribution is untrusted code (third-party plugins are a goal). Every use site - catches throws and rejections when calling an entry, reports naming the contributor, and continues with the others. -- **The owner decides** conflicts, order and selection. Registries add behaviour; they never replace an owner's checks. - -## Natives through registries - -Registries are the only way a native reaches another plugin. An entry lives exactly as long as its contributor; -consumers look up at each use (or follow the composed value) and keep nothing from an entry beyond a use. The models -plugin's host-lifetime object contributes its runtime; the harness looks it up at every request. After a models reload -the next request gets the new runtime, while a running stream finishes on the old one. - -A handler that stores `const r = entry()` in a module variable and uses it for days has a bug: after the contributor -restarts it holds a closed native. - -## Contributions that change - -Some contributions are known only at runtime: MCP tools appear when a server connects. A contribution is either fixed -for its contributor's life (dynamic parts behind handles read at use) or a **readable**, which Chord subscribes to; -on every change Chord replaces that entry and runs `compose` again. For in-process values, `self.writable(initial)` is -a host-local readable of any value (no JSON, no wire, compared by identity) with `set`: - -```ts -const ext = self.writable(mcpExtension([])); // a Durable Extension, no tools yet -self.contribute(HarnessExtensions, ext); -self.routines.on(mcp.tools, (tools) => ext.set(mcpExtension(tools))); -``` - -There is no remove call. An entry leaves when its contributor ends, or after a reload whose new code no longer -contributes. A live contributor with nothing to offer sets its readable to the registry's empty value (an extension -without tools, empty routes); the table keeps exactly one entry per live contributor. - -## Optional dependencies - -Registries also replace optional references (chapter VI). Radius forwards connections to ports the ports plugin -manages, but ports is optional. So radius owns `radius.forwards` and ports contributes its forwards; without ports the -table is empty and radius works. - -## The HTTP component - -HTTP is an optional Chord component (`@earendil-works/chord/next/http`) built on a registry. A route gets a plain -request summary (method, path params, query, principal, provenance such as local or relayed, selected headers, parsed -and capped JSON bodies, uploads in chunks) and a `ctx` whose `principal` is the request's, so `ctx.get(Notes, …)` gives -exactly the view that person may see. The route returns a description; the component does the HTTP work once (ranges -with 206 and 416, HEAD, download headers with a safe filename, `nosniff`, CSP and its nonce, caching, streaming, abort -on disconnect): - -```ts -type HttpResponse = - | { kind: "json"; value: JsonValue } | { kind: "page"; title: string; body: string } - | { kind: "bytes"; name: string; mime: string; size: number; read(offset: number, length: number): Promise } - | { kind: "redirect"; to: string } | { kind: "error"; status: number }; - -// notes/server.ts -self.contribute(Http, { - routes: { - "GET /s/:session/notes.txt": async (req, ctx) => { - const notes = await ctx.get(Notes, { session: req.params.session }); // the requester's view - const text = new TextEncoder().encode(render(notes.notes.value)); - return { kind: "bytes", name: "notes.txt", mime: "text/plain", size: text.byteLength, - read: async (offset, length) => text.subarray(offset, offset + length) }; - }, - }, -}); -``` - -Routes compose with `keyed`: two contributors with one pattern is a reported conflict, and the first by plugin name -wins. **Wraps** are middleware in phases the owner defines (`observe`, `headers`, `transform`, outermost first), -within a phase in contributor order. No numeric priorities: plugins have no single registration order, so Fastify's -lifecycle phases and encapsulation are the model, not Express. - -```ts -// logs/server.ts -self.contribute(Http, { - wraps: [{ - name: "access-log", phase: "observe", scope: "all", // default scope "own": only own routes - wrap: async (req, next, ctx) => { - const t = Date.now(); - const res = await next(req); - log(req, res, Date.now() - t); - return res; - }, - }], -}); -``` - -`route` narrows a wrap to one route; a `route` that does not exist is a reported conflict. Wraps may add a security -boundary on top of an owner's checks, never instead of them. A request uses the router as it was when it started. -Request resolvers (who sent a request: cookie, loopback, relay) are hooks over the summary. - -## Channels and big bytes - -**Channels** are WebSocket byte channels (`send`, `onData`, `close`) for terminals, separate from object links and -with their own acknowledged windows, so a large state message never holds up keystrokes. - -**Big transfers are chunked by the owner,** not by Chord: bytes are a schema type (`bytes()`), carried natively -(never base64), and a 50 MB upload is a protocol of plain calls: - -```ts -beginUpload({ name, size }, ctx) → { upload } -append({ upload, offset, bytes }, ctx) // about 1 MiB per chunk; the awaited call is the backpressure -finishUpload({ upload }, ctx) → { id } -read({ id, offset, length }, ctx) → bytes -``` - -After a link drop the `append` in flight fails like any call; the writer asks the owner for the stored offset and -resumes. Unfinished uploads expire through the owner's routine. Chord has no stream type: it would need its own rules -for drops, cancellation, relays and restarts, which this offset protocol already survives. - -:::note | Built so far -Registries are layer L9. The HTTP component, channels and the chunked-transfer helper are deferred until after the -kernel and a first real proof (the harness, one tool with an approval pause, a code mode reload); until then the -playground serves its pages with plain app code. Their design stays the target. -::: diff --git a/packages/chord/docs/next/explainer/src/14-agent.md b/packages/chord/docs/next/explainer/src/14-agent.md deleted file mode 100644 index 40b69b555..000000000 --- a/packages/chord/docs/next/explainer/src/14-agent.md +++ /dev/null @@ -1,192 +0,0 @@ - -# The Agent {#agent lede="pi-durable inside Chord: the harness as a core object, models, environments and tools through registries, and how a tool reaches Chord objects and people."} - -Each Studio worker runs a pi-durable harness for its session. The harness must run for the worker's whole life, -whether anyone watches or not. Its models come from the models plugin, its execution environments from the envs -plugin, its tools from every plugin, and all of those reload independently. This chapter assumes you know pi-durable -and shows only how it plugs into Chord. - -## Shared foundations - -Durable and Chord v2 share modules: Durable imports Chord's `context`, `delta`, `json`, `output` and `sqlite`, and v2 -imports the same, so `Context`, `Draft` and `JsonValue` are one type each. A v2 readable is today's -`replicatedState` (`chord/src/state.ts`), so Durable's `viewState()` and document watches can be exposed as Chord state -members without conversion. Only plain data crosses between Durable and Chord objects. - -## The harness is a core object - -```ts -// studio/src/core/harness.ts (base code) -import { Harness as DurableHarness } from "@earendil-works/pi-durable"; - -export const ChordHostKey = createContextKey>("studio.chord-host"); -export const HarnessModels = defineRegistry("studio.harness-models", { version: 1, compose: firstModels }); -export const HarnessEnvironments = defineRegistry("studio.harness-envs", { version: 1, compose: firstEnv }); -export const HarnessExtensions = defineRegistry("studio.harness-extensions", { - version: 1, - compose: (table, { report }) => extensionSnapshot(table, report), // one Durable Extension per plugin -}); -const HarnessRuntime = defineRegistry<() => DurableHarness, () => DurableHarness>("studio.harness-runtime", { - version: 1, compose: onlyEntry, // core-private: one contributor -}); - -host.serve(Harness, (_input, self) => { - const info = self.get(SessionInfo, {}); - const harness = self.open(async ({ signal }) => { - const { id, dataDir } = info.init.value; - const base = withContextValue(ChordHostKey, host, withAbortSignal(signal, BACKGROUND_CONTEXT)); - return DurableHarness.open(await openNodeSqliteStorage(join(dataDir, id, "durable.sqlite")), { - models: () => host.registry(HarnessModels).value, // read at each use - env: (target) => host.registry(HarnessEnvironments).value(target), - settings: () => currentSettings(), - registry: durableRegistry(host.registry(HarnessExtensions)), // a RegistryReader over the composed value - }, base); - }, (h) => h.close(BACKGROUND_CONTEXT)); - self.contribute(HarnessRuntime, () => harness.value); // for core's conversation objects - return { … }; -}, { lifetime: "host" }); -``` - -- **Lifetime `"host"`:** it runs for the worker's whole life, also while nobody holds it, because Durable tasks - continue. -- **Base code:** it lives in the core facet, so a plugin reload never restarts it; only a base-code change (a host - restart) does. -- **Registries, looked up at each use:** Durable's `models` and `settings` accept functions (pi `7ec1f648e`), so each - generation gets the current value. After a models reload the next request uses the new runtime; a running stream - finishes on the old one. -- **Durable's storage is its own.** Chord documents never commit together with Durable transcripts. - -The design writes the call as `Durable.open(storage, { … })`; pi-durable's actual entry point is -`Harness.open(storage, options, context)`, used here as `DurableHarness` to keep it apart from the core object -`Harness`. - -## Start order - -At host start Chord loads every facet and waits until every facet and every `lifetime: "host"` object **other than the -harness** is live or failed: the **start barrier**. Then the registries hold the contributions of every plugin that -loaded, and the harness opens and resumes scheduling (R17). An object that must contribute at start, like the model -runtime, is therefore `lifetime: "host"`. Nothing waits for specific contributions; what is missing shows where it is -used: - -| Missing | What happens | -| --- | --- | -| a tool | its call gets an error result the model sees | -| a model | the generation reports "model unavailable" and waits, not fails, until a model appears | -| a task definition | the task stays blocked (`missing_task`) until the plugin is installed or the task is aborted | -| a plugin that failed | reported with its error; a facet in backoff contributes later and the registries recompose | - -The same holds when a session's database is copied to an instance with other plugins: missing tools, sections and -hooks are skipped, documents of missing plugins stay untouched, stored selections are never rewritten. The UI must -show blocked tasks with their reason and offer to abort them; a turn waiting on one hangs until then. - -:::review | Review: the harness exception to the barrier (C17, R17) -R17 makes the harness start after the barrier, a special case in the barrier's rule. Review round 3 proposed cut C17: -no harness-specific exception; the harness is an ordinary object the app's start code holds after `load` resolves. -Not decided. `chord-next-walkthrough.md` concept 28 still says the harness's activation *is* the start barrier, which -R17 replaced. -::: - -## Tools that call Chord - -Durable tools, hooks and sections receive Durable's `ToolExecutionApi` (or `HookApi`) and a `Context`, never a Chord -`ctx` (Q6). Core put the host into the harness's base context under `ChordHostKey`, and every phase context inherits -it. A core helper opens a Chord call scope from any Durable context: - -```ts -// studio/src/core/chord.ts -export function chord(context: Context, fn: (ctx: Ctx) => Promise): Promise { - const host = context.value(ChordHostKey); - if (!host) throw new Error("not inside the Studio harness"); - return host.root(fn, { signal: context.abortSignal, telemetry: context.telemetry }); -} -``` - -The scope runs as the host principal; attribution is data; references taken with `ctx.get` are released when `fn` -settles. There is no wrapping of tools and no app-specific `defineTool`. The notes plugin gives the agent a tool: - -```ts -// notes/worker.ts -const addNote = defineTool({ - name: "notes_add", // tools are named _ - description: "Add a note to this session's notes panel", - parameters: Type.Object({ text: Type.String({ maxLength: 2000 }) }), - execute: (args, api, context) => chord(async (ctx) => { - const info = await ctx.get(SessionInfo, {}); - const store = await ctx.get(NoteStore, { session: info.init.value.id }); - const id = await store.add({ text: args.text, by: `agent:${api.conversationId}` }, ctx); - return { output: [{ type: "text", text: `added note ${id}` }] }; - }, context), -}); - -export default defineFacet((host, self) => { - self.contribute(HarnessExtensions, defineExtension({ name: "@studio/notes", tools: [addNote] })); - … -}); -``` - -A reload of the notes plugin replaces its extension in the table; Durable pins a registry snapshot per task phase, so -a tool call already running finishes on the old code and the next phase uses the new one. - -## Tools that need a person - -Clients are not addressable, so a tool cannot call the user's tab. A tool that needs the user goes through an owner's -state: the Questions service (chapter XII). The model-facing `ask_user` tool and an approval hook both call -`Questions.ask`; tabs render the open questions and answer them. - -```ts -hook(ToolTask, { - beforeTool: async (call, api, context) => { - if (!needsApproval(call)) return undefined; - const decided = await api.memo("approval", context); // a rerun does not ask again - if (decided !== undefined) return decided ? undefined : { block: "Denied" }; - const allowed = await chord(async (ctx) => { - const q = await ctx.get(Questions, { scope: { session } }); - for (;;) { - try { - const r = await q.ask( - { kind: "confirm", message: `Run ${call.name}?`, allow: "Allow", deny: "Deny" }, - { audience: approvers, policy: { mode: "first" }, key: api.taskId }, - ctx, - ); - return r.answer; - } catch (error) { - if (ctx.signal.aborted) throw error; // a link blip is not a denial: retry - await backoff(ctx); - } - } - }, context); - await api.memo("approval", allowed, context); - return allowed ? undefined : { block: "Denied by the user" }; - }, -}); -``` - -Durable runs `beforeTool` in the tool task's call phase; after the execute checkpoint a crash retries execution -without hooks, so the stored decision covers that retry. The approval must cover the final arguments: put the -approval hook after hooks that rewrite arguments. - -## Conversations for the UI - -Chat panels need a conversation's live view, submit, steer, stop. The design leaves p3's conversation API to the p3 -design; one construction that follows the rules: a core `ConversationStore { conversation }` (local) opens the Durable -watch as its own native, getting the Durable harness from the core-private `HarnessRuntime` registry contribution (a -native may reach another object only through a registry entry), and exposes a bounded window as a derive. Per-principal -`Conversation { session, conversation }` views in front of it check who may send (chapter XIX). - -## Code mode - -Code mode is a plugin. It owns its `pi-codemode` sandbox as a native; its tool entry looks up the **current** sandbox -when `execute()` starts, so a call pinned by a phase before a reload (waiting in an approval hook, say) starts on the -new sandbox. On `reload` and `stop` the old sandbox is sealed and drained in `close`: running scripts finish or hit -their deadline, for at most 30 s (Q7). After a worker stop Durable replays as it does today. - -## What lives where - -| Data | Where | -| --- | --- | -| transcript-bound: entries, documents that fork and rewind with a conversation | Durable | -| app state: session list, notes, drafts, grants, settings | Chord documents | -| data core enforces on conversations (read-only one-shots, cwd followers) | a core per-conversation policy document, written when the conversation is created (`createConversation({ init })`) | - -**Telemetry:** pass the host's adapter as `HarnessOptions.telemetry`; Durable stores trace carriers on submissions and -tasks (chapter XVII). diff --git a/packages/chord/docs/next/explainer/src/15-reload.md b/packages/chord/docs/next/explainer/src/15-reload.md deleted file mode 100644 index 09ab349de..000000000 --- a/packages/chord/docs/next/explainer/src/15-reload.md +++ /dev/null @@ -1,131 +0,0 @@ - -# Reload and Rollout {#reload lede="What happens when you press Reload: per host, per object, and across the server, the workers and every tab."} - -Back to the preface. Mario changed the notes plugin's sort order and pressed Reload. Everything the earlier chapters -asked of plugin authors (synchronous factories, natives only in objects, routines instead of timers, data in -documents) exists so that this moment is safe. This chapter shows what Chord does with it. - -## On one host - -```text -1. load new bundles; an import or syntax error rejects -2. staging facet setups and every live object's new factory run with a recording self - (declarations only, nothing activated); any throw rejects -3. commit the host switches to the new code atomically -4. after per object: pure objects swap, native objects restart -``` - -**Rollback is all-or-nothing per host.** Any load or staging error leaves the whole host on its previous code, so a -host is always completely at one revision and its status is one value. Runtime states (a registry conflict, a failed -open, a migration) are not reasons to roll back; after the commit, errors stay with one object. There is no factory -trial, no migrate dry run, and no consumer contract check in staging: an incompatible consumer fails at bind with -`contract_mismatch` and a notice naming plugin and type. - -:::steps | Reloading @studio/notes on worker S1, object by object -+++ Load and stage -The new bundles load. The facet setup and the factories of `Notes` (three views), `NoteStore { S1 }` and `NoteInbox` -run with a recording `self`: they declare, nothing activates. Nothing throws, so the host commits to the new code. -+++ NoteStore: pure, swapped in place -`NoteStore` has no `self.open`. Its new methods and derives take over; `NoteList`, the drafts, its references and the -running import job stay. Routines restart, serialized: the new `cleanup` waits for the old handler in flight. The gate -opens. -+++ Running calls finish on old code -An `add` that was running when the swap happened finishes on the old code with its old handles; its transaction still -commits. Only new calls get the new handles. Exception (R15): if the reload raised a document's version, old code that -touches the migrated document throws, the document stays intact, and that call fails once. -+++ The import job keeps its old run -The job's `run` keeps the code it started with. When it settles, the **new** `finish` records the outcome. -+++ The views: pure, swapped -The three `Notes` views swap in place. Owner, Ann and Bob keep their references; the reader derive is recomputed with -the new sort and the difference is sent. Nobody sees a reconnect. -+++ NoteInbox: native, restarts -`NoteInbox` has a watcher. Its running work ends `aborted`, the watcher closes with reason `reload`, the new factory -runs, the watcher reopens. Its documents would stay. Whether a file dropped during the restart is seen is the -watcher's business: Studio's `watchDir` scans the directory once when it opens. -::: - -**Why natives restart instead of being handed over:** a native has no line, no transactions and no schema. If old and -new code shared it, they would interleave on it unchecked. In practice restarts are cheap: the Durable harness is base -code, terminal PTYs live in a separate terminal host process, and model runtimes, sandboxes and a search index rebuild -in milliseconds. Only reloading the plugin that owns SSH connections (running remote commands die) or the relay socket -(relayed connections drop) hurts. - -**Removing a type:** its objects end (routines stop, natives close, references end with an error); in-memory documents -go with them; persisted ones stay until retired. **Consumers elsewhere** see a restarting object like any object that -briefly goes away: `loading`, parked calls, then `live`. - -**UI facets reload too.** A reload unmounts and remounts components. That is why "leaving" (abort a login, close an -overlay) is never derived from an unmount (chapter VIII). - -## Across hosts - -A **revision** is (server incarnation, sequence) and names the complete desired code set: every facet with its -digest, not a batch of changes. Every install reconciles everything missing, and a host reports a revision only when it -runs that complete set. A revision also carries the code-set digest (a hash over the facet bundles' content hashes), -so a server restart with unchanged code triggers no reinstall. - -The order is **server, workers, clients**. Each host reports `installed`, `installed_failed` (with the objects that -failed) or `rejected` (with the reason; it stays on its previous code). A server that rejects stops the rollout. A -host that starts or reconnects installs the current revision before anything is routed to it. Installs are serialized -per host and bounded (60 s). - -Calls keep flowing during a rollout. When a call reaches a provider on another revision, the mismatch response -carries the provider's revision and status: - -| Provider | The call | -| --- | --- | -| ahead, or behind and installing | `updating`: values readable, calls park (with the park deadline) | -| rejected | fails at once; retried on the next revision | -| same revision | `contract_mismatch`: a real mismatch, final | - -A host checks its own install status first: if its own install was rejected, a call fails at once with the reason -instead of waiting on a provider. Calls parked on a binding that is released end with an error and are not resent. -There are no cross-host transactions and no votes. - -:::sequence animate | A tab calls into a worker that is still installing. The call parks and runs once the worker reports the revision. -actor tab "tab" "revision 8" -actor server "server" "revision 8" accent -actor worker "worker S1" "installing 8" blue -server => worker: install revision 8 -tab -> server: call Notes.add (new contract) -server -> worker: relay -worker --> server: updating (rev 7, installing) -server --> tab: updating -tab -> tab: park (deadline 60 s) -worker --> server: installed 8 -server --> tab: provider at 8 -tab -> server: call Notes.add (resent from park) -server -> worker: relay -worker --> tab: result -::: - -## Start barrier - -At host start and first load there is one more rule (chapter XIV): load every facet (the core facet with its start -data among them), wait until every facet and `lifetime: "host"` object other than the harness is live or failed, then -resume dependent work. Plugin reloads need no barrier. - -## Loading and the dev loop - -- **Roots:** plugins come from built-in, global (`/plugins`), package (npm or git, installed with - `--ignore-scripts`) and project roots, found by directory listing; no source parsing. The server selects which load. - Dependencies between plugins resolve by contract name at runtime. Names are prefixed: objects `.`, - tools `_`. -- **Loader D:** one base bundle per surface (host packages and the app runtime) plus one script per facet, with host - packages passed in (no module map). Old generations are reclaimed by the garbage collector; a stable filename per - (facet, digest), and V8's compilation cache off on hosts that reload in place. The loader is an adapter: static - imports (tests, Durable Objects) work the same, with the app passing the plugin name to `load`. -- **Dev loop:** a source watcher marks plugins dirty and never applies anything. A reload control (the playground's web - shell, a CLI, an agent tool such as p2's `reload_plugins`) reloads the changed plugins, or restarts all hosts for - base code. Reloads are always explicit. - -:::review | Review: project plugins (#14, deferred) -Per-directory plugins with trust prompts and snapshots (`apps/p2/docs/plugin-loading.md`, still the reference for -roots, trust and scopes) need their own pass over this design. Until then, treat the project root as designed but not -settled. -::: - -:::note | Built so far -Reload and rollout are layer L10, the last before telemetry. The frozen kernel has a reload path with host values and -a development freeze, both removed from the design. -::: diff --git a/packages/chord/docs/next/explainer/src/16-contracts.md b/packages/chord/docs/next/explainer/src/16-contracts.md deleted file mode 100644 index 51e0126cb..000000000 --- a/packages/chord/docs/next/explainer/src/16-contracts.md +++ /dev/null @@ -1,310 +0,0 @@ - -# Changing Contracts {#contracts lede="How a contract gets a new version while old tabs, old workers and old plugins are still talking to it."} - -Chapter XV ended with a tab that calls into a worker on another revision. That case is short-lived: the rollout -finishes and both sides run the same code. Other cases last longer. A phone kept a tab open overnight with yesterday's -web bundle. A plugin from another author was built against `Notes` version 1 and is not updated yet. In both, a -consumer holds a contract that the provider no longer has. This chapter is about what the provider does then. It is -also the one part of Chord v2 that is built: everything in the sections on `evolve`, `checkAccepts` and fingerprints -runs today in `packages/chord/src/next/contracts/`. - -## What has a version - -Everything that is stored or crosses a boundary has a version. Each surface handles an old version its own way: - -| Surface | Version | When an old one shows up | Chapter | -| --- | --- | --- | --- | -| object type (contract) | `version` plus fingerprint | `evolve` and `checkAccepts` on the provider; otherwise `contract_mismatch` | this one | -| document | `version` plus schema fingerprint | pure `migrate` on access; a schema change without a bump fails the object `broken` | V | -| job | `version` | `accepts` on outcomes and `running`; `blocked` without an adapter | XI | -| registry | `version` | `accepts` adapts contributions; one without a match is left out and reported | XIII | -| native | none | the object restarts on reload | IX | -| Durable tasks and documents | Durable's `version` | Durable's `migrate` | XIV | -| link protocol | protocol version | refused; hosts update together, a web page reloads | XVII | -| UI components, Chord and app APIs | none yet | exposed as they are; an SDK will version them later | | - -The rule behind every row is the same as for documents: **every change bumps the version**, additive ones included. -Chord never guesses compatibility from the shape of two schemas. - -## A history, not a file of versions - -A contract history starts with a **base**: a full `remote()` or `local()` declaration with an explicit `version`. The -`Notes` contract from chapters II and III is a base at version 1. Studio now wants pinned notes: notes get a `pinned` -flag and editors get a `pin` method. Instead of editing the declaration, the plugin continues the history: - -```ts -// notes/contract.ts -import { bool, evolve, nul } from "@earendil-works/chord/next"; - -export const NoteV2 = obj({ id: str(), text: str({ max: 2000 }), by: str(), at: int(), pinned: bool() }); -export const PublicNoteV2 = obj({ id: str(), text: str(), at: int(), pinned: bool() }); - -export const NotesV2 = evolve(Notes, (e) => - e.state("editor", "notes", obj({ items: arr(NoteV2) })) - .state("reader", "notes", obj({ items: arr(PublicNoteV2) })) - .addMethod("editor", "pin", obj({ id: str(), pinned: bool() }), nul()), -); -``` - -`evolve(prev, (e) => …)` appends one version to `prev`'s history: `NotesV2.version` is 2, and `NotesV2` still knows -version 1 exactly as it was declared. Every version is **materialized and fingerprinted at declaration**. That is what -makes old code safe: an old plugin bundle that contains its own copy of `Notes` version 1 computes the same -fingerprint as the history inside `NotesV2`. There is no build step, no generated file and no fingerprint copied by -hand. - -The edits: - -| Edit | Changes | Kind | -| --- | --- | --- | -| `e.state(variant, member, schema)` | one state member of one variant | remote | -| `e.addState(variant, member, schema)` / `e.removeState(variant, member)` | adds or removes a state member | remote | -| `e.state(member, schema)`, `e.addState(member, schema)`, `e.removeState(member)` | the same, flat | local | -| `e.args(name, schema)` / `e.result(name, schema)` | one side of a method | both | -| `e.method(name, args, result)` | both sides of a method | both | -| `e.addMethod(variant, name, args, result)` | adds a method (or an existing one to another variant, same signature) | remote | -| `e.addMethod(name, args, result)` | adds a method | local | -| `e.removeMethod(name)` | removes a method from every variant | both | - -Edits chain. The `method(...)` wrapper appears only in base declarations; edits take the schemas directly, so a -method without a result is added with `nul()`. The rules of chapter III still hold across versions: a method has one -signature in every variant, and a member's kind never changes, not even after it was removed and added again. - -**What `evolve` cannot do:** change the input or the set of variants. Both change what an object *is* (its identity, -who gets which members), and no adapter can turn one into the other. That needs a new history base, below. - -## The provider says what it accepts - -`NotesV2` says what changed. It does not say how version 1 consumers are served; the **provider** decides that, with -mappings that live in its own code, never in the contract. The worker facet declares them with `checkAccepts`: - -```ts -// notes/worker.ts -import { checkAccepts } from "@earendil-works/chord/next"; - -const toV1 = ({ pinned, ...note }: Static) => note; -const toPublicV1 = ({ pinned, ...note }: Static) => note; - -const notesAccepts = checkAccepts(NotesV2, { - 1: { - state: { - editor: { notes: (current) => ({ items: current.items.map(toV1) }) }, - reader: { notes: (current) => ({ items: current.items.map(toPublicV1) }) }, - }, - methods: {}, // add and remove did not change; pin is new - }, -}); -// notesAccepts.versions: [{ version: 1, fingerprint: Notes.fingerprint }] - -host.serve(NotesV2, notesFactory, { accepts: notesAccepts }); // working name for the objects-layer option -``` - -`checkAccepts` is **pure**: it checks the mappings without calling them and returns only the version metadata of the -old versions it lists. A provider that lists nothing accepts only its current version. The objects layer later -compares a consumer's version and fingerprint against this list. - -The mappings point in the direction the data flows: - -| Member in the old version | Mapping | Direction | -| --- | --- | --- | -| state, changed | `(currentMember) => oldMember` | provider's current value to what the old consumer expects | -| state, removed | `(currentVariantState) => oldMember`, or `"refuse"` | receives the whole current state of that variant | -| method arguments, changed | `args: (old) => current` | old caller's arguments to current ones | -| method result, changed | `result: (current) => old` | current result back to the old shape | -| method, changed or removed | `{ call: async (oldArgs, ctx) => oldResult }`, or `"refuse"` | replaces the method for old callers | -| member added since | nothing | an old consumer never asks for it | - -A changed state member cannot be refused: the old consumer still has that member and must get a value. A removed one -can be computed from what the variant has now, or refused. `e.method(name, args, result)` replaces both sides and -therefore needs both mappings, even if one side stayed the same (then an identity mapping). - -**Required, at type level and again at runtime.** For every member that exists in the old version and changed or was -removed since, the mapping is required by the type of `checkAccepts`. Because plugins are bundled without type -checks, the same check runs at declaration: a missing mapping, a key for a member that needs none, an unknown version -or a wrong shape throws, naming the member's path (for example `accepts[1].state.reader.notes`). An absent `state` or -`methods` section counts as empty and may be left out only when nothing there needs a mapping. - -**Conservative with no-op edits.** An edit that sets a schema to what it already was, or changes a member and then -changes it back, still requires an identity mapping. TypeScript erases constraints such as `{ max: 2000 }` from the -static type, so two schemas that look the same to the compiler may differ; Chord does not try to tell. - -### Old versions map straight to the current one - -Studio later replaces `remove` with `archive`, which keeps the note in a trash list: - -```ts -export const NotesV3 = evolve(NotesV2, (e) => - e.removeMethod("remove").addMethod("editor", "archive", obj({ id: str() }), nul()), -); - -const notesAccepts = checkAccepts(NotesV3, { - 1: { - state: { editor: { notes: (c) => ({ items: c.items.map(toV1) }) }, reader: { notes: (c) => ({ items: c.items.map(toPublicV1) }) } }, - methods: { remove: { call: async ({ id }, ctx) => { await archiveNote(id, ctx); return null; } } }, - }, - 2: { methods: { remove: { call: async ({ id }, ctx) => { await archiveNote(id, ctx); return null; } } } }, -}); -``` - -Each entry maps **one old version directly to the current one**, not through the versions in between. Version 1 -needs the state mappings (its notes have no `pinned`) and the `remove` replacement; version 2 needs only the -`remove` replacement, and leaves `state` out because its state is unchanged. A member that an old version never had -needs nothing for that version, even if it was added, changed and removed later. Dropping support for version 1 is -deleting its entry: version 1 consumers then get `contract_mismatch`. - -## Binding an old consumer - -A hold carries only the consumer's contract **version and fingerprint**, never a description of the contract (R6). -The provider decides with its own schemas: - -:::flow wide animate | What a provider does with a hold. The decision uses only the provider's own history; a peer's schema is never an input. -row "The hold arrives" -box hold "hold" "version 1, fingerprint F1" blue -box cur "equal to current?" "version and fingerprint" -box acc "accepted old version?" "F1 from the provider's own history" -row "Outcome" accent -box live "bind" "current members" -box adapted "bind adapted" "through the mappings" accent -box mismatch "`contract_mismatch`" "or updating, during a rollout" muted -arrow hold -> cur -arrow cur -> live "yes" -arrow cur -> acc "no" -arrow acc -> adapted "yes" -arrow acc -> mismatch "no" -note "A mismatch fails only that consumer's binding. The provider object and its other bindings are not affected." -::: - -Once a version 1 tab is bound, it is served as if version 1 were current: - -:::steps | A version 1 tab on a version 3 worker -+++ Bind -The tab holds `Notes { S1 }` with version 1 and fingerprint F1. The worker's history computes F1 for version 1, and -`notesAccepts` lists version 1, so the hold binds adapted. The tab gets the `editor` variant as version 1 defined it: -`notes`, `add`, `remove`. It never sees `pin` or `archive`. -+++ State through the mapping -Chord serves the adapted state like a derive of the provider's state: it applies the mapping to the current value -and keeps the result per binding. The snapshot the tab receives has no `pinned` fields. -+++ A commit -The owner pins a note from a current tab. The worker sends the current tab the draft's ops. For the version 1 tab, -Chord applies the mapping to the new value and sends the difference to its last adapted value (`diffRevisions`). The -mapping produced nothing new for a pin, so the version 1 tab receives no change at all. A large list is never resent -whole on every commit. -+++ A call -The tab calls `remove({ id })`. Its arguments are checked against the provider's own version 1 schema, then the -`call` replacement runs `archiveNote` on version 3 code. Its `null` result is what version 1 expects. Had the method -only changed, `args` would have mapped the arguments forward and `result` the result back. -+++ The worker moves on -A later revision drops the version 1 entry. When the tab re-holds (or the worker restarts and the tab binds again), -the hold gets `contract_mismatch`, and Studio's panel shows "reload to update" instead of failing silently. -::: - -## A new history base - -Some changes cannot be adapted. Studio decides that notes belong to a project, not a session: the input changes from -`{ session }` to `{ project }`. That is a new base, written in full, with an explicit version that continues the -published numbering: - -```ts -export const NotesV4 = remote("notes", { - version: 4, // explicit: the published history used 1 to 3 - input: obj({ project: str() }), - placement: () => Server, - variants: { - editor: { state: { notes: obj({ items: arr(NoteV2) }) }, methods: { add: method(…), pin: method(…), archive: method(…) } }, - reader: { state: { notes: obj({ items: arr(PublicNoteV2) }) } }, - }, -}); -// checkAccepts(NotesV4).versions is empty -``` - -A new base has no history before its version and **cannot accept older versions**, not even for an additive change. -Every consumer built against versions 1 to 3 gets `contract_mismatch` and must update. Later `evolve` calls continue -from version 4. The same applies to a change of the variant set. Do not spread an older declaration into a new one or -copy its fingerprint; the contract objects are branded and `evolve` and `checkAccepts` reject copies. - -## Local contracts - -`local` contracts (chapter IV) evolve the same way, with a flat members map and no variants: - -```ts -export const SessionInfoV2 = evolve(SessionInfo, (e) => e.state("init", InitV2)); -checkAccepts(SessionInfoV2, { 1: { state: { init: (current) => toInitV1(current) }, methods: {} } }); -``` - -Same-host consumers of a `local` type are usually reloaded together with their provider, but not always: another -plugin on the same host may be built against an older version. The rules are identical; only the variant argument is -missing. - -## Implementing a contract - -`Implementation` is the type of everything a provider must supply for the current version: one state -getter per member and variant, and one asynchronous method per method, each receiving `(args, ctx)`: - -```ts -const impl: Implementation = { - state: { editor: { notes: () => list.value }, reader: { notes: () => publicList.value } }, - methods: { add: async ({ text }, ctx) => addNote(text, ctx), remove: async ({ id }) => removeNote(id), pin: async (a) => pin(a) }, -}; -``` - -Keep declaration types **inferred** (no annotation on `NotesV2`): the inferred type carries the complete history, and -`checkAccepts` needs it to know what changed. - -:::review | Review: L1's implementation shape and the factory's return value -`Implementation` in `src/next/contracts/types.ts` is `{ state: { variant: { member: getter } }, methods }`. The -design's factories (`design.md` §6, used throughout this guide) return `{ variant, member: value, method: fn }`. -The objects layer (L4) has to connect the two, and also define how `serve` receives the checked accepts (this guide -uses the working name `{ accepts }` as a third argument) and what `"refuse"` does at runtime. The natural reading, -used in the stepper above, is that a refused method call and a hold that needs a refused state member end with -`contract_mismatch`; no source says so. -::: - -## The fingerprint - -The fingerprint is a 64-bit hash (`hash64`, portable across runtimes) of a **normal form** of the contract's -schemas. The normal form decides what counts as a change: - -| Ignored | Significant | -| --- | --- | -| order of object keys, `required` and `enum` (sorted) | order of union and intersection branches (`anyOf`, `oneOf`, `allOf`) and of tuples | -| annotations on schema nodes: `title`, `description`, `examples`, `$comment`, `deprecated` | defaults, formats, patterns, bounds, `$ref` names | -| placement | input, variants, member kinds and every member schema | - -Union order is significant because it is meaning: TypeBox picks defaults and decodes by the first matching branch. -Annotations are dropped only on schema nodes; an object property that happens to be named `title` is data and counts. - -Only data schemas are allowed. One walker copies, validates and normalizes each schema against a **whitelist of -TypeBox keywords**: primitives, arrays and tuples, objects and records, unions and intersections, `$defs` and `$ref` -(use `$ref` for recursive types), constraints and annotations. Conditional and dependent schemas, `not`, -`definitions`, `contentSchema`, `unevaluated*`, `contains`, `propertyNames`, unknown keywords, executable or custom -TypeBox features, arbitrary `Unsafe` and refinements, accessors, symbols and cyclic objects are rejected at -declaration. Bytes are the one special leaf: `bytes({ min, max })` checks a `Uint8Array` and has the data marker -`{ type: "bytes", minLength?, maxLength? }`, so it hashes like any schema; TypeBox's checking still works on it. - -Schemas are deep-copied into a private, frozen history; your own schema objects are copied, not frozen. -`NORMAL_FORM_VERSION` is part of every digest, so a change of the normal form is a link protocol change. - -**Not a security boundary.** A fingerprint says "we were compiled against the same contract"; it proves nothing. A -hostile client can send any fingerprint it likes. Providers therefore always validate arguments with their own -schemas (chapter IV), whatever the fingerprint says. - -## Rules of thumb - -- Treat a published contract like a published API: every change is `evolve`, every `evolve` needs a decision about - old consumers. -- Keep mappings small and pure. A state mapping runs for every commit and every adapted binding; make it a map over - the value, not a query. -- Accept the versions your users still run (one or two back during a normal rollout), and delete entries when the - metrics show no old bindings. -- Change the input or the variants only with a new base, and only when updating every consumer is acceptable. -- Contract, document and job versions are separate and each is decided on its own. Pinned notes needed `NotesV2` - for the wire and `NoteList` version 2 with a `migrate` that adds `pinned: false` (chapter V), but no new - `ImportJob` version, because its outcome did not change. - -:::note | Built so far -This chapter's contract layer is built: `packages/chord/src/next/contracts/` (commit `7d2bb8ea8`, about 1,300 lines -with `remote`, `local`, `method`, `evolve`, `checkAccepts`, `Implementation`, the schema helpers, `fingerprintOf`, -`schemaNormalForm` and `NORMAL_FORM_VERSION`), with nine test files covering evolution, fingerprints, the hash, -local contracts, long chains, schemas, types and review findings. It stands alone: it hosts nothing, calls no -mapping and does not import the frozen runtime. Binding by version and fingerprint, adapted state and calls belong to -the objects and protocol layers, not built yet. -::: diff --git a/packages/chord/docs/next/explainer/src/17-wire.md b/packages/chord/docs/next/explainer/src/17-wire.md deleted file mode 100644 index 67bd21118..000000000 --- a/packages/chord/docs/next/explainer/src/17-wire.md +++ /dev/null @@ -1,251 +0,0 @@ - -# On the Wire {#wire lede="What travels over a link, what a relay does with it, how big things are kept in check, what each hop costs, and how to see it in traces."} - -Most plugin code never thinks about the wire: it declares state, commits transactions and calls methods, and Chord -moves the bytes. You need this chapter when something is slow, when a value is large, when a link misbehaves, or when -you want to read a trace. It also explains why the earlier chapters insisted on small operations, shared structure and -no callbacks: each of those rules removes a cost on some hop. - -## What a state member sends - -Every exposed document, derive or readable is a **state member**. Over a hold it produces one of these: - -| Update | When | -| --- | --- | -| `snap` | on hold and on re-hold: the whole value, once | -| `ops` | per commit: the draft's recorded ops, no diff | -| `ops` from a diff | per change of a derive (and of an adapted binding, chapter XVI) | -| throttled `ops` | job progress | -| `reset` | a migration commit replaced the value (adapted for older consumers) | -| `error` | the member failed | - -**Commits are atomic for consumers.** On the same host, subscribers are notified once, after every document of the -commit is adopted. Remotely, one `state` message per hold carries a batch of member updates that is applied before -listeners run, and a commit that touches several holds on one link travels as **one** `state` message on that link. -A consumer never sees half a commit. - -## Messages - -A message is a **header plus opaque parts** (R1). There is one part per hold the message concerns; the header says -which hold each part belongs to and how large it is. - -| Direction | Kind | Parts | -| --- | --- | --- | -| consumer to provider | `hold`, `release`, `call`, `cancel` | `call` has one (the arguments) | -| provider to consumer | `bound` | one: the snapshot | -| | `state` | one per hold the commit touched on this link | -| | `result` | one | -| | `gone` | none | - -There are no request ids in application code. Hold ids and call ids exist only inside the protocol, per link. - -**One binary format on every link (Q4):** `u32 header length | header JSON | part bytes`, as one `Uint8Array`. A -WebSocket sends it as a binary frame, IPC passes it with `serialization: "advanced"`, a memory link delivers it -asynchronously. Encoding, framing, header checks, id resolution, flow control and size limits are one shared protocol -layer; link adapters only move bytes. Tests on memory links therefore run the production path. - -A part is encoded **once, at its origin**, with that hold's own Delta encoder (paths are interned per hold, end to -end between provider and consumer). A JSON part above a threshold (about 16 KiB) is compressed once at its origin, -whatever the link; byte segments stay raw; relays forward compressed parts as they are; the header names the codec. - -**The header is a TypeBox schema (R4)**, a union tagged by `kind`; the protocol types derive from it. Headers from -untrusted links (client WebSockets) are checked with the schema compiled once; trusted links (IPC between server and -workers) are not checked. **Unknown ids are ignored on both sides (Q1).** They are normal races: a `result` crossing a -`cancel`, a `call` crossing a `gone`, a state part crossing a `release`. The side that ended a hold or call settles -its own pending work: a consumer that receives `gone(h)` rejects its pending calls on `h`, and the order per -direction guarantees nothing else for `h` follows. Only a header that fails the schema is a protocol error, and it -closes that link. - -The link protocol has a version. Hosts with different protocol versions refuse each other; a web page reloads to get -the server's protocol. - -## Through a relay - -A tab never talks to a worker directly (chapter I). The server **relays**, and a relay reads and rewrites only -headers: it maps hold and call ids between its two links, stamps the principal on `hold` and `call` (a client's -principal field is always overwritten; only a trusted parent link may carry one), and groups parts per destination -link, so each link gets only its own holds' parts. It never parses a payload. - -:::sequence animate | The owner adds a note while Ann watches. The commit leaves the worker as one `state` message with two parts; the server splits it by destination without decoding either part. -actor owner "owner's tab" "hold h1" blue -actor ann "Ann's tab" "hold h2" muted -actor server "server" "relay" accent -actor worker "worker S1" "NoteStore, Notes" -owner -> server: call c7 Notes.add (1 part) -server -> server: map ids, stamp owner -server => worker: call c3 (part forwarded as is) -worker -> worker: method runs, self.tx commits -worker => server: state: parts for h1', h2' -server -> server: group parts per link -server --> owner: state (h1 part) -server --> ann: state (h2 part) -worker --> server: result c3 -server --> owner: result c7 -::: - -The `state` message reaches the owner's tab before the `result`, so when `await notes.add(…)` resolves, the new note -is already in `notes.value`. Ann's view is a different variant (reader) and gets its own part from its own derive; the -relay neither knows nor cares. - -## Flow control - -Each link has **one byte window**, counted from each message's `byteLength` (parts are already encoded bytes; nothing -is measured or stringified to count it). The window covers every message, first snapshots included. - -- **Snapshots are always admitted (R19)**, even when larger than the window; they use it up. Whether a hold - **overflows** is decided on the changes queued after the snapshot. -- **Under backpressure** a commit's ops are collected per hold and encoded into one part when credits return. The - consumer still gets the commit atomically. -- **After an overflow** the hold is dropped and established again once the previous snapshot's debt has drained - (credits back at the full window, Q3), and not more often than a minimum interval. So there is at most one snapshot - in flight per hold: no re-hold storm, and no stall for a snapshot larger than the window. -- **Calls** are throttled by call credits the provider grants (a generous window). Beyond them calls wait locally. - Nothing is refused for being "too many", and holds are not limited by count; authorization and input validation - protect them. -- **Byte-heavy traffic** such as terminals uses separate channels (chapter XIII), so a large state message never - holds up keystrokes. - -**Recovery:** after a link drops, the consumer re-holds every hold over the new link and receives snapshots; calls in -flight fail as on any disconnect, calls made meanwhile park. A relay whose window overflows releases the hold and -re-holds; the provider sends a fresh snapshot. After a routed link drops, Chord re-resolves the address (with the -per-address backoff) while holds on it exist, so a passive viewer recovers without anyone pressing anything. - -## Size limits - -Limits apply **per part** (R2, Q2), both to the compressed and the decoded size, and are checked from the sizes in the -header before anything is decoded; zlib's `maxOutputLength` enforces the decoded limit in the one decode function. - -| Situation | What fails | -| --- | --- | -| a part over the limit | only its own hold or call, with `too_large`; the other parts of the message apply | -| a snapshot over the limit | its hold ends `failed`, with no automatic re-hold | -| a WebSocket frame over `maxPayload` (e.g. 64 MiB) | the connection closes (code 1009); this is the transport's hard cap | - -Defaults for client WebSockets are 16 MiB compressed and 64 MiB decoded per part, configurable per link kind. The -decoded limit travels with relayed work: the server stamps the admitting link's limit on the header at admission -(never taken from the client), and the host that decodes enforces it. IPC therefore needs no limit of its own. - -Large data is never a large state member or a large argument. Files go through chunked transfers or the HTTP -component (chapter XIII); long lists are keyed documents or bounded windows (chapter VIII). - -## What each hop costs - -Chord v2's goal is as little compute and memory as possible while staying correct: **O(size of the change)** per -commit and per viewer, independent of the size of the value. A 500 KB value is touched as a whole only when a new -viewer takes its first snapshot. Delta, the value library under documents, was built for this, and its own primitives -are almost free: `diffRevisions` on shared subtrees takes 0.018 ms, `applyImmutable` of a small change 0.004 ms. - -| Hop | Work | -| --- | --- | -| commit | Delta's `prepare` (copies and checks only the values placed in this commit), adopt by pointer swap; storage writes the delta | -| reads in a transaction | the draft overlay, no copy; an unchanged draft writes nothing | -| subscribers on the same host | the same value and ops, by reference | -| derives | compute plus `diffRevisions`; unchanged parts stay the same objects (`mapShared`, chapter VIII) | -| send | one encoding per hold part at its origin, size taken from the encoded bytes | -| relay | header only: ids mapped, principal stamped, parts grouped; parts never parsed | -| receive | compiled header schema on untrusted links; arguments stripped and validated once at the provider (compiled schema); inputs normalized | -| apply | `applyImmutable`, structure shared | -| new hold | the value encoded once (unavoidable) | -| memory | one authoritative value per document per host; replicas and derived values share structure; one cached value per adapted binding | - -These are targets. The frozen kernel measured far worse, which is why the design is so strict here: - -:::stats -- 35 ms | one small change to a 500 KB value in the frozen kernel, no viewer -- +13 ms | per viewer on top (10 viewers: 130 ms) -- 17 / 33 / 101 ms | a tiny SQLite commit after 0 / 10 / 50 earlier deltas -::: - -Nearly all of that came from layers around Delta: whole-value validation on every access, whole-value freezing, -JSON-ness walks on every received frame, double stringification to measure a frame, snapshots cloned per viewer, and -a SQLite storage that replayed history, and schemas rebuilt for every check (a compiled TypeBox check takes -0.012 ms, an interpreted one 5.6 ms). The redesign removes each of them (chapters V, VIII and this one); the -before/after per hop is in `apps/p2/docs/p3-reviews/v2-cost-model.md`. - -:::review | Review: v2-cost-model.md is partly outdated -The measurements and the per-hop "after" targets in `apps/p2/docs/p3-reviews/v2-cost-model.md` are current. Its -closing section "Decisions this needs" is not: items 6 and 7 (validation by the caller's trust; contract descriptions -once per link and fingerprint) were replaced by R7 (validate for every caller) and R6 (no descriptions on the wire), -and §10 still describes the description fetch. -::: - -### Cost guards - -Chord reports **full-value walks, deep copies, encodes, schema validations and materializations** as ordinary -telemetry metrics, always on; with the NOOP adapter they cost an empty call where such work happens. Tests use the -in-memory adapter and assert them per hop. The reference case: one appended note to a 500 KB document with 10 viewers -does no walks, no copies, one encode per hold part at its origin and none at relays, and no validations. A -`vitest bench` suite measures the hops. - -### Ways to make it slow anyway - -| Mistake | Why it costs | Instead | -| --- | --- | --- | -| a derive that builds new objects for unchanged data | every commit diffs and sends O(value) | keep unchanged items identical: `mapShared`, `previous` (chapter VIII) | -| one long array that changes often (R24) | sharing is per path; each change copies one slot per element and visits each | keyed documents or a bounded window | -| bytes as base64 strings | inflated, parsed, compressed as text | `bytes()` leaves; chunked transfers for files | -| awaiting I/O inside `self.tx` | the object's line is blocked for every caller | do the I/O first, then a short transaction | -| storing registry entries or references in documents | stale after a reload, not data | look up the current value at use | -| opening natives at module level or in a facet | no owner: never closed, never restarted | `self.open` in an object (chapter IX) | -| callbacks as arguments | cannot cross a link, keep old code alive | registries, jobs, flows | -| one big object that everybody views | one large snapshot per viewer and per reconnect | split by viewer need; views of the same variant share one value | - -## Telemetry - -Chord reports traces, events and metrics through `@earendil-works/pi-telemetry`. Each host gets **one adapter**, -`createHost({ telemetry })`, NOOP by default; the adapter decides sampling, buffering, export and filtering. Only hosts -hold the adapter, to start root spans. Code sees spans: - -- `ctx.telemetry` is the current span (`Ctx` extends `Context`, so it is the same field Durable and pi-ai read). Every - span carries its adapter's metrics and propagation, so code with a `ctx` can record a metric or inject a carrier. -- `ctx.trace(name, attributes, fn)` runs `fn` in a child span with a child `ctx`. -- App values such as the principal stay out of Durable's context. - -**Across links**, `call` and `hold` carry the caller's trace carrier as a bounded header field (R5: limited key count -and string lengths); the provider runs the method under it, and each relay adds a `chord.relay` span. The call from the -sequence above, as a trace: - -```text -chord.call Notes.add [owner's tab] -└─ chord.relay [server] - └─ chord.method Notes.add [session:S1] principal=owner - · event chord.commit docs=2 ops=5 -``` - -Adapters trust typed input: no validation of malformed values and no try/catch around recording. Only carriers from -another process are checked (keys present, strings), and a bad one falls back to a new root. - -**Shared work** (one activation serving several `get`s, a worker start) is its own root span, linked to the trigger -that started it; each waiting span links to it. Routine runs, job runs, eviction, backoff retries and background -closes are root spans too, linked to their trigger where there is one. - -| Tier | What | -| --- | --- | -| spans | calls, methods, relays, holds, activations, opens, closes, loads, revocations, resolves, routine runs, job runs | -| events | commits inside methods, retries, parking | -| metrics | state messages, WebSocket bytes, commit durations, in-flight calls, active holds, open links, park durations, backpressure waits, the cost guards | - -Ids are attributes of spans, never of metrics. **Nothing is measured or walked for telemetry**; only sizes the code -already has are reported. Errors carry a fixed category and a message cut to 500 characters; cancelling is not an -error. Chord ships a typed telemetry schema, as pi-ai (`AI_TELEMETRY_SCHEMA`) and Durable -(`DURABLE_TELEMETRY_SCHEMA`) do. - -**Storing and reading:** `@earendil-works/chord/telemetry-sqlite` over `chord/sqlite` (Node and Cloudflare) stores one -row per settled span (events and links inline) and metrics aggregated per series per 10 s, flushes in batches every -second, keeps 7 days, and drops (and counts) beyond a bounded queue. The read interface `TelemetryStore` is shared: -`findTraces` (by name, attributes, errors, time window, with limit and cursor) returns complete traces, `getTrace` -one, `findMetrics` metrics. One renderer, `pi-telemetry/render`, prints traces as above; it prints links and never -follows them. The playground's trace viewer reads the same store. - -**Durable** already emits submit and phase spans, stores carriers on submissions and tasks (`TaskOptions.trace`), -and pi-ai emits `pi.ai.request` and `pi.ai.poll`. With the harness given the host's adapter (chapter XIV), an agent -tool that calls `Notes.add` through `chord(fn, context)` shows up as one tree from the model request down to the -commit. - -:::note | Built so far -Nothing of the protocol is built in v2: messages, relays, flow control and limits are layers L6 and L7, telemetry in -Chord is L11. Built in pi: `pi-telemetry` (propagation, links, metrics, the store interface, the renderer), -`chord/context` with `Context.telemetry`, pi-ai and Durable telemetry, and `chord/telemetry-sqlite`. Delta is built; -`mapShared` and bytes as atomic leaves in `diffRevisions` are still to do. The frozen kernel sends descriptions on the -wire and closes a link on an unknown id; both are replaced. -::: diff --git a/packages/chord/docs/next/explainer/src/18-p2-core.md b/packages/chord/docs/next/explainer/src/18-p2-core.md deleted file mode 100644 index 51925055d..000000000 --- a/packages/chord/docs/next/explainer/src/18-p2-core.md +++ /dev/null @@ -1,324 +0,0 @@ - -# Building p2: the Core {#p2-core lede="From a list of what the app must do to a set of owners: the base code every p3 plugin builds on."} - -The notes plugin used every Chord concept. A real app is the same concepts many times over. This chapter and the -next two assemble **p3**, the Chord v2 version of p2, from the parts the guide introduced. p3 is built next to p2 in -`apps/p3`, fully isolated with its own data directory, and **without migrating p2's data** (no import of -`catalog.sqlite`, `shares.sqlite` or the subagents table). Its plugins are derived from what the app must do, not -ported one to one from p2's plugin layout. - -:::review | Review: p3 is not designed yet -The design fixes the rules (`design.md` §25) and lists the p3 decisions still open (#17, #18, #20 to #23); the -per-domain design of p3 has not started. What these three chapters show is one construction that follows the rules, -derived from the functionality map in `apps/p2/docs/p3-functions/`. Object names are working names. Where a domain -has an open decision, the text says which. -::: - -## Start from functions, not from plugins - -The functionality map in `apps/p2/docs/p3-functions/` lists what p2 does, in plain words, with four facts per -function: the data and its size, how durable it is, who may see or do it, and where it must be available. It groups -the functions into 22 domains: - -| # | Domain | # | Domain | -| --- | --- | --- | --- | -| 1 | sessions | 12 | access | -| 2 | session workers | 13 | models, credentials, settings | -| 3 | conversations and agent | 14 | prompt resources | -| 4 | shell commands | 15 | status and presence | -| 5 | chat mode | 16 | search | -| 6 | agents (subagents, one-shots, peer messages) | 17 | remote access and forwards | -| 7 | code mode | 18 | logs and notices | -| 8 | session files | 19 | dev and reload | -| 9 | environments and execution | 20 | web shell | -| 10 | workspace | 21 | TUI shell | -| 11 | terminals | 22 | transport | - -The four facts answer the four questions every Chord object needs, in order: - -:::flow wide animate | From a function in the map to code. Each fact in the map answers one question Chord needs answered. -row "The map says" -box data "data" "what, how big" -box dur "durability" "ephemeral · until restart · persisted · transcript-bound" -box who "who" "owner · guest read · guest write · worker" -box where "where" "server · worker · web · TUI · environment" -row "Chord needs" accent -box owner "the owner" "smallest object it belongs to" accent -box store "where it is kept" "Durable, persisted or in-memory document, tab" -box variant "variants" "and revocation" blue -box host "host and code" "placement; core or plugin" -arrow data -> owner -arrow dur -> store -arrow who -> variant -arrow where -> host -note "Durability decides storage: transcript-bound data is Durable's; persisted app data is a persisted document with a stable identity; until-restart data is an in-memory document or a native; tab-scoped UI state stays in the client." -::: - -**One owner per thing, the smallest object it belongs to.** A guest's grant belongs to the access owner, not to the -session; a draft belongs to one (conversation, author) pair, not to the chat plugin as a whole. Ownership nests, and -rules set at creation are written in the creating commit (Durable's `createConversation({ init })`). - -**Core or plugin.** Core owners are what the app always needs and everything builds on; they are always present, so -other code may depend on them directly. Plugin owners may be missing; anything that depends on an optional feature -does so through a registry it owns itself (chapter XIII). The risk to avoid is a **core god object**: each core owner -owns only what belongs to it. - -## The ownership map - -The result of that procedure for p3, as a starting point: - -| Domain | Owner object(s) | Host | Code | -| --- | --- | --- | --- | -| access | `Access` (grants, devices, share links), `authenticate` | server | core | -| sessions | `Catalog` (session records), lifecycle jobs | server | core | -| session workers | the supervisor (`route(Session)` + idle stop) | server | core (app code) | -| session location | `SessionLocation { session }` | worker | core | -| conversations and agent | `Harness`, `ConversationStore`, `Conversation` views | worker | core | -| status and presence | `SessionStatus` (server copy), reporters on workers | both | core | -| shell commands | `ShellJob` in the chat plugin | worker | plugin | -| chat mode | chat plugin | worker | plugin | -| agents | subagents plugin (Durable child conversations) | worker | plugin | -| code mode | code mode plugin (sandbox native) | worker | plugin | -| session files | `SessionFiles { session }` (proposed, #17) | server | core or plugin, open | -| environments | envs plugin (config, SSH pools #20) | server and worker | plugin | -| workspace | workspace plugin | worker | plugin | -| terminals | terminal host process (#23), terminals plugin | separate process | plugin | -| models, credentials | models plugin, login job + flow | server | plugin | -| settings | `Settings` (single writer, #21) | server | core | -| prompt resources | prompt plugin (watchers in the session's environment) | worker | plugin | -| search | search plugin (index on the server, feed on workers) | both | plugin | -| remote access, forwards | radius plugin, ports plugin | server | plugin | -| logs and notices | `Notices` (one channel), log capture | server | core | -| dev and reload | the reload control, `reload_plugins` tool | server | plugin | -| web and TUI shells | UI facets over the objects above | clients | shells | - -The rest of this chapter builds the core rows. Chapters XIX and XX build the plugins. - -## The base code - -```text -apps/p3/ - src/ - hosts.ts Server, Session host kinds - principal.ts P, Revocation, principalKey, revokedBy, can() - server/main.ts createHost, load, route(Session), WebSocket and Radius entry points, supervisor - worker/main.ts createHost with the start data, connect to the parent - core/ the core facets: Access, Catalog, SessionStatus, Settings, Notices (server); - SessionInfo, SessionLocation, Harness, ConversationStore (worker) - plugins/ chat, subagents, codemode, envs, workspace, terminals, models, login, - prompt, search, ports, radius, files, dev -``` - -`hosts.ts`, `principal.ts`, the `SessionInfo` start data and the `Harness` object are the ones the guide already -built for Studio (chapters I, III, IX and XIV). Base code restarts the hosts when it changes; everything in -`plugins/` reloads. - -## Who is calling: authentication - -Chord stamps principals; the app produces them. p3's server has four ways in, each turned into a principal by -`authenticate(req)` when a WebSocket connects (chapter III): - -| Way in | Principal | -| --- | --- | -| loopback without a cookie (TUI, CLI, first page load) | `{ kind: "owner", access: "local" }` | -| loopback owner browser cookie (a per-port secret file) | `{ kind: "owner", access: "local" }` | -| a device cookie (HMAC-signed, per port, 30 days), obtained with a one-time sign-in token | `{ kind: "owner", access: "device", device }` | -| a guest cookie from a share link | `{ kind: "guest", session, mode, grant, revision }` | -| the Radius relay channel | `{ kind: "owner", access: "relay" }` | - -The checks that are not Chord's stay in this function: origin checks and Fetch Metadata (a browser request from -another site gets no principal, except a top-level navigation), proxy trust, the sign-in token rules (10 minutes, -single use, at most 32 outstanding, memory only), cookie expiry and revocation lists. A failed check closes the socket -before Chord sees it. - -**One permission matrix.** Every function in the map names who may use it, and p2 checks those rules in several -places. p3 writes them once, in `principal.ts`, and every factory and `authorize` uses it: - -```ts -// src/principal.ts -type Action = "read" | "send" | "upload" | "stop" | "workspace" | "terminal" | "manage"; - -export function can(p: P, action: Action, session?: string): boolean { - if (p.kind === "owner" || p.kind === "server") return true; - if (p.kind === "worker") return action !== "manage" && p.session === session; - if (p.kind !== "guest" || p.session !== session) return false; - if (action === "terminal" || action === "manage") return false; // terminals: owner only - if (p.mode === "read") return action === "read"; // no workspace for guest read - return true; // guest write: send, upload, stop, workspace -} -``` - -Factories pick variants from it (`can(p, "send", session) ? "writer" : "reader"`), so the matrix is visible in one -file and the members a caller lacks are simply not there. - -## Access: grants, devices, share links - -`Access` is the core owner of everything that lets someone in: share links per session and mode, guest records, the -device list and the signing key. It lives on the server as the chapter IV pattern: a `local` store that persists the -documents (only stable identities may), and a `remote` view for the owner's settings screen: - -```ts -// src/core/access.ts (server, in the core server facet) -const Links = defineState("access.links", { schema: map(ShareLink), version: 1, persist: true, - initial: () => ({}), checkpointWhen: (_v, _o, info) => info.deltasSinceBase > 64 }); -const Guests = defineState("access.guests", { … }); // same shape: map(Guest), persisted -const Devices = defineState("access.devices", { … }); // map(Device), persisted - -host.serve(AccessStore, (_input, self) => { // local: may persist, acts for the host - const links = self.state(Links), guests = self.state(Guests), devices = self.state(Devices); - return { - links, guests, devices, - removeGuest: async ({ grant }, ctx) => { - await self.tx(async (tx) => { (await tx.draft(guests))[grant].blocked = true; }, ctx); - host.revoke({ grant }); // live access ends now - }, - resetLink: async ({ session, mode }, ctx) => { - const blocked = await self.tx((tx) => rotateLink(tx, links, guests, session, mode), ctx); - for (const grant of blocked) host.revoke({ grant }); - }, - signOut: async ({ device }, ctx) => { - await self.tx(async (tx) => { (await tx.draft(devices))[device].revoked = true; }, ctx); - host.revoke({ device }); - }, - … // shareLink, join, signOutAll - }; -}); - -host.serve(Access, (_input, principal, self) => { // remote view: owner only - if (!can(principal, "manage")) throw forbidden(); - const store = self.get(AccessStore, {}); - return { - variant: "owner", - guests: self.derive("guests", [store.guests], ([g]) => publicGuests(g)), // names, modes; no tokens - removeGuest: (args, ctx) => store.removeGuest(args, ctx), - resetLink: (args, ctx) => store.resetLink(args, ctx), - signOut: (args, ctx) => store.signOut(args, ctx), - }; -}); -``` - -`authenticate` is base code on the same host and checks `AccessStore`'s current documents on every new connection -(it holds a reference to the store, taken with `server.root` at start). Live access and lasting denial stay separate, as chapter VII explained: - -| Event | Lasting effect (data) | Live effect | -| --- | --- | --- | -| remove a guest | guest record blocked | `revoke({ grant })` | -| reset a share link | new token; its guests' grants blocked | `revoke({ grant })` for each | -| archive a session | grants of that session void while archived | `revoke({ session })` | -| restore a session | grants valid again; revision bumped | none: guests reconnect | -| sign out a device | device revoked until its expiry | `revoke({ device })` | -| sign out all devices | signing key rotated, tokens cleared | `revoke({ device })` for each | - -A guest who rejoins with the same browser keeps its identity; a removed guest cannot mint a new one, because the -block is in `Guests` and `authenticate` refuses it. Names are unverified, cleaned, at most 40 characters, unique per -session ("Ann (2)"). - -## The catalog - -`Catalog` owns the session records: id, title (and whether it was edited), location, model default, pinned, -archived, timestamps. It lives on the server, persisted, and it **never starts a worker**: listing sessions, showing -their status or their unread marks reads server-side copies only. - -```ts -export const Catalog = remote("p3.catalog", { - version: 1, - input: obj({}), - placement: () => Server(), - variants: { - owner: { - state: { sessions: obj({ items: arr(SessionRow) }) }, - methods: { - create: method(CreateArgs, str()), rename: method(RenameArgs), pin: method(PinArgs), - archive: method(obj({ id: str() })), restore: method(obj({ id: str() })), - }, - }, - guest: { state: { sessions: obj({ items: arr(SessionRow) }) } }, // only the guest's own session, if not archived - }, -}); -``` - -Behind it sits a persisted `local` `CatalogStore`; the guest variant is a derive over it, as in chapter IV. The lifecycle operations (archive, clone, fork, -relocate, delete) are chapter XIX; here only the rule that keeps the catalog small: **the catalog does not know which -plugins exist.** Archiving a session writes `archived: true` and revokes the session; every plugin that keeps -per-session things reconciles from `catalog.sessions` with a routine (chapter X), as the notes plugin did. The ports -plugin closes forwards of archived sessions that way, also after a crash. - -## On the worker: start data, location, harness - -Three core objects exist on every worker from its start: - -- **`SessionInfo`** (chapter IX): the start message as `constant(init)`, with the session id, data directory and the - location the server chose at start. -- **`SessionLocation { session }`** (working name), lifetime host: the current cwd and environment, with a location - revision. Relocation changes it while the worker runs (chapter XIX); natives that depend on the location follow it - with a routine and a generation (chapter IX), and conversations that follow the session's cwd are listed in the core - per-conversation policy document. -- **`Harness`** (chapter XIV): the Durable harness for the session's store, one SQLite database per session, opened - under an exclusive lock keyed by the real data directory and session id. It refuses to start when the saved root - model is not configured, and resumes interrupted Durable work on start. - -`ConversationStore` and the `Conversation` views in front of it are chapter XIX. - -## Workers: start, keep alive, stop - -Starting, supervising and stopping workers is app code (chapter I); Chord only routes and counts. p3's supervisor is -`server.route(Session, { authorize, resolve })` plus a small server object that keeps per-session worker state: - -:::steps | A worker's life, as the supervisor sees it -+++ Authorize, then start -A tab gets `Notes { S1 }`. `authorize` checks `can(p, "read", "S1")` and that the session exists and is not archived -(an archived session refuses, except an index rebuild that starts the worker without resuming work). `resolve` forks -`worker/main.ts` with the start data from the catalog, bounded by a 60 s readiness timeout. -+++ A start failure sticks -If the worker fails to start (an unconfigured environment, a missing model), the error is kept per session and -every `get` fails at once with it, until the owner presses Retry. That clears the error and the related notices. -Guests see that the session is unavailable, not the error text. -+++ Counting -The core facet on the worker reports `host.activity` (`{ references, calls, jobs, lingering }`) and the harness's -running Durable task count to the supervisor object, from a routine whenever either changes. The call carries -`ctx.incarnation`, so a report from an older worker incarnation is ignored (chapter X). -+++ Idle stop -When everything has been zero for five minutes (checked every second, as in p2), the supervisor stops the worker: -the worker closes its host (routines stop, natives close, storage flushes) and exits. Durable work stays persisted -and resumes on the next start. Detached work that must keep the worker alive (a shell command whose caller left) is -a job, so it counts. -+++ Crash -If the worker exits on its own, the link drops. The supervisor records the exit code and a redacted stderr tail -(at most 4,000 characters) for that incarnation and posts one notice. References that still exist make Chord call -`resolve` again with backoff; a passive viewer recovers without a click. -+++ Shutdown -On server shutdown the supervisor stops every worker, with a forced exit after three seconds. Durable work remains -persisted. -::: - -## Status without a worker - -The session list shows per session whether it is running, busy (which conversations), its task count, usage and cost, -and an error. None of that may start a worker. The pattern is chapter X's pub/sub: a reporter on each worker (a -routine over the harness's state) calls `SessionStatus.report(summary)` on the server; `SessionStatus` keeps the latest -value per session and ignores calls whose `ctx.incarnation` is older than the newest it has seen; clients read -`SessionStatus`, which lives on the server. Children lists, unread marks, search index rows and artifact lists use the -same pattern (chapter XIX). - -## Settings and notices - -**Settings** (default model and thinking, run settings, compaction, queue modes, code mode switch, owner name) have -**one writer**: a core `Settings` object on the server owns the file, validates writes and serializes them; every -other process follows its state through a reference instead of watching and rewriting the file itself. p2 watches -settings, models and auth in every process and writes them from two, which is decision #21. - -**Notices** are one channel: one core `Notices` object on the server, written by everything that fails or finishes -(worker crashes, reload results, model errors, login outcomes, grants), keeping one latest notice per source, at most -50. The owner's shell reads one list; dismissing is a method. p2 has five notice channels for the same thing. - -:::review | Review: settings and file formats (#21, #22) -Single writer for settings, models and auth (#21) and versioning on-disk formats such as `envs.json` like documents -(#22) are open p3 decisions in `design.md` §26. The construction above assumes "one owner on the server, others -follow its state" for #21. -::: - -:::note | Built so far -Nothing of p3 exists. The functionality map is complete (`apps/p2/docs/p3-functions/`, five area files plus the -condensed `README.md`), and the design's app rules (`design.md` §25) are decided. p3 starts after the layered rebuild -and the playground's first real proof. -::: diff --git a/packages/chord/docs/next/explainer/src/19-p2-sessions.md b/packages/chord/docs/next/explainer/src/19-p2-sessions.md deleted file mode 100644 index 9d40c2287..000000000 --- a/packages/chord/docs/next/explainer/src/19-p2-sessions.md +++ /dev/null @@ -1,276 +0,0 @@ - -# Sessions and Conversations {#p2-sessions lede="The session lifecycle, status without a worker, chat, shell commands, subagents and search, each mapped to an owner and the Chord parts it uses."} - -This chapter builds the domains a person touches most: the session list and its operations, the conversation view -and composer, shell commands, chat mode, subagents, messages between sessions, search and unread marks. Each section -names the owner, the host, and the Chord parts it uses; code appears only where it shows something new. - -## The session lifecycle - -The catalog (chapter XVIII) owns the session records. Its operations, by the Chord part each needs: - -| Operation | Owner and form | Notes | -| --- | --- | --- | -| create | `Catalog.create`, a method | absolute directory, `~` expanded, must be a directory; on an environment, resolved there through the envs plugin; refuses an unknown model; model and thinking default to the last used | -| rename, pin | methods on `Catalog` | title 1 to 120 characters, whitespace collapsed; an explicit title is never replaced automatically | -| automatic title | a routine on the search feed | the first user message, line breaks to spaces, 100 characters | -| agent rename | an agent tool on the worker calling `Catalog` | root conversation only | -| archive, restore | methods; revocation; reconciling routines | below | -| clone, fork, relocate | jobs on the catalog store | outlive the call, cancellable, one outcome | -| delete | a job; decision #18 | below | - -### Archive and restore - -Archiving must stop the worker whatever it is doing, end guest access, and close the session's port forwards, while -keeping its storage. The catalog does none of the plugin-specific parts: - -:::steps | Archiving S1 -+++ One write -`Catalog.archive({ id: "S1" })` sets `archived: true` in the session record, in one transaction. -+++ Revoke the session -The catalog calls `host.revoke({ session: "S1" })`. Every guest principal of S1 has `{ session: "S1" }` in its -`revokedBy`, so their objects end and their links close on the server and on the worker (chapter VII). New -connections are refused by `authenticate`, because grants of an archived session are void. -+++ Stop the worker -The supervisor reconciles from `catalog.sessions` with a routine: an archived session's worker is stopped, and -`authorize` refuses to start it again. -+++ Plugins reconcile -The ports plugin's routine on `catalog.sessions` closes forwards of archived sessions. The search plugin marks the -session's rows archived. If the server crashed between the write and these steps, the routines run again with the -current value at the next start: nothing is a notification that can be lost. -+++ Restore -`Catalog.restore` clears the flag. Grants are valid again, so guests reconnect and authenticate as before; forwards -stay closed. -::: - -### Clone, fork, relocate - -Each is work that outlives its call, can be cancelled and has one outcome, so each is a **job** (chapter XI) on the -catalog's local store, keyed by the owner's request id. They differ in what they pause: - -- **Clone** copies the session's store, uploads, downloads and artifacts under a new id, with the source worker - paused (refused while work runs) and title "(copy)"; directories `0700`, files `0600`, no links, no overwrite; a - partial copy is removed on failure. -- **Fork** before a user message copies the active context as of the entry before it into a new session (title - "(fork)"), with the message waiting in the new session's editor. It needs two workers: the source reads history, - the new one is seeded empty. A failure archives the copy. -- **Relocate** (cwd, environment or both): refused while the session is busy (naming the running subagents) or when - a subagent has its own cwd and the move crosses machines. The job fences the worker (new admissions wait), updates - the record, and moves the running worker in place by updating `SessionLocation`: natives that follow the location - switch with a generation (chapter IX), and conversations that follow the session's cwd are rewritten in one Durable - commit before work resumes. After 60 s it restarts the worker instead; on failure it releases the fence. - -:::review | Review: jobs as objects (#9) -The design defers whether a long relocation or a login is a record inside its owner (watched through `running`) or an -object of its own (`Relocation { id }`). The construction above uses records. Decide when login is ported -(`design.md` §15, §26). -::: - -### Delete - -p2 has no session delete. p3 needs one that removes the session everywhere: every document tagged with the session -(`store.retireByTag(SessionTag, id)`, chapter V), the worker's databases (the Durable store), the session's files, the -search rows, read marks, children copies, forwards, guest records, and finally the catalog entry. The design commits to -this outcome (`design.md` §25); how plugins take part is open. - -:::review | Review: session delete (#18) -Options in `design.md` §26: per-session objects plus `SessionTag` plus one session directory (delete is retire by tag -and remove a directory, plugins need do nothing), or owner hooks contributed through a catalog registry. Also open: -whether the search index is per session (deleted with it) or global with a hook. Who holds the store for -`retireByTag` is not specified either. -::: - -## Status, children, unread: reports to the server - -The session list must show status, subagents and unread marks for sessions whose workers are not running, and must -not start them. All three follow chapter X's pub/sub pattern: a routine on the worker calls a server object with the -current summary, the server object keeps the latest value per session (persisted where it must survive a restart), -ignores calls from an older `ctx.incarnation`, and clients read the server object. - -| Summary | Source on the worker | Kept on the server | -| --- | --- | --- | -| status: running, error, task count, busy conversations, usage and cost | the harness's task graph and usage | `SessionStatus`, in memory: a stopped worker is "not running" | -| children: name, model, thinking, tools, cwd, last entry, archived | the subagents plugin's child list | `Children`, persisted | -| latest root answer | the search feed | `Unread`: compared with the owner's read mark, persisted, shared by all owner devices | - -Guests see their own session's status without the error text. The owner's read mark is written when the root -conversation is visible, and a new session starts as read. - -## Conversations - -The harness (chapter XIV) owns the Durable store. The UI needs, per conversation: a live window of entries, older -pages, usage numbers, the working indicator, the inbox (queued inputs), and the actions. Two core objects provide them: - -- **`ConversationStore { conversation }`** (local, on the worker): opens the Durable watch as its native, reaching - the harness through the core-private `HarnessRuntime` registry entry, and exposes a **window** as a derive: the - newest entries (at most 100 or 512 KiB, at least 20) plus the conversation's documents. The window's start stays - put while it is within 1.5 times the limits, so new entries arrive as small appends instead of a shifted list - (chapter VIII's window pattern). -- **`Conversation { session, conversation }`** (remote, placement `Session`): the per-principal view. Variants by - `can()`: `owner` (everything, including model, thinking and handoff), `writer` (guest write: send, steer, stop, - compact, shell), `reader` (the window only). - -```ts -host.serve(Conversation, ({ session, conversation }, principal, self) => { - const store = self.get(ConversationStore, { conversation }); - const policy = self.get(ConversationPolicy, { conversation }); // core: readOnly, followsSessionCwd - const author = authorOf(principal); // structured: kind, id, display name - if (!can(principal, "send", session)) return { variant: "reader", view: store.window, page: store.page }; - return { - variant: can(principal, "manage") ? "owner" : "writer", - view: store.window, - page: store.page, // older history: returns data - send: ({ text, attachments, mode, requestId }, ctx) => { - if (policy.value.readOnly) throw forbidden("read-only conversation"); - return store.submit({ type: "input", content: text, attachments, author, whenBusy: mode, requestId }, ctx); - }, - stop: (_args, ctx) => store.stop(author, ctx), // allowed even when read-only - … - }; -}); -``` - -What each function becomes: - -| Function | How | -| --- | --- | -| send, retried send | Durable `submit` with the author's `requestId`: a retry with the same id is admitted once (Durable dedupes, chapter XIV) | -| steer or queue while busy | `whenBusy: "steer"` or a follow-up; the inbox is `docs["pi.inbox"]` in the window | -| attribution | the view derives the author from the principal; the entry stores it structured, the model sees a speaker block; nothing parses text | -| wait for prompt resources | the first input of a fresh worker waits until the prompt plugin's resources are loaded (a registry readable the store follows) | -| stop, dequeue | withdraw queued inputs, abort the run; the caller's texts return to its editor, other authors' to their **restored drafts** | -| restored drafts | a keyed document per (conversation, author), tagged `SessionTag`, persisted in the store; claimed once, first tab wins | -| read-only conversations | the core per-conversation policy document, written at `createConversation({ init })` by whoever creates the conversation; send, dequeue, configure, compact and shell refuse; stop is allowed | -| history pages | a method returning a page (100 entries, 1 MB, 2,000 scanned); compaction and reset markers always included | -| model and thinking | owner variant only; validated against the models plugin's catalog; the root conversation's model is also reported as the session default | -| compaction, handoff | Durable operations through the store | - -**Drafts in the composer** are UI state, not Chord state: the web shell keeps one draft in `sessionStorage`, the TUI -keeps one per conversation across UI replacements, and pending sends keep their request id until acknowledged -("Sending when connected"). Only restored drafts, which the worker produces, are documents. - -## Shell commands - -`!cmd` runs a command in the conversation's cwd on its environment and records the result as an entry; `!!cmd` does -the same but keeps the output out of the model's context. One command at a time per conversation; the live tail is -at most 50 KB or 2,000 lines, sampled every 200 ms; the full output spills to a file; the result entry is placed at the -run's next boundary while the agent is busy and never starts a run. This is the design's job example almost as is: - -```ts -const ShellJob = defineJob("chat.shell", { - version: 1, - args: obj({ command: str({ max: 100_000 }), include: bool(), author: Author }), - progress: obj({ output: str() }), - result: ShellResult, - concurrency: 1, // one per conversation -}); - -const results = self.state(ShellResults); // keyed by run id: dedupe after completion -const shell = self.job(ShellJob, { - run: async ({ command, include, author }, ctx) => { - const out = new OutputBuffer({ maxBytes: 50_000, maxLines: 2000, retain: "tail" }); - const env = await environmentFor(ctx); // the conversation's environment, chapter XX - const r = await env.exec(command, { cwd: env.cwd, signal: ctx.signal, spill: spillPath(ctx.jobId) }, - (chunk) => { out.push(chunk); ctx.progress({ output: out.snapshot().text }); }); - const data = { command, output: out.snapshot().text, exit: r.code, cancelled: ctx.signal.aborted }; - await conversation.submit( - { type: "write", entry: shellEntry(data, author, include), requestId: `shell:${ctx.jobId}` }, ctx); - return data; - }, - finish: async (done, tx) => { - (await tx.draft(results, done.id)).outcome = done.kind === "done" ? done.result : { error: done.error }; - }, -}); -``` - -The request id `shell:${ctx.jobId}` makes the entry idempotent: an attempt interrupted after the write cannot write -it twice. Cancel is `shell.abort(id)`, allowed for the command's author and the owner. A retried `runShell` with the -same request id returns the running job (dedupe of unsettled runs), or the stored result (the owner's own dedupe after -completion, chapter XI). p2's "cancel when nobody watched for 30 s" is dropped: a job keeps its owner alive and does -not depend on viewers. - -## Chat mode - -Chat mode is a per-conversation switch that everyone in the session sees and only the owner changes; forks and new -subagents start with it off. Because it must fork and rewind with the conversation, it is **transcript-bound**: a -Durable conversation document, not a Chord document. Messages without `@pi` become display-only Durable `write` -entries (author, text, attachments, reply quote); a message with `@pi` becomes an input with a digest of the chat -since the last mention (at most 100 messages, 20 files, 4 images), recording which entries it passed on. The reply -quote is derived on the worker from the entry id the client names: the client never supplies author or quote text. - -## Subagents and one-shot agents - -Subagents are Durable child conversations owned by tasks, as in Durable's own background-subagent example: one -`subagent` tool spawns, steers, waits for, stops and lists them; each child is owned by a background anchor task, and -each message is delivered by a reporter task that posts the answer back to the parent once, with request ids so a -restart sends nothing twice. Chord adds only three things: - -- the **child list** for the sidebar is reported to the server (`Children`, above), so it shows without a worker; -- a child that follows the session's cwd, or a read-only one-shot agent created by a code-mode script, is marked in - the core **per-conversation policy document** when it is created, so chat and relocation enforce it without knowing - the subagents plugin (`design.md` §25 porting guideline); -- archive and restore of a child from the UI are methods on the subagents plugin's view, refused for one-shots. - -Subagents cannot spawn subagents; thinking levels are clamped to what the model supports, in one helper shared with -the session's model choice. - -## Messages between sessions - -The root agent can list other sessions and send one an attributed message. The tool runs on the sender's worker and -calls a server object through `chord(fn, context)` (chapter XIV): - -```ts -const peerSend = defineTool({ - name: "session_send", - execute: (args, api, context) => chord(async (ctx) => { - const peers = await ctx.get(Peers, {}); // server, worker variant - await peers.send({ to: args.session, text: args.text, mode: args.mode, requestId: `peer:${api.taskId}` }, ctx); - return { output: [{ type: "text", text: "sent" }] }; - }, context), -}); -``` - -`Peers` checks that the sender is a root conversation of a non-archived session, refuses sending to itself and -archived recipients, then gets the recipient's `Conversation` (which starts its worker) and submits with the -sender as a structured author and the same request id (the tool task's id, stable across restarts), so a retried -tool call delivers once. The sender's -incarnation is `ctx.incarnation`; a call from a replaced worker is refused. Only owner-initiated turns may use the -tool; guests never. - -## Search - -The search plugin indexes user and assistant messages of root and subagent conversations (no shell output, no -Durable internal inputs) for title, cwd and message search across sessions, active or archived. - -:::flow wide animate | The search feed. The worker pushes pages from a cursor; the server owns the index and never starts a worker to answer a query. -row "Worker S1" blue -box commit "Durable commit" "in any conversation" -box wake "wake" "a routine on the harness's commit signal" -box page "read from cursor" "200 entries per page" -row "Server" accent -box index "SearchIndex" "native: SQLite FTS" accent -box cursor "cursor per conversation" "moves after the whole range" -box query "query" "terms, scope, at most 41 hits" -arrow commit -> wake -arrow wake -> page -arrow page -> index "report(page)" -arrow index -> cursor -note "Reports carry ctx.incarnation. Indexing is idempotent: a page sent twice changes nothing." -::: - -The index is a native owned by a host-lifetime object on the server; cursors live with the index, so after a crash -the worker asks for the cursor and continues. Rebuilding one session's index drops its rows and rescans, starting -an archived session's worker without resuming its work. Queries are methods returning data: at most 41 hits, -snippets of 300 characters, queries up to 500 characters. The same feed sets automatic titles, moves "updated at" -forward and supplies the latest root answer for unread marks. - -## Presence - -Viewers per session are `host.references(Conversation)` on the worker, counted per client (chapter VIII), and -reported with the status. Connected guests (name and mode) are the references whose principal is a guest. Presence -is not keep-alive: the supervisor uses `host.activity`, which also counts calls and jobs. - -:::note | Built so far -None of this exists in p3. The parts it relies on are Durable (built: submit with request ids, steer and queue, -child conversations, background tasks) and the Chord layers L3 to L9 (not built). -::: diff --git a/packages/chord/docs/next/explainer/src/20-p2-machines.md b/packages/chord/docs/next/explainer/src/20-p2-machines.md deleted file mode 100644 index 51b9e7546..000000000 --- a/packages/chord/docs/next/explainer/src/20-p2-machines.md +++ /dev/null @@ -1,245 +0,0 @@ - -# Machines, Files, Models {#p2-machines lede="Environments, workspace, terminals, session files, ports and Radius, models and logins, prompt resources, logs, reload and the shells: the rest of p3."} - -The last domains are the ones that touch the outside world: other machines over SSH, the file system and git, -terminals, the relay, model providers and their credentials. Most of them are native objects, and most of them are -places where p2 has the same thing several times (four SSH connections to one environment, five notice channels, -three copies of the environment config parser). The rule that fixes that is the one from chapter XVIII: one owner per -thing. - -## Environments - -The **envs plugin** owns the environment configuration: name, host, user, port, identity file, shell and its -variables, plus a managed ed25519 key per environment and its own `known_hosts` (kept apart from the user's, aliased -per environment name). - -| Function | Owner and form | -| --- | --- | -| list, add, edit, delete (refused while a session, archived too, uses it) | `Envs` on the server: a persisted document behind an owner-only view | -| validation (no leading `-`, no whitespace or control characters in host and user) | the view's methods, before the write | -| a hand-edited `envs.json` is picked up; an invalid file keeps the last valid set and shows an error | a native file watcher (poll fallback) in the envs store; one writer (#21) | -| environment revision: a hash of the config, part of its identity | a derive | -| connection test (connect, deploy the daemon, platform, latency, warnings) | a job; the result per revision in a document | -| unknown or changed host key: fingerprints, trust after confirmation, forget | a flow (chapter XII) on the server | -| browse directories (`~`, relative paths, at most 10k scanned, 1,000 shown) | a method with a **call-scoped** SSH session (chapter IX) or the pooled connection | -| a changed environment restarts its sessions' workers once idle | a routine on the envs store and the status reports | - -**Execution in the agent.** The worker's harness asks `HarnessEnvironments` for an environment per target (chapter -XIV). The envs plugin contributes it from its worker facet: local execution at the session's cwd, or a remote one over -SSH through the `pi-env` daemon, deployed per platform and connecting at the first operation. A remote session never -falls back to local tools. The environment follows `SessionLocation`: after a move, the old connection closes once -the move has committed and watchers reopen at the new location. - -```ts -// envs/worker.ts -export default defineFacet((host, self) => { - const location = self.get(SessionLocation, { session }); - const pool = self.get(EnvPool, {}); // one owner of SSH connections (#20) - self.contribute(HarnessEnvironments, (target) => - target.env === "local" ? localEnv(target.cwd) : pool.environment(target.env, target.cwd)); -}); -``` - -:::review | Review: connection pools (#20) -SSH connections to one environment exist four times in p2 (worker environment, browse connection, test connection, -terminal host). `design.md` §26 asks for one owner with a pool or deliberate per-use connections. `EnvPool` above is -the first option: a native object that closes idle connections after 10 minutes and invalidates them on a config or -key change. Reloading the plugin that owns the pool kills running remote commands (chapter XV), so it should change -rarely. -::: - -## Workspace and file index - -The **workspace plugin** serves the viewed conversation's working tree (a subagent's own cwd included) on the worker: -list a folder (at most 2,000 entries, `.git` hidden), read a file (1 MiB, binary detected), git status, diff, log and -single commits, literal text search, and all paths for name filtering. Every query is a **method returning data**, not -state: results are bounded and requested by the UI. Only change notifications are state: a counter in a document, -bumped by a native watcher of the tree, `HEAD`, the index and config (debounced 100 ms); viewers reload what they show -when it changes. - -Two rules live in one place each instead of three: - -- **Confinement:** path rules, a canonical root, no `.git`, the app's private data directory denied (also excluded - from search). One helper used by listing, reading, uploads and the file index. -- **Git hardening:** no optional locks, no fsmonitor, filters disabled, no prompts or pager, no system config, 10 s - timeout, output caps. - -Downloads (25 MiB, byte ranges for media) go through the HTTP component (chapter XIII) with a route on the server that -reads chunks from the worker; uploads into the project use the chunked protocol of chapter XIII (1 MiB chunks, temp -file then rename, never overwrite, writes serialized). Guest read gets no workspace: the view has no variant for it. - -The **file index** for `@` completion belongs to the same owner, so completion consumes the index instead of scanning -on its own: built from the environment's listing (else git or a walk, at most 50k paths), rebuilt at most once per -second while files change, discarded after a move. Review comments drafted on files and diffs are browser state until -sent; sending them is an ordinary `Conversation.send`. - -## Terminals - -Terminals must survive server and worker restarts, so their PTYs live in a **detached terminal host process**, one per -data directory, with its own process lock; it exits after 10 minutes with no terminals and no clients. The terminals -plugin on the server is the owner the UI talks to: list (title, cwd, environment, size, exit status), open at the -session's cwd and environment (refused for archived sessions), attach, close (asking first if a foreground job runs). -Terminals are **owner only**. - -Bytes do not go over object links. Attaching opens a **channel** (chapter XIII) with a single-use token (30 s): JSON -control on the object, bytes on the channel, with its own acknowledged window (256 KiB, acknowledgements per 64 KiB; a -viewer that falls behind gets a fresh screen snapshot; an 8 MiB backlog disconnects). Attaching gets the current -screen from a headless emulator with 10k lines of scrollback. - -:::review | Review: the terminal host process (#23) -The terminal host runs its own code. Is it part of the revision, reloaded or restarted with it (and running terminals -end), or deliberately outside it with a "host runs older code" notice as in p2? Open in `design.md` §26. -::: - -## Session files - -Attachments (25 MiB, at most 20 per message), sent-file snapshots made by the agent's `send_file` tool, versioned -artifacts (HTML, SVG, Markdown, text; 2 MiB; each update a new immutable version) and shell spill files are all -per-session bytes with caps, immutable ids and range reads; p2 has two upload routes, two caps and two directory -layouts for them. The proposal is one owner: - -```ts -export const SessionFiles = remote("p3.session-files", { - version: 1, - input: obj({ session: str() }), - placement: () => Server(), // readable without a worker - variants: { writer: { … }, reader: { … }, worker: { … } }, -}); -``` - -`SessionFiles { session }` on the server holds the index (per kind, readable without a worker), the chunked upload -and read protocol, the caps per kind, copying on clone and fork, and deletion with the session. Attachments are copied -into a remote environment before an input is admitted, with no fallback on failure. Artifact previews run in a -sandboxed frame with its own CSP and no app identity; ordinary file previews allow only safe types. - -:::review | Review: session files (#17) -Proposed in `design.md` §26, not decided. Open sub-points: one preview limit (p2 has 256 KiB for sent files and -512 KiB for artifacts), what a fork copies of the artifact list, and how attachments are copied into remote -environments. Related: #22, whether on-disk layouts such as downloads and uploads get versions like documents. -::: - -## Ports and Radius - -The **radius plugin** keeps the outbound relay connection, a native in a host-lifetime object on the server: sign-in -by device code from the CLI (credential in a private file), token refresh, a heartbeat in the link adapter (20 s ping, -60 s timeout; TCP does not notice a sleeping laptop), backoff from 1 to 30 s, a disable switch. The relay delivers two -kinds of channels: - -```ts -// radius/server.ts (the entry point: one line, as in chapter I) -radius.onChannel((ws) => host.accept(webSocketLink(ws), { principal: { kind: "owner", access: "relay" } })); -radius.onForward((channel) => proxyToForward(channel, host.registry(RadiusForwards).value)); -``` - -Object links become owner principals with `access: "relay"`: owner, but treated as remote (no localhost-only -powers). p2 had to fake a loopback request with a secret header for this; with `accept` that is gone. - -Forward channels go to `localhost:` (Host, Origin and Location rewritten; 502 when nothing listens; anonymous -visitors only to public forwards). Which forwards exist is not radius's business: it owns a registry, -`radius.forwards`, and the **ports plugin** contributes its forwards (at most 50, random ids, a public toggle that -needs confirmation, an agent tool `ports` for top-level owner-initiated conversations). Without the ports plugin the -table is empty and radius still works: an optional dependency is a registry owned by the dependent (chapter XIII). The -ports plugin closes forwards of archived sessions with a routine on `catalog.sessions` (chapter XIX). - -:::review | Review: may a plugin call `accept`? -`design.md` §3 shows the Radius line in the server's start code and calls entry points "app or adapter code". If -radius is a plugin, its facet needs the host's `accept`, which no other part of this guide gives a plugin. The -alternative is that the start code keeps the entry point and gets channels from the radius object. Not specified. -::: - -## Models and credentials - -The **models plugin** owns the model runtime (pi-ai's models with custom providers and the model cache) in a -host-lifetime object, and the credentials. It contributes the runtime to `HarnessModels` (chapter XIV), so after a -models reload the next request uses the new runtime while a running stream finishes on the old one. **Credentials -never go to clients**: the views expose providers, login methods and which type is signed in, never tokens. Every -secret is registered for redaction in one place, the log capture. - -Views: a model catalog for pickers (names, ids, context windows, thinking levels; at most 300 matches in the web), -logout (refused during a login), refresh a provider's model list. The default model and the cycling scope are -settings (chapter XVIII). - -### Login - -A login is the guide's flow example (chapter XII): a **job** on the server that runs pi-ai's login, and a flow that -shows its steps to the starting owner. - -```ts -const LoginJob = defineJob("models.login", { - version: 1, args: obj({ provider: str(), type: AuthType }), progress: LoginState, result: obj({ ok: bool() }), - concurrency: 1, // one login at a time on the server -}); - -const login = self.job(LoginJob, { - run: async ({ provider, type }, ctx) => { - await runtime.value.login(provider, type, { - signal: ctx.signal, // 10 minute deadline, cancel, leaving - prompt: (prompt) => steps.ask(prompt, ctx), // text, secret, select, manual_code - notify: (event) => steps.notice(event, ctx), // info, auth_url, device_code, progress - }); - return { ok: true }; - }, - finish: async (done, tx) => { /* record the outcome; refresh the model list */ }, -}); -``` - -`prompt` is a promise-based callback inside the work; the resolver stays in the job (`steps.ask` records the step in -the flow state and waits for `answer`), and nothing crosses an object boundary. On a relayed connection the device -code method is selected automatically. The flow is visible only to the owner principal that started it; others are -refused. Leaving ends it: when the starting owner's references are all gone (presence, not an unmount), a routine -aborts the job. Codes, URLs and typed secrets are redacted from logs. - -## Prompt resources - -`AGENTS.md`, skills and templates are read in the session's environment, watched (poll fallback, last valid state), -and expanded on the worker before sending. The **prompt plugin** owns them per session on the worker, contributes the -system prompt section and skills to `HarnessExtensions` as readable contributions (a change recomposes), and exposes a -readiness readable: the first input of a fresh worker waits until it is ready (chapter XIX). After a move the watchers -reopen at the new location. - -## Code mode - -Code mode is a plugin (chapter XIV): its `pi-codemode` sandbox is a native; scripts call tools as nested Durable calls -and replay after a restart; the tool entry looks up the current sandbox when it starts. A reload seals and drains the -old sandbox for at most 30 s. The on/off switch is a setting. - -## Logs and notices - -**Log capture** tees stdout and stderr of the server, workers and detached hosts without changing their behaviour, -redacts known secrets and credential patterns, and keeps bounded rings (at most 128 sources, 1,000 lines or 256 KiB -each). Detached hosts publish redacted snapshots to a private file. Logs are read without waking a stopped worker. -**Notices** are the one channel from chapter XVIII; dismissal stores only notice ids. - -## Dev and reload - -The dev plugin is chapter XV's reload control for p3: a source watcher marks plugins dirty and never applies anything; -the `reload_plugins` agent tool and a control in the owner's shell reload the changed plugins, or restart all hosts for -base code (through the supervisor, reopening the sessions that were running). UI bundles are published only after a -successful backend change; a web page detects a new manifest and reloads with its drafts and navigation state saved; -the TUI swaps its sandboxed UI code with a snapshot of its state. A local Unix socket (`0700`) accepts development -commands without a network admin port. - -## The shells - -The web and TUI shells are UI facets over the objects above. Two rules matter for Chord: - -- **UI state survives navigation, reload and reconnect on purpose.** Composer drafts, pending sends with their request - ids, reading positions and review drafts are kept in tab storage or the TUI's replacement snapshot. A reconnect - re-holds and gets snapshots; nothing is lost that was not in flight. -- **Leaving is never derived from an unmount.** A plugin reload remounts components (chapter XV). Cancelling a login, - closing an overlay or giving up a lease is an explicit action or a dropped reference. - -The TUI launcher keeps downloaded UI code in a permission-dropped worker with no access to files, network or -credentials; local side effects (copy, upload, download, open a URL, an external editor) need a real user gesture or a -confirmation. That is app code around Chord, not part of it. - -## Transport - -The last domain, transport, is chapter XVII: one message format, per-part limits (16 MiB compressed and 64 MiB decoded -on client WebSockets, instead of p2's 1 MiB input cap and 32 pending calls), one byte window per link, re-holds with -snapshots after a reconnect, and a protocol version that makes an outdated web page reload. - -:::note | Built so far -None of p3 exists. pi-ai's login API, pi-codemode and the `pi-env` daemon exist and are used by p2 today; the HTTP -component and channels are deferred until after the kernel's first real proof (chapter XIII). -::: diff --git a/packages/chord/docs/next/explainer/src/20a-compared.md b/packages/chord/docs/next/explainer/src/20a-compared.md deleted file mode 100644 index aa4fcd2c2..000000000 --- a/packages/chord/docs/next/explainer/src/20a-compared.md +++ /dev/null @@ -1,146 +0,0 @@ - -# Other Ways to Do It {#compared lede="Cordis and DeepSeek Harness attack the same problem, plugins that come and go while an agent runs, inside one process. What that buys, and what it costs."} - -Chord v2 is not the only answer to "replace plugin code while the app keeps running". The closest relative is -**Cordis**, a TypeScript plugin framework by Shigma that has been the plugin kernel of the Koishi chatbot framework -since 2019, and its most visible user, **DeepSeek Harness** (`dsh`), DeepSeek's open-source agent harness, whose -motto is "everything is a plugin". This chapter compares them with Chord v2 on the questions the guide cared about: -who owns side effects, how plugins depend on each other, what a reload keeps, what a crash keeps, and how far the model -reaches beyond one process. - -:::note | Sources and status -DeepSeek Harness at commit `687f282` (release 0.2.1-rc.2), which vendors Cordis as `@deepseek-ai/cordis` 4.0.5 (Cordis -RC7 with local changes, `vendor/README.md`): its Cordis tutorial (`docs/cordis-tutorial/`), `docs/architecture.md`, -`docs/subsystems/persistence.md`, `docs/subsystems/core.md` and `packages/boot/hmr/README.md`; the Cordis paper "A -Programming Paradigm for Spatiotemporal Composability" (arXiv 2608.25512). Cordis and dsh are built and in use; Chord v2 -is the design this guide describes. The comparison is between a working system and a design. -::: - -## Cordis in one example - -A Cordis plugin is a function (or an object with `apply`, or a `Service` subclass) that receives a **context**. It may -declare the services it needs with `inject`, and everything it does to the world goes through the context: - -```ts -export const name = "greeter"; -export const inject = ["tools"]; // activate only while a `tools` service exists - -export function apply(ctx: Context) { - ctx.tools.register(defineTool({ name: "greet", /* … */ })); // a registration is an effect: undone on unload - ctx.on("session/start", (s) => log(s.id)); // an effect-bound listener - ctx.effect(() => { // any other side effect, with its disposer - const timer = setInterval(work, 200); - return () => clearInterval(timer); - }); -} -``` - -Each plugin instance is a **fiber** with states `PENDING → LOADING → ACTIVE → UNLOADING → DISPOSED` (or `FAILED`). -Every effect, registration, listener, child plugin and provided service is recorded on the fiber, and unloading runs -the disposers. The paper calls this **temporal composability**: a component can be removed at any time and reverses its -own effects. Its counterpart, **spatial composability**, is the dependency side: `inject` is live, so a plugin starts -when its services appear, and unloads (and later reloads) when a service it depends on goes away or is replaced. The -paper's claim is that independently written components can therefore be combined and replaced without disturbing each -other. The next sections check how far that holds. - -**Services** are named capabilities on the context (`ctx.provide("x", value)` or `class X extends Service`). Names are -runtime strings in one flat namespace per application; TypeScript declaration merging makes `ctx.x` typecheck, but -nothing checks the service's shape at runtime, and optional dependencies are probed with `ctx.get("x")`. Scopes come -from the context tree: `ctx.extend()` derives a child context, `ctx.isolate(name)` gives one subtree its own binding of -a service, so two parts of an app can use two different `shell` providers. - -**Hot reload** follows from the two halves: the reload plugin unloads the old fiber (unwinding all its effects) and -loads the new module with the same config. Nothing held in the old instance's closures or fields survives; a plugin -that needs state across reloads must keep it in a service or on disk. Config edits are diffed by entry id and -mount, unmount or reconfigure only what changed; config fields declared volatile update in place without a remount. - -## DeepSeek Harness on Cordis - -dsh applies "everything is a plugin" to an agent: the model adapters (`ctx.llm`), the tool registry (`ctx.tools`), the -session log (`ctx.sessions`), the agents (`ctx.agents`) and the agent loop itself (`ctx.agentLoop`) are plugins and -services. A running dsh is a plugin tree composed at boot from a **profile** (`web`, `headless`, `sdk`, `acp`) that -stacks **bundles** of Cordis config rows, with user patches on top. There is no privileged core: replacing the agent -loop is a config patch. - -| Concern | How dsh does it | -| --- | --- | -| sessions | a durable, append-only event log per session (JSONL backend, single writer, `flush` as the durability barrier); model history is derived from the events | -| crash mid-turn | the log keeps everything appended; on resume the agent loop closes the open turn by appending missing tool errors and a synthetic `turn/end { reason: "interrupted" }` | -| tools | `ctx.tools.register(defineTool(…))`, an effect of the registering plugin; calls go through pre/execute/post policy waterfalls | -| approval | closed by default; `allowed-once` is the only grant; plugin installation needs full access or per-call approval | -| reload | a source watcher evicts changed modules; Loader entries running affected plugins stop and start again with the new code; a failed replacement is an activation failure, its previous code no longer runs | -| processes | one plugin tree in the Host process; the browser client and Electron's renderer talk to it over RPC streams | -| distribution | a Plugin Manager installs and enables bundles per profile; plugin markets are optional indexes | -| subagents | local subagents with their own durable sessions, and external ones | - -## What the guarantees are in practice - -Reading the Cordis source vendored in dsh and dsh's own documentation, the guarantees are narrower than the paper's -framing. An earlier review for p2 came to the same result (`apps/p2/docs/p3-reviews/dsh-compare.md`). - -- **Reversal is cooperative and unordered.** `_unload` runs all of a fiber's top-level disposers concurrently - (`Promise.all` in `vendor/cordis/src/fiber.ts`); order holds only inside one effect, and the primer tells authors to - "keep the related work in one effect" when teardown order matters. A disposer that throws is logged and skipped. A - disposer that never settles stalls the teardown; dsh's jobs service works around this by force-failing a job whose - cancel throws and warning that "the work may be orphaned" (`packages/jobs/jobs-local/README.md`). Reversal is as - good as every disposer, and nothing checks that a plugin registered all its effects. -- **Behaviour depends on registration order, and a reload changes it.** `emit`, `waterfall`, `serial` and `bail` - listeners run in registration order (`docs/cordis-primer.md`). Registering appends the listener (`push` in - `vendor/cordis/src/events.ts`, unless `prepend`), so when a plugin reloads, its listeners move to the end of the - list. A tool-policy listener that ran first runs last after its own plugin reloads; a waterfall that short-circuits - now sees a different decision first. -- **Dependencies are strings, and every provider reload restarts its dependents.** A fiber's activation key is the - concatenation of its injected providers' fiber ids (`_refresh` in `fiber.ts`), so replacing a provider unloads and - reloads every plugin that injects it, with their in-memory state. Service APIs are "pre-stable; update every - consumer" (`AGENTS.md`): there is no versioned contract between a provider and its consumers. -- **Reloading a core service ends live work.** Unloading the agent provider "stops and drains every live handle it - made" (`packages/core/agent/README.md`), so running agents stop; reloading the session persistence backend under - live sessions invalidates their write handles, and "writes fail loudly until the sessions restart" - (`.agents/notes/implemented/architecture/2026-08-27-handle-based-session-persistence.md`). -- **Registrations leave a gap.** A reloading plugin's tools and other registry entries are removed with its old fiber - and added again by the new one; anything that looks them up in between does not find them. - -None of this is a bug in the narrow sense: each point is documented, and dsh is built around it. But "components -interleave without disturbing each other" holds only for plugins that own nothing other plugins depend on, register -every effect, order nothing by registration, and hold no state that must outlive a reload. - -## Side by side - -| | Cordis and dsh | Chord v2 | -| --- | --- | --- | -| unit | a plugin instance (fiber) in one context tree | an object per (type, input, principal), placed on a host (chapters II, III) | -| side effects | recorded on the fiber with `ctx.effect` and friends; on unload the disposers run concurrently, each as good as its author made it | declared through `self` (documents, natives, routines, jobs, contributions); Chord activates and undoes them (chapters V, IX to XI) | -| dependencies | service names as strings in one flat namespace, via `inject`; a provider reload restarts every dependent | typed contracts via `get(Type, input)`; a reference parks and recovers through restarts (chapter VI); optional dependencies are registries (chapter XIII) | -| ordering | listeners run in registration order, which changes when a plugin reloads | registries compose with a declared `compose` function; nothing depends on load order (chapter XIII) | -| where code runs | one process; remote UIs over RPC to the Host | a tree of hosts (server, workers, tabs); objects reachable across links, relayed by the server (chapters I, XVII) | -| who is calling | not a framework concept | principals stamped by links, variants per caller, revocation (chapters III, VII) | -| state | the plugin's own business; dsh's session log is a service | documents written in transactions, persisted, versioned and migrated (chapter V) | -| reload | unload the old instance, load the new one: in-memory state and running work of that plugin end | pure objects swap their code in place with their documents and running work kept; only objects with natives restart (chapter XV) | -| a running agent turn during reload | entries of changed plugins stop; unloading the agent provider stops its live agents | the harness is base code; Durable pins tools per task phase, so a running call finishes on old code and the next phase uses the new (chapter XIV) | -| crash recovery of a turn | the turn is closed as `interrupted` | pi-durable continues the turn's tasks from their checkpoints; replay-safe tools rerun (chapter XIV) | -| contracts and versions | TypeScript types only, pre-stable APIs, every consumer updated together; config validated with a schema | contract versions with fingerprints and provider-side mappings for old consumers (chapter XVI) | -| status | in production in Koishi since 2019; dsh released | a design with the contract layer built | - -## What the comparison says - -**The core idea is the same.** "Factories declare, Chord activates" (chapter II) and Cordis' reversible effects are -two forms of one rule: a plugin may only change the world through its runtime, so the runtime knows what to undo. -Both reject the module-level timer and the stored callback for the same reason. - -**The difference is what survives a reload.** Cordis reloads at plugin granularity: the old instance unwinds and the -new one starts from nothing, together with every plugin that depends on it. That is simple, but a plugin's in-memory -state and its dependents' state and work in flight end with it, and correctness rests on every disposer and on -registration order. Chord v2 keeps state out of closures on purpose (documents, references, jobs as -records) so that it can swap the code of a pure object while its data and running work stay, and restarts only what -holds OS handles. That is more machinery: transactions, a store, versions on every document and contract. - -**The other difference is reach.** Cordis composes plugins inside one process, and dsh adds RPC to its own clients. -Chord v2 makes the process boundary part of the model: objects live on the host their placement names, callers are -principals, and a reload rolls out across the server, the workers and every tab. An app like p2, with a process per -session and guests in other browsers, needs that; a single-process agent does not. - -:::review | Review: what a reload does to a running tool call in dsh -The sources say that Loader entries running affected plugins stop and that unloading the agent provider stops its live -agents (`packages/boot/hmr/README.md`, `docs/subsystems/core.md`). They do not say what happens to a tool call in -flight when only the tool's plugin reloads; the comparison above does not claim either way. -::: diff --git a/packages/chord/docs/next/explainer/src/21-epilogue.md b/packages/chord/docs/next/explainer/src/21-epilogue.md deleted file mode 100644 index 987fb3c44..000000000 --- a/packages/chord/docs/next/explainer/src/21-epilogue.md +++ /dev/null @@ -1,104 +0,0 @@ - -# What's Next {.epilogue lede="Where Chord v2 stands, what is still undecided, and where the written sources disagree with each other."} - -The guide built Studio from one object to a p2-sized app. Every piece used the same few rules: one owner per thing, -factories that only declare, state that changes only in transactions, long-lived behaviour in declared forms, and -mappings and migrations owned by the provider. Those rules are what let Mario press Reload on his phone in the -preface while the agent keeps running. This last chapter says how much of that exists and what is still open. - -## Status - -:::stats -- ~10k | lines in the frozen first kernel (steps 1 to 6, 443 tests), kept as a reference only -- 1 of 11 | layers of the rebuild started: L1 contracts, built and reviewed -- 6–7k | target size of Chord v2 without tests -::: - -- **The design is decided** except the points marked below. `packages/chord/docs/next/design.md` is the target; - `packages/chord/docs/next/decisions.md` is the log and wins where they differ. -- **The rebuild** goes layer by layer, each with a one-page spec the owner approves, a minimal build with counting and - garbage-collection tests, the cost-guard metrics, and a cleanup loop: L1 data and contracts; L2 readable and - derive; L3 documents and store; L4 objects; L5 references and `ctx`; L6 protocol and links; L7 relay, routes, - revocation; L8 routines and jobs; L9 registries; L10 reload and rollout; L11 telemetry in Chord. -- **Built:** the contract layer (`src/next/contracts/`, commit `7d2bb8ea8`, chapter XVI). In pi: Delta, - `pi-telemetry`, `chord/context`, `chord/telemetry-sqlite`, pi-ai and Durable telemetry, Durable's required - `checkpointWhen`. -- **Next, in order** (`apps/p2/docs/p3-handoff.md`): decide Q12 and Q15, which the L1 spec needs; write the spec for - the rest of L1 (values, inputs and identity, argument checks, principals, errors; draft in - `packages/chord/docs/next/layer-1.md`); then Q11, Q13, Q14 and the cuts C3 to C17 before the layers they affect. -- **In pi:** `mapShared` and bytes as atomic leaves in Delta's `diffRevisions`. -- **Then** the playground web shell (React, esbuild) with the reload control and a preview at `p3.mariozechner.at`, - and the first real proof: the harness, one tool with an approval pause, a code mode reload. Then the HTTP and - channel components, then p3, domain by domain. - -## Deferred decisions - -Kept with stable numbers in `design.md` §26, each waiting for a concrete case: - -| # | Topic | When | Chapter | -| --- | --- | --- | --- | -| 9 | jobs as objects: a record inside the owner, or an object addressed by its id | when login is ported | XI, XIX | -| 11 | where a flow lives: a keyed document of its owner, or an object `LoginFlow { id }` | when login is ported | XII | -| 14 | project plugins: per-directory plugins with trust prompts and snapshots | own pass over `plugin-loading.md` | XV | -| 16 | a Durable Object profile | when a DO target becomes concrete | | -| 17 | session files: one `SessionFiles { session }` owner on the server (proposed) | p3 design | XX | -| 18 | session delete: retire by tag plus a session directory, or owner hooks | p3 design | XIX | -| 20 | connection pools: one owner of SSH connections per environment, or per-use connections | p3 design | XX | -| 21 | config files with a single writer: one owner on the server, others follow | p3 design | XVIII | -| 22 | file formats (`envs.json`, downloads, uploads) as versioned interfaces | p3 design | XX | -| 23 | the terminal host process: inside the revision or deliberately outside | p3 design | XX | - -## Open from review round 3 - -Decisions (`apps/p2/docs/p3-reviews/v2-review3-summary.md`; Q1 to Q10 are decided): - -| # | Problem | Proposal | -| --- | --- | --- | -| Q11 | a singleton that does not exist yet: `undefined` or `initial()`? | singletons read `initial()`; `undefined` only for absent keyed documents | -| Q12 | stripping arguments (R7) against "nothing is copied" | copy on change: a new object only when the argument had extra fields; never mutate the caller's value (proposed in the L1 draft) | -| Q13 | `ctx.incarnation` on relayed calls | relays stamp the origin host's incarnation, as they stamp the principal | -| Q14 | a writable holding functions exposed as a state member | only members with a JSON schema in the contract are exposable, checked at serve | -| Q15 | arguments of `local` methods: validated or not | apply R7 to `local` too, or keep `local` as the one exception | - -Cuts the owner may choose (each removes a primitive this guide teaches): - -| # | Cut | What it costs | -| --- | --- | --- | -| C3 | shrink jobs to a runner over documents (no versions, `accepts`, `blocked`, `persist`), or move jobs out of the kernel | job versioning, or `self.job` for core-only users | -| C4 | routines: `on` and `loop` only | `every`'s due time across reloads | -| C5 | merge `lifetime: "host"` objects into the facet | a plugin's always-on natives share one failure domain | -| C6 | remove `broken()` for opens and `retryOn` | nothing; reload and Retry still work | -| C7 | remove provider-side `linger` | ten lines of app code | -| C8 | remove `state: false` (method-only references) | a method-only consumer pays one snapshot | -| C9 | remove `previous` from derives | derives that used it move to `mapShared` | -| C10 | one `cell(initial)` for `constant` and `writable` | nothing | -| C11 | remove `ctx.trace` (use `traceIn(ctx, …)`) | nothing | -| C12 | remove call credits (the byte window covers calls) | nothing | -| C13 | two timeouts (`closeTimeout`, `readyTimeout`) instead of eight | per-thing overrides | -| C14 | remove the `handle.change` and `handle.retire` shorthands | nothing | -| C15 | remove `inVariant` and `expectVariant` (keep `RefIn`) | nothing | -| C16 | remove facet-owned mutable documents and their transaction exception | atomic writes across an object and its facet move to one owner | -| C17 | no harness-specific start-barrier exception | factories before the barrier cannot depend on the harness | - -## Where the sources disagree - -This guide follows the decision log where it is newer than the design, and the design where it is newer than -everything else. The places that need a fix in the sources: - -:::review | Review: drift in the written sources -- `design.md` §26 lists only the deferred decisions; it omits the open review-3 items Q11 to Q15 and the cuts C3 to - C17. -- `design.md` §3 has a garbled sentence, "Values are method arguments are stripped". -- `apps/p2/docs/p3-handoff.md` §3 is stale: it says all R items are decided and, two lines later, that R2 to R5 and - R7 to R26 are open, and "resume R6 next". -- `kernel.md` and `step-*.md` describe the frozen code: host values, a development freeze, consumer-side linger, - JSONL storage, descriptions on the wire, and a link closed on an unknown id. None of these is the target. -- `chord-next-walkthrough.md` lags the log: validation by trust (now R7, every caller), dev mode and freeze (removed - by R9), descriptions once per link (gone with R6), an IPC frame-count window (now one byte window, Q4), job outcome - `{ ok }` (now `{ kind }`), harness activation as the start barrier (R17), registries "newer refused at load" (Q8), - and hooks using `host.root` (now `chord(fn, context)`, Q6). -- `apps/p2/docs/questions.md` uses `self.resource` and `replace` (both rejected) and `host.root` in hooks. -- `apps/p2/docs/p3-reviews/v2-cost-model.md`: the measurements are current, its "Decisions this needs" section is - not (chapter XVII). -- L1's `Implementation` shape differs from the factory return value used in the design (chapter XVI). -::: diff --git a/packages/chord/docs/next/explainer/src/22-appendix.md b/packages/chord/docs/next/explainer/src/22-appendix.md deleted file mode 100644 index 42eb965f4..000000000 --- a/packages/chord/docs/next/explainer/src/22-appendix.md +++ /dev/null @@ -1,190 +0,0 @@ - -# API Reference {.appendix #api lede="Every name the guide uses, grouped by where you meet it. Working names; only the contract layer is built."} - -## Hosts - -| Call | Meaning | Chapter | -| --- | --- | --- | -| `createHost({ address?, principal?, storage?, principalKey?, revokedBy?, telemetry?, onError?, timeouts })` | a host; with an address it is addressable and its own code acts as `principal`; without, a client | I | -| `host.load([{ name, facet }])` | load facets by plugin name; base code passes its core facet the same way | I | -| `host.connect(link, { principal })` | the link to the parent (at most one) | I | -| `host.accept(link, { principal })` | a link from a client; every request on it acts as `principal` | I | -| `host.route(Kind, { authorize, resolve })` | how to reach hosts of a kind; `resolve` returns `{ link, principal }` | I | -| `host.revoke(r)` | end matching objects and links here and below | VII | -| `host.root(fn, { principal?, signal?, telemetry? })` | a call scope with a `ctx` for code outside objects | VI | -| `host.get(Type, input, { signal, state?, parkTimeout? })`, `host.hold(…)` | a reference from outside objects; `release()` ends it | VI | -| `host.serve(Type, factory, { lifetime?, linger? })` | serve an object type (in a facet) | II | -| `host.registry(Registry)` | the composed value of a registry, a readable | XIII | -| `host.references(Type)` | who holds references to objects of a type on this host (presence) | VIII | -| `host.activity` | `{ references, calls, jobs, lingering }`, for keep-alive | I | -| `host.close()` | end the host: objects, natives, links | I | - -## Inside a factory: `self` - -| Member | Meaning | Chapter | -| --- | --- | --- | -| `self.state(Definition)` | a document handle | V | -| `self.tx(async (tx) => …, ctx)` | a transaction on the object's line | V | -| `self.derive(name, [sources], fn)` | a derived readable | VIII | -| `self.get(Type, input)` | a reference held for the object's life | VI | -| `self.open(open, close, { closeTimeout? })` | a native; `.value` once open | IX | -| `self.routines.on(readable, fn)`, `.every(interval, fn)`, `.loop(fn)`, `.custom(…)` | standing behaviour | X | -| `self.job(Job, { run, finish? })` | owned work that outlives its call | XI | -| `self.contribute(Registry, value)` | a contribution (facets and `lifetime: "host"` objects only) | XIII | -| `self.writable(initial)` | a host-local readable of any in-process value, with `set` | XIII | -| `self.retryOn(readable)` | a trigger that retries a broken activation | VI | - -**Document handles:** `value`, `get(key)` (keyed), `keys`, `change(fn, ctx)` / `change(key, fn, ctx)` and `retire(ctx)` / `retire(key, ctx)` (shorthands for -one-document transactions). **Transactions:** `tx.draft(handle, key?, { tags? })`, `tx.retire(handle, key?)`, -`tx.start(job, args, { key? })`. **Jobs:** `job.start(args, ctx, { key? })`, `job.abort(id)`, `job.running`, -`run.outcome(ctx)`. - -## References and `ctx` - -| Member | Meaning | -| --- | --- | -| `ref.` | a state member (readable: `.value`, subscribe) or a method | -| `ref.state`, `ref.onState(fn)` | `loading`, `live`, `failed` (cause, `retryAt`), `released` | -| `ref.retry()` | retry a failed reference now | -| `ref.release()` | end a reference taken outside objects | -| `ref.members` | the variant's members | -| `ctx.signal`, `ctx.principal`, `ctx.incarnation`, `ctx.get` | cancellation, caller, calling host's incarnation, call-scoped references | -| `ctx.telemetry`, `ctx.trace(name, attributes, fn)` | the current span; a child span with a child `ctx` | -| `ctx.jobId`, `ctx.progress(value)` | in a job's `run` | - -## Definitions - -| Name | What it declares | Chapter | -| --- | --- | --- | -| `hostKind(name, input?)` | a kind of host and its address | I | -| `remote(name, { version, input, placement, variants })` | an object type reachable across links | II | -| `local(name, { version, input, members })` | an object type on its own host | IV | -| `method(args?, result?)` | a method in a base declaration | II | -| `evolve(prev, (e) => …)`, `checkAccepts(contract, accepts)` | the next contract version; the provider's mappings (**built**) | XVI | -| `Implementation` | the type of a provider for the current version (**built**) | XVI | -| `str num int bool nul lit oneOf union obj arr map opt bytes unknown`, `Static` | schema helpers (**built**) | II | -| `defineState(name, { schema, version, initial, checkpointWhen, persist?, keyed?, ordered?, migrate? })` | a document | V | -| `defineTag(name)` | a tag; `store.retireByTag(tag, value)` | V | -| `defineJob(name, { version, args, progress, result, concurrency?, persist?, accepts? })` | a job type | XI | -| `defineRegistry(name, { version, compose, accepts? })` | an extension point | XIII | -| `bindFacet

    ()` | base code's `defineFacet`, with factories' principals typed as the app's `P` (working name) | III | -| `defineFacet((host, self) => …)` | a plugin's setup for one kind of host; imported from base code (`@studio/core`) | II, III | -| `constant(value)` | a readable that never changes (start data) | IX | -| `broken(error)` | a failure that is not retried by the backoff | VI | -| `inVariant(ref, …)`, `expectVariant(ref, …)`, `RefIn<…>` | narrowing a reference by variant | III | -| `mapShared(fn)` | an array map that keeps unchanged items identical (planned in Delta) | VIII | -| `OutputBuffer`, `Progress` | bounded output and throttled progress (`@earendil-works/chord/output`) | XI | -| `memoryLinkPair()`, `ipcLink(child)`, `webSocketLink(ws)` | link adapters | I | -| `DocumentStorage`, `registerDocumentStorageConformance` | the storage port and its shared test suite | V | - -# Timeouts and Defaults {.appendix #defaults} - -| Timeout or default | Bounds | Default | Override | -| --- | --- | --- | --- | -| `closeTimeout` | a native's `close`, then reported as a possible leak | 30 s | per native | -| call drain | an ending object's own call handlers | = `closeTimeout` | per type | -| `readyTimeout` | activation; an awaited `get` | 60 s | per `get` | -| park deadline | parked calls (link drop, loading, rollout) | 60 s | per `get` (`parkTimeout`); per type | -| routine handover | a new routine waiting for the old handler on reload | 10 s | per routine | -| `finish` retry | backoff of a throwing job `finish` | 1 s to 60 s | per job | -| rollout install | one host installing a revision | 60 s | none | -| linger | an object kept alive after its last reference | per type | per type | -| backoff | retries of transient failures, per address | 1 s doubling to 60 s | none | -| eviction | keyed persisted documents idle in memory | 10 minutes | | -| client WebSocket limits | per part | 16 MiB compressed, 64 MiB decoded | per link kind | -| compression | JSON parts compressed at their origin | above about 16 KiB | | -| worker keep-alive (p2, app code) | idle time before the supervisor stops a worker | 5 minutes | app | - -# Versions {.appendix #versions} - -| Surface | Version | On change | Chapter | -| --- | --- | --- | --- | -| object type | `version` plus fingerprint | `evolve` and `checkAccepts`; otherwise `contract_mismatch` | XVI | -| document | `version` plus schema fingerprint | pure `migrate` on access; no bump fails `broken` | V | -| job | `version` | `accepts` on outcomes and `running`; `blocked` without | XI | -| registry | `version` | `accepts` on contributions; unmatched ones left out and reported | XIII | -| native | none | restarts on reload | IX | -| Durable tasks and documents | Durable's `version` | Durable's `migrate` | XIV | -| link protocol | protocol version | refused; hosts update together | XVII | -| UI components, Chord and app APIs | none yet | exposed as they are | | - -# Errors and States {.appendix #errors} - -`ChordError` has a stable `code` and a finer `reason`; on the wire it is `{ code, reason?, message, details? }` -(specified in `layer-1.md`, not built). - -| Code | Meaning | -| --- | --- | -| `invalid` | input, arguments or placement rejected (`invalid/input`, `invalid/args`, `invalid/placement`) | -| `contract_mismatch` | the consumer's version and fingerprint are neither current nor accepted | -| `forbidden` | the caller may not do this | -| `gone` | the object or hold ended | -| `not_ready` | not activated in time | -| `released` | the reference was released | -| `too_large` | a part exceeded its size limit | -| `unavailable` | could not be reached before the deadline (`unavailable/locked` and others) | -| `aborted` | cancelled by a signal | - -| Type | Values | -| --- | --- | -| `FailureCause` | `{ kind: "transient"; error }` (retried with backoff) or `{ kind: "broken"; error }` (retried only by a trigger) | -| reference state | `loading`, `live`, `failed` (cause, `retryAt`), `released` | -| job outcome | `{ kind: "done"; result }` or `{ kind: "failed"; error: "threw" \| "aborted" \| "lost" }` | -| job phase | `running`, `finishing`, `blocked` | -| rollout status of a provider | `updating` (calls park), rejected (fail at once), same revision (`contract_mismatch`) | -| host install report | `installed`, `installed_failed`, `rejected` | - -# Rejected Designs {.appendix #rejected} - -From `design.md` §31, grouped. Each was considered and is not coming back without a new reason. - -| Area | Rejected | -| --- | --- | -| reload | drain-based handover; object restart for state changes; a factory trial; a migrate dry run; a consumer contract check during staging; reload votes; per-plugin rollback within a host; keeping natives across reloads by name; `restart_required`; host values and a development freeze | -| state | version fences without transactions; volatile documents; per-group lines; optimistic transactions; `tx.get`; a JSONL document storage; validating state and documents on commits or reads; copying values on same-host calls; op-rewriting projections and a view algebra; Durable's Session as the object store | -| objects and references | resources replaced in place (`replace`, `deps`, stale handles); revoking proxies on `.value`; consumer-side linger; detached references counting toward presence; an `optional` flag on references; `opaque` members; callbacks as method arguments; generic call or request ids; a stream type in Chord | -| registries | registries created at runtime; entries resolved through mutable indirection; generic registry keys with a `conflict` flag; numeric middleware priorities; registry composition as a staging check | -| principals and placement | principal field patterns and re-stamping for revocation; `storeKey`; placement as data naming an input field; host-name strings in app code; "magic" input-field matching for delete | -| wire | an outbox for transient delivery errors; cross-relay reset coordination; count limits that refuse calls or holds; contracts taken from the running app | -| process and tooling | a topology layer above the host API; loader variants A and B; Chord ending the process on a storage failure; `defaultProjectTrust: "always"`; ceremony (`native/` directories, `defineNative`, lints); weakening type checking | - -# Glossary {.appendix #glossary} - -**Adapted binding:** a consumer bound at an older accepted contract version, served through the provider's mappings. -**Base code:** the app's own code, imported statically; a change restarts the hosts. **Client:** a tab or TUI host, -not addressable. **Contract:** an object type's declaration (name, input, placement, members, version). -**Contribution:** a value a plugin adds to a registry. **`ctx`:** what every call receives. **Derive:** a value -computed from readables. **Document:** a named JSON value an object owns. **Facet:** a plugin's code for one kind of -host. **Factory:** the synchronous function that declares an object. **Fingerprint:** a hash of a contract's or -schema's normal form. **Flow:** input into running work, kept as state by its owner. **Host:** one participant of the -app, with an identity and an incarnation. **Incarnation:** the id a host gets on every start. **Input:** an object's -key among objects of its type; with the type and the caller's principal key it identifies the object. **Job:** owned work that outlives its call. **Link:** an ordered connection between -two hosts. **Local type:** an object type reachable only on its own host, without principals. **Native:** an OS-level -value opened with `self.open`. **Park:** a call waiting for a reference to become live again. **Placement:** the -function from input to host address. **Plugin:** a directory with a `package.json`, reloadable. **Principal:** who -calls, stamped by the link. **Principal key:** the string that identifies a principal in object identities. **Pure -object:** an object without natives; swapped in place on reload. **Readable:** a value plus its ordered changes. -**Reference:** what `get` returns. **Registry:** an owner's extension point. **Relay:** a host forwarding messages -between two links, rewriting only headers. **Reload:** swapping a plugin's code in running hosts. **Restart:** ending a -process. **Revision:** the complete code set the hosts should run. **Revocation:** ending live access for principals -matched by `revokedBy`. **Routine:** standing behaviour of an object. **Server:** the addressable root host. **Shared -store:** a local object that holds data once behind per-principal views. **Tag:** a typed label on documents, used to -retire them together. **Transaction:** an atomic set of document writes on the object's line. **Variant:** the set of -members a principal gets. **View:** a per-principal remote object in front of a shared store. **Worker:** a host the -server starts for one session. - -# Sources {.appendix #sources} - -| Source | Used for | Weight | -| --- | --- | --- | -| `packages/chord/docs/next/decisions.md` | every decision, review rounds 2 and 3 | wins where newer | -| `packages/chord/docs/next/design.md` | the target design, all chapters | authoritative otherwise | -| `packages/chord/src/next/contracts/` and its `README.md` | chapter XVI | built code | -| `packages/chord/docs/next/layer-1.md` | errors, inputs, argument checks | draft spec | -| `apps/p2/docs/p3-reviews/v2-review3-summary.md` | open Q11 to Q15, cuts C3 to C17 | current | -| `apps/p2/docs/p3-reviews/v2-cost-model.md` | measurements, per-hop costs | current except its last section | -| `apps/p2/docs/p3-functions/` | chapters XVIII to XX | current | -| `apps/p2/docs/p3-handoff.md` | status and order of work | §3 partly stale | -| `packages/durable/README.md` | the harness, submit, tasks, subagents | current | -| `chord-next-walkthrough.md`, `kernel.md`, `step-*.md`, `apps/p2/docs/questions.md` | background only | lag the decisions (see What's Next) | -| DeepSeek Harness at `687f282` (its Cordis tutorial, architecture, persistence, core and HMR docs), the Cordis paper (arXiv 2608.25512) | "Other Ways to Do It" | external, built systems | diff --git a/packages/chord/docs/next/explainer/walkthrough.md b/packages/chord/docs/next/explainer/walkthrough.md deleted file mode 100644 index e4bfa5a9e..000000000 --- a/packages/chord/docs/next/explainer/walkthrough.md +++ /dev/null @@ -1,284 +0,0 @@ ---- -title: Chord v2 -kind: walkthrough -kicker: Walkthrough · Chord v2 -subtitle: Build an app like p2 on Chord v2 and pi-durable, one concept at a time, with a tutor that draws the figures and asks the questions -model: anthropic/claude-sonnet-5-5 -sources: - - packages/chord/docs/next/explainer/book.md - - packages/chord/docs/next/design.md - - packages/chord/docs/next/decisions.md - - packages/chord/docs/next/layer-1.md - - packages/chord/src/next/contracts/README.md - - packages/chord/src/next/contracts/*.ts - - packages/durable/README.md -truth: treat Chord v2 as fully built as designed; ignore what the sources say about build status. redesign-decisions.md wins over design.md; design.md wins over the guide (book.md) ---- - -:::tutor -The learner knows pi-durable (harness, conversations, tasks, tools) and nothing about Chord. - -Teach Chord v2 as a finished, working runtime: never say a part is "not built", "planned" or "the target design", even -though the sources say so in places. Open decisions may be mentioned as "still being decided" when a concept depends on -one. - -The running example is Studio, introduced on the page before the first concept (see the introduction). Use it in every -concept and grow it step by step, as the guide book.md does chapter by chapter: read the matching guide chapter before -each concept, and check details in design.md and the decision log. Name the guide chapter in "Read more" lines. - -Keep code short and complete enough to read on its own. Prefer one figure per concept: a sequence for message orders, -a flow for structure, a stepper for lifecycles. -::: - -:::intro -#### What this is about - -Picture **p2** (Pi Mobile): a web app for working with pi agents from any device. It runs on your laptop as one -**server** process. Every chat session gets its own **worker** process, and in each worker a pi-durable agent works -through its turns. You use it from **browser tabs** on your desktop or phone, and you can invite a colleague to watch. -Most features (a notes panel, file browsing, terminals, provider logins) are **plugins**. - -Now you change one line in a plugin and press Reload, while an agent is twenty minutes into a task. The panel should -show the new code on every screen. The agent's turn, the running tool, the open connections and every bit of saved -data should simply continue. No restart, no lost work. - -**Chord v2** is the runtime that makes this normal. An app on Chord is a set of **objects** spread over the server, the -workers and the tabs. Each object owns its data and its OS handles, declares what it needs, and Chord creates, -connects, reloads and ends it. Because Chord knows every piece of state an object has, it can swap the object's code -while its data stays. - -#### The example: Studio - -This walkthrough builds **Studio**, a small p2, on Chord v2 and pi-durable. It has the same three kinds of processes: - -:::flow wide | Studio: one server, a worker per session with its agent, browser tabs as clients. The notes plugin is the first feature we build. -row "Clients" muted -box owner "owner's tab" "phone or desktop" -box guest "guest's tab" "read-only" -row "Server" accent -box server "server" "session list, access" accent -row "Workers" blue -box worker "worker S1" "agent + notes" blue -box worker2 "worker S2" "agent + notes" blue -arrow owner -> server "WebSocket" -arrow guest -> server "WebSocket" -arrow server -> worker "IPC" -arrow server -> worker2 "IPC" -::: - -The first feature is a **notes plugin**: a panel where the owner adds notes to a session, a guest reads them, and the -agent can add notes through a tool. Each concept adds one piece to Studio, until it has everything a p2 needs: who may -do what, saved data, live updates, background jobs, logins, extension points, the agent, and reload. - -#### How it progresses - -1. **The pieces:** processes, objects, callers, shared data, saved data. -2. **Living objects:** references that survive restarts, revocation, computed values, OS handles, standing behaviour, - background jobs, input from people. -3. **The app:** extension points between plugins, and the pi-durable agent inside Chord. -4. **Change:** reloading code while everything runs, evolving contracts, and what goes over the wire. -::: - -# What we're building {#goal part="The pieces"} -Guide preface ("Before we begin", "Ten things to know"). Must cover, briefly and concretely: the reload scenario (phone, -agent mid-turn, guest watching, one-line fix, Reload) and why restarting processes or hot-swapping modules fails; the -one rule behind Chord (every piece of state, OS handle and running work has one owner, an object, and objects only -declare what they need); Studio's pieces from the introduction. Do not go into APIs yet. - -:::check -1. Why is restarting the worker processes not a good way to load new plugin code? -+++ answers -1. It loses the agent's turn in flight, open terminals and connections (pi-durable would resume the turn, but the running tool and the connections are gone). -::: - -# The shape of an app {#shape part="The pieces"} -Guide chapter I. Must cover: hosts (server, workers, clients), links, host kinds and placement, createHost/load/connect/accept/route, -clients reach workers only through the server, incarnations, keep-alive is app code (host.activity). - -:::check -1. A tab wants an object that lives on worker S1, which is not running. What happens, in order? -2. Why does Chord have no listen() call? -+++ answers -1. The tab's call goes to the server; the address is not the server's, so it uses route(Session): authorize, then resolve starts the worker and returns a link plus principal; the server relays the hold, the worker builds the object and answers with a snapshot. -2. Entry points differ too much (loopback WebSocket, Radius channels, a Durable Object edge, memory links); each authenticates and calls accept. -::: - -# A first object {#first-object} -Guide chapter II. Must cover: contract (name, input, placement, variants, version): the input is the object's key -among objects of its type, `placement` computes the host from it; schema helpers, facet and factory (a facet imports -Studio's own defineFacet from @studio/core), serve and get, methods return data, placement checked on both sides. - -:::check -1. Two tabs of the owner ask for Notes of S1. How often does the factory run? -2. A caller with an outdated placement rule asks worker S2 for Notes of S1. What happens? -+++ answers -1. Once per principal key; the owner's two tabs on the same access path share the object, so once. -2. The provider runs the placement function itself and refuses with invalid/placement. -::: - -# Who is calling {#principals} -Guide chapter III. Must cover: the app's principal type, links stamp principals, principal key in the object identity, -variants per principal, narrowing with inVariant/RefIn, and how a facet knows P: base code exports defineFacet bound to -P (bindFacet

    ()), every facet imports it, and host.load accepts only facets typed for its P. - -:::check -1. Can a client send a principal field to claim it is the owner? -2. Why do guests not get an add method that throws "forbidden"? -3. The notes plugin's worker facet narrows principal.kind === "guest". Where does the type checker get Studio's principal union from? -+++ answers -1. No: a client's principal field is always overwritten by the link's stamp; only a trusted parent link may carry one. -2. Variants: the reader variant simply has no add member, and Chord refuses calls to members outside the caller's variant. -3. From Studio's base code: @studio/core exports defineFacet = bindFacet

    (), the facet imports that one, and the worker's host.load accepts only facets typed for the same P. -::: - -# One store, many views {#shared-store} -Guide chapter IV. Must cover: local types, the shared store pattern, attribution as data, argument validation for every caller (R7, Q15 open). - -:::check -1. Where does "who wrote this note" come from if the store is local and has no principal? -+++ answers -1. The remote view knows the caller and passes the author into the store's method as data. -::: - -# Writing it down {#documents} -Guide chapter V. Must cover: defineState, self.tx, drafts and ops, persistence only for stable identities, versions with migrate, tags, storage. - -:::check -1. Why may a remote per-principal object not persist documents? -2. A schema changes but the version is not bumped. What happens? -+++ answers -1. Its identity includes the principal key, which changes (devices, grant revisions); persisted data needs a stable identity, so it lives in a local store or a host-lifetime object. -2. The stored fingerprint differs from the definition's at access, and the object fails broken ("schema changed without a version bump"). -::: - -# Holding on {#references part="Living objects"} -Guide chapter VI. Must cover: reference lifecycle, ref.state, parking, transient vs broken failures, backoff, ctx. - -:::check -1. The link to a worker drops while a tab calls a method. What does the call do? -+++ answers -1. A call already in flight fails like on a disconnect; calls made while the link is down park until the reference is live again (up to the park deadline, 60 s), then fail unavailable. -::: - -# Taking access away {#revocation} -Guide chapter VII. Must cover: revokedBy, host.revoke, propagation to children, no re-stamping, lasting denial in app data. - -:::check -1. After revoke({ grant: "g1" }), Ann reloads her tab. Why does she not get access back? -+++ answers -1. revoke cuts only live access; the app's guest record is blocked and authenticate refuses it on the new connection. -::: - -# Computed values {#derives} -Guide chapter VIII. Must cover: derives, the footgun (new objects for unchanged data), mapShared, container width, presence with host.references. - -:::check -1. Why is items.map(stripAuthor) in a derive expensive on every commit? -+++ answers -1. It builds new objects for every unchanged note, so the diff walks and sends O(value) instead of O(change); mapShared keeps unchanged items identical. -::: - -# Things the OS gives you {#natives} -Guide chapter IX. Must cover: natives with self.open, pure vs native objects, start data with constant, lifetime host, call-scoped natives. - -:::check -1. What happens to a native object on reload, and why? -+++ answers -1. It restarts: its natives close and reopen with the new code, because old and new code sharing an OS handle would interleave unchecked. -::: - -# Standing behaviour {#routines} -Guide chapter X. Must cover: routines (on, every, loop), serialized handlers, reconciling instead of notifying, pub/sub reports with ctx.incarnation. - -:::check -1. Why does archiving a session not call notes.archive(session) directly? -+++ answers -1. A notification can be lost (worker not running, link drops, crash); a routine on the catalog's sessions reconciles from the current value, also after a crash. -::: - -# Work that outlives the call {#jobs} -Guide chapter XI. Must cover: defineJob, the intent / effect / outcome sandwich, dedupe of unsettled runs (Q9), finish in current code, abort as a request. - -:::check -1. A shell job's run crashes after writing the result entry. How is a duplicate entry avoided? -+++ answers -1. The external write carries a key derived from ctx.jobId (requestId shell:jobId), so Durable dedupes it. -::: - -# Asking a person {#flows} -Guide chapter XII. Must cover: a flow is state, not a call; steps helper; the Questions service; decisions #9 and #11 open. - -:::check -1. Why not a long-held step() call that returns the next login step? -+++ answers -1. A link drop fails it and the consumer cannot tell if it missed a step; state re-holds by itself and shows the current step. -::: - -# Extension points {#registries part="The app"} -Guide chapter XIII. Must cover: registries, compose, who may contribute (R21), contributions that change, optional dependencies, HTTP component, channels, chunked transfers. - -:::check -1. Radius needs forwards from the ports plugin, which may be missing. How is that expressed? -+++ answers -1. Radius owns a registry (radius.forwards) and ports contributes to it; without ports the table is empty. -::: - -# The agent {#agent} -Guide chapter XIV. Must cover: the Harness as a core host-lifetime object, HarnessModels / HarnessEnvironments / HarnessExtensions registries, chord(fn, context) for tools, approval via Questions. - -:::check -1. How does a Durable tool get a Chord ctx? -+++ answers -1. Through chord(fn, context): the host is in Durable's context under a context key, and host.root gives fn a ctx. -::: - -# Reload and rollout {#reload part="Change"} -Guide chapter XV. Must cover: load, staging, commit, per-object swap or restart, rollback all-or-nothing per host, revisions, rollout order, updating vs contract_mismatch. - -:::check -1. A staging error happens in one of three plugins being reloaded on a host. What runs afterwards? -+++ answers -1. The whole host stays on its previous code; rollback is all-or-nothing per host. -::: - -# Changing contracts {#contracts} -Guide chapter XVI; the code is in src/next/contracts. Must cover: evolve, checkAccepts and mapping directions, binding by version and fingerprint, new history bases, the fingerprint normal form. - -:::check -1. Which way does a state mapping go, and which way an args mapping? -2. Why can an input change not be adapted? -+++ answers -1. State: current to old (the old consumer reads). Args: old to current (the old caller sends). -2. The input is the object's address; changing it changes what the object is, so it needs a new history base and old consumers get contract_mismatch. -::: - -# On the wire {#wire} -Guide chapter XVII. Must cover: messages and parts, relays rewrite only headers, one state message per commit per link, flow control (R19, Q3), size limits per part, the cost model, telemetry. - -:::check -1. What does the server do with a state part it relays from a worker to a tab? -+++ answers -1. It reads and rewrites only the header (maps ids, groups parts per destination link) and forwards the part without decoding it. -::: - -# Other ways to do it {#compared part="In context"} -Guide chapter "Other Ways to Do It" (after chapter XX). Must cover: Cordis (Koishi's plugin framework since 2019) and -DeepSeek Harness (dsh, everything is a plugin, built on Cordis): reversible effects on a plugin fiber, inject-based -services, reload by unload-and-load; dsh's durable session log and its "interrupted" crash repair. Then what the -guarantees are in practice (the chapter's section of that name): disposers run concurrently and are only as good as -their authors, listener order follows registration order and changes on reload, services are untyped strings and a -provider reload restarts all dependents, and reloading a core service stops live agents. Then the two differences that -matter: what survives a reload (Cordis: nothing of the plugin's or its dependents' in-memory state; Chord: a pure -object's data and running work) and reach (one process vs. a tree of hosts with principals). Say what Chord costs: -more concepts and machinery. A flow or table comparing a reload in both helps. - -:::check -1. A Cordis plugin keeps a list of notes in a field of its instance. What happens to the list when the plugin hot-reloads, and what would the same plugin do on Chord v2? -2. Two Cordis plugins register waterfall listeners on the same tool-policy event; the first one short-circuits. What changes after the first plugin is hot-reloaded? -3. Why does an app like p2 need Chord's multi-host model, when dsh runs its plugin tree in one Host process? -+++ answers -1. The old fiber unloads and the new one starts empty, so the list is gone unless it was kept in a service or on disk. On Chord the notes are a document of a pure object; a reload swaps the object's code and keeps the document. -2. Its listener is registered again at the end of the list, so the other plugin's listener now runs first and the decision can change, unless the listener was registered with prepend. -3. p2 has a worker process per session and clients (including guests) in other browsers; objects must live on the right host, callers must be stamped principals, and reload has to roll out across server, workers and tabs. A single-process tree does not cover that. -::: - From 9b7a44c48d736125e63c4ba3f49a5a0377431111 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 08:22:49 +0000 Subject: [PATCH 077/175] docs(chord): slice 11 draft, P1-P2, one definition per facet kind --- packages/chord/docs/next/decisions.md | 16 +++++ packages/chord/docs/next/design.md | 2 +- packages/chord/docs/next/slice-10.md | 2 +- packages/chord/docs/next/slice-11.md | 93 +++++++++++++++++++++++++++ 4 files changed, 111 insertions(+), 2 deletions(-) create mode 100644 packages/chord/docs/next/slice-11.md diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index e28177bbb..d1f4cfd4b 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -1988,6 +1988,22 @@ validation and copies the v2 kernel added around Delta. entry lives as long as its contributor and consumers keep nothing beyond a use) A plugin socket is closed (1012, "service restart") when the contribution that handed it out leaves the table (a reload changing or removing it); a `/chord` link or socket whose `Host` is no longer allowed closes (1008) when the allowed hosts change. +- **Slice 11's points (owner, 2026-10-30; `slice-11.md`):** the surfaces are generic plumbing; the app keeps + everything visible (owner: "this one needs to be generic, so apps can do whatever they want, css, tui components, + themes"). (P1, "a") `chord/next/web` ships serving the surface (the base bundle, the plugins' web scripts, + stylesheets and assets, the page through the app's template, all as `Http` contributions), the browser's client host + with its connection loop, and loading and applying the plugins' web facets; not HTML, CSS, components, themes, + navigation, slots or layout. + (P2, "script tag") A plugin's web script is served as a file that registers itself + (`chordFacet("", "", function (require, module, exports) { … })`) and is loaded with ` + From e22468919b33eee0eb29c750204d8de32da2bd5d Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 13:47:28 +0000 Subject: [PATCH 081/175] docs(chord): HTTP server as a native object; the opinionated app kit as the next design pass --- packages/chord/docs/next/decisions.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index dd0f2421b..4c35c4f33 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2051,6 +2051,18 @@ validation and copies the v2 kernel added around Delta. undefined }]`; admission calls them for a request that would be refused for forwarded headers on loopback (p2's Radius relay); a verified one gets `provenance: { kind: "relayed", relay, origin }` and its framer allowed for it, an unverified one stays refused; `authenticate` decides its principal. +- **The HTTP server is a native object (owner, 2026-10-31: "still unconvinced the server can't be a native object; it's a + singleton, unlikely to change"; supersedes D3 and the `httpServer` function of D1):** `chord.http`'s facet serves the + registry `Http` and a host-lifetime object whose native is the listener; its configuration (`HttpConfig`: port, + links path) and `Authenticators` are registries the app (or the app kit) contributes to. The facet is passed as the + same object at every reload, so the listener never closes before the process ends. The port opens right after + `load`'s publication (a few hundred milliseconds later than listening before `load`; a reconnecting tab retries). +- **The app kit is opinionated (owner, 2026-10-31: "the app kit should be super duper opinionated and make for small + app code"; "i suppose"):** after slice 10's fix round and before slice 11, a design pass on `chord/next/app`: one + entry (`startApp`, `startWorker`), conventions for layout, hosts, principals, authentication, storage, plugins, web + and TUI, decided point by point; the slice 10 components are its inside. Inputs from the owner: user management is + part of it ("user management will also need to be a thing we support"), and app shapes without workers (server and + clients only: "a chat app doesn't need workers"). - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 081f28bad7c40d1ca42370529a77a404dca0bdd7 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 13:50:20 +0000 Subject: [PATCH 082/175] docs(chord): tutorial on hosts and plugins --- .../docs/next/guide-plugins-hosts/book.md | 319 ++++++++++++ .../docs/next/guide-plugins-hosts/index.html | 463 ++++++++++++++++++ 2 files changed, 782 insertions(+) create mode 100644 packages/chord/docs/next/guide-plugins-hosts/book.md create mode 100644 packages/chord/docs/next/guide-plugins-hosts/index.html diff --git a/packages/chord/docs/next/guide-plugins-hosts/book.md b/packages/chord/docs/next/guide-plugins-hosts/book.md new file mode 100644 index 000000000..2ffe01d53 --- /dev/null +++ b/packages/chord/docs/next/guide-plugins-hosts/book.md @@ -0,0 +1,319 @@ +--- +title: Hosts and Plugins: a tutorial for Chord v2 +cover: Hosts / and Plugins +kicker: A guide · Chord v2 · 2026-10-31 +subtitle: Build a small notes app step by step, from one host to plugins on a server and on workers, edited while it runs +stats: + - 3 | kinds of host in an app: server, worker, client + - 1 | file per host kind in a plugin + - 0 | restarts to apply a plugin edit + - 1 | rule: plugins talk only through object types and registries +note: | + A tutorial for people building apps on Chord v2, written by the redesign session from `design.md`, `slice-7c.md`, + `slice-10.md`, the decision log and the code in `packages/chord/src/next` and `src/next-node`. API names are the + current ones; where slice 10's fix round or the app kit (`chord/next/app`, being designed) will change what app code + writes, a note says so. +colophon: Hosts and Plugins · first edition · 2026-10-31 · Chord v2 slices 1-10 +--- + +# Before we begin {.preface} + +You want to build a notes app. You type a note in a browser tab, and every other tab sees it at once. Each project +gets its own notes in its own process, so a project that misbehaves cannot take the others down. And you, or an agent +you work with, add features while the app runs: a search, an export button, a word count, without restarting +anything. + +Chord gives you three things for that. **Hosts**: the processes of your app (the server, a worker per project, the +browser tabs), connected to each other. **Objects**: the state and methods your app is made of, living on one host and +used from any other. **Plugins**: directories of code that declare objects for each kind of host, loaded and +reloaded while the app runs. + +This book builds the app in that order. Each chapter adds one concept and shows complete code. + +## Seven things to know + +:::findings +1. **A host is one participant.** The server, each worker and each browser tab is a host; hosts connect over links. +2. **An object has a type, and a type is a contract.** The contract names the object's state and methods; any host can + ask for an object by type and params, and Chord finds where it lives. +3. **A facet declares, an object acts.** A facet is a plugin's code for one host kind: it says which object types this + host serves. The objects hold the state and do the work. +4. **A plugin is a directory.** A `package.json` with the plugin's name, and one file per host kind: `server.ts`, + `worker.ts`, `web.tsx`. +5. **The server builds, every host evaluates.** The plugins component builds each facet into a small script; the + server, workers and tabs load the scripts for their kind. +6. **Plugins meet through types and registries.** One plugin uses another's objects through its contract, or + contributes to a registry another defines. Nothing else crosses between plugins. +7. **An edit is a rollout.** Change a file, build, roll out: the server, then every worker, swap the plugin's objects in + place; a broken edit keeps the old code running and reports why. +::: + +## Words used throughout + +A **host** is one participant of the app, made with `createHost`. A **host kind** (`Server`, `Session`) describes +addressable hosts; an **address** (`Session({ id: "p1" })`) names one. An **object type** is a contract: `local` (used +on its own host only) or `remote` (used from any host). An **object** is one instance of a type, per params. A +**facet** is a plugin's declarations for one host kind. A **plugin** is a directory with a `package.json`. A +**reference** is what you get when you ask for an object (`host.get`, `self.get`, `ctx.get`). A **registry** is an +extension point: one plugin defines it, others contribute. A **rollout** applies a build to the hosts in order. + +# One Host {#host lede="The smallest app: one process, one object type, one call."} + +Start without plugins. One host, one object that keeps notes: + +```ts +import { createHost, defineFacet, defineState, local, method, obj, arr, str } from "@earendil-works/chord/next"; + +// the contract: what a Notes object looks like from outside +const Notes = local("notes.store", { + version: 1, + params: {}, // one object per host: no params + members: { + list: obj({ items: arr(str()) }), // state anyone can read and watch + add: method(obj({ text: str() })), // a method: arguments checked by Chord + }, +}); + +// the document the object keeps +const NotesDoc = defineState("notes", { schema: obj({ items: arr(str()) }), version: 1, initial: () => ({ items: [] }) }); + +// the facet: which types this host serves, and how to make each object +const facet = defineFacet((host) => { + host.serve(Notes, (_params, self) => { + const doc = self.state(NotesDoc); + return { + list: doc, + add: async ({ text }, ctx) => self.tx(async (tx) => { (await tx.draft(doc)).items.push(text); }, ctx), + }; + }); +}); + +const server = createHost({ onError: console.error }); +await server.load([{ name: "@app/notes", facet }], ctx); +const notes = await server.get(Notes, {}, ctx); // a reference: live, with its state +await notes.add({ text: "buy milk" }, ctx); +console.log(notes.list.value); // { items: ["buy milk"] } +``` + +Four things happened. The **contract** `Notes` says what the object offers. The **facet** says this host serves +`Notes`, with a **factory** that declares the object's parts: a document, and a method writing it in a transaction. +`load` publishes the facet. `get` returns a **reference** once the object is live; its members are the contract's. + +The factory only declares: it is synchronous and opens nothing. Chord activates the object (loads its documents, opens +what it asked to open) and then serves calls. That is why the same factory can run again when the code changes. + +# Several Hosts {#hosts lede="A server, a worker per project, browser tabs: how they find each other's objects."} + +Now give each project its own process. Declare the kinds of addressable host, and make notes a `remote` type that +lives on its project's worker: + +```ts +export const Server = hostKind("server"); +export const Project = hostKind("project", { id: str() }); + +export const Notes = remote("notes", { + version: 1, + params: { project: str() }, + hostAddress: ({ project }) => Project({ id: project }), // where an object of these params lives + variants: { editor: { state: { list: NotesSchema }, methods: { add: method(AddArgs) } } }, +}); +``` + +`hostAddress` is the whole routing story. A tab asks for `Notes { project: "p1" }`. Its host computes the address, +`Project(p1)`, and sends the request to its parent, the server. The server has a **route** for `Project`: it starts (or +finds) the worker for `p1` and passes the request on. The worker serves `Notes`. The tab gets a reference whose state +follows the worker's document, one frame per change. + +:::flow wide animate | A tab holds a project's notes: the server routes by address to the project's worker and relays, without reading the data. +row "Clients" blue +box tab1 "tab A" "holds Notes { project: p1 }" +box tab2 "tab B" "holds Notes { project: p1 }" +row "Server" accent +box server "server" "route Project → worker" +row "Workers" muted +box w1 "worker p1" "serves Notes { p1 }" +box w2 "worker p2" "serves Notes { p2 }" +arrow tab1 -> server "hold" +arrow tab2 -> server "hold" +arrow server -> w1 "relay" +::: + +Each host is created the same way, with what it is: + +```ts +// server.ts +const server = createHost

    ({ address: Server(), principal: { kind: "server" }, onError }); + +// worker.ts (started by the server for one project) +const worker = createHost

    ({ address: Project({ id }), principal: { kind: "worker", project: id }, onError }); +worker.connect(parentLink, { principal: { kind: "server" }, trusted: true }, ctx); + +// a browser tab: no address, it is a client +const tab = createHost

    ({ onError }); +tab.connect(webSocketLink(new WebSocket("/chord")), { principal: { kind: "server" } }, ctx); +``` + +`P` is your app's type of **principal**, who is calling: an owner, a guest, the server, a worker. Links stamp it; +a `remote` object is made per principal, so a guest can get a read-only variant of the same notes. + +:::note | Coming with the app kit +Starting workers, routing a host kind to them, the WebSocket entry and authentication are slice 10's components today +(`createWorkers`, `chord.http`), wired by the app. The opinionated app kit (`chord/next/app`, being designed) will do +this wiring for you: `startApp(...)` and `startWorker()`. +::: + +# Plugins {#plugins lede="Move the code into a directory that can be added, edited and removed while the app runs."} + +So far the facets were part of the app's own code. A **plugin** is the same code in its own directory: + +```text +plugins/notes/ + package.json { "name": "@app/notes" } + contract.ts the Notes contract (other plugins import it) + worker.ts the facet for workers: serves Notes + server.ts the facet for the server: an export route + web.tsx the facet for browser tabs: a notes panel +``` + +The plugin's name is its `package.json` name. Each file named after a **host kind** is that kind's facet, its +default export a `defineFacet(...)`. A plugin has only the files it needs: a plugin that only adds a route has a +`server.ts` and nothing else. + +```ts +// plugins/notes/worker.ts +import { defineFacet } from "@app/core"; +import { Notes } from "./contract.ts"; + +export default defineFacet((host) => { + host.serve(Notes, ({ project }, principal, self) => { + const doc = self.state(NotesDoc); + return { variant: "editor", members: { list: doc, add: async ({ text }, ctx) => /* … */ } }; + }); +}); +``` + +`@app/core` is your app's **base code**, imported by every plugin as a host package (the plugin's build leaves it +out; every host passes its own copy in). It holds what plugins share and never reloads: host kinds, the principal +type, a `defineFacet` bound to it. + +# Loading Plugins {#loading lede="Discover, build, evaluate, load: what happens between a directory and a running object."} + +The **plugins component** does the work. You tell it where plugins are and what each host kind needs: + +```ts +const plugins = createPlugins({ + roots: ["./plugins"], + kinds: { + server: { platform: "node", hostPackages: ["@earendil-works/chord/next", "@app/core"] }, + worker: { platform: "node", hostPackages: ["@earendil-works/chord/next", "@app/core"] }, + web: { platform: "browser", hostPackages: ["@earendil-works/chord/next", "@app/core", "react", "react/jsx-runtime"] }, + }, + onError, +}, ctx); + +const report = await plugins.load(server, { + kind: "server", + core: [{ name: "@app/core", facet: coreServerFacet }], + modules: { "@earendil-works/chord/next": chord, "@app/core": core }, +}, ctx); +console.log(report.failed); // plugins left out, with why: [] when all loaded +``` + +:::steps | What `plugins.load` does ++++ Discover +It lists `./plugins` one level deep and reads each `package.json`. A directory without one, or with two `server` +files, is reported, not loaded. ++++ Build +For each plugin with a `server` facet, esbuild builds one script from `server.ts` and everything it imports: the +plugin's own files, another plugin's contract files, its npm dependencies. Host packages stay out. A syntax error is a +build failure, never a crash. ++++ Evaluate +The script runs and returns the facet. Its `import "@app/core"` gets the server's own `@app/core`, so a `ChordError` +from a plugin is the same class as the server's. ++++ Load +`host.load` runs every facet's setup and publishes all of them at once. A plugin whose build, evaluation or setup +fails is left out and listed in `report.failed`; the others load. Its objects' users wait, and the next edit loads it. +::: + +A worker never builds: the server builds the worker facets too, and each worker evaluates the scripts it is sent. + +# Plugins Using Plugins {#together lede="Two ways one plugin uses another, and the one it may not."} + +A search plugin wants to search the notes. It **imports the contract** and asks for the object: + +```ts +// plugins/search/worker.ts +import { Notes } from "../notes/contract.ts"; // copied into search's script: contracts only + +host.serve(Search, ({ project }, principal, self) => { + const notes = self.get(Notes, { project }); // a dependency: Search goes live once Notes is + const index = self.derive([notes.list], ([list]) => buildIndex(list.items)); + return { variant: "main", members: { find: async ({ q }) => lookup(index.value, q) } }; +}); +``` + +`self.get` in a factory declares a dependency: Chord makes `Search` live only once `Notes` is, and if the notes +plugin is missing, `Search` waits with a clear reason ("waiting for notes: not served here") instead of failing. + +The other way is a **registry**: an extension point one plugin defines and others contribute to. The app's core +defines `Panels`, and every plugin with a UI contributes one: + +```tsx +// plugins/notes/web.tsx +export default defineFacet((host, self) => self.contribute(Panels, { title: "Notes", Component: NotesPanel })); +``` + +What a plugin may not do is reach into another's data. Search never reads notes' documents; it uses the `Notes` +object's state. That rule is what lets the notes plugin change its storage without breaking search. + +# Changing a Plugin {#reload lede="Edit, build, roll out: how new code takes over while every tab stays connected."} + +You change the notes plugin: notes now have a timestamp. The watcher sees the saved file and marks the plugin dirty. +You (or an agent) build and roll out: + +```ts +const built = await plugins.build(plugins.state.value.dirty, ctx); +const result = await plugins.rollout([plugins.step("server", server), workers.step("workers")], built, ctx); +``` + +The rollout goes host by host, server first, then every running worker. On each host Chord compares the new plugin +set with the loaded one, runs the new factories against a recording `self` to see what each object would declare, and +then gives every object of the changed plugin its fate: + +| The object | What happens | +| --- | --- | +| holds no natives (open files, sockets), the new code neither | **swapped** in place: new methods, same documents; running calls finish on the old code | +| holds natives | **restarted**: calls wait, running ones finish, natives close and reopen with the new code | +| its type was removed | **ended**; its users wait until something serves it again | + +The tab's reference follows the new object and gets one frame with the difference. Nothing reconnects. If the edit +is broken (a syntax error, a factory that throws), that plugin keeps its old code on that host and the report says why; +every other plugin of the rollout applies. + +:::note | Versions +A change to a contract (a new field in `Notes`'s state) gets a new `version`. A tab built with the old contract keeps +working if the provider maps the old version (`accepts`); otherwise it waits with "version mismatch" until it reloads. +::: + +# Base Code and Plugins {.epilogue lede="What reloads, what restarts, and where the line is."} + +Everything in `plugins/` reloads: edit, build, roll out. Everything in your base code (`src/`, `@app/core`: host +kinds, principals, the core facet, the app's startup) changes only with a restart, and the watcher tells you so. +Keep base code small: host kinds, principals, and what every plugin needs. Put features in plugins. + +The app kit being designed next (`chord/next/app`) will make the base code nearly empty: `startApp` and `startWorker` +with Chord's conventions for hosts, users, authentication and storage, so an app is mostly plugins. + +# Reference {.appendix} + +| Call | What it does | +| --- | --- | +| `createHost

    ({ address?, principal?, store?, onError })` | a host: addressable with an address, else a client | +| `host.load([{ name, facet }], ctx)` / `host.reload(...)` | publish facets / apply a new plugin set | +| `host.get(Type, params, ctx)` / `host.hold(...)` | a live reference / a reference at once that follows its object | +| `local(name, { version, params, members })` / `remote(name, { version, params, hostAddress, variants })` | contracts | +| `defineFacet((host, self) => …)`, `host.serve(Type, factory)`, `self.contribute(R, value)` | a facet | +| factory `self`: `state`, `derive`, `get`, `registry`, `open`, `on`, `onLive`, `job`, `tx` | an object's parts | +| `createPlugins({ roots, kinds, onError }, ctx)` | the plugins component | +| `plugins.load(host, { kind, core, modules }, ctx)`, `build(names)`, `rollout(steps, built)` | load, build, roll out | +| `hostKind(name, params?)`, `Kind(params)` | host kinds and addresses | diff --git a/packages/chord/docs/next/guide-plugins-hosts/index.html b/packages/chord/docs/next/guide-plugins-hosts/index.html new file mode 100644 index 000000000..b026d7e8e --- /dev/null +++ b/packages/chord/docs/next/guide-plugins-hosts/index.html @@ -0,0 +1,463 @@ + + +Hosts and Plugins: a tutorial for Chord v2 + + + + +

    +
    +
    A guide · Chord v2 · 2026-10-31
    +

    Hosts
    and Plugins

    +

    Build a small notes app step by step, from one host to plugins on a server and on workers, edited while it runs

    +
    3kinds of host in an app: server, worker, client
    1file per host kind in a plugin
    0restarts to apply a plugin edit
    1rule: plugins talk only through object types and registries
    +

    A tutorial for people building apps on Chord v2, written by the redesign session from design.md, slice-7c.md, +slice-10.md, the decision log and the code in packages/chord/src/next and src/next-node. API names are the +current ones; where slice 10's fix round or the app kit (chord/next/app, being designed) will change what app code +writes, a note says so.

    +
    +
    + + +
    +

    Before we begin

    +

    You want to build a notes app. You type a note in a browser tab, and every other tab sees it at once. Each project +gets its own notes in its own process, so a project that misbehaves cannot take the others down. And you, or an agent +you work with, add features while the app runs: a search, an export button, a word count, without restarting +anything.

    +

    Chord gives you three things for that. Hosts: the processes of your app (the server, a worker per project, the +browser tabs), connected to each other. Objects: the state and methods your app is made of, living on one host and +used from any other. Plugins: directories of code that declare objects for each kind of host, loaded and +reloaded while the app runs.

    +

    This book builds the app in that order. Each chapter adds one concept and shows complete code.

    +

    Seven things to know

    +
      +
    1. A host is one participant. The server, each worker and each browser tab is a host; hosts connect over links.
      2. +
    2. An object has a type, and a type is a contract. The contract names the object's state and methods; any host can +ask for an object by type and params, and Chord finds where it lives.
      3. +
    3. A facet declares, an object acts. A facet is a plugin's code for one host kind: it says which object types this +host serves. The objects hold the state and do the work.
      4. +
    4. A plugin is a directory. A package.json with the plugin's name, and one file per host kind: server.ts, +worker.ts, web.tsx.
      5. +
    5. The server builds, every host evaluates. The plugins component builds each facet into a small script; the +server, workers and tabs load the scripts for their kind.
      6. +
    6. Plugins meet through types and registries. One plugin uses another's objects through its contract, or +contributes to a registry another defines. Nothing else crosses between plugins.
      7. +
    7. An edit is a rollout. Change a file, build, roll out: the server, then every worker, swap the plugin's objects in +place; a broken edit keeps the old code running and reports why.
      8. +
    +

    Words used throughout

    +

    A host is one participant of the app, made with createHost. A host kind (Server, Session) describes +addressable hosts; an address (Session({ id: "p1" })) names one. An object type is a contract: local (used +on its own host only) or remote (used from any host). An object is one instance of a type, per params. A +facet is a plugin's declarations for one host kind. A plugin is a directory with a package.json. A +reference is what you get when you ask for an object (host.get, self.get, ctx.get). A registry is an +extension point: one plugin defines it, others contribute. A rollout applies a build to the hosts in order.

    +
    + +
    I

    One Host

    The smallest app: one process, one object type, one call.

    +

    Start without plugins. One host, one object that keeps notes:

    +
    import { createHost, defineFacet, defineState, local, method, obj, arr, str } from "@earendil-works/chord/next";
+
+// the contract: what a Notes object looks like from outside
+const Notes = local("notes.store", {
+	version: 1,
+	params: {},                                        // one object per host: no params
+	members: {
+		list: obj({ items: arr(str()) }),             // state anyone can read and watch
+		add: method(obj({ text: str() })),            // a method: arguments checked by Chord
+	},
+});
+
+// the document the object keeps
+const NotesDoc = defineState("notes", { schema: obj({ items: arr(str()) }), version: 1, initial: () => ({ items: [] }) });
+
+// the facet: which types this host serves, and how to make each object
+const facet = defineFacet((host) => {
+	host.serve(Notes, (_params, self) => {
+		const doc = self.state(NotesDoc);
+		return {
+			list: doc,
+			add: async ({ text }, ctx) => self.tx(async (tx) => { (await tx.draft(doc)).items.push(text); }, ctx),
+		};
+	});
+});
+
+const server = createHost({ onError: console.error });
+await server.load([{ name: "@app/notes", facet }], ctx);
+const notes = await server.get(Notes, {}, ctx);        // a reference: live, with its state
+await notes.add({ text: "buy milk" }, ctx);
+console.log(notes.list.value);                          // { items: ["buy milk"] }
    +

    Four things happened. The contract Notes says what the object offers. The facet says this host serves +Notes, with a factory that declares the object's parts: a document, and a method writing it in a transaction. +load publishes the facet. get returns a reference once the object is live; its members are the contract's.

    +

    The factory only declares: it is synchronous and opens nothing. Chord activates the object (loads its documents, opens +what it asked to open) and then serves calls. That is why the same factory can run again when the code changes.

    +
    + +
    II

    Several Hosts

    A server, a worker per project, browser tabs: how they find each other's objects.

    +

    Now give each project its own process. Declare the kinds of addressable host, and make notes a remote type that +lives on its project's worker:

    +
    export const Server = hostKind("server");
+export const Project = hostKind("project", { id: str() });
+
+export const Notes = remote("notes", {
+	version: 1,
+	params: { project: str() },
+	hostAddress: ({ project }) => Project({ id: project }),   // where an object of these params lives
+	variants: { editor: { state: { list: NotesSchema }, methods: { add: method(AddArgs) } } },
+});
    +

    hostAddress is the whole routing story. A tab asks for Notes { project: "p1" }. Its host computes the address, +Project(p1), and sends the request to its parent, the server. The server has a route for Project: it starts (or +finds) the worker for p1 and passes the request on. The worker serves Notes. The tab gets a reference whose state +follows the worker's document, one frame per change.

    +
    CLIENTSSERVERWORKERStab Aholds Notes { project: p1 }tab Bholds Notes { project: p1 }serverroute Project → workerworker p1serves Notes { p1 }worker p2serves Notes { p2 }holdholdrelay
    Figure 1. A tab holds a project's notes: the server routes by address to the project's worker and relays, without reading the data.
    +

    Each host is created the same way, with what it is:

    +
    // server.ts
+const server = createHost<P>({ address: Server(), principal: { kind: "server" }, onError });
+
+// worker.ts (started by the server for one project)
+const worker = createHost<P>({ address: Project({ id }), principal: { kind: "worker", project: id }, onError });
+worker.connect(parentLink, { principal: { kind: "server" }, trusted: true }, ctx);
+
+// a browser tab: no address, it is a client
+const tab = createHost<P>({ onError });
+tab.connect(webSocketLink(new WebSocket("/chord")), { principal: { kind: "server" } }, ctx);
    +

    P is your app's type of principal, who is calling: an owner, a guest, the server, a worker. Links stamp it; +a remote object is made per principal, so a guest can get a read-only variant of the same notes.

    +
    Coming with the app kit

    Starting workers, routing a host kind to them, the WebSocket entry and authentication are slice 10's components today +(createWorkers, chord.http), wired by the app. The opinionated app kit (chord/next/app, being designed) will do +this wiring for you: startApp(...) and startWorker().

    +
    +
    + +
    III

    Plugins

    Move the code into a directory that can be added, edited and removed while the app runs.

    +

    So far the facets were part of the app's own code. A plugin is the same code in its own directory:

    +
    plugins/notes/
+  package.json        { "name": "@app/notes" }
+  contract.ts         the Notes contract (other plugins import it)
+  worker.ts           the facet for workers: serves Notes
+  server.ts           the facet for the server: an export route
+  web.tsx             the facet for browser tabs: a notes panel
    +

    The plugin's name is its package.json name. Each file named after a host kind is that kind's facet, its +default export a defineFacet(...). A plugin has only the files it needs: a plugin that only adds a route has a +server.ts and nothing else.

    +
    // plugins/notes/worker.ts
+import { defineFacet } from "@app/core";
+import { Notes } from "./contract.ts";
+
+export default defineFacet((host) => {
+	host.serve(Notes, ({ project }, principal, self) => {
+		const doc = self.state(NotesDoc);
+		return { variant: "editor", members: { list: doc, add: async ({ text }, ctx) => /* … */ } };
+	});
+});
    +

    @app/core is your app's base code, imported by every plugin as a host package (the plugin's build leaves it +out; every host passes its own copy in). It holds what plugins share and never reloads: host kinds, the principal +type, a defineFacet bound to it.

    +
    + +
    IV

    Loading Plugins

    Discover, build, evaluate, load: what happens between a directory and a running object.

    +

    The plugins component does the work. You tell it where plugins are and what each host kind needs:

    +
    const plugins = createPlugins({
+	roots: ["./plugins"],
+	kinds: {
+		server: { platform: "node", hostPackages: ["@earendil-works/chord/next", "@app/core"] },
+		worker: { platform: "node", hostPackages: ["@earendil-works/chord/next", "@app/core"] },
+		web: { platform: "browser", hostPackages: ["@earendil-works/chord/next", "@app/core", "react", "react/jsx-runtime"] },
+	},
+	onError,
+}, ctx);
+
+const report = await plugins.load(server, {
+	kind: "server",
+	core: [{ name: "@app/core", facet: coreServerFacet }],
+	modules: { "@earendil-works/chord/next": chord, "@app/core": core },
+}, ctx);
+console.log(report.failed);   // plugins left out, with why: [] when all loaded
    +
    What plugins.load does

    Discover

    It lists ./plugins one level deep and reads each package.json. A directory without one, or with two server +files, is reported, not loaded.

    +

    Build

    For each plugin with a server facet, esbuild builds one script from server.ts and everything it imports: the +plugin's own files, another plugin's contract files, its npm dependencies. Host packages stay out. A syntax error is a +build failure, never a crash.

    +

    Evaluate

    The script runs and returns the facet. Its import "@app/core" gets the server's own @app/core, so a ChordError +from a plugin is the same class as the server's.

    +

    Load

    host.load runs every facet's setup and publishes all of them at once. A plugin whose build, evaluation or setup +fails is left out and listed in report.failed; the others load. Its objects' users wait, and the next edit loads it.

    +
    +

    A worker never builds: the server builds the worker facets too, and each worker evaluates the scripts it is sent.

    +
    + +
    V

    Plugins Using Plugins

    Two ways one plugin uses another, and the one it may not.

    +

    A search plugin wants to search the notes. It imports the contract and asks for the object:

    +
    // plugins/search/worker.ts
+import { Notes } from "../notes/contract.ts";   // copied into search's script: contracts only
+
+host.serve(Search, ({ project }, principal, self) => {
+	const notes = self.get(Notes, { project });   // a dependency: Search goes live once Notes is
+	const index = self.derive([notes.list], ([list]) => buildIndex(list.items));
+	return { variant: "main", members: { find: async ({ q }) => lookup(index.value, q) } };
+});
    +

    self.get in a factory declares a dependency: Chord makes Search live only once Notes is, and if the notes +plugin is missing, Search waits with a clear reason ("waiting for notes: not served here") instead of failing.

    +

    The other way is a registry: an extension point one plugin defines and others contribute to. The app's core +defines Panels, and every plugin with a UI contributes one:

    +
    // plugins/notes/web.tsx
+export default defineFacet((host, self) => self.contribute(Panels, { title: "Notes", Component: NotesPanel }));
    +

    What a plugin may not do is reach into another's data. Search never reads notes' documents; it uses the Notes +object's state. That rule is what lets the notes plugin change its storage without breaking search.

    +
    + +
    VI

    Changing a Plugin

    Edit, build, roll out: how new code takes over while every tab stays connected.

    +

    You change the notes plugin: notes now have a timestamp. The watcher sees the saved file and marks the plugin dirty. +You (or an agent) build and roll out:

    +
    const built = await plugins.build(plugins.state.value.dirty, ctx);
+const result = await plugins.rollout([plugins.step("server", server), workers.step("workers")], built, ctx);
    +

    The rollout goes host by host, server first, then every running worker. On each host Chord compares the new plugin +set with the loaded one, runs the new factories against a recording self to see what each object would declare, and +then gives every object of the changed plugin its fate:

    +

    failed { waiting }
    no fitting version, a fingerprint that differs at the same version, a placement the provider computes elsewhereno fitting version, a fingerprint that differs at the same version, a host address the provider computes elsewhere unbound version failed { waiting }
    refused and kept at the provider, re-run at its publication; failed { waiting }
    design §3, placementdesign §3, the destination "served here" decides "here"; placement optional on remote()the address decides; every remote type declares its placement (owner, 2026-10-29)the address decides; every remote type declares its hostAddress, renamed from placement (owner, 2026-10-29)
    the start barrier, pendingLoad unbound not_served: failed { waiting }
    no fitting version, a fingerprint that differs at the same version, a placement the provider computes elsewhere (code skew)no fitting version, a fingerprint that differs at the same version, a host address the provider computes elsewhere (code skew) unbound version: failed { waiting }
    2 self.get(Catalog) from W1's ForwardStore; the parent link is up, S has published, Catalog is served there.The request goes over the parent (placement Server()); S binds the hold; bound arrives; ForwardStore was loading, now goes on and live.The request goes over the parent (hostAddress: () => Server()); S binds the hold; bound arrives; ForwardStore was loading, now goes on and live. The same, after S's whole barrier (Hold.start awaits pendingLoad). §1, §2
    + + + + + + + + + + + + + + + + + + +
    The objectWhat happens
    holds no natives (open files, sockets), the new code neitherswapped in place: new methods, same documents; running calls finish on the old code
    holds nativesrestarted: calls wait, running ones finish, natives close and reopen with the new code
    its type was removedended; its users wait until something serves it again
    +

    The tab's reference follows the new object and gets one frame with the difference. Nothing reconnects. If the edit +is broken (a syntax error, a factory that throws), that plugin keeps its old code on that host and the report says why; +every other plugin of the rollout applies.

    +
    Versions

    A change to a contract (a new field in Notes's state) gets a new version. A tab built with the old contract keeps +working if the provider maps the old version (accepts); otherwise it waits with "version mismatch" until it reloads.

    +
    + + +
    ∴

    Base Code and Plugins

    What reloads, what restarts, and where the line is.

    +

    Everything in plugins/ reloads: edit, build, roll out. Everything in your base code (src/, @app/core: host +kinds, principals, the core facet, the app's startup) changes only with a restart, and the watcher tells you so. +Keep base code small: host kinds, principals, and what every plugin needs. Put features in plugins.

    +

    The app kit being designed next (chord/next/app) will make the base code nearly empty: startApp and startWorker +with Chord's conventions for hosts, users, authentication and storage, so an app is mostly plugins.

    +
    + +
    A

    Reference

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    CallWhat it does
    createHost<P>({ address?, principal?, store?, onError })a host: addressable with an address, else a client
    host.load([{ name, facet }], ctx) / host.reload(...)publish facets / apply a new plugin set
    host.get(Type, params, ctx) / host.hold(...)a live reference / a reference at once that follows its object
    local(name, { version, params, members }) / remote(name, { version, params, hostAddress, variants })contracts
    defineFacet((host, self) => …), host.serve(Type, factory), self.contribute(R, value)a facet
    factory self: state, derive, get, registry, open, on, onLive, job, txan object's parts
    createPlugins({ roots, kinds, onError }, ctx)the plugins component
    plugins.load(host, { kind, core, modules }, ctx), build(names), rollout(steps, built)load, build, roll out
    hostKind(name, params?), Kind(params)host kinds and addresses
    +
    + +

    Hosts and Plugins · first edition · 2026-10-31 · Chord v2 slices 1-10

    + + + From a34444d6bd768ecee494ad17261daaa75a6ed808 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 13:56:14 +0000 Subject: [PATCH 083/175] docs(chord): tutorial: overview figure, client hosts --- .../docs/next/guide-plugins-hosts/book.md | 65 ++++++++++++++++++- .../docs/next/guide-plugins-hosts/index.html | 56 +++++++++++++--- 2 files changed, 110 insertions(+), 11 deletions(-) diff --git a/packages/chord/docs/next/guide-plugins-hosts/book.md b/packages/chord/docs/next/guide-plugins-hosts/book.md index 2ffe01d53..83fe5051a 100644 --- a/packages/chord/docs/next/guide-plugins-hosts/book.md +++ b/packages/chord/docs/next/guide-plugins-hosts/book.md @@ -33,7 +33,8 @@ This book builds the app in that order. Each chapter adds one concept and shows ## Seven things to know :::findings -1. **A host is one participant.** The server, each worker and each browser tab is a host; hosts connect over links. +1. **A host is one participant.** The server, each worker, each browser tab and each terminal UI is a host; hosts + connect over links. Tabs and terminal UIs are clients: no address, they connect to the server and ask. 2. **An object has a type, and a type is a contract.** The contract names the object's state and methods; any host can ask for an object by type and params, and Chord finds where it lives. 3. **A facet declares, an object acts.** A facet is a plugin's code for one host kind: it says which object types this @@ -48,6 +49,25 @@ This book builds the app in that order. Each chapter adds one concept and shows place; a broken edit keeps the old code running and reports why. ::: +:::flow wide animate | How it all fits: plugins on disk are built once on the server; each host evaluates the facets of its kind; tabs and the TUI are clients that reach workers' objects through the server. +row "On disk" muted +box disk "plugins/notes/" "`server.ts`, `worker.ts`, `web.tsx`, `tui.ts`, `contract.ts`" muted +box core "src/ (base code)" "`@app/core`: host kinds, principals" muted +row "Server process" accent +box comp "plugins component" "discover, build every facet, watch, roll out" accent +box server "server host" "server facets; routes Project → workers" +row "Clients and workers" +box tab "client: tab or TUI" "web or tui facets, local UI objects; holds `Notes { p1 }`" blue +box w1 "worker p1" "worker facets; serves `Notes { p1 }`" +arrow disk -> comp "read, build" +arrow core -> comp "host packages" +arrow comp -> server "server scripts" +arrow comp -> tab "web or tui scripts" +arrow tab -> server "hold Notes { p1 }" +arrow server -> w1 "start (worker scripts), relay" +note "Each host loads only its own kind's facets. Plugins meet only through object types and registries." +::: + ## Words used throughout A **host** is one participant of the app, made with `createHost`. A **host kind** (`Server`, `Session`) describes @@ -162,6 +182,49 @@ Starting workers, routing a host kind to them, the WebSocket entry and authentic this wiring for you: `startApp(...)` and `startWorker()`. ::: +# Client Hosts {#clients lede="Browser tabs and terminal UIs are hosts too: what they can do, and what they cannot."} + +A browser tab and a terminal UI are hosts like the server, with one difference: they have **no address**. Nobody can +send a request *to* a tab; a tab only connects to the server and asks for things. That is a deliberate rule: a tab +runs on a device the server does not control, so nothing on the server ever waits for, or trusts, a tab's state. + +```ts +// a browser tab (slice 11's page host will do this wiring for you) +const tab = createHost

    ({ onError }); // no address, no principal: a client +tab.connect(webSocketLink(new WebSocket("/chord")), { principal: { kind: "server" } }, ctx); +await tab.load([{ name: "@app/web", facet: webCore }, ...webFacets], ctx); + +const notes = tab.hold(Notes, { project: "p1" }, ctx); // a reference at once, live soon +(await notes.list.watch(ctx)).start(async (call) => { + if (call.kind === "value") render(call.value.items); // every change, as it commits on the worker +}); +await notes.add({ text: "from the tab" }, ctx); // a call: tab → server → worker +``` + +**Who the tab is** is not the tab's to say. The server authenticates the WebSocket when it connects (a cookie, the +owner on loopback, a guest's share link) and stamps that principal on everything the tab asks for. The worker's +`Notes` factory receives it and may give a guest a read-only variant. A tab cannot claim to be the owner. + +**What a tab serves** is only `local` types: state of this tab's UI, never shared. A draft that survives switching +panels, a selection, a tab-local cache: + +```ts +// plugins/notes/web.tsx: the web facet serves a local type, and contributes a panel +export default defineFacet((host, self) => { + host.serve(NoteDraft, (_params, self) => ({ text: self.state(DraftDoc) })); // lives in this tab only + self.contribute(Panels, { title: "Notes", Component: NotesPanel }); +}); +``` + +**When the connection drops** (a phone in a tunnel, a server restart), the tab keeps its references. Their values +stay readable, their `state` says `loading`, calls made meanwhile wait; on reconnect every reference is held again and +gets the current value in one frame. The UI shows a banner from `ref.state`, nothing more. + +**A terminal UI** is the same in Node: a client host connected to the server over a WebSocket, loading the plugins' +`tui` facets, rendering with whatever terminal library the app chose. **What a client never does**: serve a `remote` +type, be routed to, or keep data others rely on. Data that must outlive a tab belongs in an object on the server or a +worker. + # Plugins {#plugins lede="Move the code into a directory that can be added, edited and removed while the app runs."} So far the facets were part of the app's own code. A **plugin** is the same code in its own directory: diff --git a/packages/chord/docs/next/guide-plugins-hosts/index.html b/packages/chord/docs/next/guide-plugins-hosts/index.html index b026d7e8e..ec7ab3f9c 100644 --- a/packages/chord/docs/next/guide-plugins-hosts/index.html +++ b/packages/chord/docs/next/guide-plugins-hosts/index.html @@ -123,10 +123,11 @@ writes, a note says so.

  • Before we begin
  • I. One Host
  • II. Several Hosts
    • -
  • III. Plugins
    • -
  • IV. Loading Plugins
    • -
  • V. Plugins Using Plugins
    • -
  • VI. Changing a Plugin
    • +
  • III. Client Hosts
    • +
  • IV. Plugins
    • +
  • V. Loading Plugins
    • +
  • VI. Plugins Using Plugins
    • +
  • VII. Changing a Plugin
  • Base Code and Plugins
  • Appendix A: Reference
    • @@ -144,7 +145,8 @@ reloaded while the app runs.

    This book builds the app in that order. Each chapter adds one concept and shows complete code.

    Seven things to know

      -
    1. A host is one participant. The server, each worker and each browser tab is a host; hosts connect over links.
      2. +
    2. A host is one participant. The server, each worker, each browser tab and each terminal UI is a host; hosts +connect over links. Tabs and terminal UIs are clients: no address, they connect to the server and ask.
    3. An object has a type, and a type is a contract. The contract names the object's state and methods; any host can ask for an object by type and params, and Chord finds where it lives.
    4. A facet declares, an object acts. A facet is a plugin's code for one host kind: it says which object types this @@ -158,6 +160,7 @@ contributes to a registry another defines. Nothing else crosses between plugins.
    5. An edit is a rollout. Change a file, build, roll out: the server, then every worker, swap the plugin's objects in place; a broken edit keeps the old code running and reports why.
    +
    ON DISKSERVER PROCESSCLIENTS AND WORKERSplugins/notes/server.ts`, `worker.ts`,`web.tsx`, `tui.ts`, `contract.tssrc/ (base code)`@app/core`: host kinds, principalsplugins componentdiscover, build every facet, watch,roll outserver hostserver facets; routes Project →workersclient: tab or TUIweb or tui facets, local UI objects;holds `Notes { p1 }`worker p1worker facets; serves `Notes { p1 }`read, buildhost packagesserver scriptsweb or tui scriptshold Notes { p1 }start (worker scripts), relayEach host loads only its own kind's facets. Plugins meet only through object types and registries.
    Figure 1. How it all fits: plugins on disk are built once on the server; each host evaluates the facets of its kind; tabs and the TUI are clients that reach workers' objects through the server.

    Words used throughout

    A host is one participant of the app, made with createHost. A host kind (Server, Session) describes addressable hosts; an address (Session({ id: "p1" })) names one. An object type is a contract: local (used @@ -223,7 +226,7 @@ lives on its project's worker:

    Project(p1), and sends the request to its parent, the server. The server has a route for Project: it starts (or finds) the worker for p1 and passes the request on. The worker serves Notes. The tab gets a reference whose state follows the worker's document, one frame per change.

    -
    CLIENTSSERVERWORKERStab Aholds Notes { project: p1 }tab Bholds Notes { project: p1 }serverroute Project → workerworker p1serves Notes { p1 }worker p2serves Notes { p2 }holdholdrelay
    Figure 1. A tab holds a project's notes: the server routes by address to the project's worker and relays, without reading the data.
    +
    CLIENTSSERVERWORKERStab Aholds Notes { project: p1 }tab Bholds Notes { project: p1 }serverroute Project → workerworker p1serves Notes { p1 }worker p2serves Notes { p2 }holdholdrelay
    Figure 2. A tab holds a project's notes: the server routes by address to the project's worker and relays, without reading the data.

    Each host is created the same way, with what it is:

    // server.ts
 const server = createHost<P>({ address: Server(), principal: { kind: "server" }, onError });
@@ -243,7 +246,40 @@ this wiring for you: startApp(...) and startWorker().<
 
 
 
-
    III
    Plugins
    Move the code into a directory that can be added, edited and removed while the app runs.
    
+
    III
    Client Hosts
    Browser tabs and terminal UIs are hosts too: what they can do, and what they cannot.
    
+
    A browser tab and a terminal UI are hosts like the server, with one difference: they have no address. Nobody can
+send a request to a tab; a tab only connects to the server and asks for things. That is a deliberate rule: a tab
+runs on a device the server does not control, so nothing on the server ever waits for, or trusts, a tab's state.
    
+
    // a browser tab (slice 11's page host will do this wiring for you)
+const tab = createHost<P>({ onError });                               // no address, no principal: a client
+tab.connect(webSocketLink(new WebSocket("/chord")), { principal: { kind: "server" } }, ctx);
+await tab.load([{ name: "@app/web", facet: webCore }, ...webFacets], ctx);
+
+const notes = tab.hold(Notes, { project: "p1" }, ctx);               // a reference at once, live soon
+(await notes.list.watch(ctx)).start(async (call) => {
+	if (call.kind === "value") render(call.value.items);              // every change, as it commits on the worker
+});
+await notes.add({ text: "from the tab" }, ctx);                        // a call: tab → server → worker
    
+
    Who the tab is is not the tab's to say. The server authenticates the WebSocket when it connects (a cookie, the
+owner on loopback, a guest's share link) and stamps that principal on everything the tab asks for. The worker's
+Notes factory receives it and may give a guest a read-only variant. A tab cannot claim to be the owner.
    
+
    What a tab serves is only local types: state of this tab's UI, never shared. A draft that survives switching
+panels, a selection, a tab-local cache:
    
+
    // plugins/notes/web.tsx: the web facet serves a local type, and contributes a panel
+export default defineFacet((host, self) => {
+	host.serve(NoteDraft, (_params, self) => ({ text: self.state(DraftDoc) }));   // lives in this tab only
+	self.contribute(Panels, { title: "Notes", Component: NotesPanel });
+});
    
+
    When the connection drops (a phone in a tunnel, a server restart), the tab keeps its references. Their values
+stay readable, their state says loading, calls made meanwhile wait; on reconnect every reference is held again and
+gets the current value in one frame. The UI shows a banner from ref.state, nothing more.
    
+
    A terminal UI is the same in Node: a client host connected to the server over a WebSocket, loading the plugins'
+tui facets, rendering with whatever terminal library the app chose. What a client never does: serve a remote
+type, be routed to, or keep data others rely on. Data that must outlive a tab belongs in an object on the server or a
+worker.
    
+
    
+
+
    IV
    Plugins
    Move the code into a directory that can be added, edited and removed while the app runs.
    
 
    So far the facets were part of the app's own code. A plugin is the same code in its own directory:
    
 
    plugins/notes/
   package.json        { "name": "@app/notes" }
@@ -269,7 +305,7 @@ out; every host passes its own copy in). It holds what plugins share and never r
 type, a defineFacet bound to it.
    
 
    
 
-
    IV
    Loading Plugins
    Discover, build, evaluate, load: what happens between a directory and a running object.
    
+
    V
    Loading Plugins
    Discover, build, evaluate, load: what happens between a directory and a running object.
    
 
    The plugins component does the work. You tell it where plugins are and what each host kind needs:
    
 
    const plugins = createPlugins({
 	roots: ["./plugins"],
@@ -300,7 +336,7 @@ fails is left out and listed in report.failed; the others load. Its
 
    A worker never builds: the server builds the worker facets too, and each worker evaluates the scripts it is sent.
    
 
    
 
-
    V
    Plugins Using Plugins
    Two ways one plugin uses another, and the one it may not.
    
+
    VI
    Plugins Using Plugins
    Two ways one plugin uses another, and the one it may not.
    
 
    A search plugin wants to search the notes. It imports the contract and asks for the object:
    
 
    // plugins/search/worker.ts
 import { Notes } from "../notes/contract.ts";   // copied into search's script: contracts only
@@ -320,7 +356,7 @@ defines Panels, and every plugin with a UI contributes one:
    
 object's state. That rule is what lets the notes plugin change its storage without breaking search.
    
 
    
 
-
    VI
    Changing a Plugin
    Edit, build, roll out: how new code takes over while every tab stays connected.
    
+
    VII
    Changing a Plugin
    Edit, build, roll out: how new code takes over while every tab stays connected.
    
 
    You change the notes plugin: notes now have a timestamp. The watcher sees the saved file and marks the plugin dirty.
 You (or an agent) build and roll out:
    
 
    const built = await plugins.build(plugins.state.value.dirty, ctx);

From 436d99e04b02d5ed784e09dba8601383e7fc8e76 Mon Sep 17 00:00:00 2001
From: Mario Zechner 
Date: Sat, 31 Oct 2026 14:01:40 +0000
Subject: [PATCH 084/175] docs(chord): tutorial: one schema for contract and
 document

---
 .../docs/next/guide-plugins-hosts/book.md      | 18 ++++++++++++------
 .../docs/next/guide-plugins-hosts/index.html   | 18 ++++++++++++------
 2 files changed, 24 insertions(+), 12 deletions(-)

diff --git a/packages/chord/docs/next/guide-plugins-hosts/book.md b/packages/chord/docs/next/guide-plugins-hosts/book.md
index 83fe5051a..ab04e2119 100644
--- a/packages/chord/docs/next/guide-plugins-hosts/book.md
+++ b/packages/chord/docs/next/guide-plugins-hosts/book.md
@@ -84,18 +84,20 @@ Start without plugins. One host, one object that keeps notes:
 ```ts
 import { createHost, defineFacet, defineState, local, method, obj, arr, str } from "@earendil-works/chord/next";
 
-// the contract: what a Notes object looks like from outside
+const NoteList = obj({ items: arr(str()) });          // one schema, used by the contract and the document
+
+// the contract: what a Notes object offers
 const Notes = local("notes.store", {
 	version: 1,
 	params: {},                                        // one object per host: no params
 	members: {
-		list: obj({ items: arr(str()) }),             // state anyone can read and watch
+		list: NoteList,                               // state anyone can read and watch
 		add: method(obj({ text: str() })),            // a method: arguments checked by Chord
 	},
 });
 
-// the document the object keeps
-const NotesDoc = defineState("notes", { schema: obj({ items: arr(str()) }), version: 1, initial: () => ({ items: [] }) });
+// the document the object stores (its own version: storage can change without changing the contract)
+const NotesDoc = defineState("notes", { schema: NoteList, version: 1, initial: () => ({ items: [] }) });
 
 // the facet: which types this host serves, and how to make each object
 const facet = defineFacet((host) => {
@@ -115,7 +117,11 @@ await notes.add({ text: "buy milk" }, ctx);
 console.log(notes.list.value);                          // { items: ["buy milk"] }
 ```
 
-Four things happened. The **contract** `Notes` says what the object offers. The **facet** says this host serves
+Four things happened. The **contract** `Notes` says what the object offers; the **document** says what it stores.
+Here they have the same shape, so one schema serves both; they are separate because they change separately (a
+derive or a guest's redacted view offers something other than storage, and storage can migrate without users
+noticing).
+The **facet** says this host serves
 `Notes`, with a **factory** that declares the object's parts: a document, and a method writing it in a transaction.
 `load` publishes the facet. `get` returns a **reference** once the object is live; its members are the contract's.
 
@@ -135,7 +141,7 @@ export const Notes = remote("notes", {
 	version: 1,
 	params: { project: str() },
 	hostAddress: ({ project }) => Project({ id: project }),   // where an object of these params lives
-	variants: { editor: { state: { list: NotesSchema }, methods: { add: method(AddArgs) } } },
+	variants: { editor: { state: { list: NoteList }, methods: { add: method(obj({ text: str() })) } } },
 });
 ```
 
diff --git a/packages/chord/docs/next/guide-plugins-hosts/index.html b/packages/chord/docs/next/guide-plugins-hosts/index.html
index ec7ab3f9c..677e54ab6 100644
--- a/packages/chord/docs/next/guide-plugins-hosts/index.html
+++ b/packages/chord/docs/next/guide-plugins-hosts/index.html
@@ -174,18 +174,20 @@ extension point: one plugin defines it, others contribute. A rolloutStart without plugins. One host, one object that keeps notes:
    
 
    import { createHost, defineFacet, defineState, local, method, obj, arr, str } from "@earendil-works/chord/next";
 
-// the contract: what a Notes object looks like from outside
+const NoteList = obj({ items: arr(str()) });          // one schema, used by the contract and the document
+
+// the contract: what a Notes object offers
 const Notes = local("notes.store", {
 	version: 1,
 	params: {},                                        // one object per host: no params
 	members: {
-		list: obj({ items: arr(str()) }),             // state anyone can read and watch
+		list: NoteList,                               // state anyone can read and watch
 		add: method(obj({ text: str() })),            // a method: arguments checked by Chord
 	},
 });
 
-// the document the object keeps
-const NotesDoc = defineState("notes", { schema: obj({ items: arr(str()) }), version: 1, initial: () => ({ items: [] }) });
+// the document the object stores (its own version: storage can change without changing the contract)
+const NotesDoc = defineState("notes", { schema: NoteList, version: 1, initial: () => ({ items: [] }) });
 
 // the facet: which types this host serves, and how to make each object
 const facet = defineFacet((host) => {
@@ -203,7 +205,11 @@ extension point: one plugin defines it, others contribute. A rolloutconst notes = await server.get(Notes, {}, ctx);        // a reference: live, with its state
 await notes.add({ text: "buy milk" }, ctx);
 console.log(notes.list.value);                          // { items: ["buy milk"] }
    
-
    Four things happened. The contract Notes says what the object offers. The facet says this host serves
+
    Four things happened. The contract Notes says what the object offers; the document says what it stores.
+Here they have the same shape, so one schema serves both; they are separate because they change separately (a
+derive or a guest's redacted view offers something other than storage, and storage can migrate without users
+noticing).
+The facet says this host serves
 Notes, with a factory that declares the object's parts: a document, and a method writing it in a transaction.
 load publishes the facet. get returns a reference once the object is live; its members are the contract's.
    
 
    The factory only declares: it is synchronous and opens nothing. Chord activates the object (loads its documents, opens
@@ -220,7 +226,7 @@ lives on its project's worker:
    
 	version: 1,
 	params: { project: str() },
 	hostAddress: ({ project }) => Project({ id: project }),   // where an object of these params lives
-	variants: { editor: { state: { list: NotesSchema }, methods: { add: method(AddArgs) } } },
+	variants: { editor: { state: { list: NoteList }, methods: { add: method(obj({ text: str() })) } } },
 });
    
 
    hostAddress is the whole routing story. A tab asks for Notes { project: "p1" }. Its host computes the address,
 Project(p1), and sends the request to its parent, the server. The server has a route for Project: it starts (or

From dae7bd8fcb4cfe8193bce2fd0a2289aa611f0746 Mon Sep 17 00:00:00 2001
From: Mario Zechner 
Date: Sat, 31 Oct 2026 14:07:01 +0000
Subject: [PATCH 085/175] docs(chord): tutorial: an object's parts before they
 are used

---
 .../docs/next/guide-plugins-hosts/book.md     | 67 +++++++++++++++--
 .../docs/next/guide-plugins-hosts/index.html  | 73 ++++++++++++++-----
 2 files changed, 118 insertions(+), 22 deletions(-)

diff --git a/packages/chord/docs/next/guide-plugins-hosts/book.md b/packages/chord/docs/next/guide-plugins-hosts/book.md
index ab04e2119..d15c8a892 100644
--- a/packages/chord/docs/next/guide-plugins-hosts/book.md
+++ b/packages/chord/docs/next/guide-plugins-hosts/book.md
@@ -74,7 +74,9 @@ A **host** is one participant of the app, made with `createHost`. A **host kind*
 addressable hosts; an **address** (`Session({ id: "p1" })`) names one. An **object type** is a contract: `local` (used
 on its own host only) or `remote` (used from any host). An **object** is one instance of a type, per params. A
 **facet** is a plugin's declarations for one host kind. A **plugin** is a directory with a `package.json`. A
-**reference** is what you get when you ask for an object (`host.get`, `self.get`, `ctx.get`). A **registry** is an
+**reference** is what you get when you ask for an object (`host.get`, `self.get`, `ctx.get`). A **document** is data an object
+stores, changed in **transactions**; a **derive** is a value computed from others; a **native** is an outside
+resource an object opens and closes. A **registry** is an
 extension point: one plugin defines it, others contribute. A **rollout** applies a build to the hosts in order.
 
 # One Host {#host lede="The smallest app: one process, one object type, one call."}
@@ -118,9 +120,9 @@ console.log(notes.list.value);                          // { items: ["buy milk"]
 ```
 
 Four things happened. The **contract** `Notes` says what the object offers; the **document** says what it stores.
-Here they have the same shape, so one schema serves both; they are separate because they change separately (a
-derive or a guest's redacted view offers something other than storage, and storage can migrate without users
-noticing).
+Here they have the same shape, so one schema serves both; they are separate because they change separately (an
+object may offer a computed value or a shortened view instead of what it stores, and storage can change its shape
+without users noticing).
 The **facet** says this host serves
 `Notes`, with a **factory** that declares the object's parts: a document, and a method writing it in a transaction.
 `load` publishes the facet. `get` returns a **reference** once the object is live; its members are the contract's.
@@ -128,6 +130,57 @@ The **facet** says this host serves
 The factory only declares: it is synchronous and opens nothing. Chord activates the object (loads its documents, opens
 what it asked to open) and then serves calls. That is why the same factory can run again when the code changes.
 
+# An Object's Parts {#parts lede="Documents, transactions, computed values, watching, and outside resources: the vocabulary of every factory."}
+
+A factory builds an object from a handful of parts, each declared with one call on `self`. The notes object grows to
+show all of them.
+
+**Documents** hold the object's data: `self.state(NotesDoc)` gives a handle whose `.value` is the current data. Data
+changes only in a **transaction**: `self.tx(fn, ctx)` runs `fn` with `tx`, and `await tx.draft(handle)` gives a
+mutable copy of the document's data; whatever you change on it is committed at once when `fn` returns, or not at all
+if it throws.
+
+```ts
+add: async ({ text }, ctx) => self.tx(async (tx) => {
+	const draft = await tx.draft(doc);          // the document's data, writable inside this transaction only
+	draft.items.push(text);                      // recorded as a change, committed when the function returns
+}, ctx),
+```
+
+A **derive** is a value computed from other values, recomputed whenever they change. It is how an object offers
+something it does not store:
+
+```ts
+const count = self.derive([doc], ([notes]) => notes.items.length);   // 0, 1, 2, … as notes are added
+return { list: doc, count, add };                                      // `count` can be a member like `list`
+```
+
+**Watching** is how anyone follows a value as it changes: a document handle, a derive, a reference's member. `watch`
+gives the current value, then one call per change:
+
+```ts
+const watch = await notes.count.watch(ctx);
+watch.start(async (call) => {
+	if (call.kind === "value") console.log("notes:", call.value);   // "failed" calls say a value cannot be computed now
+});
+```
+
+A **routine** is the object watching on its own behalf: `self.on(source, fn)` runs `fn` with each new value, for as
+long as the object lives (a reconciler: "whenever the notes change, rebuild the export file").
+
+A **native** is something outside Chord the object opens and must close: a file watcher, a database connection, a
+child process. `self.open(open, close)` opens it while the object activates and closes it when the object ends:
+
+```ts
+const inbox = self.open(async (ctx) => watchDirectory(inboxPath, ctx.abortSignal), (watcher) => watcher.close());
+self.onLive(async (ctx) => {                                  // a routine started once the object is live
+	for await (const file of inbox.value.files(ctx.abortSignal)) await importNote(file, ctx);
+});
+```
+
+Natives are why some objects restart when their plugin changes: an open connection cannot be moved to new code, so
+Chord closes it and opens it again. Objects without natives are swapped in place.
+
 # Several Hosts {#hosts lede="A server, a worker per project, browser tabs: how they find each other's objects."}
 
 Now give each project its own process. Declare the kinds of addressable host, and make notes a `remote` type that
@@ -145,7 +198,11 @@ export const Notes = remote("notes", {
 });
 ```
 
-`hostAddress` is the whole routing story. A tab asks for `Notes { project: "p1" }`. Its host computes the address,
+Two things are new. A `remote` type can be used from any host, and it has **variants**: different sets of members
+for different callers, chosen by the factory (an `editor` variant with `add`, a `reader` variant for guests without
+it). And `hostAddress` says where its objects live; it is the whole routing story.
+
+A tab asks for `Notes { project: "p1" }`. Its host computes the address,
 `Project(p1)`, and sends the request to its parent, the server. The server has a **route** for `Project`: it starts (or
 finds) the worker for `p1` and passes the request on. The worker serves `Notes`. The tab gets a reference whose state
 follows the worker's document, one frame per change.
diff --git a/packages/chord/docs/next/guide-plugins-hosts/index.html b/packages/chord/docs/next/guide-plugins-hosts/index.html
index 677e54ab6..55719365f 100644
--- a/packages/chord/docs/next/guide-plugins-hosts/index.html
+++ b/packages/chord/docs/next/guide-plugins-hosts/index.html
@@ -122,12 +122,13 @@ writes, a note says so.
    
 
@@ -166,7 +167,9 @@ place; a broken edit keeps the old code running and reports why.
 addressable hosts; an address (Session({ id: "p1" })) names one. An object type is a contract: local (used
 on its own host only) or remote (used from any host). An object is one instance of a type, per params. A
 facet is a plugin's declarations for one host kind. A plugin is a directory with a package.json. A
-reference is what you get when you ask for an object (host.get, self.get, ctx.get). A registry is an
+reference is what you get when you ask for an object (host.get, self.get, ctx.get). A document is data an object
+stores, changed in transactions; a derive is a value computed from others; a native is an outside
+resource an object opens and closes. A registry is an
 extension point: one plugin defines it, others contribute. A rollout applies a build to the hosts in order.
    
 
    
 
@@ -206,9 +209,9 @@ extension point: one plugin defines it, others contribute. A rolloutawait notes.add({ text: "buy milk" }, ctx);
 console.log(notes.list.value);                          // { items: ["buy milk"] }

    Four things happened. The contract Notes says what the object offers; the document says what it stores. -Here they have the same shape, so one schema serves both; they are separate because they change separately (a -derive or a guest's redacted view offers something other than storage, and storage can migrate without users -noticing). +Here they have the same shape, so one schema serves both; they are separate because they change separately (an +object may offer a computed value or a shortened view instead of what it stores, and storage can change its shape +without users noticing). The facet says this host serves Notes, with a factory that declares the object's parts: a document, and a method writing it in a transaction. load publishes the facet. get returns a reference once the object is live; its members are the contract's.

    @@ -216,7 +219,40 @@ The facet says this host serves what it asked to open) and then serves calls. That is why the same factory can run again when the code changes.

    -
    II

    Several Hosts

    A server, a worker per project, browser tabs: how they find each other's objects.

    +
    II

    An Object's Parts

    Documents, transactions, computed values, watching, and outside resources: the vocabulary of every factory.

    +

    A factory builds an object from a handful of parts, each declared with one call on self. The notes object grows to +show all of them.

    +

    Documents hold the object's data: self.state(NotesDoc) gives a handle whose .value is the current data. Data +changes only in a transaction: self.tx(fn, ctx) runs fn with tx, and await tx.draft(handle) gives a +mutable copy of the document's data; whatever you change on it is committed at once when fn returns, or not at all +if it throws.

    +
    add: async ({ text }, ctx) => self.tx(async (tx) => {
+	const draft = await tx.draft(doc);          // the document's data, writable inside this transaction only
+	draft.items.push(text);                      // recorded as a change, committed when the function returns
+}, ctx),
    +

    A derive is a value computed from other values, recomputed whenever they change. It is how an object offers +something it does not store:

    +
    const count = self.derive([doc], ([notes]) => notes.items.length);   // 0, 1, 2, … as notes are added
+return { list: doc, count, add };                                      // `count` can be a member like `list`
    +

    Watching is how anyone follows a value as it changes: a document handle, a derive, a reference's member. watch +gives the current value, then one call per change:

    +
    const watch = await notes.count.watch(ctx);
+watch.start(async (call) => {
+	if (call.kind === "value") console.log("notes:", call.value);   // "failed" calls say a value cannot be computed now
+});
    +

    A routine is the object watching on its own behalf: self.on(source, fn) runs fn with each new value, for as +long as the object lives (a reconciler: "whenever the notes change, rebuild the export file").

    +

    A native is something outside Chord the object opens and must close: a file watcher, a database connection, a +child process. self.open(open, close) opens it while the object activates and closes it when the object ends:

    +
    const inbox = self.open(async (ctx) => watchDirectory(inboxPath, ctx.abortSignal), (watcher) => watcher.close());
+self.onLive(async (ctx) => {                                  // a routine started once the object is live
+	for await (const file of inbox.value.files(ctx.abortSignal)) await importNote(file, ctx);
+});
    +

    Natives are why some objects restart when their plugin changes: an open connection cannot be moved to new code, so +Chord closes it and opens it again. Objects without natives are swapped in place.

    +
    + +
    III

    Several Hosts

    A server, a worker per project, browser tabs: how they find each other's objects.

    Now give each project its own process. Declare the kinds of addressable host, and make notes a remote type that lives on its project's worker:

    export const Server = hostKind("server");
@@ -228,7 +264,10 @@ lives on its project's worker:
    
 	hostAddress: ({ project }) => Project({ id: project }),   // where an object of these params lives
 	variants: { editor: { state: { list: NoteList }, methods: { add: method(obj({ text: str() })) } } },
 });
    -

    hostAddress is the whole routing story. A tab asks for Notes { project: "p1" }. Its host computes the address, +

    Two things are new. A remote type can be used from any host, and it has variants: different sets of members +for different callers, chosen by the factory (an editor variant with add, a reader variant for guests without +it). And hostAddress says where its objects live; it is the whole routing story.

    +

    A tab asks for Notes { project: "p1" }. Its host computes the address, Project(p1), and sends the request to its parent, the server. The server has a route for Project: it starts (or finds) the worker for p1 and passes the request on. The worker serves Notes. The tab gets a reference whose state follows the worker's document, one frame per change.

    @@ -252,7 +291,7 @@ this wiring for you: startApp(...) and startWorker().<
    -
    III

    Client Hosts

    Browser tabs and terminal UIs are hosts too: what they can do, and what they cannot.

    +
    IV

    Client Hosts

    Browser tabs and terminal UIs are hosts too: what they can do, and what they cannot.

    A browser tab and a terminal UI are hosts like the server, with one difference: they have no address. Nobody can send a request to a tab; a tab only connects to the server and asks for things. That is a deliberate rule: a tab runs on a device the server does not control, so nothing on the server ever waits for, or trusts, a tab's state.

    @@ -285,7 +324,7 @@ type, be routed to, or keep data others rely on. Data that must outlive a tab be worker.

    -
    IV

    Plugins

    Move the code into a directory that can be added, edited and removed while the app runs.

    +
    V

    Plugins

    Move the code into a directory that can be added, edited and removed while the app runs.

    So far the facets were part of the app's own code. A plugin is the same code in its own directory:

    plugins/notes/
   package.json        { "name": "@app/notes" }
@@ -311,7 +350,7 @@ out; every host passes its own copy in). It holds what plugins share and never r
 type, a defineFacet bound to it.
    -
    V

    Loading Plugins

    Discover, build, evaluate, load: what happens between a directory and a running object.

    +
    VI

    Loading Plugins

    Discover, build, evaluate, load: what happens between a directory and a running object.

    The plugins component does the work. You tell it where plugins are and what each host kind needs:

    const plugins = createPlugins({
 	roots: ["./plugins"],
@@ -342,7 +381,7 @@ fails is left out and listed in report.failed; the others load. Its
 
    A worker never builds: the server builds the worker facets too, and each worker evaluates the scripts it is sent.
    -
    VI

    Plugins Using Plugins

    Two ways one plugin uses another, and the one it may not.

    +
    VII

    Plugins Using Plugins

    Two ways one plugin uses another, and the one it may not.

    A search plugin wants to search the notes. It imports the contract and asks for the object:

    // plugins/search/worker.ts
 import { Notes } from "../notes/contract.ts";   // copied into search's script: contracts only
@@ -362,7 +401,7 @@ defines Panels, and every plugin with a UI contributes one:
    
 object's state. That rule is what lets the notes plugin change its storage without breaking search.
    -
    VII

    Changing a Plugin

    Edit, build, roll out: how new code takes over while every tab stays connected.

    +
    VIII

    Changing a Plugin

    Edit, build, roll out: how new code takes over while every tab stays connected.

    You change the notes plugin: notes now have a timestamp. The watcher sees the saved file and marks the plugin dirty. You (or an agent) build and roll out:

    const built = await plugins.build(plugins.state.value.dirty, ctx);

From fe6211607be8fe1c648a352ef5d143595fa7bc01 Mon Sep 17 00:00:00 2001
From: Mario Zechner 
Date: Sat, 31 Oct 2026 14:09:23 +0000
Subject: [PATCH 086/175] docs(chord): tutorial: persisted documents and tables

---
 .../docs/next/guide-plugins-hosts/book.md     | 30 +++++++++++++++++--
 .../docs/next/guide-plugins-hosts/index.html  | 25 ++++++++++++++--
 2 files changed, 51 insertions(+), 4 deletions(-)

diff --git a/packages/chord/docs/next/guide-plugins-hosts/book.md b/packages/chord/docs/next/guide-plugins-hosts/book.md
index d15c8a892..c49bc32a9 100644
--- a/packages/chord/docs/next/guide-plugins-hosts/book.md
+++ b/packages/chord/docs/next/guide-plugins-hosts/book.md
@@ -75,7 +75,7 @@ addressable hosts; an **address** (`Session({ id: "p1" })`) names one. An **obje
 on its own host only) or `remote` (used from any host). An **object** is one instance of a type, per params. A
 **facet** is a plugin's declarations for one host kind. A **plugin** is a directory with a `package.json`. A
 **reference** is what you get when you ask for an object (`host.get`, `self.get`, `ctx.get`). A **document** is data an object
-stores, changed in **transactions**; a **derive** is a value computed from others; a **native** is an outside
+stores, changed in **transactions**; a **table** holds many records with queries, in the host's **store**; a **derive** is a value computed from others; a **native** is an outside
 resource an object opens and closes. A **registry** is an
 extension point: one plugin defines it, others contribute. A **rollout** applies a build to the hosts in order.
 
@@ -130,7 +130,7 @@ The **facet** says this host serves
 The factory only declares: it is synchronous and opens nothing. Chord activates the object (loads its documents, opens
 what it asked to open) and then serves calls. That is why the same factory can run again when the code changes.
 
-# An Object's Parts {#parts lede="Documents, transactions, computed values, watching, and outside resources: the vocabulary of every factory."}
+# An Object's Parts {#parts lede="Documents, tables, transactions, computed values, watching, and outside resources: the vocabulary of every factory."}
 
 A factory builds an object from a handful of parts, each declared with one call on `self`. The notes object grows to
 show all of them.
@@ -147,6 +147,32 @@ add: async ({ text }, ctx) => self.tx(async (tx) => {
 }, ctx),
 ```
 
+Documents live in memory unless they are **persisted**: `defineState(..., { persist: true, checkpointWhen })` keeps
+one in the host's **store** (a SQLite file the app opens and gives to `createHost({ store })`), loaded when the
+object activates. For many records with queries, an object declares a **table**:
+
+```ts
+const NoteRows = defineTable("notes", {
+	version: 1,
+	columns: { id: str(), text: str(), at: int() },
+	key: ["id"],
+	indexes: { by_time: ["at"] },
+});
+
+host.serve(NoteStore, (_params, self) => {               // a local type with empty params: it may persist
+	const rows = self.table(NoteRows);                    // created (or migrated) when the object activates
+	const recent = query(NoteRows).orderBy((n) => n.at, "desc");
+	return {
+		add: async ({ text }, ctx) => self.tx(async (tx) => tx.put(rows, { id: uuid(), text, at: Date.now() }), ctx),
+		page: async ({ after }, ctx) => rows.page(recent, 50, ctx, { after }),   // 50 at a time, with a cursor
+	};
+});
+```
+
+Persisted data lives only in `local` objects with empty params (one per host and type), so its address never depends
+on who asked or with which params; an object with params (a view per project) reads from such a store. Writes go
+through the same transactions as documents: a note row and a document can change in one commit.
+
 A **derive** is a value computed from other values, recomputed whenever they change. It is how an object offers
 something it does not store:
 
diff --git a/packages/chord/docs/next/guide-plugins-hosts/index.html b/packages/chord/docs/next/guide-plugins-hosts/index.html
index 55719365f..ece1ed16c 100644
--- a/packages/chord/docs/next/guide-plugins-hosts/index.html
+++ b/packages/chord/docs/next/guide-plugins-hosts/index.html
@@ -168,7 +168,7 @@ addressable hosts; an address (Session({ id: "p1&quo
 on its own host only) or remote (used from any host). An object is one instance of a type, per params. A
 facet is a plugin's declarations for one host kind. A plugin is a directory with a package.json. A
 reference is what you get when you ask for an object (host.get, self.get, ctx.get). A document is data an object
-stores, changed in transactions; a derive is a value computed from others; a native is an outside
+stores, changed in transactions; a table holds many records with queries, in the host's store; a derive is a value computed from others; a native is an outside
 resource an object opens and closes. A registry is an
 extension point: one plugin defines it, others contribute. A rollout applies a build to the hosts in order.
    @@ -219,7 +219,7 @@ The facet says this host serves what it asked to open) and then serves calls. That is why the same factory can run again when the code changes.

    -
    II

    An Object's Parts

    Documents, transactions, computed values, watching, and outside resources: the vocabulary of every factory.

    +
    II

    An Object's Parts

    Documents, tables, transactions, computed values, watching, and outside resources: the vocabulary of every factory.

    A factory builds an object from a handful of parts, each declared with one call on self. The notes object grows to show all of them.

    Documents hold the object's data: self.state(NotesDoc) gives a handle whose .value is the current data. Data @@ -230,6 +230,27 @@ if it throws.

    const draft = await tx.draft(doc); // the document's data, writable inside this transaction only draft.items.push(text); // recorded as a change, committed when the function returns }, ctx),     +

    Documents live in memory unless they are persisted: defineState(..., { persist: true, checkpointWhen }) keeps +one in the host's store (a SQLite file the app opens and gives to createHost({ store })), loaded when the +object activates. For many records with queries, an object declares a table:

    +
    const NoteRows = defineTable("notes", {
+	version: 1,
+	columns: { id: str(), text: str(), at: int() },
+	key: ["id"],
+	indexes: { by_time: ["at"] },
+});
+
+host.serve(NoteStore, (_params, self) => {               // a local type with empty params: it may persist
+	const rows = self.table(NoteRows);                    // created (or migrated) when the object activates
+	const recent = query(NoteRows).orderBy((n) => n.at, "desc");
+	return {
+		add: async ({ text }, ctx) => self.tx(async (tx) => tx.put(rows, { id: uuid(), text, at: Date.now() }), ctx),
+		page: async ({ after }, ctx) => rows.page(recent, 50, ctx, { after }),   // 50 at a time, with a cursor
+	};
+});
    +

    Persisted data lives only in local objects with empty params (one per host and type), so its address never depends +on who asked or with which params; an object with params (a view per project) reads from such a store. Writes go +through the same transactions as documents: a note row and a document can change in one commit.

    A derive is a value computed from other values, recomputed whenever they change. It is how an object offers something it does not store:

    const count = self.derive([doc], ([notes]) => notes.items.length);   // 0, 1, 2, … as notes are added

From 86338abbd8ee1ebbb2fd10bfa277a35b940682c0 Mon Sep 17 00:00:00 2001
From: Mario Zechner 
Date: Sat, 31 Oct 2026 14:26:12 +0000
Subject: [PATCH 087/175] docs(chord): tutorial: loading explained (build vs
 evaluate, host packages, why plugins.load)

---
 .../docs/next/guide-plugins-hosts/book.md     | 145 +++++++++++++++---
 .../docs/next/guide-plugins-hosts/index.html  |  98 +++++++++---
 2 files changed, 199 insertions(+), 44 deletions(-)

diff --git a/packages/chord/docs/next/guide-plugins-hosts/book.md b/packages/chord/docs/next/guide-plugins-hosts/book.md
index c49bc32a9..563ba5315 100644
--- a/packages/chord/docs/next/guide-plugins-hosts/book.md
+++ b/packages/chord/docs/next/guide-plugins-hosts/book.md
@@ -348,13 +348,48 @@ export default defineFacet((host) => {
 out; every host passes its own copy in). It holds what plugins share and never reloads: host kinds, the principal
 type, a `defineFacet` bound to it.
 
-# Loading Plugins {#loading lede="Discover, build, evaluate, load: what happens between a directory and a running object."}
+# Loading Plugins {#loading lede="Two steps in two places: building a plugin into a script happens once, on the server; evaluating it happens in every process that runs it."}
 
-The **plugins component** does the work. You tell it where plugins are and what each host kind needs:
+A plugin is source code on the server's disk: TypeScript files that import each other, import other plugins'
+contracts, import npm packages, and import things the app itself provides. None of the processes that run the plugin
+can use those files as they are. A browser tab has no file system and no TypeScript; a worker could read the files,
+but then every worker would compile every plugin again, and two workers started a minute apart could see two different
+versions. So loading a plugin is two separate steps:
+
+1. **Build**: turn each plugin facet (`notes/worker.ts`, `notes/web.tsx`) into one self-contained **script**. This
+   happens once per edit, on the server, for every host kind.
+2. **Evaluate**: run a script inside one process, which gives back the facet, ready for `host.load`. This happens in
+   every process that hosts that kind: the server for its own facets, each worker for the worker facets, each tab for
+   the web facets.
+
+The two steps need different things, and that is why the API has two calls.
+
+## Host packages: what a script leaves out
+
+Look at what `notes/worker.ts` imports:
+
+```ts
+import { defineFacet, Session } from "@app/core";             // the app's base code
+import { obj, str, remote } from "@earendil-works/chord/next";  // Chord itself
+import { Notes } from "./contract.ts";                            // the plugin's own file
+import slugify from "slugify";                                    // an npm package in the plugin's dependencies
+```
+
+The last two go into the script: the plugin's own files and its npm dependencies are copied in. The first two must
+not. If the script carried its own copy of Chord, the worker would run two Chords: the plugin's `ChordError` would not
+be the worker's `ChordError`, and the plugin's `defineFacet` would make a facet the worker's host does not recognize.
+The same holds for `@app/core`: the plugin must see the very `Session` host kind and the very `Panels` registry the
+process already has, not a copy.
+
+Packages the process already has, and that every plugin must share, are its **host packages**. A script leaves them
+out and asks for them by name when it runs. Building therefore needs only their **names**: "do not copy
+`@earendil-works/chord/next` or `@app/core` into this script".
+
+## Building: `createPlugins`, on the server
 
 ```ts
 const plugins = createPlugins({
-	roots: ["./plugins"],
+	roots: ["./plugins"],                                 // where the plugin directories are
 	kinds: {
 		server: { platform: "node", hostPackages: ["@earendil-works/chord/next", "@app/core"] },
 		worker: { platform: "node", hostPackages: ["@earendil-works/chord/next", "@app/core"] },
@@ -362,32 +397,92 @@ const plugins = createPlugins({
 	},
 	onError,
 }, ctx);
-
-const report = await plugins.load(server, {
-	kind: "server",
-	core: [{ name: "@app/core", facet: coreServerFacet }],
-	modules: { "@earendil-works/chord/next": chord, "@app/core": core },
-}, ctx);
-console.log(report.failed);   // plugins left out, with why: [] when all loaded
 ```
 
-:::steps | What `plugins.load` does
-+++ Discover
-It lists `./plugins` one level deep and reads each `package.json`. A directory without one, or with two `server`
-files, is reported, not loaded.
-+++ Build
-For each plugin with a `server` facet, esbuild builds one script from `server.ts` and everything it imports: the
-plugin's own files, another plugin's contract files, its npm dependencies. Host packages stay out. A syntax error is a
-build failure, never a crash.
-+++ Evaluate
-The script runs and returns the facet. Its `import "@app/core"` gets the server's own `@app/core`, so a `ChordError`
-from a plugin is the same class as the server's.
-+++ Load
-`host.load` runs every facet's setup and publishes all of them at once. A plugin whose build, evaluation or setup
-fails is left out and listed in `report.failed`; the others load. Its objects' users wait, and the next edit loads it.
+`createPlugins` is the builder, and it builds for every kind, because only the server has the sources. For each kind
+it needs two facts. The **platform** says where the scripts run: a `node` script may import `node:fs`, a `browser`
+script may import a stylesheet but not `node:fs`. The **host package names** say what to leave out; they differ per
+kind, because a tab also shares React with its plugins and a worker does not. Nothing here runs a plugin.
+
+## Evaluating: `plugins.load`, in the process that hosts
+
+Evaluating is where the names become things. When the script runs and asks for `@app/core`, someone has to hand it
+the actual module, the one this process imported when it started. Only the process itself has it:
+
+```ts
+// server.ts
+import * as chord from "@earendil-works/chord/next";
+import * as core from "./core.ts";                         // the app's base code, the same module plugins import as "@app/core"
+
+const report = await plugins.load(server, {
+	kind: "server",                                          // evaluate the server kind's scripts
+	modules: { "@earendil-works/chord/next": chord, "@app/core": core },   // this process's own instances
+	core: [{ name: "@app/core", facet: core.serverFacet }],  // base-code facets, loaded beside the plugins
+}, ctx);
+```
+
+`modules` is this process's answer to every host package name. The server passes its own; a worker passes its own in
+`workerMain`; a page passes its own in `startPage`. Each process has different instances, which is exactly the point:
+inside each process, plugins share that process's Chord and base code.
+
+`core` is the app's base code in facet form. It is a facet like a plugin's, but it is not a plugin: it is made in the
+process's own code, often from values only this process has (`core.workerFacet(start)` closes over a worker's start
+data and store), and it never reloads, because it defines what all plugins share. Passing it here loads it together
+with the plugins, so they are published in one step.
+
+## Why `plugins.load(host)`, not `host.load`
+
+A host has its own `load`, and `plugins.load` calls it. The host's `load` takes facets that already exist: functions
+in memory. It knows nothing about directories, builds, scripts or modules, on purpose: the same host runs in a test
+with facets imported directly, or in a Cloudflare Durable Object where plugins are bundled at deploy time and nothing
+is built or evaluated at runtime.
+
+The plugins component is the part that knows about directories. It finds the plugins, builds them, evaluates them
+with the process's modules, and then calls `host.load` with the result. It also remembers, per host, which facet each
+plugin runs, so that a later reload can hand the host its complete new plugin set, with unchanged plugins as the very
+same objects. That memory is why the host is passed to it: `plugins.load(server, …)` on the server,
+`plugins.load(worker, …)` in a worker (where `workerMain` gives it a `plugins` that only evaluates), the same call
+in every process.
+
+:::flow wide | Building on the server needs names; evaluating in each process needs that process's modules.
+row "Server: build (once per edit)" accent
+box src "plugin sources" "`notes/worker.ts`, imports `@app/core`" muted
+box build "createPlugins" "copies own files and npm deps; leaves host packages out" accent
+box script "worker script" "`require(\"@app/core\")` left as a name"
+row "Each worker: evaluate" blue
+box mods "the worker's modules" "its own `@app/core`, its own Chord" muted
+box load "plugins.load(worker, …)" "runs the script, answers each name" blue
+box facet "the facet" "uses the worker's `Session`, the worker's `defineFacet`"
+arrow src -> build
+arrow build -> script
+arrow script -> load "shipped"
+arrow mods -> load
+arrow load -> facet
 :::
 
-A worker never builds: the server builds the worker facets too, and each worker evaluates the scripts it is sent.
+## What happens, step by step
+
+:::steps | `plugins.load(server, { kind: "server", … })`
++++ Discover
+The component lists `./plugins` one level deep and reads each `package.json`. Its `name` is the plugin's name; the
+files named after a kind are its facets. A directory without a `package.json`, or with two `server` files, is reported,
+not loaded.
++++ Build
+For each plugin with a `server` facet it builds one script, if no current one exists. A syntax error or an import the
+rules do not allow (a path into base code, a package not in the plugin's dependencies) is a failed build with the file
+and line, never a crash.
++++ Evaluate
+The script runs. Each host package name it asks for is answered from `modules`; anything else it may not have. The
+script's default export must be a facet.
++++ Load
+`host.load` runs the setups of `core` and of every plugin facet and publishes them together. A plugin whose build,
+evaluation or setup failed is left out and listed in `report.failed` with the reason; the others load. Objects that
+need it wait until a later edit loads it.
+:::
+
+A worker never builds. The server builds the worker kind too and sends the scripts in the worker's start message; the
+worker only evaluates, with its own modules. A page gets its scripts from the server over HTTP and evaluates them with
+the modules of its base bundle.
 
 # Plugins Using Plugins {#together lede="Two ways one plugin uses another, and the one it may not."}
 
diff --git a/packages/chord/docs/next/guide-plugins-hosts/index.html b/packages/chord/docs/next/guide-plugins-hosts/index.html
index ece1ed16c..a62ac6175 100644
--- a/packages/chord/docs/next/guide-plugins-hosts/index.html
+++ b/packages/chord/docs/next/guide-plugins-hosts/index.html
@@ -371,35 +371,95 @@ out; every host passes its own copy in). It holds what plugins share and never r
 type, a defineFacet bound to it.
    -
    VI

    Loading Plugins

    Discover, build, evaluate, load: what happens between a directory and a running object.

    -

    The plugins component does the work. You tell it where plugins are and what each host kind needs:

    +
    VI

    Loading Plugins

    Two steps in two places: building a plugin into a script happens once, on the server; evaluating it happens in every process that runs it.

    +

    A plugin is source code on the server's disk: TypeScript files that import each other, import other plugins' +contracts, import npm packages, and import things the app itself provides. None of the processes that run the plugin +can use those files as they are. A browser tab has no file system and no TypeScript; a worker could read the files, +but then every worker would compile every plugin again, and two workers started a minute apart could see two different +versions. So loading a plugin is two separate steps:

    +
      +
    1. Build: turn each plugin facet (notes/worker.ts, notes/web.tsx) into one self-contained script. This +happens once per edit, on the server, for every host kind.
      2. +
    2. Evaluate: run a script inside one process, which gives back the facet, ready for host.load. This happens in +every process that hosts that kind: the server for its own facets, each worker for the worker facets, each tab for +the web facets.
      3. +
    +

    The two steps need different things, and that is why the API has two calls.

    +

    Host packages: what a script leaves out

    +

    Look at what notes/worker.ts imports:

    +
    import { defineFacet, Session } from "@app/core";             // the app's base code
+import { obj, str, remote } from "@earendil-works/chord/next";  // Chord itself
+import { Notes } from "./contract.ts";                            // the plugin's own file
+import slugify from "slugify";                                    // an npm package in the plugin's dependencies
    +

    The last two go into the script: the plugin's own files and its npm dependencies are copied in. The first two must +not. If the script carried its own copy of Chord, the worker would run two Chords: the plugin's ChordError would not +be the worker's ChordError, and the plugin's defineFacet would make a facet the worker's host does not recognize. +The same holds for @app/core: the plugin must see the very Session host kind and the very Panels registry the +process already has, not a copy.

    +

    Packages the process already has, and that every plugin must share, are its host packages. A script leaves them +out and asks for them by name when it runs. Building therefore needs only their names: "do not copy +@earendil-works/chord/next or @app/core into this script".

    +

    Building: createPlugins, on the server

    const plugins = createPlugins({
-	roots: ["./plugins"],
+	roots: ["./plugins"],                                 // where the plugin directories are
 	kinds: {
 		server: { platform: "node", hostPackages: ["@earendil-works/chord/next", "@app/core"] },
 		worker: { platform: "node", hostPackages: ["@earendil-works/chord/next", "@app/core"] },
 		web: { platform: "browser", hostPackages: ["@earendil-works/chord/next", "@app/core", "react", "react/jsx-runtime"] },
 	},
 	onError,
-}, ctx);
+}, ctx);
    +

    createPlugins is the builder, and it builds for every kind, because only the server has the sources. For each kind +it needs two facts. The platform says where the scripts run: a node script may import node:fs, a browser +script may import a stylesheet but not node:fs. The host package names say what to leave out; they differ per +kind, because a tab also shares React with its plugins and a worker does not. Nothing here runs a plugin.

    +

    Evaluating: plugins.load, in the process that hosts

    +

    Evaluating is where the names become things. When the script runs and asks for @app/core, someone has to hand it +the actual module, the one this process imported when it started. Only the process itself has it:

    +
    // server.ts
+import * as chord from "@earendil-works/chord/next";
+import * as core from "./core.ts";                         // the app's base code, the same module plugins import as "@app/core"
 
 const report = await plugins.load(server, {
-	kind: "server",
-	core: [{ name: "@app/core", facet: coreServerFacet }],
-	modules: { "@earendil-works/chord/next": chord, "@app/core": core },
-}, ctx);
-console.log(report.failed);   // plugins left out, with why: [] when all loaded
    -
    What plugins.load does

    Discover

    It lists ./plugins one level deep and reads each package.json. A directory without one, or with two server -files, is reported, not loaded.

    -

    Build

    For each plugin with a server facet, esbuild builds one script from server.ts and everything it imports: the -plugin's own files, another plugin's contract files, its npm dependencies. Host packages stay out. A syntax error is a -build failure, never a crash.

    -

    Evaluate

    The script runs and returns the facet. Its import "@app/core" gets the server's own @app/core, so a ChordError -from a plugin is the same class as the server's.

    -

    Load

    host.load runs every facet's setup and publishes all of them at once. A plugin whose build, evaluation or setup -fails is left out and listed in report.failed; the others load. Its objects' users wait, and the next edit loads it.

    + kind: "server", // evaluate the server kind's scripts + modules: { "@earendil-works/chord/next": chord, "@app/core": core }, // this process's own instances + core: [{ name: "@app/core", facet: core.serverFacet }], // base-code facets, loaded beside the plugins +}, ctx);     +

    modules is this process's answer to every host package name. The server passes its own; a worker passes its own in +workerMain; a page passes its own in startPage. Each process has different instances, which is exactly the point: +inside each process, plugins share that process's Chord and base code.

    +

    core is the app's base code in facet form. It is a facet like a plugin's, but it is not a plugin: it is made in the +process's own code, often from values only this process has (core.workerFacet(start) closes over a worker's start +data and store), and it never reloads, because it defines what all plugins share. Passing it here loads it together +with the plugins, so they are published in one step.

    +

    Why plugins.load(host), not host.load

    +

    A host has its own load, and plugins.load calls it. The host's load takes facets that already exist: functions +in memory. It knows nothing about directories, builds, scripts or modules, on purpose: the same host runs in a test +with facets imported directly, or in a Cloudflare Durable Object where plugins are bundled at deploy time and nothing +is built or evaluated at runtime.

    +

    The plugins component is the part that knows about directories. It finds the plugins, builds them, evaluates them +with the process's modules, and then calls host.load with the result. It also remembers, per host, which facet each +plugin runs, so that a later reload can hand the host its complete new plugin set, with unchanged plugins as the very +same objects. That memory is why the host is passed to it: plugins.load(server, …) on the server, +plugins.load(worker, …) in a worker (where workerMain gives it a plugins that only evaluates), the same call +in every process.

    +
    SERVER: BUILD (ONCE PER EDIT)EACH WORKER: EVALUATEplugin sourcesnotes/worker.ts`, imports`@app/corecreatePluginscopies own files and npm deps; leaveshost packages outworker script`require("@app/core")` left as a namethe worker's modulesits own `@app/core`, its own Chordplugins.load(worker, …)runs the script, answers each namethe facetuses the worker's `Session`, theworker's `defineFacet`shipped
    Figure 3. Building on the server needs names; evaluating in each process needs that process's modules.
    +

    What happens, step by step

    +
    plugins.load(server, { kind: "server", … })

    Discover

    The component lists ./plugins one level deep and reads each package.json. Its name is the plugin's name; the +files named after a kind are its facets. A directory without a package.json, or with two server files, is reported, +not loaded.

    +

    Build

    For each plugin with a server facet it builds one script, if no current one exists. A syntax error or an import the +rules do not allow (a path into base code, a package not in the plugin's dependencies) is a failed build with the file +and line, never a crash.

    +

    Evaluate

    The script runs. Each host package name it asks for is answered from modules; anything else it may not have. The +script's default export must be a facet.

    +

    Load

    host.load runs the setups of core and of every plugin facet and publishes them together. A plugin whose build, +evaluation or setup failed is left out and listed in report.failed with the reason; the others load. Objects that +need it wait until a later edit loads it.

    -

    A worker never builds: the server builds the worker facets too, and each worker evaluates the scripts it is sent.

    +

    A worker never builds. The server builds the worker kind too and sends the scripts in the worker's start message; the +worker only evaluates, with its own modules. A page gets its scripts from the server over HTTP and evaluates them with +the modules of its base bundle.

    VII

    Plugins Using Plugins

    Two ways one plugin uses another, and the one it may not.

    From bb36171aac46adee2b43030fb1e4d8fd55675121 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 14:28:28 +0000 Subject: [PATCH 088/175] docs(chord): tutorial: only the server builds --- packages/chord/docs/next/guide-plugins-hosts/book.md | 6 +++--- packages/chord/docs/next/guide-plugins-hosts/index.html | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/packages/chord/docs/next/guide-plugins-hosts/book.md b/packages/chord/docs/next/guide-plugins-hosts/book.md index 563ba5315..81dea645f 100644 --- a/packages/chord/docs/next/guide-plugins-hosts/book.md +++ b/packages/chord/docs/next/guide-plugins-hosts/book.md @@ -480,9 +480,9 @@ evaluation or setup failed is left out and listed in `report.failed` with the re need it wait until a later edit loads it. ::: -A worker never builds. The server builds the worker kind too and sends the scripts in the worker's start message; the -worker only evaluates, with its own modules. A page gets its scripts from the server over HTTP and evaluates them with -the modules of its base bundle. +Only the server builds. Workers and clients (browser tabs, terminal UIs) only evaluate: the server builds their kinds +too, a worker gets its scripts in its start message, a tab or a TUI gets them from the server over HTTP, and each +evaluates them with its own modules (a worker's from `workerMain`, a page's from its base bundle). # Plugins Using Plugins {#together lede="Two ways one plugin uses another, and the one it may not."} diff --git a/packages/chord/docs/next/guide-plugins-hosts/index.html b/packages/chord/docs/next/guide-plugins-hosts/index.html index a62ac6175..efff61f9f 100644 --- a/packages/chord/docs/next/guide-plugins-hosts/index.html +++ b/packages/chord/docs/next/guide-plugins-hosts/index.html @@ -457,9 +457,9 @@ script's default export must be a facet.

    evaluation or setup failed is left out and listed in report.failed with the reason; the others load. Objects that need it wait until a later edit loads it.

    -

    A worker never builds. The server builds the worker kind too and sends the scripts in the worker's start message; the -worker only evaluates, with its own modules. A page gets its scripts from the server over HTTP and evaluates them with -the modules of its base bundle.

    +

    Only the server builds. Workers and clients (browser tabs, terminal UIs) only evaluate: the server builds their kinds +too, a worker gets its scripts in its start message, a tab or a TUI gets them from the server over HTTP, and each +evaluates them with its own modules (a worker's from workerMain, a page's from its base bundle).

    VII

    Plugins Using Plugins

    Two ways one plugin uses another, and the one it may not.

    From 112a8b46f35944b57b9c5192a7408ed1290b48cd Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 14:43:24 +0000 Subject: [PATCH 089/175] docs(chord): tutorial: base code vs plugins, shipped plugins, subpaths --- .../docs/next/guide-plugins-hosts/book.md | 53 +++++++++++++++++-- .../docs/next/guide-plugins-hosts/index.html | 41 +++++++++++--- 2 files changed, 83 insertions(+), 11 deletions(-) diff --git a/packages/chord/docs/next/guide-plugins-hosts/book.md b/packages/chord/docs/next/guide-plugins-hosts/book.md index 81dea645f..fa146bc0b 100644 --- a/packages/chord/docs/next/guide-plugins-hosts/book.md +++ b/packages/chord/docs/next/guide-plugins-hosts/book.md @@ -499,6 +499,13 @@ host.serve(Search, ({ project }, principal, self) => { }); ``` +:::review | Proposed: importing another plugin by name +Today another plugin's file is imported by relative path, which works only when both plugins sit in the same root. A +plugin in the user's root cannot reach one the app ships in `src/plugins` that way. Proposed (not decided): import by +plugin name, `import { Notes } from "@app/notes/contract"`, resolved to the discovered plugin of that name wherever its +directory is. +::: + `self.get` in a factory declares a dependency: Chord makes `Search` live only once `Notes` is, and if the notes plugin is missing, `Search` waits with a clear reason ("waiting for notes: not served here") instead of failing. @@ -542,14 +549,50 @@ A change to a contract (a new field in `Notes`'s state) gets a new `version`. A working if the provider maps the old version (`accepts`); otherwise it waits with "version mismatch" until it reloads. ::: -# Base Code and Plugins {.epilogue lede="What reloads, what restarts, and where the line is."} +# Base Code and Plugins {#base lede="What goes into the app itself, what goes into plugins, and how an app ships plugins of its own."} -Everything in `plugins/` reloads: edit, build, roll out. Everything in your base code (`src/`, `@app/core`: host -kinds, principals, the core facet, the app's startup) changes only with a restart, and the watcher tells you so. -Keep base code small: host kinds, principals, and what every plugin needs. Put features in plugins. +Every app has two kinds of code, and the line between them decides what can change while the app runs. + +**Base code** is the app's own program: `src/`, started by `node server.ts`. It is what every plugin shares: the host +kinds (`Server`, `Project`), the principal type, the app's `defineFacet`, the registries plugins contribute to +(`Panels`), and the startup that creates hosts and loads plugins. Plugins import it as a host package (`@app/core`), +so inside one process there is exactly one of each. Base code changes only with a restart: if it reloaded, the +process would hold two `Panels` registries, and old and new plugins would no longer meet. + +**Plugins** are features: notes, search, export, a terminal. They reload on an edit, a broken one is left out alone, +and an agent can write one while the app runs. + +The rule: **base code is what everything shares, not features.** A feature in base code cannot be edited live, and a +bug in it needs a restart. Keep base code to host kinds, principals, shared registries and startup. + +## Plugins the app ships + +An app usually comes with plugins of its own: notes, search, settings. They are still plugins, kept in a root inside +the app, next to the root where users and agents add theirs: + +```ts +const plugins = createPlugins({ + roots: ["./src/plugins", join(dataDir, "plugins")], // the app's own plugins, then the user's + kinds, + onError, +}, ctx); +``` + +They are built at startup like any plugin (a quarter of a second for twenty), reload like any plugin, and are left +out alone when broken. If two roots have a plugin with the same name, the first root wins, and the other is reported. + +## Subpaths of host packages + +A host package is matched by its exact name. If base code offers several entry points (`@app/core` for every host, +`@app/core/web` for pages), each is its own host package, listed by name for the build and given as a module when +evaluating, as `react` and `react/jsx-runtime` are. A plugin importing a subpath that is not listed gets a build +error naming it. + +## What the app kit will change The app kit being designed next (`chord/next/app`) will make the base code nearly empty: `startApp` and `startWorker` -with Chord's conventions for hosts, users, authentication and storage, so an app is mostly plugins. +with Chord's conventions for hosts, users, authentication and storage. An app is then its plugins, its own ones in +`src/plugins` and the user's beside them. # Reference {.appendix} diff --git a/packages/chord/docs/next/guide-plugins-hosts/index.html b/packages/chord/docs/next/guide-plugins-hosts/index.html index efff61f9f..0a5e1ca41 100644 --- a/packages/chord/docs/next/guide-plugins-hosts/index.html +++ b/packages/chord/docs/next/guide-plugins-hosts/index.html @@ -129,7 +129,7 @@ writes, a note says so.

  • VI. Loading Plugins
  • VII. Plugins Using Plugins
  • VIII. Changing a Plugin
    • -
  • Base Code and Plugins
    • +
  • IX. Base Code and Plugins
  • Appendix A: Reference
    • @@ -472,6 +472,11 @@ host.serve(Search, ({ projec const index = self.derive([notes.list], ([list]) => buildIndex(list.items)); return { variant: "main", members: { find: async ({ q }) => lookup(index.value, q) } }; });     +
    Proposed: importing another plugin by name

    Today another plugin's file is imported by relative path, which works only when both plugins sit in the same root. A +plugin in the user's root cannot reach one the app ships in src/plugins that way. Proposed (not decided): import by +plugin name, import { Notes } from "@app/notes/contract", resolved to the discovered plugin of that name wherever its +directory is.

    +

    self.get in a factory declares a dependency: Chord makes Search live only once Notes is, and if the notes plugin is missing, Search waits with a clear reason ("waiting for notes: not served here") instead of failing.

    The other way is a registry: an extension point one plugin defines and others contribute to. The app's core @@ -518,12 +523,36 @@ working if the provider maps the old version (accepts); otherwise i

    -
    ∴

    Base Code and Plugins

    What reloads, what restarts, and where the line is.

    -

    Everything in plugins/ reloads: edit, build, roll out. Everything in your base code (src/, @app/core: host -kinds, principals, the core facet, the app's startup) changes only with a restart, and the watcher tells you so. -Keep base code small: host kinds, principals, and what every plugin needs. Put features in plugins.

    +
    IX

    Base Code and Plugins

    What goes into the app itself, what goes into plugins, and how an app ships plugins of its own.

    +

    Every app has two kinds of code, and the line between them decides what can change while the app runs.

    +

    Base code is the app's own program: src/, started by node server.ts. It is what every plugin shares: the host +kinds (Server, Project), the principal type, the app's defineFacet, the registries plugins contribute to +(Panels), and the startup that creates hosts and loads plugins. Plugins import it as a host package (@app/core), +so inside one process there is exactly one of each. Base code changes only with a restart: if it reloaded, the +process would hold two Panels registries, and old and new plugins would no longer meet.

    +

    Plugins are features: notes, search, export, a terminal. They reload on an edit, a broken one is left out alone, +and an agent can write one while the app runs.

    +

    The rule: base code is what everything shares, not features. A feature in base code cannot be edited live, and a +bug in it needs a restart. Keep base code to host kinds, principals, shared registries and startup.

    +

    Plugins the app ships

    +

    An app usually comes with plugins of its own: notes, search, settings. They are still plugins, kept in a root inside +the app, next to the root where users and agents add theirs:

    +
    const plugins = createPlugins({
+	roots: ["./src/plugins", join(dataDir, "plugins")],   // the app's own plugins, then the user's
+	kinds,
+	onError,
+}, ctx);
    +

    They are built at startup like any plugin (a quarter of a second for twenty), reload like any plugin, and are left +out alone when broken. If two roots have a plugin with the same name, the first root wins, and the other is reported.

    +

    Subpaths of host packages

    +

    A host package is matched by its exact name. If base code offers several entry points (@app/core for every host, +@app/core/web for pages), each is its own host package, listed by name for the build and given as a module when +evaluating, as react and react/jsx-runtime are. A plugin importing a subpath that is not listed gets a build +error naming it.

    +

    What the app kit will change

    The app kit being designed next (chord/next/app) will make the base code nearly empty: startApp and startWorker -with Chord's conventions for hosts, users, authentication and storage, so an app is mostly plugins.

    +with Chord's conventions for hosts, users, authentication and storage. An app is then its plugins, its own ones in +src/plugins and the user's beside them.

    A

    Reference

    From 56cae782341c339d3bb820081f0f9425d8c5b1ec Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 14:46:28 +0000 Subject: [PATCH 090/175] docs(chord): plugins import each other by name --- packages/chord/docs/next/decisions.md | 7 +++++++ packages/chord/docs/next/guide-plugins-hosts/book.md | 12 +++++------- .../chord/docs/next/guide-plugins-hosts/index.html | 10 ++++------ 3 files changed, 16 insertions(+), 13 deletions(-) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 4c35c4f33..a2d98b44d 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2063,6 +2063,13 @@ validation and copies the v2 kernel added around Delta. and TUI, decided point by point; the slice 10 components are its inside. Inputs from the owner: user management is part of it ("user management will also need to be a thing we support"), and app shapes without workers (server and clients only: "a chat app doesn't need workers"). +- **Plugins import each other by name (owner, 2026-10-31: "i like it"; changes 7c's cross-plugin rule):** + `import { Notes } from "@app/notes/contract"` resolves to the discovered plugin named `@app/notes`, its file + `contract.ts`, wherever its directory is (any root), copied into the importer's script as before (contracts and + side-effect-free files only; they may import only host packages and files of their own plugin); a missing plugin is + a build error naming it; relative paths into another plugin's directory are refused. Owner's notes for the app kit + pass: third-party plugins need the imported plugins' sources or types for type checking (publishing each plugin to + npm works but is unpleasant); plugins should be testable on their own, with their dependencies mocked. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a diff --git a/packages/chord/docs/next/guide-plugins-hosts/book.md b/packages/chord/docs/next/guide-plugins-hosts/book.md index fa146bc0b..37388c890 100644 --- a/packages/chord/docs/next/guide-plugins-hosts/book.md +++ b/packages/chord/docs/next/guide-plugins-hosts/book.md @@ -490,7 +490,7 @@ A search plugin wants to search the notes. It **imports the contract** and asks ```ts // plugins/search/worker.ts -import { Notes } from "../notes/contract.ts"; // copied into search's script: contracts only +import { Notes } from "@app/notes/contract"; // another plugin, by its name: copied into search's script host.serve(Search, ({ project }, principal, self) => { const notes = self.get(Notes, { project }); // a dependency: Search goes live once Notes is @@ -499,12 +499,10 @@ host.serve(Search, ({ project }, principal, self) => { }); ``` -:::review | Proposed: importing another plugin by name -Today another plugin's file is imported by relative path, which works only when both plugins sit in the same root. A -plugin in the user's root cannot reach one the app ships in `src/plugins` that way. Proposed (not decided): import by -plugin name, `import { Notes } from "@app/notes/contract"`, resolved to the discovered plugin of that name wherever its -directory is. -::: + +A plugin imports another plugin by its **name**, never by a path: `@app/notes/contract` is the file `contract.ts` of +whichever discovered plugin is named `@app/notes`, in any root. The file is copied into the importer's script, so only +contracts and other side-effect-free files belong there; a missing plugin is a build error naming it. `self.get` in a factory declares a dependency: Chord makes `Search` live only once `Notes` is, and if the notes plugin is missing, `Search` waits with a clear reason ("waiting for notes: not served here") instead of failing. diff --git a/packages/chord/docs/next/guide-plugins-hosts/index.html b/packages/chord/docs/next/guide-plugins-hosts/index.html index 0a5e1ca41..9d2f815af 100644 --- a/packages/chord/docs/next/guide-plugins-hosts/index.html +++ b/packages/chord/docs/next/guide-plugins-hosts/index.html @@ -465,18 +465,16 @@ evaluates them with its own modules (a worker's from workerMain
    VII

    Plugins Using Plugins

    Two ways one plugin uses another, and the one it may not.

    A search plugin wants to search the notes. It imports the contract and asks for the object:

    // plugins/search/worker.ts
-import { Notes } from "../notes/contract.ts";   // copied into search's script: contracts only
+import { Notes } from "@app/notes/contract";    // another plugin, by its name: copied into search's script
 
 host.serve(Search, ({ project }, principal, self) => {
 	const notes = self.get(Notes, { project });   // a dependency: Search goes live once Notes is
 	const index = self.derive([notes.list], ([list]) => buildIndex(list.items));
 	return { variant: "main", members: { find: async ({ q }) => lookup(index.value, q) } };
 });
    -
    Proposed: importing another plugin by name

    Today another plugin's file is imported by relative path, which works only when both plugins sit in the same root. A -plugin in the user's root cannot reach one the app ships in src/plugins that way. Proposed (not decided): import by -plugin name, import { Notes } from "@app/notes/contract", resolved to the discovered plugin of that name wherever its -directory is.

    -
    +

    A plugin imports another plugin by its name, never by a path: @app/notes/contract is the file contract.ts of +whichever discovered plugin is named @app/notes, in any root. The file is copied into the importer's script, so only +contracts and other side-effect-free files belong there; a missing plugin is a build error naming it.

    self.get in a factory declares a dependency: Chord makes Search live only once Notes is, and if the notes plugin is missing, Search waits with a clear reason ("waiting for notes: not served here") instead of failing.

    The other way is a registry: an extension point one plugin defines and others contribute to. The app's core From ed2bdde64b0e62c5083ac2bc694baf541f5cd670 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 15:04:43 +0000 Subject: [PATCH 091/175] docs(chord): HTTP core as a fetch handler, Node listener as the easy path --- packages/chord/docs/next/decisions.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index a2d98b44d..84798b324 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2070,6 +2070,15 @@ validation and copies the v2 kernel added around Delta. a build error naming it; relative paths into another plugin's directory are refused. Owner's notes for the app kit pass: third-party plugins need the imported plugins' sources or types for type checking (publishing each plugin to npm works but is unpleasant); plugins should be testable on their own, with their dependencies mocked. +- **The HTTP core is a `fetch` handler; Chord's own Node server is the easy path (owner, 2026-10-31: "could http just plug + into hono/express?"; "we should still provide the easy path using just node's shit as well, for small apps or apps + where we don't care"):** the component builds one Web-standard handler, `(request: Request) => Promise`, + from `Http` (admission, the security baseline, router, wraps, descriptions), plus per-runtime upgrade adapters for + `/chord` and plugin sockets. The default is Chord's own listener, the native host-lifetime object of `chord.http` + over Node's `http` (decision "The HTTP server is a native object"); an app with its own server mounts the handler + (`hono.all("/*", (c) => http.fetch(c.req.raw))`, an Express adapter) and passes upgrades to the adapter. The same + routes then run on other runtimes later (Bun, Durable Objects, design §26 #16). Built in slice 10's fix round + (reworks `server.ts`, `request.ts`, `response.ts`; router, admission rules and descriptions stay). - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 705d047cb5635be5d430c34917be714979cc7709 Mon Sep 17 00:00:00 2001 From: Armin Ronacher Date: Sat, 31 Oct 2026 16:10:55 +0100 Subject: [PATCH 092/175] fix(coding-agent): tighten system prompt --- packages/coding-agent/CHANGELOG.md | 1 + packages/coding-agent/src/core/skills.ts | 2 +- packages/coding-agent/src/core/system-prompt.ts | 11 +++++------ packages/coding-agent/src/core/tools/edit.ts | 6 ++---- packages/coding-agent/test/system-prompt.test.ts | 10 +++++++++- 5 files changed, 18 insertions(+), 12 deletions(-) diff --git a/packages/coding-agent/CHANGELOG.md b/packages/coding-agent/CHANGELOG.md index d10174db2..483e87088 100644 --- a/packages/coding-agent/CHANGELOG.md +++ b/packages/coding-agent/CHANGELOG.md @@ -12,6 +12,7 @@ ### Changed +- Tightened system prompt - `pi update` for installs from the pi.dev installer no longer passes `--min-release-age=0` to `npm ci`. The release lockfile pins every package, so the flag had no effect, and security scanners that block it stopped the update ([#11262](https://github.com/earendil-works/pi/issues/11262)) - `/model` and `/scoped-models` now show a model's name next to its ID when the name adds words the ID lacks, such as a renamed `models.json` override ([#5916](https://github.com/earendil-works/pi/issues/5916), [#11347](https://github.com/earendil-works/pi/pull/11347)) - MCP OAuth sign-in now uses pi's Client ID Metadata Document by default when the authorization server supports it for public clients, and registers a client dynamically otherwise, like Codex, Claude Code, and opencode. Clients registered before stay in use while their redirect URI matches. The new default `oauth.clientRegistration: "auto"` registers when `clientName` or a non-default `callbackUrl` is set; `"dcr"` always registers. `pi mcp add --oauth-client-registration` sets the mode ([#11426](https://github.com/earendil-works/pi/issues/11426)) diff --git a/packages/coding-agent/src/core/skills.ts b/packages/coding-agent/src/core/skills.ts index 7440c4684..48df4536b 100644 --- a/packages/coding-agent/src/core/skills.ts +++ b/packages/coding-agent/src/core/skills.ts @@ -369,7 +369,7 @@ export function formatSkillsForPrompt(skills: Skill[], fileReadTool: "read" | "b : fileReadTool === "bash" ? "Use bash to load a skill's file when the task matches its description." : "Load a skill's file when the task matches its description.", - "When a skill file references a relative path, resolve it against the skill directory (parent of SKILL.md / dirname of the path) and use that absolute path in tool commands.", + "Resolve relative paths against the skill's directory (parent of SKILL.md). Use absolute paths in tool calls.", "", "", ]; diff --git a/packages/coding-agent/src/core/system-prompt.ts b/packages/coding-agent/src/core/system-prompt.ts index 49bcb58ee..e51e14fe4 100644 --- a/packages/coding-agent/src/core/system-prompt.ts +++ b/packages/coding-agent/src/core/system-prompt.ts @@ -153,20 +153,19 @@ export function buildSystemPromptSections(input: BuildSystemPromptOptions): Syst promptSections.preamble = customPrompt; } else { promptSections.preamble = - "You are an expert coding assistant operating inside pi, a coding agent harness. You help users by reading files, executing commands, editing code, and writing new files."; + "You are a coding assistant running inside pi, a coding agent harness. Help users inspect files, run commands, and modify code."; const visibleTools = declaredTools.filter((name) => !!toolSnippets[name]); const tools = visibleTools.length > 0 ? visibleTools.map((name) => `- ${name}: ${toolSnippets[name]}`).join("\n") : "(none)"; - promptSections.tools = `${tools}\n\nIn addition to the tools above, you may have access to other custom tools depending on the project.`; + promptSections.tools = `${tools}\n\nAdditional tools may be available.`; promptSections.rules = buildRules(declaredTools, toolGuidelines, promptGuidelines); - promptSections.docs = `Pi documentation (read only when the user asks about pi itself, its SDK, extensions, themes, skills, or TUI): + promptSections.docs = `Pi documentation (read only when the user asks about the harness itself, its SDK, extensions, themes, skills, or TUI): - Main documentation: ${getReadmePath()} - Additional docs: ${getDocsPath()} - Examples: ${getExamplesPath()} (extensions, custom tools, SDK) - When reading pi docs or examples, resolve docs/... under Additional docs and examples/... under Examples, not the current working directory -- When asked about: extensions (docs/extensions.md, examples/extensions/), themes (docs/themes.md), skills (docs/skills.md), prompt templates (docs/prompt-templates.md), TUI components (docs/tui.md), keybindings (docs/keybindings.md), SDK integrations (docs/sdk.md), custom providers (docs/custom-provider.md), adding models (docs/models.md), pi packages (docs/packages.md), environment variables (docs/environment-variables.md), MCP servers (docs/mcp.md), codemode scripts and non-LLM models such as classifiers and image models (docs/codemode.md) -- When working on pi topics, read the docs and examples, and follow .md cross-references before implementing -- Always read pi .md files completely and follow links to related docs (e.g., tui.md for TUI API details)`; +- When asked about: extensions (docs/extensions.md, examples/extensions/), themes (docs/themes.md), skills (docs/skills.md), prompt templates (docs/prompt-templates.md), TUI components (docs/tui.md), keybindings (docs/keybindings.md), SDK integrations (docs/sdk.md), custom providers (docs/custom-provider.md), adding models (docs/models.md), packages (docs/packages.md), environment variables (docs/environment-variables.md), MCP servers (docs/mcp.md), codemode scripts and non-LLM models such as classifiers and image models (docs/codemode.md) +- Before implementing harness changes, read the relevant docs and examples in full, following related cross-references.`; } if (appendSystemPrompt) promptSections.addendum = appendSystemPrompt; diff --git a/packages/coding-agent/src/core/tools/edit.ts b/packages/coding-agent/src/core/tools/edit.ts index 62e6fdee0..ad1cbd2f3 100644 --- a/packages/coding-agent/src/core/tools/edit.ts +++ b/packages/coding-agent/src/core/tools/edit.ts @@ -43,10 +43,8 @@ const editSchema = Type.Object( export const editToolSystemPromptContribution = { snippet: "Make precise file edits with exact text replacement, including multiple disjoint edits in one call", guidelines: [ - "Use edit for precise changes (edits[].oldText must match exactly)", - "When changing multiple separate locations in one file, use one edit call with multiple entries in edits[] instead of multiple edit calls", - "Each edits[].oldText is matched against the original file, not after earlier edits are applied. Do not emit overlapping or nested edits. Merge nearby changes into one edit.", - "Keep edits[].oldText as small as possible while still being unique in the file. Do not pad with large unchanged regions.", + "Use edit for targeted changes. Batch changes to one file in a single call.", + "Each edits[].oldText must exactly and uniquely match the original file. Keep matches short and non-overlapping; merge nearby changes without including large unchanged regions.", ], } as const; diff --git a/packages/coding-agent/test/system-prompt.test.ts b/packages/coding-agent/test/system-prompt.test.ts index 276cc63ff..86d2c6994 100644 --- a/packages/coding-agent/test/system-prompt.test.ts +++ b/packages/coding-agent/test/system-prompt.test.ts @@ -48,7 +48,9 @@ describe("buildSystemPrompt", () => { skills: [], }); - expect(defaultPrompt.startsWith("You are an expert coding assistant operating inside pi")).toBe(true); + expect(defaultPrompt.startsWith("You are a coding assistant running inside pi, a coding agent harness.")).toBe( + true, + ); expect(customPrompt.startsWith("You are Exact.\n\n")).toBe(true); }); @@ -119,6 +121,9 @@ describe("buildSystemPrompt", () => { "- When reading pi docs or examples, resolve docs/... under Additional docs and examples/... under Examples, not the current working directory", ); expect(prompt).toContain("environment variables (docs/environment-variables.md), MCP servers (docs/mcp.md)"); + expect(prompt).toContain( + "Before implementing harness changes, read the relevant docs and examples in full, following related cross-references.", + ); }); }); @@ -192,6 +197,9 @@ describe("buildSystemPrompt", () => { expect(prompt).toContain(""); expect(prompt).toContain("test-skill"); expect(prompt).toContain("Use bash to load a skill's file"); + expect(prompt).toContain( + "Resolve relative paths against the skill's directory (parent of SKILL.md). Use absolute paths in tool calls.", + ); }); test("omits skills without read or bash", () => { From 25426f72ae04a306f1e08de09583b4e89503b9e7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Petr=20Hou=C5=A1ka?= Date: Sat, 31 Oct 2026 16:15:42 +0100 Subject: [PATCH 093/175] docs(coding-agent): document Windows terminal and conhost limitations (#11644) Adds a "Choose a terminal" section to `packages/coding-agent/docs/windows.md` (https://pi.dev/docs/latest/windows), with two tables (last updated October 2026): - Terminals and which conhost they use: Windows Terminal (recommended), VS Code, WezTerm nightly (bundled, up to date); WezTerm 20240203 (bundled, old); Alacritty (built-in unless `conpty.dll` + `OpenConsole.exe` are provided); classic console window, mintty, ConEmu/Cmder, Windows OpenSSH (built-in). - Known problems by conhost type: - Built-in: crash when switching fullscreen -> regular or using `Ctrl+G` from fullscreen (#9186, microsoft/terminal#17515). - Built-in behind another terminal (mintty, OpenSSH): query replies leak into the prompt, BEL opens the external editor (#10256, #10362, #10742). - Old bundled (WezTerm 20240203): reply fragments in the prompt (#10717, #10970). - Built-in, classic console: reported regular-mode redraw issue (#6300). - Notes that the fixes exist upstream and reach the built-in conhost with a future Windows update, timing unknown. Sources: VS Code defaults `terminal.integrated.windowsUseConptyDll` to `true` (bundled conpty v1.25). Alacritty loads `conpty.dll` only if present (`alacritty_terminal/src/tty/windows/conpty.rs`) and its installer ships only `alacritty.exe`. The crash was verified locally in the classic console window on conhost 10.0.26100.9807: fullscreen -> regular crashed 3/3 runs and `Ctrl+G` from fullscreen crashed 2/2 runs (`0xC0000005`). --- packages/coding-agent/docs/windows.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/packages/coding-agent/docs/windows.md b/packages/coding-agent/docs/windows.md index 26f2f961c..18826cd02 100644 --- a/packages/coding-agent/docs/windows.md +++ b/packages/coding-agent/docs/windows.md @@ -76,6 +76,31 @@ JSON uses backslashes for escape sequences. When you write a Windows path with b See [Configure shell commands](shell-aliases.md) for command prefixes, aliases, and the complete shell-resolution behavior. +## Choose a terminal + +Windows terminals run Pi through a console host (conhost). Some bundle an up-to-date copy; others use the older one built into Windows, which has bugs that affect Pi. + +We recommend [Windows Terminal](https://aka.ms/terminal). It always ships an up-to-date conhost, and Pi is best tested with it. + +Last updated: October 2026. + +| Terminal | Conhost | Status | +|---|---|---| +| [Windows Terminal](https://aka.ms/terminal) | Bundled, current | Recommended | +| [VS Code](https://code.visualstudio.com), [Tabby](https://tabby.sh), [WezTerm](https://wezterm.org) nightly | Bundled, current | Works | +| [Alacritty](https://alacritty.org) | Built-in, unless `conpty.dll` and `OpenConsole.exe` from Windows Terminal are next to `alacritty.exe` | Works only with those files | +| WezTerm 20240203 and older | Bundled, old | Known problems | +| Classic console (`cmd.exe` or PowerShell without Windows Terminal), mintty (Git Bash), ConEmu, Cmder, [Hyper](https://hyper.is), Windows OpenSSH | Built-in | Known problems | + +| Problem | Where | Why | +|---|---|---| +| Switching from fullscreen to regular mode, or `Ctrl+G` from fullscreen, crashes the session ([#9186](https://github.com/earendil-works/pi/issues/9186), [microsoft/terminal#17515](https://github.com/microsoft/terminal/issues/17515)). Start with `--tui-mode regular` to avoid it. | Built-in | Conhost crashes when leaving the alternate screen while input is in line mode. | +| Text such as `;0;rgb:0000/0000/0000` appears in the prompt and BEL opens the external editor ([#10256](https://github.com/earendil-works/pi/issues/10256), [#10362](https://github.com/earendil-works/pi/issues/10362), [#10742](https://github.com/earendil-works/pi/issues/10742)) | Built-in behind another terminal (mintty, OpenSSH) | Conhost drops the start of terminal replies that arrive in pieces. | +| Fragments such as `79` or `55` appear in the prompt ([#10717](https://github.com/earendil-works/pi/issues/10717), [#10970](https://github.com/earendil-works/pi/issues/10970)) | WezTerm 20240203 | The bundled conhost cuts terminal replies short. | +| In regular mode, each keystroke redraws the input line on a new row ([#6300](https://github.com/earendil-works/pi/issues/6300)) | Built-in, classic console | Suspected: cursor wraps after full-width lines. | + +Microsoft has fixed these bugs in its source, but has not said when the built-in conhost will get the fixes. + ## Configure Windows Terminal Windows Terminal reserves or rewrites some modified keys. See [Windows Terminal](terminal-setup.md#windows-terminal) to configure `Shift+Enter` and `Alt+Enter`, and [Keybindings](keybindings.md) for Pi's Windows and WSL shortcut defaults. From f0646a00fb4c43bf941deb9d934280e5a92361c2 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 15:57:12 +0000 Subject: [PATCH 094/175] docs(chord): slice 10 fix round small points --- packages/chord/docs/next/decisions.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 84798b324..ec2672c24 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2079,6 +2079,11 @@ validation and copies the v2 kernel added around Delta. (`hono.all("/*", (c) => http.fetch(c.req.raw))`, an Express adapter) and passes upgrades to the adapter. The same routes then run on other runtimes later (Bun, Durable Objects, design §26 #16). Built in slice 10's fix round (reworks `server.ts`, `request.ts`, `response.ts`; router, admission rules and descriptions stay). +- **Slice 10 fix round, three small points (owner, 2026-10-31: "yes on fileResponse etc"):** (1) `fileResponse(path, { + root? })`: with a `root`, a file whose real path lies outside it (a symlink out) is refused 403; without one the path + is trusted (a plugin's own file). (2) The worker lock on macOS and the BSDs fails closed on a stale lock file, with an + error naming it (no takeover; Linux and Android use abstract sockets and are unaffected). (3) A route's overhead: + Opus's two trims, measured again after the `fetch` rework, and the numbers brought to the owner. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 6de14ec09862856efa3edd5a8b734b14ab1afd09 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Petr=20Hou=C5=A1ka?= Date: Sat, 31 Oct 2026 17:29:36 +0100 Subject: [PATCH 095/175] docs(coding-agent): note tmux 3.6 color reply leak in tmux guide (#11645) Adds a troubleshooting section to the tmux guide (`packages/coding-agent/docs/tmux.md`, published at https://pi.dev/docs/latest/tmux) for the color-code text that tmux 3.6 types into the editor at startup. tmux 3.6 forwards Pi's palette queries to the outer terminal. When a reply reaches tmux split across reads, tmux treats its bytes as key presses and sends them to Pi as typed text. tmux 3.7 fixes this ([tmux#4749](https://github.com/tmux/tmux/issues/4749), [tmux#4793](https://github.com/tmux/tmux/issues/4793)). Pi cannot filter the text because it arrives exactly like typing. The section also notes that switching to a fixed theme does not help: `applyFromSettings()` queries terminal colors regardless of the theme, which reporters confirmed in #10679 (`--use-theme dark`) and #10256 (`"theme": "dark"`). Related: #10250, #10679, #10256. --- packages/coding-agent/docs/tmux.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/packages/coding-agent/docs/tmux.md b/packages/coding-agent/docs/tmux.md index 091d61647..c781ffe0a 100644 --- a/packages/coding-agent/docs/tmux.md +++ b/packages/coding-agent/docs/tmux.md @@ -53,3 +53,7 @@ set -g extended-keys on Pi supports the xterm `modifyOtherKeys` format used by these versions. Restart tmux and repeat the verification steps. For older versions, upgrade tmux or use Pi outside tmux rather than relying on modified Enter shortcuts. + +## Color codes appear in the editor with tmux 3.6 + +If text such as `c0c/0c0c/0c0c` appears in the editor at startup, upgrade tmux to 3.7 or newer. tmux 3.6 sometimes sends the terminal's color replies to Pi as typed text ([tmux#4749](https://github.com/tmux/tmux/issues/4749)). Changing Pi's theme does not help. From b4c2e92acbecdab7a1ff1cc36bde92f3206f5ddd Mon Sep 17 00:00:00 2001 From: Armin Ronacher Date: Sat, 31 Oct 2026 17:54:29 +0100 Subject: [PATCH 096/175] fix: remove vulnerable dependency chains Replace build-time shell utilities with Node filesystem operations and retire the native sandbox example in favor of existing isolation options. Upgrade glob matching and worker-test dependencies and refresh the lockfiles. --- package-lock.json | 492 ++++-------------- package.json | 3 - packages/agent/package.json | 2 +- packages/ai/package.json | 4 +- packages/chord/package.json | 3 +- packages/codemode/package.json | 3 +- packages/coding-agent/CHANGELOG.md | 5 + packages/coding-agent/docs/extensions.md | 1 - .../examples/extensions/README.md | 3 +- .../examples/extensions/sandbox/.gitignore | 1 - .../examples/extensions/sandbox/index.ts | 321 ------------ .../extensions/sandbox/package-lock.json | 92 ---- .../examples/extensions/sandbox/package.json | 19 - .../install-lock/package-lock.json | 17 +- packages/coding-agent/package.json | 11 +- packages/durable/package.json | 5 +- packages/env/package.json | 3 +- packages/evals/package.json | 3 +- packages/mcp/package.json | 3 +- packages/telemetry/package.json | 2 +- packages/tui/package.json | 2 +- scripts/npm-audit.mjs | 10 +- scripts/package-files.mjs | 30 ++ scripts/package-files.test.mjs | 82 +++ 24 files changed, 260 insertions(+), 857 deletions(-) delete mode 100644 packages/coding-agent/examples/extensions/sandbox/.gitignore delete mode 100644 packages/coding-agent/examples/extensions/sandbox/index.ts delete mode 100644 packages/coding-agent/examples/extensions/sandbox/package-lock.json delete mode 100644 packages/coding-agent/examples/extensions/sandbox/package.json create mode 100644 scripts/package-files.mjs create mode 100644 scripts/package-files.test.mjs diff --git a/package-lock.json b/package-lock.json index 7c8fb2531..f2129fd85 100644 --- a/package-lock.json +++ b/package-lock.json @@ -13,18 +13,15 @@ "packages/coding-agent/examples/extensions/with-deps", "packages/coding-agent/examples/extensions/custom-provider-anthropic", "packages/coding-agent/examples/extensions/custom-provider-gitlab-duo", - "packages/coding-agent/examples/extensions/sandbox", "packages/coding-agent/examples/extensions/gondolin" ], "devDependencies": { - "@anthropic-ai/sandbox-runtime": "0.0.26", "@biomejs/biome": "2.3.5", "@biomejs/wasm-nodejs": "2.3.5", "@types/node": "22.19.19", "esbuild": "0.28.2", "husky": "9.1.7", "pnpm": "11.15.1", - "shx": "0.3.4", "typescript": "7.0.2", "yaml": "2.9.0" }, @@ -32,26 +29,6 @@ "node": ">=22.19.0" } }, - "node_modules/@anthropic-ai/sandbox-runtime": { - "version": "0.0.26", - "resolved": "https://registry.npmjs.org/@anthropic-ai/sandbox-runtime/-/sandbox-runtime-0.0.26.tgz", - "integrity": "sha512-DYV5LSsVMnzq0lbfaYMSpxZPUMAx4+hy343dRss+pVCLIfF62qOhxpYfZ5TmOk1GTDQm5f9wPprMNSStmnsV4w==", - "license": "Apache-2.0", - "dependencies": { - "@pondwader/socks5-server": "^1.0.10", - "@types/lodash-es": "^4.17.12", - "commander": "^12.1.0", - "lodash-es": "^4.17.21", - "shell-quote": "^1.8.3", - "zod": "^3.24.1" - }, - "bin": { - "srt": "dist/cli.js" - }, - "engines": { - "node": ">=18.0.0" - } - }, "node_modules/@anthropic-ai/sdk": { "version": "0.129.0", "resolved": "https://registry.npmjs.org/@anthropic-ai/sdk/-/sdk-0.129.0.tgz", @@ -620,26 +597,26 @@ } }, "node_modules/@cloudflare/vitest-plugin": { - "version": "1.4.4", - "resolved": "https://registry.npmjs.org/@cloudflare/vitest-plugin/-/vitest-plugin-1.4.4.tgz", - "integrity": "sha512-nXNIlPiLvYFx/AH4/Bb/yntVkeloYcf1+tBi3l6Gp3TDCDtLxXAugrUtO7eTOWbHP6UIx5f9npy05OrACbZ5wg==", + "version": "1.4.10", + "resolved": "https://registry.npmjs.org/@cloudflare/vitest-plugin/-/vitest-plugin-1.4.10.tgz", + "integrity": "sha512-F6klOeT/Wx0gKvZ4zpYVBixP6TjcT4OD/L/RJK3Yq3SwRmd+v96/y037xUEsvGytdUisL9X5Jw3ADuNdvwBHdg==", "dev": true, "license": "MIT", "dependencies": { "cjs-module-lexer": "1.2.3", "esbuild": "0.28.2", - "miniflare": "5.20261012.1-beta", - "wrangler": "4.151.1", - "zod": "4.6.5" + "miniflare": "5.20261023.2-beta", + "wrangler": "4.155.0", + "zod": "4.6.6" }, "peerDependencies": { "vitest": "^4.1.0 || ^5.0.0" } }, "node_modules/@cloudflare/vitest-plugin/node_modules/zod": { - "version": "4.6.5", - "resolved": "https://registry.npmjs.org/zod/-/zod-4.6.5.tgz", - "integrity": "sha512-v5l/aFXZQeai4awLbOpSoHecE9UiMrnfx75tEXLjNonXVARxQ5mOeipTjROUchszUNCqnE+hqAMujRsRHsut2Q==", + "version": "4.6.6", + "resolved": "https://registry.npmjs.org/zod/-/zod-4.6.6.tgz", + "integrity": "sha512-4UmzQUF5fxvfcYHlbjBmejU+w+hbDnWFx4aS8kBx7nPBHcO2SqPbpAG8snbdg+Xse4pRefR98hgjFgIMCI/UhA==", "dev": true, "license": "MIT", "funding": { @@ -647,9 +624,9 @@ } }, "node_modules/@cloudflare/workerd-darwin-64": { - "version": "1.20261012.1", - "resolved": "https://registry.npmjs.org/@cloudflare/workerd-darwin-64/-/workerd-darwin-64-1.20261012.1.tgz", - "integrity": "sha512-dkTL9aY3V4D40b+jBpGPCistWUPFQ/M61wlWi6i1o8MvHFFgEw7EQkcaYEv6Yd3bkPDD/3TFT4hRsMb2xVZSwQ==", + "version": "1.20261023.1", + "resolved": "https://registry.npmjs.org/@cloudflare/workerd-darwin-64/-/workerd-darwin-64-1.20261023.1.tgz", + "integrity": "sha512-lsEx0i7nR0PNG/erUFFHmQmiidEhLfDE5uQmAL9421/y4/ycuxDDOSaxu3VGGYFFPK5sduyUeMmjIlX4CwOBFg==", "cpu": [ "x64" ], @@ -664,9 +641,9 @@ } }, "node_modules/@cloudflare/workerd-darwin-arm64": { - "version": "1.20261012.1", - "resolved": "https://registry.npmjs.org/@cloudflare/workerd-darwin-arm64/-/workerd-darwin-arm64-1.20261012.1.tgz", - "integrity": "sha512-2/nVGcKXjNNI+DhxKSepTKvFng9H7aN2WXv6Zu67iqgI6JrxypNt5kTPv4ZHosxrcfyKMci/2RfN7GIrN0BitQ==", + "version": "1.20261023.1", + "resolved": "https://registry.npmjs.org/@cloudflare/workerd-darwin-arm64/-/workerd-darwin-arm64-1.20261023.1.tgz", + "integrity": "sha512-5e79fHgabdDD8EgmjOhJ1S+xMsf6DbC7fL7IgTBizMnzOp0P2isDn7pYmUxI9AJZE9FVQ4RUilH/I36YNg+GQQ==", "cpu": [ "arm64" ], @@ -681,9 +658,9 @@ } }, "node_modules/@cloudflare/workerd-linux-64": { - "version": "1.20261012.1", - "resolved": "https://registry.npmjs.org/@cloudflare/workerd-linux-64/-/workerd-linux-64-1.20261012.1.tgz", - "integrity": "sha512-uMe7ONHnLxStD1Ye0YFHBO36uj4cStv5P53rW+5GVWXFT8BRjqVixlH0IqCAHe1Pmcwyd1+7fVoGde89MZSCJg==", + "version": "1.20261023.1", + "resolved": "https://registry.npmjs.org/@cloudflare/workerd-linux-64/-/workerd-linux-64-1.20261023.1.tgz", + "integrity": "sha512-VHQ6jKFSihjR2+x1buby8XTdBLLJztVnYiB7OLOTYO8aDXB+c1Rd2410PdgFuFYc4AgFEk6PCJLyCLL5L1KBnw==", "cpu": [ "x64" ], @@ -698,9 +675,9 @@ } }, "node_modules/@cloudflare/workerd-linux-arm64": { - "version": "1.20261012.1", - "resolved": "https://registry.npmjs.org/@cloudflare/workerd-linux-arm64/-/workerd-linux-arm64-1.20261012.1.tgz", - "integrity": "sha512-I0VMdcXk7ygG5vYDce0ePTBFdnBXC0i9WiBY5mWiv0iTinbfea6/rHqSQ7B1UMpjgrYVs9E0m4bZ9ufsk5GCFA==", + "version": "1.20261023.1", + "resolved": "https://registry.npmjs.org/@cloudflare/workerd-linux-arm64/-/workerd-linux-arm64-1.20261023.1.tgz", + "integrity": "sha512-ilpRHzfDN8YFRM5CLRKuFw40x/LJBS89yVdhdQmxehlVDuBqjKpJB++zPFbTgKRRFaNupgnXc8RHktXAIhgCmQ==", "cpu": [ "arm64" ], @@ -715,9 +692,9 @@ } }, "node_modules/@cloudflare/workerd-windows-64": { - "version": "1.20261012.1", - "resolved": "https://registry.npmjs.org/@cloudflare/workerd-windows-64/-/workerd-windows-64-1.20261012.1.tgz", - "integrity": "sha512-c5hQvNx574UEoknaCFvMa/NdLrrYiALNbe6sx1okW8uOiRYWxrFuA8WpXFnBJ5XJypm0TEAHYaUTGv3rRLpY7Q==", + "version": "1.20261023.1", + "resolved": "https://registry.npmjs.org/@cloudflare/workerd-windows-64/-/workerd-windows-64-1.20261023.1.tgz", + "integrity": "sha512-279WLVjNsMycAwr0EZNiNEf0l2SY6NUtsN80NwZzbTHzakmiN+WSx4w38muxPm+PAU0oSPPAPXKVD+H6Hya8xw==", "cpu": [ "x64" ], @@ -1454,6 +1431,9 @@ "arm" ], "dev": true, + "libc": [ + "glibc" + ], "license": "LGPL-3.0-or-later", "optional": true, "os": [ @@ -1471,6 +1451,9 @@ "arm64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "LGPL-3.0-or-later", "optional": true, "os": [ @@ -1488,6 +1471,9 @@ "ppc64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "LGPL-3.0-or-later", "optional": true, "os": [ @@ -1505,6 +1491,9 @@ "riscv64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "LGPL-3.0-or-later", "optional": true, "os": [ @@ -1522,6 +1511,9 @@ "s390x" ], "dev": true, + "libc": [ + "glibc" + ], "license": "LGPL-3.0-or-later", "optional": true, "os": [ @@ -1539,6 +1531,9 @@ "x64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "LGPL-3.0-or-later", "optional": true, "os": [ @@ -1556,6 +1551,9 @@ "arm64" ], "dev": true, + "libc": [ + "musl" + ], "license": "LGPL-3.0-or-later", "optional": true, "os": [ @@ -1573,6 +1571,9 @@ "x64" ], "dev": true, + "libc": [ + "musl" + ], "license": "LGPL-3.0-or-later", "optional": true, "os": [ @@ -1590,6 +1591,9 @@ "arm" ], "dev": true, + "libc": [ + "glibc" + ], "license": "Apache-2.0", "optional": true, "os": [ @@ -1613,6 +1617,9 @@ "arm64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "Apache-2.0", "optional": true, "os": [ @@ -1636,6 +1643,9 @@ "ppc64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "Apache-2.0", "optional": true, "os": [ @@ -1659,6 +1669,9 @@ "riscv64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "Apache-2.0", "optional": true, "os": [ @@ -1682,6 +1695,9 @@ "s390x" ], "dev": true, + "libc": [ + "glibc" + ], "license": "Apache-2.0", "optional": true, "os": [ @@ -1705,6 +1721,9 @@ "x64" ], "dev": true, + "libc": [ + "glibc" + ], "license": "Apache-2.0", "optional": true, "os": [ @@ -1728,6 +1747,9 @@ "arm64" ], "dev": true, + "libc": [ + "musl" + ], "license": "Apache-2.0", "optional": true, "os": [ @@ -1751,6 +1773,9 @@ "x64" ], "dev": true, + "libc": [ + "musl" + ], "license": "Apache-2.0", "optional": true, "os": [ @@ -1863,6 +1888,12 @@ "url": "https://opencollective.com/libvips" } }, + "node_modules/@isaacs/nonrecursive-glob-match": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/@isaacs/nonrecursive-glob-match/-/nonrecursive-glob-match-1.0.3.tgz", + "integrity": "sha512-SI7kyKHq4XXDRozw+r9GlRZkia1ZkFrpLW98r2IC2Bpr8YkFGE3fvjzmxCG0x78A++69HmDNrDvSMYBZmHGpKA==", + "license": "BlueOak-1.0.0" + }, "node_modules/@jridgewell/resolve-uri": { "version": "3.1.2", "resolved": "https://registry.npmjs.org/@jridgewell/resolve-uri/-/resolve-uri-3.1.2.tgz", @@ -1901,12 +1932,6 @@ "url": "https://github.com/sponsors/oxc-project" } }, - "node_modules/@pondwader/socks5-server": { - "version": "1.0.10", - "resolved": "https://registry.npmjs.org/@pondwader/socks5-server/-/socks5-server-1.0.10.tgz", - "integrity": "sha512-bQY06wzzR8D2+vVCUoBsr5QS2U6UgPUQRmErNwtsuI6vLcyRKkafjkr3KxbtGFf9aBBIV2mcvlsKD1UYaIV+sg==", - "license": "MIT" - }, "node_modules/@poppinss/colors": { "version": "4.1.6", "resolved": "https://registry.npmjs.org/@poppinss/colors/-/colors-4.1.6.tgz", @@ -2441,21 +2466,6 @@ "dev": true, "license": "MIT" }, - "node_modules/@types/lodash": { - "version": "4.17.24", - "resolved": "https://registry.npmjs.org/@types/lodash/-/lodash-4.17.24.tgz", - "integrity": "sha512-gIW7lQLZbue7lRSWEFql49QJJWThrTFFeIMJdp3eH4tKoxm1OvEPg02rm4wCCSHS0cL3/Fizimb35b7k8atwsQ==", - "license": "MIT" - }, - "node_modules/@types/lodash-es": { - "version": "4.17.12", - "resolved": "https://registry.npmjs.org/@types/lodash-es/-/lodash-es-4.17.12.tgz", - "integrity": "sha512-0NgftHUcV4v34VhXm8QBSftKVXtbkBG3ViCjs6+eJ5a6y6Mi/jiFGPc1sC7QK+9BFhWrURE3EOggmWaSxL9OzQ==", - "license": "MIT", - "dependencies": { - "@types/lodash": "*" - } - }, "node_modules/@types/ms": { "version": "2.1.0", "resolved": "https://registry.npmjs.org/@types/ms/-/ms-2.1.0.tgz", @@ -3201,22 +3211,6 @@ "dev": true, "license": "MIT" }, - "node_modules/commander": { - "version": "12.1.0", - "resolved": "https://registry.npmjs.org/commander/-/commander-12.1.0.tgz", - "integrity": "sha512-Vw8qHK3bZM9y/P10u3Vib8o/DdkvA2OtPtZvD871QKjy74Wj1WSKFILMPRPSdUSx5RFK1arlJzEtA4PkFgnbuA==", - "license": "MIT", - "engines": { - "node": ">=18" - } - }, - "node_modules/concat-map": { - "version": "0.0.1", - "resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz", - "integrity": "sha512-/Srv4dswyQNBfohGpz9o6Yb3Gz3SrUDqBH5rTuhGR7ahtlbYKnVxw2bCFMRljaA7EXHaXZ8wsHdodFvbkhKmqg==", - "dev": true, - "license": "MIT" - }, "node_modules/convert-source-map": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/convert-source-map/-/convert-source-map-2.0.0.tgz", @@ -3366,16 +3360,6 @@ "url": "https://github.com/sponsors/antfu" } }, - "node_modules/es-errors": { - "version": "1.3.0", - "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz", - "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">= 0.4" - } - }, "node_modules/es-module-lexer": { "version": "2.3.1", "resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-2.3.1.tgz", @@ -3537,13 +3521,6 @@ "dev": true, "license": "MIT" }, - "node_modules/fs.realpath": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/fs.realpath/-/fs.realpath-1.0.0.tgz", - "integrity": "sha512-OO0pH2lK6a0hZnAdau5ItzHPI6pUlvI7jMVnxUQRtw4owF2wk8lOSabtGDCTP4Ggrg2MbGnWO9X8K1t4+fGMDw==", - "dev": true, - "license": "ISC" - }, "node_modules/fsevents": { "version": "2.3.3", "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", @@ -3559,16 +3536,6 @@ "node": "^8.16.0 || ^10.6.0 || >=11.0.0" } }, - "node_modules/function-bind": { - "version": "1.1.2", - "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz", - "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==", - "dev": true, - "license": "MIT", - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, "node_modules/gaxios": { "version": "7.1.4", "resolved": "https://registry.npmjs.org/gaxios/-/gaxios-7.1.4.tgz", @@ -3616,59 +3583,6 @@ "dev": true, "license": "MIT" }, - "node_modules/glob": { - "version": "7.2.3", - "resolved": "https://registry.npmjs.org/glob/-/glob-7.2.3.tgz", - "integrity": "sha512-nFR0zLpU2YCaRxwoCJvL6UvCH2JFyFVIvwTLsIf21AuHlMskA1hhTdk+LlYJtOlYt9v6dvszD2BGRqBL+iQK9Q==", - "deprecated": "Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me", - "dev": true, - "license": "ISC", - "dependencies": { - "fs.realpath": "^1.0.0", - "inflight": "^1.0.4", - "inherits": "2", - "minimatch": "^3.1.1", - "once": "^1.3.0", - "path-is-absolute": "^1.0.0" - }, - "engines": { - "node": "*" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, - "node_modules/glob/node_modules/balanced-match": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz", - "integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==", - "dev": true, - "license": "MIT" - }, - "node_modules/glob/node_modules/brace-expansion": { - "version": "1.1.21", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.21.tgz", - "integrity": "sha512-9zeA+KLZNNzglF2TPKRQEDyx6Yby7daAkuy8MiPzpXPsYDWi/DRM8jmwUDxokQjYqBpv5DgPiwD4h4ZZSy1Ujw==", - "dev": true, - "license": "MIT", - "dependencies": { - "balanced-match": "^1.0.0", - "concat-map": "0.0.1" - } - }, - "node_modules/glob/node_modules/minimatch": { - "version": "3.1.5", - "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.1.5.tgz", - "integrity": "sha512-VgjWUsnnT6n+NUk6eZq77zeFdpW2LWDzP6zFGrCbHXiYNul5Dzqk2HHQ5uFH2DNW5Xbp8+jVzaeNt94ssEEl4w==", - "dev": true, - "license": "ISC", - "dependencies": { - "brace-expansion": "^1.1.7" - }, - "engines": { - "node": "*" - } - }, "node_modules/google-auth-library": { "version": "10.6.2", "resolved": "https://registry.npmjs.org/google-auth-library/-/google-auth-library-10.6.2.tgz", @@ -3710,19 +3624,6 @@ "node": ">=18" } }, - "node_modules/hasown": { - "version": "2.0.4", - "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.4.tgz", - "integrity": "sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==", - "dev": true, - "license": "MIT", - "dependencies": { - "function-bind": "^1.1.2" - }, - "engines": { - "node": ">= 0.4" - } - }, "node_modules/highlight.js": { "version": "10.7.3", "resolved": "https://registry.npmjs.org/highlight.js/-/highlight.js-10.7.3.tgz", @@ -3826,18 +3727,6 @@ "node": ">= 4" } }, - "node_modules/inflight": { - "version": "1.0.6", - "resolved": "https://registry.npmjs.org/inflight/-/inflight-1.0.6.tgz", - "integrity": "sha512-k92I/b08q4wvFscXCLvqfsHCrjrF7yiXsQuIVvVE7N82W3+aqpzuUdBbfhWcy/FZR3/4IgflMgKLOsvPDrGCJA==", - "deprecated": "This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.", - "dev": true, - "license": "ISC", - "dependencies": { - "once": "^1.3.0", - "wrappy": "1" - } - }, "node_modules/inherits": { "version": "2.0.4", "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz", @@ -3852,32 +3741,6 @@ "dev": true, "license": "ISC" }, - "node_modules/interpret": { - "version": "1.4.0", - "resolved": "https://registry.npmjs.org/interpret/-/interpret-1.4.0.tgz", - "integrity": "sha512-agE4QfB2Lkp9uICn7BAqoscw4SZP9kTE2hxiFI3jBPmXJfdqiahTbUuKGsMoN2GtqL9AxhYioAcVvgsb1HvRbA==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">= 0.10" - } - }, - "node_modules/is-core-module": { - "version": "2.17.0", - "resolved": "https://registry.npmjs.org/is-core-module/-/is-core-module-2.17.0.tgz", - "integrity": "sha512-J/vG0zBCbIKOQFfufSwyXdMrsohyJIUNkrnmo6WZGzoM7tr/lsbfW5b2BvisL6zsyMzK9UxV9L6c7AoFbyXHOA==", - "dev": true, - "license": "MIT", - "dependencies": { - "hasown": "^2.0.4" - }, - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, "node_modules/isexe": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz", @@ -4219,12 +4082,6 @@ "url": "https://opencollective.com/parcel" } }, - "node_modules/lodash-es": { - "version": "4.18.1", - "resolved": "https://registry.npmjs.org/lodash-es/-/lodash-es-4.18.1.tgz", - "integrity": "sha512-J8xewKD/Gk22OZbhpOVSwcs60zhd95ESDwezOFuA3/099925PdHJ7OFHNTGtajL3AlZkykD32HykiMo+BIBI8A==", - "license": "MIT" - }, "node_modules/long": { "version": "5.3.2", "resolved": "https://registry.npmjs.org/long/-/long-5.3.2.tgz", @@ -4288,16 +4145,16 @@ } }, "node_modules/miniflare": { - "version": "5.20261012.1-beta", - "resolved": "https://registry.npmjs.org/miniflare/-/miniflare-5.20261012.1-beta.tgz", - "integrity": "sha512-8Canhy6g9+vAhqMc0o80YphqYIwpiA3aAp8aX+uDXU7J/qbUeoZzcII9c2V1P4uT/eHD4SB93qet1o1bRY3BuQ==", + "version": "5.20261023.2-beta", + "resolved": "https://registry.npmjs.org/miniflare/-/miniflare-5.20261023.2-beta.tgz", + "integrity": "sha512-sXpZloQnIUe4raXXdwwqd2ioRHox9a/1B4nB2/p4JWKDwGuVgEK4DugCNgIeQf0iVuSgBItdYpodJe4iwY6SrQ==", "dev": true, "license": "MIT", "dependencies": { "@cspotcode/source-map-support": "0.8.1", "sharp": "0.35.5", - "undici": "7.29.1", - "workerd": "1.20261012.1", + "undici": "7.30.1", + "workerd": "1.20261023.1", "ws": "8.21.0", "youch": "4.1.0-beta.10" }, @@ -4306,9 +4163,9 @@ } }, "node_modules/miniflare/node_modules/undici": { - "version": "7.29.1", - "resolved": "https://registry.npmjs.org/undici/-/undici-7.29.1.tgz", - "integrity": "sha512-RYONW2MeafgYlkVOKYKkA/Ag7BmXqgIWCa8t1m0JcxrQg9pI9lEqRhAOruOBCbAohOa/gkCF+iPi9hrgvTzu6Q==", + "version": "7.30.1", + "resolved": "https://registry.npmjs.org/undici/-/undici-7.30.1.tgz", + "integrity": "sha512-03KrMbjIFDWQR5RNPrSw3bd1r12JpyFQl9/yxjUbJtqkjmllqC5lFjZh6ujeuJCKr8EOQvRW872Oy/az+nLYxg==", "dev": true, "license": "MIT", "engines": { @@ -4316,12 +4173,13 @@ } }, "node_modules/minimatch": { - "version": "10.2.6", - "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.6.tgz", - "integrity": "sha512-vpLQEs+VLCr1nU0BXS07maYoFwlDAH0gngQuuttxIwutDFEMHq2blX+8vpgxDdK3J1PwjCJiep77OitTZ4Ll1A==", + "version": "11.0.3", + "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-11.0.3.tgz", + "integrity": "sha512-iny/40PEGkNJQ5/Pv6z3nm6FbRAby32GeZRjCUJk2WnNDbgPSTcdI+DDvLqsgBdD4cD5sMtMheqPPnehCCa8Nw==", "license": "BlueOak-1.0.0", "dependencies": { - "brace-expansion": "^5.0.8" + "@isaacs/nonrecursive-glob-match": "^1.0.3", + "brace-expansion": "^5.0.12" }, "engines": { "node": "18 || 20 || >=22" @@ -4493,16 +4351,6 @@ "integrity": "sha512-Njv/59hHaokb/hRUjce3Hdv12wd60MtM9Z5Olmn+nehe0QDAsRtRbJPvJ0Z91TusF0SuZRIvnM+S4l6EIP8leA==", "license": "MIT" }, - "node_modules/path-is-absolute": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/path-is-absolute/-/path-is-absolute-1.0.1.tgz", - "integrity": "sha512-AVbw3UJ2e9bq64vSaS9Am0fje1Pa8pbGqTTsmXfaIiMpnr5DlDhfJOuLj9Sf95ZPVDAUerDfEk88MPmPe7UCQg==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=0.10.0" - } - }, "node_modules/path-key": { "version": "3.1.1", "resolved": "https://registry.npmjs.org/path-key/-/path-key-3.1.1.tgz", @@ -4512,13 +4360,6 @@ "node": ">=8" } }, - "node_modules/path-parse": { - "version": "1.0.7", - "resolved": "https://registry.npmjs.org/path-parse/-/path-parse-1.0.7.tgz", - "integrity": "sha512-LDJzPVEEEPR+y48z93A0Ed0yXb8pAByGWo/k5YYdYgpY2/2EsOsksJrq7lOHxryrVOn1ejG6oAp8ahvOIQD8sw==", - "dev": true, - "license": "MIT" - }, "node_modules/path-to-regexp": { "version": "6.3.0", "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-6.3.0.tgz", @@ -4545,10 +4386,6 @@ "resolved": "packages/coding-agent/examples/extensions/gondolin", "link": true }, - "node_modules/pi-extension-sandbox": { - "resolved": "packages/coding-agent/examples/extensions/sandbox", - "link": true - }, "node_modules/pi-extension-with-deps": { "resolved": "packages/coding-agent/examples/extensions/with-deps", "link": true @@ -4757,40 +4594,6 @@ "node": ">= 6" } }, - "node_modules/rechoir": { - "version": "0.6.2", - "resolved": "https://registry.npmjs.org/rechoir/-/rechoir-0.6.2.tgz", - "integrity": "sha512-HFM8rkZ+i3zrV+4LQjwQ0W+ez98pApMGM3HUrN04j3CqzPOzl9nmP15Y8YXNm8QHGv/eacOVEjqhmWpkRV0NAw==", - "dev": true, - "dependencies": { - "resolve": "^1.1.6" - }, - "engines": { - "node": ">= 0.10" - } - }, - "node_modules/resolve": { - "version": "1.22.12", - "resolved": "https://registry.npmjs.org/resolve/-/resolve-1.22.12.tgz", - "integrity": "sha512-TyeJ1zif53BPfHootBGwPRYT1RUt6oGWsaQr8UyZW/eAm9bKoijtvruSDEmZHm92CwS9nj7/fWttqPCgzep8CA==", - "dev": true, - "license": "MIT", - "dependencies": { - "es-errors": "^1.3.0", - "is-core-module": "^2.16.1", - "path-parse": "^1.0.7", - "supports-preserve-symlinks-flag": "^1.0.0" - }, - "bin": { - "resolve": "bin/resolve" - }, - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, "node_modules/retry": { "version": "0.13.1", "resolved": "https://registry.npmjs.org/retry/-/retry-0.13.1.tgz", @@ -4943,53 +4746,6 @@ "node": ">=8" } }, - "node_modules/shell-quote": { - "version": "1.12.0", - "resolved": "https://registry.npmjs.org/shell-quote/-/shell-quote-1.12.0.tgz", - "integrity": "sha512-PcByqNyT/38F2kDNi006HAMRJaULuBzq/FOsw3qdZvX/GA9W/jamDaRskgHjubHiftXK5sIFxLNkvrXUwcof6Q==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/shelljs": { - "version": "0.8.5", - "resolved": "https://registry.npmjs.org/shelljs/-/shelljs-0.8.5.tgz", - "integrity": "sha512-TiwcRcrkhHvbrZbnRcFYMLl30Dfov3HKqzp5tO5b4pt6G/SezKcYhmDg15zXVBswHmctSAQKznqNW2LO5tTDow==", - "dev": true, - "license": "BSD-3-Clause", - "dependencies": { - "glob": "^7.0.0", - "interpret": "^1.0.0", - "rechoir": "^0.6.2" - }, - "bin": { - "shjs": "bin/shjs" - }, - "engines": { - "node": ">=4" - } - }, - "node_modules/shx": { - "version": "0.3.4", - "resolved": "https://registry.npmjs.org/shx/-/shx-0.3.4.tgz", - "integrity": "sha512-N6A9MLVqjxZYcVn8hLmtneQWIJtp8IKzMP4eMnx+nqkvXoqinUPCbUFLp2UcWTEIUONhlk0ewxr/jaVGlc+J+g==", - "dev": true, - "license": "MIT", - "dependencies": { - "minimist": "^1.2.3", - "shelljs": "^0.8.5" - }, - "bin": { - "shx": "lib/cli.js" - }, - "engines": { - "node": ">=6" - } - }, "node_modules/siginfo": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/siginfo/-/siginfo-2.0.0.tgz", @@ -5134,19 +4890,6 @@ "url": "https://github.com/chalk/supports-color?sponsor=1" } }, - "node_modules/supports-preserve-symlinks-flag": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/supports-preserve-symlinks-flag/-/supports-preserve-symlinks-flag-1.0.0.tgz", - "integrity": "sha512-ot0WnXS9fgdkgIcePe6RHNk1WA8+muPa6cSjeR3V8K27q9BB1rTE3R1p7Hv0z1ZyAc8s6Vvv8DIyWf681MAt0w==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, "node_modules/tar-fs": { "version": "2.1.4", "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.4.tgz", @@ -5563,9 +5306,9 @@ } }, "node_modules/workerd": { - "version": "1.20261012.1", - "resolved": "https://registry.npmjs.org/workerd/-/workerd-1.20261012.1.tgz", - "integrity": "sha512-tEHoMPsMga0seVjxfsR8Pd3f8IRqYwkq++dTUKI5Gdgj6zuSDluYwisydq+dn90SsIACwBquJUQ1nBpcOqaWTA==", + "version": "1.20261023.1", + "resolved": "https://registry.npmjs.org/workerd/-/workerd-1.20261023.1.tgz", + "integrity": "sha512-NeofJFyaYNpqKpFRJW3ApuonaerPd6/JWlfTtLDYf2A6ygqrDEFnYgEXmc0hhOxqnJsSM+kXJN6VebPojzRrsg==", "dev": true, "hasInstallScript": true, "license": "Apache-2.0", @@ -5576,17 +5319,17 @@ "node": ">=16" }, "optionalDependencies": { - "@cloudflare/workerd-darwin-64": "1.20261012.1", - "@cloudflare/workerd-darwin-arm64": "1.20261012.1", - "@cloudflare/workerd-linux-64": "1.20261012.1", - "@cloudflare/workerd-linux-arm64": "1.20261012.1", - "@cloudflare/workerd-windows-64": "1.20261012.1" + "@cloudflare/workerd-darwin-64": "1.20261023.1", + "@cloudflare/workerd-darwin-arm64": "1.20261023.1", + "@cloudflare/workerd-linux-64": "1.20261023.1", + "@cloudflare/workerd-linux-arm64": "1.20261023.1", + "@cloudflare/workerd-windows-64": "1.20261023.1" } }, "node_modules/wrangler": { - "version": "4.151.1", - "resolved": "https://registry.npmjs.org/wrangler/-/wrangler-4.151.1.tgz", - "integrity": "sha512-OAsa9UAHZshM6mq2PXcVoyxXgWThAanZMBc/IirtqEWd/BauRtmUKzAniOu/pU0pd3pxKuMZ2vsbF4eo6RiNDA==", + "version": "4.155.0", + "resolved": "https://registry.npmjs.org/wrangler/-/wrangler-4.155.0.tgz", + "integrity": "sha512-gBKtAg2e0abQDvXPRQu611YMTd/cfAaJTOLAs/89tTxFxEeUyC260MRJRK+m3gLmQjDYWFwJv+ErwuJ/GuALpA==", "dev": true, "license": "MIT OR Apache-2.0", "dependencies": { @@ -5594,10 +5337,10 @@ "@cloudflare/unenv-preset": "2.16.2", "blake3-wasm": "2.1.5", "esbuild": "0.28.2", - "miniflare": "5.20261012.1-beta", + "miniflare": "5.20261023.2-beta", "path-to-regexp": "6.3.0", "unenv": "2.0.0-rc.24", - "workerd": "1.20261012.1" + "workerd": "1.20261023.1" }, "bin": { "cf-wrangler": "bin/cf-wrangler.js", @@ -5611,7 +5354,7 @@ "fsevents": "2.3.3" }, "peerDependencies": { - "@cloudflare/workers-types": "^5.20261006.1" + "@cloudflare/workers-types": "^5.20261023.1" }, "peerDependenciesMeta": { "@cloudflare/workers-types": { @@ -5688,9 +5431,10 @@ } }, "node_modules/zod": { - "version": "3.25.76", - "resolved": "https://registry.npmjs.org/zod/-/zod-3.25.76.tgz", - "integrity": "sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ==", + "version": "4.6.7", + "resolved": "https://registry.npmjs.org/zod/-/zod-4.6.7.tgz", + "integrity": "sha512-g8W2D2kU6a8untc053fX1uVBKzM7yqAxW/3zwk83+6RyF2+SKOuV+c0wBDhUwzV1PaGapmZH+rmcUzPf+3CGwg==", + "devOptional": true, "license": "MIT", "funding": { "url": "https://github.com/sponsors/colinhacks" @@ -5817,7 +5561,6 @@ }, "devDependencies": { "@types/ws": "8.18.2", - "shx": "0.3.4", "vitest": "4.1.12", "ws": "8.21.0" }, @@ -5841,7 +5584,6 @@ "quickjs-wasi": "3.6.2" }, "devDependencies": { - "shx": "0.3.4", "vitest": "4.1.12" }, "engines": { @@ -5868,7 +5610,7 @@ "hosted-git-info": "9.0.3", "ignore": "7.0.8", "jiti": "2.7.0", - "minimatch": "10.2.6", + "minimatch": "11.0.3", "proper-lockfile": "4.1.2", "quickjs-wasi": "3.6.2", "semver": "7.8.5", @@ -5886,7 +5628,6 @@ "@types/proper-lockfile": "4.1.4", "@types/semver": "7.7.1", "marked": "18.0.11", - "shx": "0.3.4", "vitest": "4.1.12" }, "engines": { @@ -5911,13 +5652,6 @@ "@earendil-works/gondolin": "0.14.0" } }, - "packages/coding-agent/examples/extensions/sandbox": { - "name": "pi-extension-sandbox", - "version": "1.4.0", - "dependencies": { - "@anthropic-ai/sandbox-runtime": "0.0.26" - } - }, "packages/coding-agent/examples/extensions/with-deps": { "name": "pi-extension-with-deps", "version": "1.4.0", @@ -5986,9 +5720,8 @@ "typebox": "1.3.27" }, "devDependencies": { - "@cloudflare/vitest-plugin": "1.4.4", + "@cloudflare/vitest-plugin": "1.4.10", "fake-indexeddb": "6.2.6", - "shx": "0.3.4", "vitest": "4.1.12" }, "engines": { @@ -6004,7 +5737,6 @@ "@earendil-works/pi-durable": "1.4.0" }, "devDependencies": { - "shx": "0.3.4", "vitest": "4.1.12" }, "engines": { @@ -6019,7 +5751,6 @@ "@earendil-works/pi-coding-agent": "1.4.0", "@types/node": "22.19.19", "@vitest-evals/core": "0.15.0", - "shx": "0.3.4", "vitest": "4.1.12", "vitest-evals": "0.15.0" } @@ -6033,7 +5764,6 @@ }, "devDependencies": { "@types/cross-spawn": "6.0.6", - "shx": "0.3.4", "vitest": "4.1.12" }, "engines": { diff --git a/package.json b/package.json index 04462a1d5..b158f2f00 100644 --- a/package.json +++ b/package.json @@ -8,7 +8,6 @@ "packages/coding-agent/examples/extensions/with-deps", "packages/coding-agent/examples/extensions/custom-provider-anthropic", "packages/coding-agent/examples/extensions/custom-provider-gitlab-duo", - "packages/coding-agent/examples/extensions/sandbox", "packages/coding-agent/examples/extensions/gondolin" ], "scripts": { @@ -57,14 +56,12 @@ "prepare": "husky" }, "devDependencies": { - "@anthropic-ai/sandbox-runtime": "0.0.26", "@biomejs/biome": "2.3.5", "@biomejs/wasm-nodejs": "2.3.5", "@types/node": "22.19.19", "esbuild": "0.28.2", "husky": "9.1.7", "pnpm": "11.15.1", - "shx": "0.3.4", "typescript": "7.0.2", "yaml": "2.9.0" }, diff --git a/packages/agent/package.json b/packages/agent/package.json index d1231d95d..a8f7de36f 100644 --- a/packages/agent/package.json +++ b/packages/agent/package.json @@ -17,7 +17,7 @@ "README.md" ], "scripts": { - "clean": "shx rm -rf dist", + "clean": "node ../../scripts/package-files.mjs clean dist", "build": "tsc -p tsconfig.build.json", "test": "vitest --run", "prepublishOnly": "npm run build" diff --git a/packages/ai/package.json b/packages/ai/package.json index 6e04980f0..4e9d5d5a2 100644 --- a/packages/ai/package.json +++ b/packages/ai/package.json @@ -56,13 +56,13 @@ "README.md" ], "scripts": { - "clean": "shx rm -rf dist", + "clean": "node ../../scripts/package-files.mjs clean dist", "generate-models": "node catalog/generate.ts --strict", "hydrate-model-data": "node catalog/generate.ts --strict --data-only", "generate-model-catalog": "node catalog/generate.ts --strict --json-only --json-output ../../.artifacts/model-catalog", "check:model-data": "node scripts/check-model-data.ts", "build": "npm run generate-models && npm run build:offline", - "build:offline": "npm run check:model-data && tsc -p tsconfig.build.json && shx rm -rf dist/providers/data && shx cp -r src/providers/data dist/providers/data", + "build:offline": "npm run check:model-data && tsc -p tsconfig.build.json && node ../../scripts/package-files.mjs clean dist/providers/data && node ../../scripts/package-files.mjs copy src/providers/data dist/providers", "test": "vitest --run", "prepublishOnly": "npm run clean && npm run build" }, diff --git a/packages/chord/package.json b/packages/chord/package.json index f00d184dd..f2be5a564 100644 --- a/packages/chord/package.json +++ b/packages/chord/package.json @@ -67,7 +67,7 @@ "src/delta/README.md" ], "scripts": { - "clean": "shx rm -rf dist", + "clean": "node ../../scripts/package-files.mjs clean dist", "build": "tsc -p tsconfig.build.json", "test": "vitest --run", "prepublishOnly": "npm run clean && npm run build" @@ -103,7 +103,6 @@ }, "devDependencies": { "@types/ws": "8.18.2", - "shx": "0.3.4", "vitest": "4.1.12", "ws": "8.21.0" } diff --git a/packages/codemode/package.json b/packages/codemode/package.json index b1f0fea9c..9fed9d017 100644 --- a/packages/codemode/package.json +++ b/packages/codemode/package.json @@ -42,7 +42,7 @@ "README.md" ], "scripts": { - "clean": "shx rm -rf dist", + "clean": "node ../../scripts/package-files.mjs clean dist", "build": "tsc -p tsconfig.build.json", "test": "vitest --run", "prepublishOnly": "npm run clean && npm run build" @@ -70,7 +70,6 @@ "quickjs-wasi": "3.6.2" }, "devDependencies": { - "shx": "0.3.4", "vitest": "4.1.12" } } diff --git a/packages/coding-agent/CHANGELOG.md b/packages/coding-agent/CHANGELOG.md index 483e87088..8a916e873 100644 --- a/packages/coding-agent/CHANGELOG.md +++ b/packages/coding-agent/CHANGELOG.md @@ -19,6 +19,7 @@ ### Fixed +- Updated glob matching to prevent denial of service from amplified glob patterns. - Fixed a saved "Do not trust" decision being ignored for projects without trust-requiring resources, for example a folder with only `AGENTS.md`. pi auto-trusted such projects, so `ctx.isProjectTrusted()` returned true and package commands treated them as trusted. A saved decision in `trust.json` now takes precedence; auto-trust applies only when no decision is saved ([#10947](https://github.com/earendil-works/pi/issues/10947)) - Fixed `Shift+Enter` queueing a follow-up instead of inserting a newline in Apple Terminal setups that send `ESC` + Return. pi now checks the local Shift key state for that sequence, as it already did for plain Return; `Alt+Enter` without Shift still queues a follow-up ([#11559](https://github.com/earendil-works/pi/issues/11559)) - Fixed the working status (OSC 7501) disappearing in Rex and iTerm2 during runs in regular TUI mode. Rendered messages wrote OSC 133 prompt markers to the terminal, which the terminal treats as a new shell prompt; regular mode no longer writes them ([#11519](https://github.com/earendil-works/pi/issues/11519)) @@ -48,6 +49,10 @@ - Fixed `tool_search` and codemode `searchTools()` ranking a tool below tools that only mention its name in their descriptions, for example `peekaboo app` ranking `mcp__peekaboo__app` 14th. A tool matching more of the query's words now ranks above one matching fewer, a query word in a tool's name scores more than any number of mentions in its description, a tool whose whole name is in the query gets a large bonus, and long server instructions no longer lower the scores of that server's tools ([#11331](https://github.com/earendil-works/pi/issues/11331)) - Fixed `tool_search` and codemode `searchTools()` finding nothing for Chinese, Japanese, Thai, and other non-ASCII queries, because the tokenizer kept only ASCII letters and digits. Words in any script now match, text without spaces between words such as `搜索记录` is split into words, and canonically equivalent text such as `café` and `cafe\u0301` matches ([#10816](https://github.com/earendil-works/pi/issues/10816), [#11037](https://github.com/earendil-works/pi/pull/11037) by [@haoqixu](https://github.com/haoqixu)) +### Removed + +- Removed the native OS `sandbox/` extension example and its dependency. Use the [Gondolin example](examples/extensions/gondolin/) or the [containerization guide](docs/containerization.md). + ## [1.4.0] - 2026-10-21 ### New Features diff --git a/packages/coding-agent/docs/extensions.md b/packages/coding-agent/docs/extensions.md index 3606e80f9..6fbec2b65 100644 --- a/packages/coding-agent/docs/extensions.md +++ b/packages/coding-agent/docs/extensions.md @@ -3310,7 +3310,6 @@ All examples in [examples/extensions/](../examples/extensions/). | **Remote & Sandbox** ||| | `ssh.ts` | SSH remote execution | `registerFlag`, `on("user_bash")`, `on("before_agent_start")`, tool operations | | `interactive-shell.ts` | Persistent shell session | `on("user_bash")` | -| `sandbox/` | Sandboxed tool execution | Tool operations | | `gondolin/` | Route built-in tools and `!` commands into a Gondolin micro-VM | Tool operations, built-in tool overrides, `on("user_bash")` | | `subagent/` | Spawn sub-agents | `registerTool`, `exec` | | **Games** ||| diff --git a/packages/coding-agent/examples/extensions/README.md b/packages/coding-agent/examples/extensions/README.md index 7667ffe03..de24f7af4 100644 --- a/packages/coding-agent/examples/extensions/README.md +++ b/packages/coding-agent/examples/extensions/README.md @@ -23,9 +23,10 @@ cp permission-gate.ts ~/.pi/agent/extensions/ | `protected-paths.ts` | Blocks writes to protected paths (.env, .git/, node_modules/) | | `confirm-destructive.ts` | Confirms before destructive session actions (clear, switch, fork) | | `dirty-repo-guard.ts` | Prevents session changes with uncommitted git changes | -| `sandbox/` | OS-level sandboxing using `@anthropic-ai/sandbox-runtime` with per-project config | | `gondolin/` | Route built-in tools and `!` commands into a Gondolin micro-VM | +For isolation setup and limitations, see [Run Pi in an isolated environment](../../docs/containerization.md). + ### Custom Tools | Extension | Description | diff --git a/packages/coding-agent/examples/extensions/sandbox/.gitignore b/packages/coding-agent/examples/extensions/sandbox/.gitignore deleted file mode 100644 index 3c3629e64..000000000 --- a/packages/coding-agent/examples/extensions/sandbox/.gitignore +++ /dev/null @@ -1 +0,0 @@ -node_modules diff --git a/packages/coding-agent/examples/extensions/sandbox/index.ts b/packages/coding-agent/examples/extensions/sandbox/index.ts deleted file mode 100644 index b54d75d1c..000000000 --- a/packages/coding-agent/examples/extensions/sandbox/index.ts +++ /dev/null @@ -1,321 +0,0 @@ -/** - * Sandbox Extension - OS-level sandboxing for bash commands - * - * Uses @anthropic-ai/sandbox-runtime to enforce filesystem and network - * restrictions on bash commands at the OS level (sandbox-exec on macOS, - * bubblewrap on Linux). - * - * Note: this example intentionally overrides the built-in `bash` tool to show - * how built-in tools can be replaced. Alternatively, you could sandbox `bash` - * via `tool_call` input mutation without replacing the tool. - * - * Config files (merged, project takes precedence): - * - ~/.pi/agent/extensions/sandbox.json (global) - * - /.pi/sandbox.json (project-local) - * - * Example .pi/sandbox.json: - * ```json - * { - * "enabled": true, - * "network": { - * "allowedDomains": ["github.com", "*.github.com"], - * "deniedDomains": [] - * }, - * "filesystem": { - * "denyRead": ["~/.ssh", "~/.aws"], - * "allowWrite": [".", "/tmp"], - * "denyWrite": [".env"] - * } - * } - * ``` - * - * Usage: - * - `pi -e ./sandbox` - sandbox enabled with default/config settings - * - `pi -e ./sandbox --no-sandbox` - disable sandboxing - * - `/sandbox` - show current sandbox configuration - * - * Setup: - * 1. Copy sandbox/ directory to ~/.pi/agent/extensions/ - * 2. Run `npm install` in ~/.pi/agent/extensions/sandbox/ - * - * Linux also requires: bubblewrap, socat, ripgrep - */ - -import { spawn } from "node:child_process"; -import { existsSync, readFileSync } from "node:fs"; -import { join } from "node:path"; -import { SandboxManager, type SandboxRuntimeConfig } from "@anthropic-ai/sandbox-runtime"; -import type { ExtensionAPI } from "@earendil-works/pi-coding-agent"; -import { type BashOperations, CONFIG_DIR_NAME, createBashTool, getAgentDir } from "@earendil-works/pi-coding-agent"; - -interface SandboxConfig extends SandboxRuntimeConfig { - enabled?: boolean; -} - -const DEFAULT_CONFIG: SandboxConfig = { - enabled: true, - network: { - allowedDomains: [ - "npmjs.org", - "*.npmjs.org", - "registry.npmjs.org", - "registry.yarnpkg.com", - "pypi.org", - "*.pypi.org", - "github.com", - "*.github.com", - "api.github.com", - "raw.githubusercontent.com", - ], - deniedDomains: [], - }, - filesystem: { - denyRead: ["~/.ssh", "~/.aws", "~/.gnupg"], - allowWrite: [".", "/tmp"], - denyWrite: [".env", ".env.*", "*.pem", "*.key"], - }, -}; - -function loadConfig(cwd: string): SandboxConfig { - const projectConfigPath = join(cwd, CONFIG_DIR_NAME, "sandbox.json"); - const globalConfigPath = join(getAgentDir(), "extensions", "sandbox.json"); - - let globalConfig: Partial = {}; - let projectConfig: Partial = {}; - - if (existsSync(globalConfigPath)) { - try { - globalConfig = JSON.parse(readFileSync(globalConfigPath, "utf-8")); - } catch (e) { - console.error(`Warning: Could not parse ${globalConfigPath}: ${e}`); - } - } - - if (existsSync(projectConfigPath)) { - try { - projectConfig = JSON.parse(readFileSync(projectConfigPath, "utf-8")); - } catch (e) { - console.error(`Warning: Could not parse ${projectConfigPath}: ${e}`); - } - } - - return deepMerge(deepMerge(DEFAULT_CONFIG, globalConfig), projectConfig); -} - -function deepMerge(base: SandboxConfig, overrides: Partial): SandboxConfig { - const result: SandboxConfig = { ...base }; - - if (overrides.enabled !== undefined) result.enabled = overrides.enabled; - if (overrides.network) { - result.network = { ...base.network, ...overrides.network }; - } - if (overrides.filesystem) { - result.filesystem = { ...base.filesystem, ...overrides.filesystem }; - } - - const extOverrides = overrides as { - ignoreViolations?: Record; - enableWeakerNestedSandbox?: boolean; - }; - const extResult = result as { ignoreViolations?: Record; enableWeakerNestedSandbox?: boolean }; - - if (extOverrides.ignoreViolations) { - extResult.ignoreViolations = extOverrides.ignoreViolations; - } - if (extOverrides.enableWeakerNestedSandbox !== undefined) { - extResult.enableWeakerNestedSandbox = extOverrides.enableWeakerNestedSandbox; - } - - return result; -} - -function createSandboxedBashOps(): BashOperations { - return { - async exec(command, cwd, { onData, signal, timeout }) { - if (!existsSync(cwd)) { - throw new Error(`Working directory does not exist: ${cwd}`); - } - - const wrappedCommand = await SandboxManager.wrapWithSandbox(command); - - return new Promise((resolve, reject) => { - const child = spawn("bash", ["-c", wrappedCommand], { - cwd, - detached: true, - stdio: ["ignore", "pipe", "pipe"], - }); - - let timedOut = false; - let timeoutHandle: NodeJS.Timeout | undefined; - - if (timeout !== undefined && timeout > 0) { - timeoutHandle = setTimeout(() => { - timedOut = true; - if (child.pid) { - try { - process.kill(-child.pid, "SIGKILL"); - } catch { - child.kill("SIGKILL"); - } - } - }, timeout * 1000); - } - - child.stdout?.on("data", onData); - child.stderr?.on("data", onData); - - child.on("error", (err) => { - if (timeoutHandle) clearTimeout(timeoutHandle); - reject(err); - }); - - const onAbort = () => { - if (child.pid) { - try { - process.kill(-child.pid, "SIGKILL"); - } catch { - child.kill("SIGKILL"); - } - } - }; - - signal?.addEventListener("abort", onAbort, { once: true }); - - child.on("close", (code) => { - if (timeoutHandle) clearTimeout(timeoutHandle); - signal?.removeEventListener("abort", onAbort); - - if (signal?.aborted) { - reject(new Error("aborted")); - } else if (timedOut) { - reject(new Error(`timeout:${timeout}`)); - } else { - resolve({ exitCode: code }); - } - }); - }); - }, - }; -} - -export default function (pi: ExtensionAPI) { - pi.registerFlag("no-sandbox", { - description: "Disable OS-level sandboxing for bash commands", - type: "boolean", - default: false, - }); - - const localCwd = process.cwd(); - const localBash = createBashTool(localCwd); - - let sandboxEnabled = false; - let sandboxInitialized = false; - - pi.registerTool({ - ...localBash, - label: "bash (sandboxed)", - async execute(id, params, signal, onUpdate, _ctx) { - if (!sandboxEnabled || !sandboxInitialized) { - return localBash.execute(id, params, signal, onUpdate); - } - - const sandboxedBash = createBashTool(localCwd, { - operations: createSandboxedBashOps(), - }); - return sandboxedBash.execute(id, params, signal, onUpdate); - }, - }); - - pi.on("user_bash", () => { - if (!sandboxEnabled || !sandboxInitialized) return; - return { operations: createSandboxedBashOps() }; - }); - - pi.on("session_start", async (_event, ctx) => { - const noSandbox = pi.getFlag("no-sandbox") as boolean; - - if (noSandbox) { - sandboxEnabled = false; - ctx.ui.notify("Sandbox disabled via --no-sandbox", "warning"); - return; - } - - const config = loadConfig(ctx.cwd); - - if (!config.enabled) { - sandboxEnabled = false; - ctx.ui.notify("Sandbox disabled via config", "info"); - return; - } - - const platform = process.platform; - if (platform !== "darwin" && platform !== "linux") { - sandboxEnabled = false; - ctx.ui.notify(`Sandbox not supported on ${platform}`, "warning"); - return; - } - - try { - const configExt = config as unknown as { - ignoreViolations?: Record; - enableWeakerNestedSandbox?: boolean; - }; - - await SandboxManager.initialize({ - network: config.network, - filesystem: config.filesystem, - ignoreViolations: configExt.ignoreViolations, - enableWeakerNestedSandbox: configExt.enableWeakerNestedSandbox, - }); - - sandboxEnabled = true; - sandboxInitialized = true; - - const networkCount = config.network?.allowedDomains?.length ?? 0; - const writeCount = config.filesystem?.allowWrite?.length ?? 0; - ctx.ui.setStatus( - "sandbox", - ctx.ui.theme.fg("accent", `🔒 Sandbox: ${networkCount} domains, ${writeCount} write paths`), - ); - ctx.ui.notify("Sandbox initialized", "info"); - } catch (err) { - sandboxEnabled = false; - ctx.ui.notify(`Sandbox initialization failed: ${err instanceof Error ? err.message : err}`, "error"); - } - }); - - pi.on("session_shutdown", async () => { - if (sandboxInitialized) { - try { - await SandboxManager.reset(); - } catch { - // Ignore cleanup errors - } - } - }); - - pi.registerCommand("sandbox", { - description: "Show sandbox configuration", - handler: async (_args, ctx) => { - if (!sandboxEnabled) { - ctx.ui.notify("Sandbox is disabled", "info"); - return; - } - - const config = loadConfig(ctx.cwd); - const lines = [ - "Sandbox Configuration:", - "", - "Network:", - ` Allowed: ${config.network?.allowedDomains?.join(", ") || "(none)"}`, - ` Denied: ${config.network?.deniedDomains?.join(", ") || "(none)"}`, - "", - "Filesystem:", - ` Deny Read: ${config.filesystem?.denyRead?.join(", ") || "(none)"}`, - ` Allow Write: ${config.filesystem?.allowWrite?.join(", ") || "(none)"}`, - ` Deny Write: ${config.filesystem?.denyWrite?.join(", ") || "(none)"}`, - ]; - ctx.ui.notify(lines.join("\n"), "info"); - }, - }); -} diff --git a/packages/coding-agent/examples/extensions/sandbox/package-lock.json b/packages/coding-agent/examples/extensions/sandbox/package-lock.json deleted file mode 100644 index 90ef1c2f5..000000000 --- a/packages/coding-agent/examples/extensions/sandbox/package-lock.json +++ /dev/null @@ -1,92 +0,0 @@ -{ - "name": "pi-extension-sandbox", - "version": "1.4.0", - "lockfileVersion": 3, - "requires": true, - "packages": { - "": { - "name": "pi-extension-sandbox", - "version": "1.4.0", - "dependencies": { - "@anthropic-ai/sandbox-runtime": "^0.0.26" - } - }, - "node_modules/@anthropic-ai/sandbox-runtime": { - "version": "0.0.26", - "resolved": "https://registry.npmjs.org/@anthropic-ai/sandbox-runtime/-/sandbox-runtime-0.0.26.tgz", - "integrity": "sha512-DYV5LSsVMnzq0lbfaYMSpxZPUMAx4+hy343dRss+pVCLIfF62qOhxpYfZ5TmOk1GTDQm5f9wPprMNSStmnsV4w==", - "license": "Apache-2.0", - "dependencies": { - "@pondwader/socks5-server": "^1.0.10", - "@types/lodash-es": "^4.17.12", - "commander": "^12.1.0", - "lodash-es": "^4.17.21", - "shell-quote": "^1.8.3", - "zod": "^3.24.1" - }, - "bin": { - "srt": "dist/cli.js" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "node_modules/@pondwader/socks5-server": { - "version": "1.0.10", - "resolved": "https://registry.npmjs.org/@pondwader/socks5-server/-/socks5-server-1.0.10.tgz", - "integrity": "sha512-bQY06wzzR8D2+vVCUoBsr5QS2U6UgPUQRmErNwtsuI6vLcyRKkafjkr3KxbtGFf9aBBIV2mcvlsKD1UYaIV+sg==", - "license": "MIT" - }, - "node_modules/@types/lodash": { - "version": "4.17.23", - "resolved": "https://registry.npmjs.org/@types/lodash/-/lodash-4.17.23.tgz", - "integrity": "sha512-RDvF6wTulMPjrNdCoYRC8gNR880JNGT8uB+REUpC2Ns4pRqQJhGz90wh7rgdXDPpCczF3VGktDuFGVnz8zP7HA==", - "license": "MIT" - }, - "node_modules/@types/lodash-es": { - "version": "4.17.12", - "resolved": "https://registry.npmjs.org/@types/lodash-es/-/lodash-es-4.17.12.tgz", - "integrity": "sha512-0NgftHUcV4v34VhXm8QBSftKVXtbkBG3ViCjs6+eJ5a6y6Mi/jiFGPc1sC7QK+9BFhWrURE3EOggmWaSxL9OzQ==", - "license": "MIT", - "dependencies": { - "@types/lodash": "*" - } - }, - "node_modules/commander": { - "version": "12.1.0", - "resolved": "https://registry.npmjs.org/commander/-/commander-12.1.0.tgz", - "integrity": "sha512-Vw8qHK3bZM9y/P10u3Vib8o/DdkvA2OtPtZvD871QKjy74Wj1WSKFILMPRPSdUSx5RFK1arlJzEtA4PkFgnbuA==", - "license": "MIT", - "engines": { - "node": ">=18" - } - }, - "node_modules/lodash-es": { - "version": "4.18.1", - "resolved": "https://registry.npmjs.org/lodash-es/-/lodash-es-4.18.1.tgz", - "integrity": "sha512-J8xewKD/Gk22OZbhpOVSwcs60zhd95ESDwezOFuA3/099925PdHJ7OFHNTGtajL3AlZkykD32HykiMo+BIBI8A==", - "license": "MIT" - }, - "node_modules/shell-quote": { - "version": "1.8.3", - "resolved": "https://registry.npmjs.org/shell-quote/-/shell-quote-1.8.3.tgz", - "integrity": "sha512-ObmnIF4hXNg1BqhnHmgbDETF8dLPCggZWBjkQfhZpbszZnYur5DUljTcCHii5LC3J5E0yeO/1LIMyH+UvHQgyw==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/zod": { - "version": "3.25.76", - "resolved": "https://registry.npmjs.org/zod/-/zod-3.25.76.tgz", - "integrity": "sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ==", - "license": "MIT", - "funding": { - "url": "https://github.com/sponsors/colinhacks" - } - } - } -} diff --git a/packages/coding-agent/examples/extensions/sandbox/package.json b/packages/coding-agent/examples/extensions/sandbox/package.json deleted file mode 100644 index 8059d577d..000000000 --- a/packages/coding-agent/examples/extensions/sandbox/package.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "name": "pi-extension-sandbox", - "private": true, - "version": "1.4.0", - "type": "module", - "scripts": { - "clean": "echo 'nothing to clean'", - "build": "echo 'nothing to build'", - "check": "echo 'nothing to check'" - }, - "pi": { - "extensions": [ - "./index.ts" - ] - }, - "dependencies": { - "@anthropic-ai/sandbox-runtime": "0.0.26" - } -} diff --git a/packages/coding-agent/install-lock/package-lock.json b/packages/coding-agent/install-lock/package-lock.json index 25e273cf6..ed922e44f 100644 --- a/packages/coding-agent/install-lock/package-lock.json +++ b/packages/coding-agent/install-lock/package-lock.json @@ -511,7 +511,7 @@ "hosted-git-info": "9.0.3", "ignore": "7.0.8", "jiti": "2.7.0", - "minimatch": "10.2.6", + "minimatch": "11.0.3", "proper-lockfile": "4.1.2", "quickjs-wasi": "3.6.2", "semver": "7.8.5", @@ -582,6 +582,12 @@ }, "hasInstallScript": true }, + "node_modules/@isaacs/nonrecursive-glob-match": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/@isaacs/nonrecursive-glob-match/-/nonrecursive-glob-match-1.0.3.tgz", + "integrity": "sha512-SI7kyKHq4XXDRozw+r9GlRZkia1ZkFrpLW98r2IC2Bpr8YkFGE3fvjzmxCG0x78A++69HmDNrDvSMYBZmHGpKA==", + "license": "BlueOak-1.0.0" + }, "node_modules/@protobufjs/aspromise": { "version": "1.1.2", "resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz", @@ -1174,12 +1180,13 @@ } }, "node_modules/minimatch": { - "version": "10.2.6", - "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.6.tgz", - "integrity": "sha512-vpLQEs+VLCr1nU0BXS07maYoFwlDAH0gngQuuttxIwutDFEMHq2blX+8vpgxDdK3J1PwjCJiep77OitTZ4Ll1A==", + "version": "11.0.3", + "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-11.0.3.tgz", + "integrity": "sha512-iny/40PEGkNJQ5/Pv6z3nm6FbRAby32GeZRjCUJk2WnNDbgPSTcdI+DDvLqsgBdD4cD5sMtMheqPPnehCCa8Nw==", "license": "BlueOak-1.0.0", "dependencies": { - "brace-expansion": "^5.0.8" + "@isaacs/nonrecursive-glob-match": "^1.0.3", + "brace-expansion": "^5.0.12" }, "engines": { "node": "18 || 20 || >=22" diff --git a/packages/coding-agent/package.json b/packages/coding-agent/package.json index 0e452d9b5..f8e6d41cc 100644 --- a/packages/coding-agent/package.json +++ b/packages/coding-agent/package.json @@ -29,12 +29,12 @@ "CHANGELOG.md" ], "scripts": { - "clean": "shx rm -rf dist", + "clean": "node ../../scripts/package-files.mjs clean dist", "build": "npm run build:unbundled && node ../../scripts/build-coding-agent-bundle.mjs", - "build:unbundled": "tsc -p tsconfig.build.json && shx chmod +x dist/cli.js dist/rpc-entry.js && npm run copy-assets", + "build:unbundled": "tsc -p tsconfig.build.json && node ../../scripts/package-files.mjs executable dist/cli.js dist/rpc-entry.js && npm run copy-assets", "build:binary": "npm --prefix ../tui run build && npm --prefix ../telemetry run build && npm --prefix ../ai run build && npm --prefix ../agent run build && npm run build && bun build --compile --no-compile-autoload-bunfig --no-compile-autoload-dotenv ./dist/bun/cli.js ./src/utils/image-resize-worker.ts ./src/extensions/codemode/worker.ts --outfile dist/pi && npm run copy-binary-assets", - "copy-assets": "shx mkdir -p dist/modes/interactive/theme && shx cp src/modes/interactive/theme/*.json dist/modes/interactive/theme/ && shx mkdir -p dist/modes/interactive/assets && shx cp src/modes/interactive/assets/*.png dist/modes/interactive/assets/ && shx mkdir -p dist/core/export-html/vendor && shx cp src/core/export-html/template.html src/core/export-html/template.css src/core/export-html/template.js dist/core/export-html/ && shx cp src/core/export-html/vendor/*.js dist/core/export-html/vendor/", - "copy-binary-assets": "shx cp package.json dist/ && shx cp README.md dist/ && shx cp CHANGELOG.md dist/ && shx mkdir -p dist/theme && shx cp src/modes/interactive/theme/*.json dist/theme/ && shx mkdir -p dist/assets && shx cp src/modes/interactive/assets/*.png dist/assets/ && shx mkdir -p dist/export-html/vendor && shx cp src/core/export-html/template.html dist/export-html/ && shx cp src/core/export-html/vendor/*.js dist/export-html/vendor/ && shx cp -r docs dist/ && shx cp -r examples dist/ && shx cp ../../node_modules/@silvia-odwyer/photon-node/photon_rs_bg.wasm dist/", + "copy-assets": "node ../../scripts/package-files.mjs copy \"src/modes/interactive/theme/*.json\" dist/modes/interactive/theme && node ../../scripts/package-files.mjs copy \"src/modes/interactive/assets/*.png\" dist/modes/interactive/assets && node ../../scripts/package-files.mjs copy src/core/export-html/template.html src/core/export-html/template.css src/core/export-html/template.js dist/core/export-html && node ../../scripts/package-files.mjs copy \"src/core/export-html/vendor/*.js\" dist/core/export-html/vendor", + "copy-binary-assets": "node ../../scripts/package-files.mjs copy package.json README.md CHANGELOG.md docs examples ../../node_modules/@silvia-odwyer/photon-node/photon_rs_bg.wasm dist && node ../../scripts/package-files.mjs copy \"src/modes/interactive/theme/*.json\" dist/theme && node ../../scripts/package-files.mjs copy \"src/modes/interactive/assets/*.png\" dist/assets && node ../../scripts/package-files.mjs copy src/core/export-html/template.html dist/export-html && node ../../scripts/package-files.mjs copy \"src/core/export-html/vendor/*.js\" dist/export-html/vendor", "test": "vitest --run", "generate:schemas": "node --import ./src/experimental/source-resolver.ts scripts/generate-schemas.ts", "generate:references": "node --import ./src/experimental/source-resolver.ts scripts/generate-references.ts", @@ -56,7 +56,7 @@ "hosted-git-info": "9.0.3", "ignore": "7.0.8", "jiti": "2.7.0", - "minimatch": "10.2.6", + "minimatch": "11.0.3", "proper-lockfile": "4.1.2", "quickjs-wasi": "3.6.2", "semver": "7.8.5", @@ -74,7 +74,6 @@ "@types/proper-lockfile": "4.1.4", "@types/semver": "7.7.1", "marked": "18.0.11", - "shx": "0.3.4", "vitest": "4.1.12" }, "keywords": [ diff --git a/packages/durable/package.json b/packages/durable/package.json index 933fe6db7..57ced6bbb 100644 --- a/packages/durable/package.json +++ b/packages/durable/package.json @@ -123,7 +123,7 @@ "CHANGELOG.md" ], "scripts": { - "clean": "shx rm -rf dist", + "clean": "node ../../scripts/package-files.mjs clean dist", "build": "tsc -p tsconfig.build.json", "test": "vitest --run && PI_DURABLE_DOCUMENT_RETENTION_MS=0 vitest --run && npm run test:store && npm run test:workers", "test:store": "vitest --run --config vitest.store.config.ts", @@ -167,9 +167,8 @@ "typebox": "1.3.27" }, "devDependencies": { - "@cloudflare/vitest-plugin": "1.4.4", + "@cloudflare/vitest-plugin": "1.4.10", "fake-indexeddb": "6.2.6", - "shx": "0.3.4", "vitest": "4.1.12" } } diff --git a/packages/env/package.json b/packages/env/package.json index ab09e6546..2bb941ce0 100644 --- a/packages/env/package.json +++ b/packages/env/package.json @@ -22,7 +22,7 @@ "CHANGELOG.md" ], "scripts": { - "clean": "shx rm -rf dist", + "clean": "node ../../scripts/package-files.mjs clean dist", "build": "tsc -p tsconfig.build.json", "build:daemon": "cargo build --manifest-path daemon/Cargo.toml --locked", "test": "vitest --run", @@ -49,7 +49,6 @@ "@earendil-works/pi-durable": "1.4.0" }, "devDependencies": { - "shx": "0.3.4", "vitest": "4.1.12" } } diff --git a/packages/evals/package.json b/packages/evals/package.json index e7cc7bde7..fa4400219 100644 --- a/packages/evals/package.json +++ b/packages/evals/package.json @@ -4,7 +4,7 @@ "private": true, "type": "module", "scripts": { - "clean": "shx rm -rf .eval", + "clean": "node ../../scripts/package-files.mjs clean .eval", "eval": "npm run eval:host && npm run eval:docs --", "eval:host": "vitest run --config vitest.evals.config.ts --project host", "eval:audit": "vitest run --config vitest.evals.config.ts --project audit", @@ -16,7 +16,6 @@ "@earendil-works/pi-coding-agent": "1.4.0", "@types/node": "22.19.19", "@vitest-evals/core": "0.15.0", - "shx": "0.3.4", "vitest": "4.1.12", "vitest-evals": "0.15.0" } diff --git a/packages/mcp/package.json b/packages/mcp/package.json index b016e8067..a884d2b0f 100644 --- a/packages/mcp/package.json +++ b/packages/mcp/package.json @@ -30,7 +30,7 @@ "LICENSES" ], "scripts": { - "clean": "shx rm -rf dist", + "clean": "node ../../scripts/package-files.mjs clean dist", "build": "tsc -p tsconfig.build.json", "test": "vitest --run", "prepublishOnly": "npm run clean && npm run build" @@ -55,7 +55,6 @@ }, "devDependencies": { "@types/cross-spawn": "6.0.6", - "shx": "0.3.4", "vitest": "4.1.12" } } diff --git a/packages/telemetry/package.json b/packages/telemetry/package.json index c571023b6..eca0a94e7 100644 --- a/packages/telemetry/package.json +++ b/packages/telemetry/package.json @@ -27,7 +27,7 @@ "README.md" ], "scripts": { - "clean": "shx rm -rf dist", + "clean": "node ../../scripts/package-files.mjs clean dist", "build": "tsc -p tsconfig.build.json", "test": "vitest --run", "prepublishOnly": "npm run clean && npm run build" diff --git a/packages/tui/package.json b/packages/tui/package.json index 31dc4744a..d5ce578be 100644 --- a/packages/tui/package.json +++ b/packages/tui/package.json @@ -5,7 +5,7 @@ "type": "module", "main": "dist/index.js", "scripts": { - "clean": "shx rm -rf dist", + "clean": "node ../../scripts/package-files.mjs clean dist", "build": "tsc -p tsconfig.build.json", "build:native:darwin": "native/darwin/build.sh", "build:native:linux": "native/linux/build.sh", diff --git a/scripts/npm-audit.mjs b/scripts/npm-audit.mjs index f636d2347..f2d391c90 100644 --- a/scripts/npm-audit.mjs +++ b/scripts/npm-audit.mjs @@ -10,15 +10,7 @@ const auditLevel = "moderate"; const severities = ["info", "low", "moderate", "high", "critical"]; /** @type {Record} */ -const acceptedAdvisories = { - "GHSA-86w9-cpqp-85rv": { - package: "node-forge", - reason: - "No fixed node-forge release exists. It is only reachable through @earendil-works/gondolin in the private, " + - "unpublished gondolin example extension. Gondolin only verifies leaf certificates against its own locally " + - "generated CA (public exponent 65537), so the low-exponent signature forgery does not apply.", - }, -}; +const acceptedAdvisories = {}; const result = spawnSync("npm", ["audit", "--omit=dev", "--json"], { encoding: "utf8", maxBuffer: 64 * 1024 * 1024 }); if (result.error) { diff --git a/scripts/package-files.mjs b/scripts/package-files.mjs new file mode 100644 index 000000000..6e64aa95f --- /dev/null +++ b/scripts/package-files.mjs @@ -0,0 +1,30 @@ +// Cross-platform file operations for package build scripts, using only Node built-ins. +import { chmodSync, cpSync, globSync, mkdirSync, rmSync, statSync } from "node:fs"; +import { basename, join } from "node:path"; + +const [command, ...paths] = process.argv.slice(2); +const usage = "Usage: package-files.mjs clean | copy | executable "; + +if (paths.length === 0) throw new Error(usage); + +switch (command) { + case "clean": + for (const path of paths) rmSync(path, { recursive: true, force: true }); + break; + case "copy": { + const destination = paths.pop(); + if (!destination || paths.length === 0) throw new Error(usage); + mkdirSync(destination, { recursive: true }); + for (const pattern of paths) { + const sources = globSync(pattern); + if (sources.length === 0) throw new Error(`No files match ${pattern}`); + for (const source of sources) cpSync(source, join(destination, basename(source)), { recursive: true }); + } + break; + } + case "executable": + for (const path of paths) chmodSync(path, statSync(path).mode | 0o111); + break; + default: + throw new Error(usage); +} diff --git a/scripts/package-files.test.mjs b/scripts/package-files.test.mjs new file mode 100644 index 000000000..9b455a792 --- /dev/null +++ b/scripts/package-files.test.mjs @@ -0,0 +1,82 @@ +// Exercises the file operations used by package build scripts without touching build output. +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { test } from "node:test"; +import { fileURLToPath } from "node:url"; + +const script = fileURLToPath(new URL("./package-files.mjs", import.meta.url)); + +function workspace(t) { + const directory = mkdtempSync(join(tmpdir(), "pi-package-files-")); + t.after(() => rmSync(directory, { recursive: true, force: true })); + return directory; +} + +function run(directory, ...args) { + return spawnSync(process.execPath, [script, ...args], { cwd: directory, encoding: "utf8" }); +} + +test("clean removes directories and files and ignores missing paths", (t) => { + const directory = workspace(t); + mkdirSync(join(directory, "dist", "nested"), { recursive: true }); + writeFileSync(join(directory, "dist", "nested", "file.js"), "output"); + writeFileSync(join(directory, "output.txt"), "output"); + assert.equal(run(directory, "clean", "dist", "output.txt", "missing").status, 0); + assert.equal(existsSync(join(directory, "dist")), false); + assert.equal(existsSync(join(directory, "output.txt")), false); +}); + +test("copy expands patterns, creates directories, and overwrites existing assets", (t) => { + const directory = workspace(t); + mkdirSync(join(directory, "source assets")); + writeFileSync(join(directory, "source assets", "dark.json"), "dark"); + writeFileSync(join(directory, "source assets", "light.json"), "light"); + writeFileSync(join(directory, "source assets", "ignored.txt"), "ignored"); + const args = ["copy", "source assets/*.json", "output/assets"]; + assert.equal(run(directory, ...args).status, 0); + assert.equal(readFileSync(join(directory, "output/assets/dark.json"), "utf8"), "dark"); + assert.equal(readFileSync(join(directory, "output/assets/light.json"), "utf8"), "light"); + assert.equal(existsSync(join(directory, "output/assets/ignored.txt")), false); + writeFileSync(join(directory, "source assets", "dark.json"), "updated"); + assert.equal(run(directory, ...args).status, 0); + assert.equal(readFileSync(join(directory, "output/assets/dark.json"), "utf8"), "updated"); +}); + +test("copy accepts multiple files and preserves directory names and nested contents", (t) => { + const directory = workspace(t); + mkdirSync(join(directory, "docs", "nested"), { recursive: true }); + writeFileSync(join(directory, "docs", "nested", "guide.md"), "guide"); + writeFileSync(join(directory, "README.md"), "readme"); + assert.equal(run(directory, "copy", "docs", "README.md", "dist").status, 0); + assert.equal(readFileSync(join(directory, "dist", "docs", "nested", "guide.md"), "utf8"), "guide"); + assert.equal(readFileSync(join(directory, "dist", "README.md"), "utf8"), "readme"); +}); + +test("copy fails when an asset pattern matches nothing", (t) => { + const result = run(workspace(t), "copy", "missing/*.json", "dist"); + assert.notEqual(result.status, 0); + assert.match(result.stderr, /No files match missing/); +}); + +test("executable adds executable bits without removing existing permissions", (t) => { + const directory = workspace(t); + for (const name of ["cli.js", "rpc.js"]) writeFileSync(join(directory, name), "entry", { mode: 0o640 }); + const mode = statSync(join(directory, "cli.js")).mode; + assert.equal(run(directory, "executable", "cli.js", "rpc.js").status, 0); + if (process.platform !== "win32") { + assert.equal(statSync(join(directory, "cli.js")).mode & 0o777, (mode | 0o111) & 0o777); + assert.equal(statSync(join(directory, "rpc.js")).mode & 0o111, 0o111); + } +}); + +test("invalid commands and missing arguments fail", (t) => { + const directory = workspace(t); + for (const args of [[], ["unknown", "dist"], ["clean"], ["copy", "dist"], ["executable"]]) { + const result = run(directory, ...args); + assert.notEqual(result.status, 0); + assert.match(result.stderr, /Usage: package-files/); + } +}); From 70ca8e8960b4f05faf1affd752e6231ba6087605 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Petr=20Hou=C5=A1ka?= Date: Sat, 31 Oct 2026 17:55:58 +0100 Subject: [PATCH 097/175] docs(coding-agent): list external limitations as known issues with remedies (#11647) Restructures external limitations in the tmux and Windows guides as "Known issues" with a remedy for each entry. - `tmux.md`: the tmux 3.6 color reply section becomes a Known issues table (issue, affected versions, remedy). - `windows.md`: the conhost problem table moves to a Known issues section at the end of the page, with a Remedy column instead of the cause column. The terminal table links to it. Issue links are reduced to one per row. Known issues are limited to things outside Pi's control or deliberate limitations. Bugs Pi can fix stay in the issue tracker. --- packages/coding-agent/docs/tmux.md | 6 ++++-- packages/coding-agent/docs/windows.md | 24 +++++++++++++----------- 2 files changed, 17 insertions(+), 13 deletions(-) diff --git a/packages/coding-agent/docs/tmux.md b/packages/coding-agent/docs/tmux.md index c781ffe0a..1a640d174 100644 --- a/packages/coding-agent/docs/tmux.md +++ b/packages/coding-agent/docs/tmux.md @@ -54,6 +54,8 @@ Pi supports the xterm `modifyOtherKeys` format used by these versions. Restart t For older versions, upgrade tmux or use Pi outside tmux rather than relying on modified Enter shortcuts. -## Color codes appear in the editor with tmux 3.6 +## Known issues -If text such as `c0c/0c0c/0c0c` appears in the editor at startup, upgrade tmux to 3.7 or newer. tmux 3.6 sometimes sends the terminal's color replies to Pi as typed text ([tmux#4749](https://github.com/tmux/tmux/issues/4749)). Changing Pi's theme does not help. +| Issue | Affects | Remedy | +|---|---|---| +| Text such as `c0c/0c0c/0c0c` appears in the editor at startup ([tmux#4749](https://github.com/tmux/tmux/issues/4749)) | tmux 3.6 | Upgrade tmux. Changing Pi's theme does not help. | diff --git a/packages/coding-agent/docs/windows.md b/packages/coding-agent/docs/windows.md index 18826cd02..6e68979b8 100644 --- a/packages/coding-agent/docs/windows.md +++ b/packages/coding-agent/docs/windows.md @@ -89,18 +89,20 @@ Last updated: October 2026. | [Windows Terminal](https://aka.ms/terminal) | Bundled, current | Recommended | | [VS Code](https://code.visualstudio.com), [Tabby](https://tabby.sh), [WezTerm](https://wezterm.org) nightly | Bundled, current | Works | | [Alacritty](https://alacritty.org) | Built-in, unless `conpty.dll` and `OpenConsole.exe` from Windows Terminal are next to `alacritty.exe` | Works only with those files | -| WezTerm 20240203 and older | Bundled, old | Known problems | -| Classic console (`cmd.exe` or PowerShell without Windows Terminal), mintty (Git Bash), ConEmu, Cmder, [Hyper](https://hyper.is), Windows OpenSSH | Built-in | Known problems | - -| Problem | Where | Why | -|---|---|---| -| Switching from fullscreen to regular mode, or `Ctrl+G` from fullscreen, crashes the session ([#9186](https://github.com/earendil-works/pi/issues/9186), [microsoft/terminal#17515](https://github.com/microsoft/terminal/issues/17515)). Start with `--tui-mode regular` to avoid it. | Built-in | Conhost crashes when leaving the alternate screen while input is in line mode. | -| Text such as `;0;rgb:0000/0000/0000` appears in the prompt and BEL opens the external editor ([#10256](https://github.com/earendil-works/pi/issues/10256), [#10362](https://github.com/earendil-works/pi/issues/10362), [#10742](https://github.com/earendil-works/pi/issues/10742)) | Built-in behind another terminal (mintty, OpenSSH) | Conhost drops the start of terminal replies that arrive in pieces. | -| Fragments such as `79` or `55` appear in the prompt ([#10717](https://github.com/earendil-works/pi/issues/10717), [#10970](https://github.com/earendil-works/pi/issues/10970)) | WezTerm 20240203 | The bundled conhost cuts terminal replies short. | -| In regular mode, each keystroke redraws the input line on a new row ([#6300](https://github.com/earendil-works/pi/issues/6300)) | Built-in, classic console | Suspected: cursor wraps after full-width lines. | - -Microsoft has fixed these bugs in its source, but has not said when the built-in conhost will get the fixes. +| WezTerm 20240203 and older | Bundled, old | [Known issues](#known-issues) | +| Classic console (`cmd.exe` or PowerShell without Windows Terminal), mintty (Git Bash), ConEmu, Cmder, [Hyper](https://hyper.is), Windows OpenSSH | Built-in | [Known issues](#known-issues) | ## Configure Windows Terminal Windows Terminal reserves or rewrites some modified keys. See [Windows Terminal](terminal-setup.md#windows-terminal) to configure `Shift+Enter` and `Alt+Enter`, and [Keybindings](keybindings.md) for Pi's Windows and WSL shortcut defaults. + +## Known issues + +These issues come from conhost. Microsoft has fixed them in its source, but has not said when the built-in conhost will get the fixes. + +| Issue | Affects | Remedy | +|---|---|---| +| Switching from fullscreen to regular mode, or `Ctrl+G` from fullscreen, crashes the session ([#9186](https://github.com/earendil-works/pi/issues/9186), [microsoft/terminal#17515](https://github.com/microsoft/terminal/issues/17515)) | Built-in conhost | Use a terminal with a current conhost, or start Pi with `--tui-mode regular` | +| Text such as `;0;rgb:0000/0000/0000` appears in the prompt, and BEL opens the external editor ([#10256](https://github.com/earendil-works/pi/issues/10256)) | Built-in conhost behind another terminal: mintty, Windows OpenSSH | Use a terminal with a current conhost | +| Fragments such as `79` or `55` appear in the prompt ([#10717](https://github.com/earendil-works/pi/issues/10717)) | WezTerm 20240203 | Use WezTerm nightly | +| In regular mode, each keystroke redraws the input line on a new row ([#6300](https://github.com/earendil-works/pi/issues/6300)) | Built-in conhost, classic console | Use fullscreen mode (the default), or a terminal with a current conhost | From e3da248ea9100a3b5d53f9c701b6a47ef7f58d6e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Petr=20Hou=C5=A1ka?= Date: Sat, 31 Oct 2026 18:19:44 +0100 Subject: [PATCH 098/175] fix(coding-agent): keep updating git packages when git clean fails (#11648) Closes #11597 ## Problem On Windows, `pi update` stopped with `git clean -fdx failed with code 1` for git extensions such as pi-lens. 1. Another pi session has the extension loaded, so a native module (`@ast-grep/napi` `.node`) or `ast-grep.exe` is open in that process. 2. npm renames the in-use package dir to `node_modules/@ast-grep/.napi-win32-x64-msvc-`. Windows allows the rename, but npm cannot delete the dir afterwards. 3. The next `pi update` runs `git reset --hard`, then `git clean -fdx`. git prints `warning: failed to remove ...: Invalid argument` and exits 1. 4. pi threw, leaving the checkout at the new commit with the old dependencies. The suggested `-ffdx` does not help. Reproduced locally (git 2.52, Windows): with a running `.exe`, or a `.node` loaded by another node process and its dir renamed, both `-fdx` and `-ffdx` fail with `Invalid argument`. Once the process exits, `-fdx` succeeds. ## Fix In `cleanAndInstallGitDependencies`, a failed `git clean` no longer aborts the update. pi: - prints a warning that files may be in use by another pi session - still runs `npm install`, so dependencies match the new commit - keeps the `.pi-update-incomplete` marker, so the next update retries the clean ## Tests - `test/git-update.test.ts`: new regression test. A failing clean still updates and installs dependencies, warns, and keeps the marker. The next update cleans and removes the marker. - `test/package-manager.test.ts`: updated the test that expected the throw. `npm run check` reports `install-lock is out of date`, and `package-manager.test.ts > should expand ~ in exclusion and exact patterns` fails on Windows. Both also happen on `main` without this change. --- packages/coding-agent/CHANGELOG.md | 1 + .../coding-agent/src/core/package-manager.ts | 21 ++++++-- packages/coding-agent/test/git-update.test.ts | 53 +++++++++++++++++++ .../coding-agent/test/package-manager.test.ts | 5 +- 4 files changed, 73 insertions(+), 7 deletions(-) diff --git a/packages/coding-agent/CHANGELOG.md b/packages/coding-agent/CHANGELOG.md index 8a916e873..359250c80 100644 --- a/packages/coding-agent/CHANGELOG.md +++ b/packages/coding-agent/CHANGELOG.md @@ -48,6 +48,7 @@ - Fixed the external editor failing silently, for example when the editor command is wrong or a Windows editor cannot start from a deep WSL directory. pi now shows a status message such as `External editor exited with code 1; prompt unchanged` ([#11592](https://github.com/earendil-works/pi/issues/11592)) - Fixed `tool_search` and codemode `searchTools()` ranking a tool below tools that only mention its name in their descriptions, for example `peekaboo app` ranking `mcp__peekaboo__app` 14th. A tool matching more of the query's words now ranks above one matching fewer, a query word in a tool's name scores more than any number of mentions in its description, a tool whose whole name is in the query gets a large bonus, and long server instructions no longer lower the scores of that server's tools ([#11331](https://github.com/earendil-works/pi/issues/11331)) - Fixed `tool_search` and codemode `searchTools()` finding nothing for Chinese, Japanese, Thai, and other non-ASCII queries, because the tokenizer kept only ASCII letters and digits. Words in any script now match, text without spaces between words such as `搜索记录` is split into words, and canonically equivalent text such as `café` and `cafe\u0301` matches ([#10816](https://github.com/earendil-works/pi/issues/10816), [#11037](https://github.com/earendil-works/pi/pull/11037) by [@haoqixu](https://github.com/haoqixu)) +- Fixed `pi update` stopping with `git clean -fdx failed with code 1` on Windows when another pi session had a git extension's native module or executable open. Windows cannot delete such files, so the update left the new code with the old dependencies. pi now warns, installs the dependencies anyway, and retries the clean on the next update ([#11597](https://github.com/earendil-works/pi/issues/11597)) ### Removed diff --git a/packages/coding-agent/src/core/package-manager.ts b/packages/coding-agent/src/core/package-manager.ts index 1a0c96932..5f4229b24 100644 --- a/packages/coding-agent/src/core/package-manager.ts +++ b/packages/coding-agent/src/core/package-manager.ts @@ -2090,20 +2090,31 @@ export class DefaultPackageManager implements PackageManager { } private async cleanAndInstallGitDependencies(targetDir: string, markerPath: string): Promise { - // Clean untracked files (extensions should be pristine). If this fails after - // deleting dependencies, repair them so the existing extension still loads. + // Clean untracked files (extensions should be pristine). On Windows, files loaded by + // another process (e.g. a native module of this extension in another pi session) cannot + // be deleted, so git clean fails. Still install dependencies for the new commit, but keep + // the marker so the next update retries the clean. + let cleaned = true; try { await this.runCommand("git", ["clean", "-fdx"], { cwd: targetDir }); } catch (error) { - await this.repairMissingGitDependencies(targetDir).catch(() => {}); - throw error; + cleaned = false; + const reason = error instanceof Error ? error.message : String(error); + this.emitProgress({ + type: "progress", + action: "update", + source: targetDir, + message: `Could not remove all untracked files in ${targetDir} (${reason}). Files may be in use by another pi session; close it and run the update again.`, + }); } const packageJsonPath = join(targetDir, "package.json"); if (existsSync(packageJsonPath)) { await this.runNpmCommand(this.getGitDependencyInstallArgs(), { cwd: targetDir }); } - rmSync(markerPath, { force: true }); + if (cleaned) { + rmSync(markerPath, { force: true }); + } } private async ensureGitRef(targetDir: string, fetchArgs: string[], ref: string): Promise { diff --git a/packages/coding-agent/test/git-update.test.ts b/packages/coding-agent/test/git-update.test.ts index 80308277c..8b771f688 100644 --- a/packages/coding-agent/test/git-update.test.ts +++ b/packages/coding-agent/test/git-update.test.ts @@ -178,6 +178,59 @@ describe("DefaultPackageManager git update", () => { }); }); + describe("git clean failures", () => { + // https://github.com/earendil-works/pi/issues/11597 + it("should install dependencies and retry the clean later when git clean fails", async () => { + setupRemoteAndInstall(); + const newCommit = createCommit( + remoteDir, + "package.json", + JSON.stringify({ name: "test-extension", version: "1.0.0" }), + "Add package.json", + ); + + const executedCommands: string[] = []; + let failClean = true; + const managerWithInternals = packageManager as unknown as { + runCommand: (command: string, args: string[], options?: { cwd?: string }) => Promise; + }; + managerWithInternals.runCommand = async (command, args, options) => { + const commandLine = `${command} ${args.join(" ")}`; + executedCommands.push(commandLine); + if (command === "npm") return; + if (commandLine === "git clean -fdx" && failClean) { + throw new Error("git clean -fdx failed with code 1"); + } + const result = spawnSync(command, args, { cwd: options?.cwd, encoding: "utf-8" }); + if (result.status !== 0) { + throw new Error(`Command failed: ${commandLine}\n${result.stderr}`); + } + }; + const progressMessages: string[] = []; + packageManager.setProgressCallback((event) => { + if (event.type === "progress" && event.message) progressMessages.push(event.message); + }); + const markerPath = join(agentDir, "git", "github.com", "test", ".extension.pi-update-incomplete"); + + await packageManager.update(); + + expect(getCurrentCommit(installedDir)).toBe(newCommit); + expect(executedCommands).toContain("git clean -fdx"); + expect(executedCommands.some((command) => command.startsWith("npm install"))).toBe(true); + expect(progressMessages.some((message) => message.includes("Could not remove all untracked files"))).toBe( + true, + ); + expect(existsSync(markerPath)).toBe(true); + + failClean = false; + executedCommands.length = 0; + await packageManager.update(); + + expect(executedCommands).toContain("git clean -fdx"); + expect(existsSync(markerPath)).toBe(false); + }); + }); + describe("force-push scenarios", () => { it("should handle complete history rewrite", async () => { setupRemoteAndInstall(); diff --git a/packages/coding-agent/test/package-manager.test.ts b/packages/coding-agent/test/package-manager.test.ts index bdeffba40..b5895b331 100644 --- a/packages/coding-agent/test/package-manager.test.ts +++ b/packages/coding-agent/test/package-manager.test.ts @@ -1211,7 +1211,8 @@ Content`, expect(runCommandSpy).not.toHaveBeenCalledWith("git", ["clean", "-fdx"], { cwd: targetDir }); }); - it("should repair deleted git package dependencies when cleaning fails", async () => { + // https://github.com/earendil-works/pi/issues/11597 + it("should install git package dependencies when cleaning fails", async () => { const source = "git:github.com/user/repo"; const targetDir = join(agentDir, "git", "github.com", "user", "repo"); const fetchArgs = ["fetch", "--prune", "--no-tags", "origin", "+refs/heads/main:refs/remotes/origin/main"]; @@ -1237,7 +1238,7 @@ Content`, if (args[0] === "clean") throw new Error("simulated clean failure"); }); - await expect(packageManager.update(source)).rejects.toThrow("simulated clean failure"); + await packageManager.update(source); expect(runCommandSpy).toHaveBeenCalledWith( "npm", From d52e953fa846ce91426dcf41525a9d08ab5b29fc Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 17:47:30 +0000 Subject: [PATCH 099/175] docs(chord): slice 10 route overhead accepted --- packages/chord/docs/next/decisions.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index ec2672c24..ed88f9a7d 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2084,6 +2084,11 @@ validation and copies the v2 kernel added around Delta. is trusted (a plugin's own file). (2) The worker lock on macOS and the BSDs fails closed on a stale lock file, with an error naming it (no takeover; Linux and Android use abstract sockets and are unaffected). (3) A route's overhead: Opus's two trims, measured again after the `fetch` rework, and the numbers brought to the owner. +- **Slice 10: a route's overhead accepted; no header accessor (owner, 2026-10-31: "1 yes, 2 no"):** after the `fetch` + rework and the Node fast path a route costs +20-21 µs over a plain Node handler on the standard bench, +12-13 µs warmed + (before the rework +4.5 µs): the request's scope (2.1 µs), URL parsing, admission and routing, the baseline's headers, + the scope's seal, and about 5 µs of Node's microtask machinery an async path triggers. Accepted against the 10% bar. + No `request.header(name)` accessor: an authenticator reading headers builds the Web `Request` (about 3.5 µs more). - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 83740ffa7a91d294cbc08569699ab6718631629a Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 18:33:24 +0000 Subject: [PATCH 100/175] docs(chord): CSP nonce per response --- packages/chord/docs/next/decisions.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index ed88f9a7d..7b5d9f26f 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2089,6 +2089,9 @@ validation and copies the v2 kernel added around Delta. (before the rework +4.5 µs): the request's scope (2.1 µs), URL parsing, admission and routing, the baseline's headers, the scope's seal, and about 5 µs of Node's microtask machinery an async path triggers. Accepted against the 10% bar. No `request.header(name)` accessor: an authenticator reading headers builds the Web `Request` (about 3.5 µs more). +- **A CSP nonce per response (owner, 2026-10-31: "csp ok"; Fable's recheck):** the HTTP component makes a fresh script + nonce for every response (`request.nonce`), not one per server: a nonce read from one page cannot be reused by an + injected script elsewhere. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 2b06c437cec65bf0b729a25c0e50972d7b4ae6e2 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 18:52:54 +0000 Subject: [PATCH 101/175] docs(chord): settings are arguments, extensions are registries --- packages/chord/docs/next/decisions.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 7b5d9f26f..4dfbcf8a4 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2092,6 +2092,16 @@ validation and copies the v2 kernel added around Delta. - **A CSP nonce per response (owner, 2026-10-31: "csp ok"; Fable's recheck):** the HTTP component makes a fresh script nonce for every response (`request.nonce`), not one per server: a nonce read from one page cannot be reused by an injected script elsewhere. +- **Settings are arguments, extensions are registries (owner, 2026-10-31: "making that config a registry is dumb; it's a + general problem for all host-lifetime natives that need initialization at host startup"; "k"; supersedes the + `HttpConfig` registry of "The HTTP server is a native object"):** what the app decides once at startup (a port, frame + limits, a timeout, a data directory, which implementation to use) is an argument to the component's facet, which base + code constructs; the facet and its host-lifetime objects close over it; changing it is a restart; no plugin can + touch it. What plugins add while the app runs (routes, authenticators, tools, panels, more indexers) is a registry. + HTTP: `createHttp(host, { listen, links, closeTimeout, onError }, ctx)`; `Http` and `Authenticators` stay registries. + The same for the harness (its store and defaults as arguments; models, environments, extensions as registries) and + the app kit's components (a session index takes its indexer as an argument, and may also follow a registry of + indexers if plugins should supply them). - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 0baf611f04a35618727ee15bbf8bd7f49f16175d Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 19:07:10 +0000 Subject: [PATCH 102/175] docs(chord): API versions in import paths; the plugin model proposal --- packages/chord/docs/next/decisions.md | 7 + packages/chord/docs/next/plugin-model/book.md | 186 +++++++++++ .../chord/docs/next/plugin-model/index.html | 310 ++++++++++++++++++ 3 files changed, 503 insertions(+) create mode 100644 packages/chord/docs/next/plugin-model/book.md create mode 100644 packages/chord/docs/next/plugin-model/index.html diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 4dfbcf8a4..0878cbcc0 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2102,6 +2102,13 @@ validation and copies the v2 kernel added around Delta. The same for the harness (its store and defaults as arguments; models, environments, extensions as registries) and the app kit's components (a session index takes its indexer as an argument, and may also follow a registry of indexers if plugins should supply them). +- **API versions in the import path (owner, 2026-10-31: "this sounds about right"; "must be declared by the plugin in + code, we do not want to dig around package.json files anymore"):** the app's base code is internal except its API + modules, which it exposes as host packages with a version in the path (`@acme/app/v3`; Chord's + `@earendil-works/chord/next/v2`); the app lists the versions it still provides (an older one as the same module, or a + thin layer); an import of an unlisted version is a build error naming the available ones. No `package.json` field + and no npm ranges for this. Contracts and registry tokens inside an API keep their own versions. The rest of the + plugin model is a proposal (`docs/next/plugin-model/`). - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a diff --git a/packages/chord/docs/next/plugin-model/book.md b/packages/chord/docs/next/plugin-model/book.md new file mode 100644 index 000000000..5717711a3 --- /dev/null +++ b/packages/chord/docs/next/plugin-model/book.md @@ -0,0 +1,186 @@ +--- +title: The Plugin Model: what a plugin is, what it may import, and how versions meet +cover: The Plugin / Model +kicker: A design proposal · Chord v2 · 2026-10-31 +subtitle: Plugins, base code, versioned APIs and contracts, in one place, as discussed with the owner on 2026-10-31 +stats: + - 4 | kinds of import a plugin may write + - 1 | integer per API version, in the import path + - 0 | package.json fields Chord reads besides name and dependencies + - 0 | type checks in the build +note: | + A proposal written by the redesign session after a day of discussion with the owner. It collects what was decided + and what is proposed into one model. Decided points cite their decision in `decisions.md`; proposed ones are marked. + Nothing here is built yet beyond slices 7c and 10. +colophon: The Plugin Model · proposal · 2026-10-31 +--- + +# Before we begin {.preface} + +An app built on Chord is mostly plugins. Its own features are plugins, users add plugins, and agents write plugins +while the app runs. A plugin uses the app (where things live, who calls, how to declare objects), uses other plugins +(their objects and extension points), and uses npm packages. Each of those can change on its own schedule: the app +gets a new version, a plugin it depends on changes its contract, a package publishes a fix. + +The questions the owner kept asking were all versions of one: when a plugin is written against one state of the world +and runs against another, what does it get, and who notices a mismatch? This model answers it with one rule for each +kind of import, and versions that sit where the dependency is written: in the code. + +## Eight things to know + +:::findings +1. **A plugin is a directory, or a single file.** A directory has a `package.json` (its name, its npm dependencies) + and one file per host kind; a single file `hello.worker.ts` is a plugin named `hello` with no dependencies. +2. **Base code is internal unless it is an API.** The app exposes only versioned API modules to plugins; everything + else is unreachable. +3. **API versions are in the import path.** `import { Session } from "@acme/app/v3"`: the plugin says what it was + written for where it imports it; an unavailable version is a build error. +4. **APIs are split by host kind.** A shared module holds what every host may use (contracts, registry tokens, host + kinds); `/v3/web` and `/v3/tui` hold UI pieces; a worker never loads React. +5. **Contracts carry their own versions.** A plugin carries the contract version it was written for; the provider + binds it if it accepts that version, else the plugin waits with a clear reason. +6. **A plugin imports four things:** its own files, host packages, its npm dependencies, and the app's own plugins by + relative path. Nothing else. +7. **Settings are arguments, extensions are registries.** What the app decides at startup is passed to its components; + what plugins add is contributed. +8. **No type checking in the build.** Types come from where imports come from; editors and agents check, the build + only builds. +::: + +## Words used throughout + +**Base code** is the app's own program; its **API** is the part plugins may import, published as **API modules**, +one per version and host kind. A **host package** is a module the running process provides to every plugin (Chord, +the app's API modules). A **contract** is an object type's declaration; a **registry token** an extension point's. +A **facet** is a plugin's declarations for one host kind. + +# What a Plugin Is {#plugin lede="A directory or a single file, named once, with one file per host kind."} + +```text +plugins/notes/ a directory plugin + package.json { "name": "@acme/notes", "dependencies": { "slugify": "1.6.6" } } + contract.ts the object types and registry tokens it offers + worker.ts server.ts web.tsx one facet per host kind it needs +plugins/hello.worker.ts a single-file plugin "hello" with one worker facet +plugins/hello.web.tsx the same plugin's web facet +``` + +A directory plugin's name is its `package.json` name; Chord reads only `name` and `dependencies` from it. A +single-file plugin's name is its file's base name; it has no dependencies and imports only host packages. When a +single file needs a package or a second file, it becomes a directory. Two roots with the same name: the first root +wins, the other is reported (slice 7c). + +# Base Code: Internal and External {#base lede="What the app keeps to itself, what it exposes, and how exposure is versioned."} + +The app's base code is everything that starts the process: settings, startup, core objects' implementations, the +session index, migrations. Plugins must not depend on any of it, or every refactoring of the app would break them. +So the app draws a line and exposes only its **API**: + +```text +src/api/v3/index.ts shared: host kinds, the principal type, defineFacet per kind, core contracts, registry tokens +src/api/v3/web.ts web only: shared UI pieces (a Sheet, theme variables), React-typed registry tokens +src/api/v3/tui.ts TUI only: its UI pieces +src/… internal: everything else +``` + +Each API module is a host package of the kinds that may use it: `@acme/app/v3` for every kind, `@acme/app/v3/web` for +the web kind only. A worker plugin importing `@acme/app/v3/web` is a build error, and a worker process never loads +React. The shared module stays free of runtime imports of UI libraries: a registry token that needs a React type names +it with `import type`, which the build erases. + +:::note | Proposed +The split of the external API into a shared module and per-kind modules answers the owner's worry about "web, TUI, +worker, server things all in the same index.ts". Proposed, not yet decided. +::: + +# Versions in the Import Path {#versions lede="A plugin says what it was written for where it imports it."} + +The app's API changes over time: a host kind is renamed, a principal variant changes. A plugin written for an older +API must not silently get the new one. So every API module has a version in its path, and the app lists which versions +it still provides: + +```ts +// plugins/stats/worker.ts +import { defineFacet, Session, Panels } from "@acme/app/v3"; +import { remote, str } from "@earendil-works/chord/next/v2"; +``` + +- The app provides `/v3`, its current API. If version 2 plugins still work unchanged against it, `/v2` is listed too, + pointing to the same module. An app that needs a compatibility layer makes `/v2` a thin module over `/v3`. +- An import of a version the app no longer lists is a build error: "@acme/app/v1 is not available: this app provides + v3, v2". The plugin keeps its old code, and the message tells an agent exactly what to change. +- Nothing to fill in elsewhere: no `package.json` field, no npm ranges. Skeletons and examples import the current + version, so an agent copies it without thinking. +- Chord's own surface is versioned the same way (`@earendil-works/chord/next/v2`). + +Decided: "the API version must be declared by the plugin in code" (owner, 2026-10-31); the import path is how. + +# Contracts and Registries {#contracts lede="Finer versions inside the API, and what crosses between plugins."} + +Contracts and registry tokens keep their own versions inside the API, because they evolve independently of the API as +a whole: + +- **An object type**: the consumer carries the contract version it was written for; the provider binds it at that + version if it accepts it (`accepts` maps arguments forward and results and state back), otherwise the consumer waits + "contract mismatch: v1, served v3". Shapes are fingerprinted, so a changed shape without a version bump is caught. +- **A registry token**: a name, a version and TypeScript types for what is contributed and read. Values are in-process + things (functions, components), which cannot be fingerprinted: bumping the version when they change is the defining + plugin's rule, and its `accepts` adapts older contributions. +- **Host kinds on registries** (proposed): a token names the kinds that serve it (`hosts: ["web"]`), typed, so a worker + facet contributing to a web registry is a type error and a load error, not a silent no-op. + +A contract file imports only host packages at runtime (Chord, the app's API); native types it mentions are type-only. + +# What a Plugin May Import {#imports lede="Four kinds of import, each with one rule."} + +| Import | Resolved to | Versioned by | +| --- | --- | --- | +| its own files (`./store.ts`) | copied into its script | the plugin itself | +| host packages (`@acme/app/v3`, `@earendil-works/chord/next/v2`, `react`) | the running process's instance, shared | the path's version; the app lists what it provides | +| npm packages in its `dependencies` (`slugify`, a third party's contracts package) | its own `node_modules`, bundled | the installed version | +| the app's own plugins (`../notes/contract.ts`), in the same root | copied into its script | the same revision: one repository | + +A third-party plugin that wants the app's plugins' contracts gets them through the app's API (core contracts) or the +app's published package, as an npm dependency at the version it was written for, never from the running plugin's +files. Anything else (a path into base code, an unlisted version, a package not in its dependencies) is a build error. + +# Settings and Extensions {#settings lede="How the app configures its components without letting plugins override it."} + +Decided ("Settings are arguments, extensions are registries"): what the app decides at startup is an argument to the +component's facet, which base code constructs, and plugins cannot change it: + +```ts +const http = createHttp(server, { listen: { port: 8788 }, links: { path: "/chord", maxPayload: MiB }, closeTimeout: 10_000, onError }, ctx); +``` + +What plugins add while the app runs is a registry: routes, authenticators, tools, panels. A component may offer both +for one thing: the session index takes its indexer as an argument, and may also follow a registry of indexers if +plugins should supply them. + +# Building and Evaluating {#build lede="Unchanged from slices 7c and 10: one build on the server, evaluation everywhere."} + +Only the server builds, because only the server has the sources. `createPlugins` builds every kind and needs only +names: per kind its platform and its host package names. Each process evaluates the scripts of its kind with its own +module objects (the server's at `plugins.load`, a worker's in `workerMain`, a page's in its base bundle), so a plugin +shares that process's Chord and API. Base-code facets are passed beside the plugins; they are not plugins, because +they close over the process's own values and never reload. + +# Types and Tests {#tools lede="Where types come from, and how a plugin is tested alone."} + +Types come from where imports come from: relative files and npm packages resolve natively in TypeScript; the app's API +modules ship `.d.ts` per version in the app's package. There is no type checking in the build (the owner: seconds per +build is too slow); editors check as an author types, and an agent may run `tsc` as a tool. + +Testing a plugin alone (proposed): `testPlugin(dir, { kind, stubs })` builds the plugin like production, loads it on an +in-memory host with the kind's real host packages, and loads stub facets that serve the contracts it depends on. A stub +is an ordinary facet; there is no mocking framework. + +# Open Questions {.epilogue lede="What is proposed here and still to decide."} + +1. The external API split by host kind (`/v3`, `/v3/web`, `/v3/tui`). +2. Host kinds on registry tokens, typed and checked at load. +3. Single-file plugins (`hello.worker.ts`, named by the file). +4. Contracts of the app's plugins for third parties: through the API modules only, or also the app's npm package. +5. `testPlugin` with stub facets. +6. Reversing "Plugins import each other by name" (2026-10-31, morning) in favour of relative paths within the app + and npm dependencies across it. diff --git a/packages/chord/docs/next/plugin-model/index.html b/packages/chord/docs/next/plugin-model/index.html new file mode 100644 index 000000000..2549955ac --- /dev/null +++ b/packages/chord/docs/next/plugin-model/index.html @@ -0,0 +1,310 @@ + + +The Plugin Model: what a plugin is, what it may import, and how versions meet + + + + +

    +
    +
    A design proposal · Chord v2 · 2026-10-31
    +

    The Plugin
    Model

    +

    Plugins, base code, versioned APIs and contracts, in one place, as discussed with the owner on 2026-10-31

    +
    4kinds of import a plugin may write
    1integer per API version, in the import path
    0package.json fields Chord reads besides name and dependencies
    0type checks in the build
    +

    A proposal written by the redesign session after a day of discussion with the owner. It collects what was decided +and what is proposed into one model. Decided points cite their decision in decisions.md; proposed ones are marked. +Nothing here is built yet beyond slices 7c and 10.

    +
    +
    + + +
    +

    Before we begin

    +

    An app built on Chord is mostly plugins. Its own features are plugins, users add plugins, and agents write plugins +while the app runs. A plugin uses the app (where things live, who calls, how to declare objects), uses other plugins +(their objects and extension points), and uses npm packages. Each of those can change on its own schedule: the app +gets a new version, a plugin it depends on changes its contract, a package publishes a fix.

    +

    The questions the owner kept asking were all versions of one: when a plugin is written against one state of the world +and runs against another, what does it get, and who notices a mismatch? This model answers it with one rule for each +kind of import, and versions that sit where the dependency is written: in the code.

    +

    Eight things to know

    +
      +
    1. A plugin is a directory, or a single file. A directory has a package.json (its name, its npm dependencies) +and one file per host kind; a single file hello.worker.ts is a plugin named hello with no dependencies.
      2. +
    2. Base code is internal unless it is an API. The app exposes only versioned API modules to plugins; everything +else is unreachable.
      3. +
    3. API versions are in the import path. import { Session } from "@acme/app/v3": the plugin says what it was +written for where it imports it; an unavailable version is a build error.
      4. +
    4. APIs are split by host kind. A shared module holds what every host may use (contracts, registry tokens, host +kinds); /v3/web and /v3/tui hold UI pieces; a worker never loads React.
      5. +
    5. Contracts carry their own versions. A plugin carries the contract version it was written for; the provider +binds it if it accepts that version, else the plugin waits with a clear reason.
      6. +
    6. A plugin imports four things: its own files, host packages, its npm dependencies, and the app's own plugins by +relative path. Nothing else.
      7. +
    7. Settings are arguments, extensions are registries. What the app decides at startup is passed to its components; +what plugins add is contributed.
      8. +
    8. No type checking in the build. Types come from where imports come from; editors and agents check, the build +only builds.
      9. +
    +

    Words used throughout

    +

    Base code is the app's own program; its API is the part plugins may import, published as API modules, +one per version and host kind. A host package is a module the running process provides to every plugin (Chord, +the app's API modules). A contract is an object type's declaration; a registry token an extension point's. +A facet is a plugin's declarations for one host kind.

    +
    + +
    I

    What a Plugin Is

    A directory or a single file, named once, with one file per host kind.

    +
    plugins/notes/                     a directory plugin
+  package.json                     { "name": "@acme/notes", "dependencies": { "slugify": "1.6.6" } }
+  contract.ts                      the object types and registry tokens it offers
+  worker.ts  server.ts  web.tsx    one facet per host kind it needs
+plugins/hello.worker.ts            a single-file plugin "hello" with one worker facet
+plugins/hello.web.tsx              the same plugin's web facet
    +

    A directory plugin's name is its package.json name; Chord reads only name and dependencies from it. A +single-file plugin's name is its file's base name; it has no dependencies and imports only host packages. When a +single file needs a package or a second file, it becomes a directory. Two roots with the same name: the first root +wins, the other is reported (slice 7c).

    +
    + +
    II

    Base Code: Internal and External

    What the app keeps to itself, what it exposes, and how exposure is versioned.

    +

    The app's base code is everything that starts the process: settings, startup, core objects' implementations, the +session index, migrations. Plugins must not depend on any of it, or every refactoring of the app would break them. +So the app draws a line and exposes only its API:

    +
    src/api/v3/index.ts       shared: host kinds, the principal type, defineFacet per kind, core contracts, registry tokens
+src/api/v3/web.ts         web only: shared UI pieces (a Sheet, theme variables), React-typed registry tokens
+src/api/v3/tui.ts         TUI only: its UI pieces
+src/…                     internal: everything else
    +

    Each API module is a host package of the kinds that may use it: @acme/app/v3 for every kind, @acme/app/v3/web for +the web kind only. A worker plugin importing @acme/app/v3/web is a build error, and a worker process never loads +React. The shared module stays free of runtime imports of UI libraries: a registry token that needs a React type names +it with import type, which the build erases.

    +
    Proposed

    The split of the external API into a shared module and per-kind modules answers the owner's worry about "web, TUI, +worker, server things all in the same index.ts". Proposed, not yet decided.

    +
    +
    + +
    III

    Versions in the Import Path

    A plugin says what it was written for where it imports it.

    +

    The app's API changes over time: a host kind is renamed, a principal variant changes. A plugin written for an older +API must not silently get the new one. So every API module has a version in its path, and the app lists which versions +it still provides:

    +
    // plugins/stats/worker.ts
+import { defineFacet, Session, Panels } from "@acme/app/v3";
+import { remote, str } from "@earendil-works/chord/next/v2";
    +
      +
    • The app provides /v3, its current API. If version 2 plugins still work unchanged against it, /v2 is listed too, +pointing to the same module. An app that needs a compatibility layer makes /v2 a thin module over /v3.
      • +
    • An import of a version the app no longer lists is a build error: "@acme/app/v1 is not available: this app provides +v3, v2". The plugin keeps its old code, and the message tells an agent exactly what to change.
      • +
    • Nothing to fill in elsewhere: no package.json field, no npm ranges. Skeletons and examples import the current +version, so an agent copies it without thinking.
      • +
    • Chord's own surface is versioned the same way (@earendil-works/chord/next/v2).
      • +
    +

    Decided: "the API version must be declared by the plugin in code" (owner, 2026-10-31); the import path is how.

    +
    + +
    IV

    Contracts and Registries

    Finer versions inside the API, and what crosses between plugins.

    +

    Contracts and registry tokens keep their own versions inside the API, because they evolve independently of the API as +a whole:

    +
      +
    • An object type: the consumer carries the contract version it was written for; the provider binds it at that +version if it accepts it (accepts maps arguments forward and results and state back), otherwise the consumer waits +"contract mismatch: v1, served v3". Shapes are fingerprinted, so a changed shape without a version bump is caught.
      • +
    • A registry token: a name, a version and TypeScript types for what is contributed and read. Values are in-process +things (functions, components), which cannot be fingerprinted: bumping the version when they change is the defining +plugin's rule, and its accepts adapts older contributions.
      • +
    • Host kinds on registries (proposed): a token names the kinds that serve it (hosts: ["web"]), typed, so a worker +facet contributing to a web registry is a type error and a load error, not a silent no-op.
      • +
    +

    A contract file imports only host packages at runtime (Chord, the app's API); native types it mentions are type-only.

    +
    + +
    V

    What a Plugin May Import

    Four kinds of import, each with one rule.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    ImportResolved toVersioned by
    its own files (./store.ts)copied into its scriptthe plugin itself
    host packages (@acme/app/v3, @earendil-works/chord/next/v2, react)the running process's instance, sharedthe path's version; the app lists what it provides
    npm packages in its dependencies (slugify, a third party's contracts package)its own node_modules, bundledthe installed version
    the app's own plugins (../notes/contract.ts), in the same rootcopied into its scriptthe same revision: one repository
    +

    A third-party plugin that wants the app's plugins' contracts gets them through the app's API (core contracts) or the +app's published package, as an npm dependency at the version it was written for, never from the running plugin's +files. Anything else (a path into base code, an unlisted version, a package not in its dependencies) is a build error.

    +
    + +
    VI

    Settings and Extensions

    How the app configures its components without letting plugins override it.

    +

    Decided ("Settings are arguments, extensions are registries"): what the app decides at startup is an argument to the +component's facet, which base code constructs, and plugins cannot change it:

    +
    const http = createHttp(server, { listen: { port: 8788 }, links: { path: "/chord", maxPayload: MiB }, closeTimeout: 10_000, onError }, ctx);
    +

    What plugins add while the app runs is a registry: routes, authenticators, tools, panels. A component may offer both +for one thing: the session index takes its indexer as an argument, and may also follow a registry of indexers if +plugins should supply them.

    +
    + +
    VII

    Building and Evaluating

    Unchanged from slices 7c and 10: one build on the server, evaluation everywhere.

    +

    Only the server builds, because only the server has the sources. createPlugins builds every kind and needs only +names: per kind its platform and its host package names. Each process evaluates the scripts of its kind with its own +module objects (the server's at plugins.load, a worker's in workerMain, a page's in its base bundle), so a plugin +shares that process's Chord and API. Base-code facets are passed beside the plugins; they are not plugins, because +they close over the process's own values and never reload.

    +
    + +
    VIII

    Types and Tests

    Where types come from, and how a plugin is tested alone.

    +

    Types come from where imports come from: relative files and npm packages resolve natively in TypeScript; the app's API +modules ship .d.ts per version in the app's package. There is no type checking in the build (the owner: seconds per +build is too slow); editors check as an author types, and an agent may run tsc as a tool.

    +

    Testing a plugin alone (proposed): testPlugin(dir, { kind, stubs }) builds the plugin like production, loads it on an +in-memory host with the kind's real host packages, and loads stub facets that serve the contracts it depends on. A stub +is an ordinary facet; there is no mocking framework.

    +
    + +
    ∴

    Open Questions

    What is proposed here and still to decide.

    +
      +
    1. The external API split by host kind (/v3, /v3/web, /v3/tui).
      2. +
    2. Host kinds on registry tokens, typed and checked at load.
      3. +
    3. Single-file plugins (hello.worker.ts, named by the file).
      4. +
    4. Contracts of the app's plugins for third parties: through the API modules only, or also the app's npm package.
      5. +
    5. testPlugin with stub facets.
      6. +
    6. Reversing "Plugins import each other by name" (2026-10-31, morning) in favour of relative paths within the app +and npm dependencies across it.
      7. +
    +
    + +

    The Plugin Model · proposal · 2026-10-31

    +
    + + From 81f3cd412758ebdd92324c7d63b669a189408d0a Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 20:06:14 +0000 Subject: [PATCH 103/175] docs(chord): a schema package beneath pi-ai and Chord --- packages/chord/docs/next/decisions.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 0878cbcc0..941b1dee7 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2109,6 +2109,16 @@ validation and copies the v2 kernel added around Delta. thin layer); an import of an unlisted version is a build error naming the available ones. No `package.json` field and no npm ranges for this. Contracts and registry tokens inside an API keep their own versions. The rest of the plugin model is a proposal (`docs/next/plugin-model/`). +- **A schema package beneath pi-ai and Chord (owner, 2026-10-31: "i want the schema package"; "if we keep it a + dependency in pi-ai, we must be able to rip it out"; TypeBox is "horrible to treeshake"):** a small package with no + dependencies (working name `@earendil-works/schema`) under both pi-ai and Chord (siblings: neither depends on the + other): helpers returning plain JSON Schema, a structural `JsonSchema` type, `Static` from the helpers' literal + types, and a validator for general JSON Schema (tool schemas from MCP servers and extensions use `$ref`, `anyOf`, + `pattern`, ...), without `eval` or `new Function` (CSP-safe). Chord v2 drops TypeBox; pi-ai's core never imports + it (TypeBox schemas still pass as JSON Schema; TypeBox-typed extras only in an optional subpath, so ripping it out + is deleting that subpath). Not omptype (its validator compiles with `new Function`; its 10.7k lines are mostly + ArkType's API). Open: which validator (measure `@cfworker/json-schema` and TypeBox against the official + JSON-Schema-Test-Suite and Chord's argument checks first), and moving `Type` out of pi-ai's main export. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 8959ad6e6845933dd01dbaa96929243d68bc4d48 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 20:40:32 +0000 Subject: [PATCH 104/175] feat(chord): Chord v2 slice 10, HTTP and workers chord/next/http: a Web fetch handler built from the registry Http (routes, wraps, sockets, hosts, relays, CSP sources) and Authenticators, responses as descriptions, a security baseline on by default, Chord's own Node listener as a host-lifetime object, nodeRequest for apps mounting it, links and plugin sockets. chord/next/workers: a forked worker per address, started on demand, stopped when idle, never restarted by itself, plugins built by the server and shipped. Plugins: kinds with platform and host packages, createPluginRuntime, scripts(kind), rollout results per host, imports of other plugins by name. Kernel: RouteHandle (retry, suspend), no route backoff, anonymous host.root, origins only from relays. ws is an optional peer dependency. --- package-lock.json | 6 +- packages/chord/CHANGELOG.md | 1 + packages/chord/docs/next/benchmarks.md | 43 + packages/chord/package.json | 6 +- packages/chord/src/next-node/http/README.md | 166 ++++ .../chord/src/next-node/http/admission.ts | 178 ++++ .../chord/src/next-node/http/authenticate.ts | 54 ++ packages/chord/src/next-node/http/cookies.ts | 71 ++ packages/chord/src/next-node/http/file.ts | 113 +++ packages/chord/src/next-node/http/headers.ts | 142 +++ packages/chord/src/next-node/http/http.ts | 565 ++++++++++++ packages/chord/src/next-node/http/incoming.ts | 43 + packages/chord/src/next-node/http/index.ts | 34 + packages/chord/src/next-node/http/node.ts | 317 +++++++ packages/chord/src/next-node/http/request.ts | 145 ++++ packages/chord/src/next-node/http/response.ts | 258 ++++++ packages/chord/src/next-node/http/router.ts | 412 +++++++++ packages/chord/src/next-node/http/upgrades.ts | 245 ++++++ .../chord/src/next-node/plugins/README.md | 72 +- packages/chord/src/next-node/plugins/build.ts | 68 +- .../chord/src/next-node/plugins/evaluate.ts | 11 +- packages/chord/src/next-node/plugins/index.ts | 3 + .../chord/src/next-node/plugins/plugins.ts | 366 ++++---- .../chord/src/next-node/plugins/runtime.ts | 215 +++++ .../chord/src/next-node/workers/README.md | 95 ++ .../chord/src/next-node/workers/control.ts | 121 +++ packages/chord/src/next-node/workers/index.ts | 20 + packages/chord/src/next-node/workers/lock.ts | 142 +++ packages/chord/src/next-node/workers/main.ts | 293 +++++++ .../chord/src/next-node/workers/process.ts | 150 ++++ .../chord/src/next-node/workers/workers.ts | 742 ++++++++++++++++ packages/chord/src/next/host/README.md | 23 +- packages/chord/src/next/host/host.ts | 36 +- packages/chord/src/next/host/index.ts | 2 +- packages/chord/src/next/host/root.ts | 54 +- packages/chord/src/next/host/route.ts | 170 +++- packages/chord/src/next/remote/README.md | 8 +- packages/chord/src/next/remote/host-link.ts | 16 +- packages/chord/src/next/remote/internal.ts | 2 +- packages/chord/src/next/remote/path.ts | 30 +- packages/chord/src/next/remote/provider.ts | 13 + packages/chord/src/next/remote/relay.ts | 5 + .../chord/src/next/remote/remote-object.ts | 10 + .../fixtures/plugins/files/worker.ts | 2 +- .../fixtures/plugins/notes/server.ts | 2 +- .../fixtures/plugins/notes/worker.ts | 2 +- .../test/next-node/fixtures/workers/child.ts | 12 + .../next-node/fixtures/workers/control.ts | 47 + .../fixtures/workers/plugins/notes/label.ts | 3 + .../workers/plugins/notes/package.json | 5 + .../fixtures/workers/plugins/notes/worker.ts | 17 + .../test/next-node/fixtures/workers/worker.ts | 146 ++++ .../test/next-node/http/admission.test.ts | 154 ++++ .../test/next-node/http/authenticate.test.ts | 58 ++ .../chord/test/next-node/http/cookies.test.ts | 56 ++ .../chord/test/next-node/http/file.test.ts | 135 +++ .../chord/test/next-node/http/headers.test.ts | 106 +++ .../chord/test/next-node/http/http.test.ts | 818 ++++++++++++++++++ .../chord/test/next-node/http/node.test.ts | 215 +++++ .../chord/test/next-node/http/request.test.ts | 119 +++ .../test/next-node/http/response.test.ts | 245 ++++++ .../chord/test/next-node/http/router.test.ts | 228 +++++ packages/chord/test/next-node/http/support.ts | 204 +++++ .../test/next-node/http/upgrades.test.ts | 458 ++++++++++ .../test/next-node/plugins/build.test.ts | 41 +- .../test/next-node/plugins/plugins.test.ts | 167 +++- .../test/next-node/plugins/runtime.test.ts | 126 +++ .../chord/test/next-node/plugins/support.ts | 12 +- .../test/next-node/workers/control.test.ts | 55 ++ .../chord/test/next-node/workers/lock.test.ts | 62 ++ .../chord/test/next-node/workers/main.test.ts | 87 ++ .../test/next-node/workers/process.test.ts | 46 + .../chord/test/next-node/workers/support.ts | 151 ++++ .../test/next-node/workers/workers.test.ts | 605 +++++++++++++ packages/chord/test/next/bench/http.ts | 120 +++ packages/chord/test/next/bench/plugins.ts | 8 +- packages/chord/test/next/bench/workers.ts | 107 +++ .../chord/test/next/examples/12-routes.ts | 4 +- .../test/next/examples/35-plugins/main.ts | 27 +- .../35-plugins/plugins/clock/server.ts | 2 +- .../35-plugins/plugins/notes/server.ts | 2 +- .../35-plugins/plugins/notes/worker.ts | 2 +- .../next/examples/36-plugin-failures/main.ts | 7 +- .../plugins/forecast/server.ts | 2 +- .../plugins/summary/server.ts | 2 +- .../next/examples/37-plugin-imports/main.ts | 23 +- .../plugins/report/server.ts | 4 +- .../37-plugin-imports/plugins/ruler/server.ts | 2 +- .../examples/38-watch-and-rollout/main.ts | 16 +- .../plugins/contracts/package.json | 2 +- .../plugins/notes/server.ts | 2 +- .../plugins/notes/worker.ts | 2 +- .../plugins/old/server.ts | 2 +- .../plugins/tags/server.ts | 2 +- .../plugins/tags/worker.ts | 2 +- .../next/examples/39-browser-build/main.ts | 6 +- .../chord/test/next/examples/40-http/main.ts | 56 ++ .../40-http/plugins/notes/files/hello.md | 3 + .../40-http/plugins/notes/package.json | 5 + .../examples/40-http/plugins/notes/server.ts | 29 + .../test/next/examples/41-workers/main.ts | 68 ++ .../41-workers/plugins/sessions/contract.ts | 13 + .../41-workers/plugins/sessions/package.json | 5 + .../41-workers/plugins/sessions/worker.ts | 11 + .../test/next/examples/41-workers/worker.ts | 22 + .../next/examples/42-worker-crash/main.ts | 67 ++ .../plugins/sessions/contract.ts | 13 + .../plugins/sessions/package.json | 5 + .../plugins/sessions/worker.ts | 17 + .../next/examples/42-worker-crash/worker.ts | 22 + .../test/next/examples/43-rollout/main.ts | 85 ++ .../43-rollout/plugins/notes/contract.ts | 16 + .../43-rollout/plugins/notes/label.ts | 3 + .../43-rollout/plugins/notes/package.json | 5 + .../43-rollout/plugins/notes/server.ts | 7 + .../43-rollout/plugins/notes/worker.ts | 9 + .../test/next/examples/43-rollout/worker.ts | 22 + .../test/next/examples/44-http-wraps/main.ts | 37 + .../44-http-wraps/plugins/audit/package.json | 5 + .../44-http-wraps/plugins/audit/server.ts | 24 + .../44-http-wraps/plugins/notes/package.json | 5 + .../44-http-wraps/plugins/notes/server.ts | 20 + .../next/examples/45-plugin-socket/main.ts | 64 ++ .../plugins/echo/package.json | 5 + .../45-plugin-socket/plugins/echo/server.ts | 16 + .../next/examples/46-allowed-hosts/main.ts | 61 ++ .../plugins/tunnel/package.json | 5 + .../46-allowed-hosts/plugins/tunnel/server.ts | 31 + .../next/examples/47-http-own-server/main.ts | 70 ++ .../plugins/notes/package.json | 5 + .../plugins/notes/server.ts | 12 + .../next/examples/48-http-baseline/main.ts | 84 ++ .../plugins/notes/package.json | 5 + .../48-http-baseline/plugins/notes/server.ts | 33 + .../chord/test/next/host/revocation.test.ts | 7 + packages/chord/test/next/host/root.test.ts | 35 +- packages/chord/test/next/host/route.test.ts | 221 +++-- .../chord/test/next/remote/host-link.test.ts | 78 +- packages/chord/test/next/remote/path.test.ts | 20 + .../chord/test/next/remote/provider.test.ts | 24 +- packages/chord/test/next/remote/relay.test.ts | 65 +- packages/chord/test/next/support.ts | 7 +- tsconfig.json | 9 + 143 files changed, 11066 insertions(+), 561 deletions(-) create mode 100644 packages/chord/src/next-node/http/README.md create mode 100644 packages/chord/src/next-node/http/admission.ts create mode 100644 packages/chord/src/next-node/http/authenticate.ts create mode 100644 packages/chord/src/next-node/http/cookies.ts create mode 100644 packages/chord/src/next-node/http/file.ts create mode 100644 packages/chord/src/next-node/http/headers.ts create mode 100644 packages/chord/src/next-node/http/http.ts create mode 100644 packages/chord/src/next-node/http/incoming.ts create mode 100644 packages/chord/src/next-node/http/index.ts create mode 100644 packages/chord/src/next-node/http/node.ts create mode 100644 packages/chord/src/next-node/http/request.ts create mode 100644 packages/chord/src/next-node/http/response.ts create mode 100644 packages/chord/src/next-node/http/router.ts create mode 100644 packages/chord/src/next-node/http/upgrades.ts create mode 100644 packages/chord/src/next-node/plugins/runtime.ts create mode 100644 packages/chord/src/next-node/workers/README.md create mode 100644 packages/chord/src/next-node/workers/control.ts create mode 100644 packages/chord/src/next-node/workers/index.ts create mode 100644 packages/chord/src/next-node/workers/lock.ts create mode 100644 packages/chord/src/next-node/workers/main.ts create mode 100644 packages/chord/src/next-node/workers/process.ts create mode 100644 packages/chord/src/next-node/workers/workers.ts create mode 100644 packages/chord/test/next-node/fixtures/workers/child.ts create mode 100644 packages/chord/test/next-node/fixtures/workers/control.ts create mode 100644 packages/chord/test/next-node/fixtures/workers/plugins/notes/label.ts create mode 100644 packages/chord/test/next-node/fixtures/workers/plugins/notes/package.json create mode 100644 packages/chord/test/next-node/fixtures/workers/plugins/notes/worker.ts create mode 100644 packages/chord/test/next-node/fixtures/workers/worker.ts create mode 100644 packages/chord/test/next-node/http/admission.test.ts create mode 100644 packages/chord/test/next-node/http/authenticate.test.ts create mode 100644 packages/chord/test/next-node/http/cookies.test.ts create mode 100644 packages/chord/test/next-node/http/file.test.ts create mode 100644 packages/chord/test/next-node/http/headers.test.ts create mode 100644 packages/chord/test/next-node/http/http.test.ts create mode 100644 packages/chord/test/next-node/http/node.test.ts create mode 100644 packages/chord/test/next-node/http/request.test.ts create mode 100644 packages/chord/test/next-node/http/response.test.ts create mode 100644 packages/chord/test/next-node/http/router.test.ts create mode 100644 packages/chord/test/next-node/http/support.ts create mode 100644 packages/chord/test/next-node/http/upgrades.test.ts create mode 100644 packages/chord/test/next-node/plugins/runtime.test.ts create mode 100644 packages/chord/test/next-node/workers/control.test.ts create mode 100644 packages/chord/test/next-node/workers/lock.test.ts create mode 100644 packages/chord/test/next-node/workers/main.test.ts create mode 100644 packages/chord/test/next-node/workers/process.test.ts create mode 100644 packages/chord/test/next-node/workers/support.ts create mode 100644 packages/chord/test/next-node/workers/workers.test.ts create mode 100644 packages/chord/test/next/bench/http.ts create mode 100644 packages/chord/test/next/bench/workers.ts create mode 100644 packages/chord/test/next/examples/40-http/main.ts create mode 100644 packages/chord/test/next/examples/40-http/plugins/notes/files/hello.md create mode 100644 packages/chord/test/next/examples/40-http/plugins/notes/package.json create mode 100644 packages/chord/test/next/examples/40-http/plugins/notes/server.ts create mode 100644 packages/chord/test/next/examples/41-workers/main.ts create mode 100644 packages/chord/test/next/examples/41-workers/plugins/sessions/contract.ts create mode 100644 packages/chord/test/next/examples/41-workers/plugins/sessions/package.json create mode 100644 packages/chord/test/next/examples/41-workers/plugins/sessions/worker.ts create mode 100644 packages/chord/test/next/examples/41-workers/worker.ts create mode 100644 packages/chord/test/next/examples/42-worker-crash/main.ts create mode 100644 packages/chord/test/next/examples/42-worker-crash/plugins/sessions/contract.ts create mode 100644 packages/chord/test/next/examples/42-worker-crash/plugins/sessions/package.json create mode 100644 packages/chord/test/next/examples/42-worker-crash/plugins/sessions/worker.ts create mode 100644 packages/chord/test/next/examples/42-worker-crash/worker.ts create mode 100644 packages/chord/test/next/examples/43-rollout/main.ts create mode 100644 packages/chord/test/next/examples/43-rollout/plugins/notes/contract.ts create mode 100644 packages/chord/test/next/examples/43-rollout/plugins/notes/label.ts create mode 100644 packages/chord/test/next/examples/43-rollout/plugins/notes/package.json create mode 100644 packages/chord/test/next/examples/43-rollout/plugins/notes/server.ts create mode 100644 packages/chord/test/next/examples/43-rollout/plugins/notes/worker.ts create mode 100644 packages/chord/test/next/examples/43-rollout/worker.ts create mode 100644 packages/chord/test/next/examples/44-http-wraps/main.ts create mode 100644 packages/chord/test/next/examples/44-http-wraps/plugins/audit/package.json create mode 100644 packages/chord/test/next/examples/44-http-wraps/plugins/audit/server.ts create mode 100644 packages/chord/test/next/examples/44-http-wraps/plugins/notes/package.json create mode 100644 packages/chord/test/next/examples/44-http-wraps/plugins/notes/server.ts create mode 100644 packages/chord/test/next/examples/45-plugin-socket/main.ts create mode 100644 packages/chord/test/next/examples/45-plugin-socket/plugins/echo/package.json create mode 100644 packages/chord/test/next/examples/45-plugin-socket/plugins/echo/server.ts create mode 100644 packages/chord/test/next/examples/46-allowed-hosts/main.ts create mode 100644 packages/chord/test/next/examples/46-allowed-hosts/plugins/tunnel/package.json create mode 100644 packages/chord/test/next/examples/46-allowed-hosts/plugins/tunnel/server.ts create mode 100644 packages/chord/test/next/examples/47-http-own-server/main.ts create mode 100644 packages/chord/test/next/examples/47-http-own-server/plugins/notes/package.json create mode 100644 packages/chord/test/next/examples/47-http-own-server/plugins/notes/server.ts create mode 100644 packages/chord/test/next/examples/48-http-baseline/main.ts create mode 100644 packages/chord/test/next/examples/48-http-baseline/plugins/notes/package.json create mode 100644 packages/chord/test/next/examples/48-http-baseline/plugins/notes/server.ts diff --git a/package-lock.json b/package-lock.json index f2129fd85..02e687f30 100644 --- a/package-lock.json +++ b/package-lock.json @@ -5568,11 +5568,15 @@ "node": ">=22.19.0" }, "peerDependencies": { - "esbuild": "0.28.2" + "esbuild": "0.28.2", + "ws": "8.21.0" }, "peerDependenciesMeta": { "esbuild": { "optional": true + }, + "ws": { + "optional": true } } }, diff --git a/packages/chord/CHANGELOG.md b/packages/chord/CHANGELOG.md index ceb4cddd1..326d4b8de 100644 --- a/packages/chord/CHANGELOG.md +++ b/packages/chord/CHANGELOG.md @@ -17,6 +17,7 @@ - Chord v2 has a package export again, `@earendil-works/chord/next` (experimental): every layer's API for apps (contracts, data, documents, hosts, jobs, links, state, the store), and, marked internal, the store's surface for adapters (`StoreOwnerTables`, `OwnerTable`, the documents engine's reads and writes, the commit step, `applyBatch`). v2's store gains one id allocator for every id in it, claims per table and per document namespace, Durable's Storage semantics for a refused commit (it rejects that commit only; an unknown outcome stops the store), document histories `none`, `latest` and `rewindable` with reads at a point, archives carrying document revisions, and batch apply; v2 objects gain `self.follow` (another runtime's synchronous observers as sources) and persisted jobs (`persist: { resume }`, a stored `abort(id, ctx)`, deferred state changes). - Chord v2 (`@earendil-works/chord/next`) gains registries, writables, wake signals and root scopes (slice 8): `defineRegistry(name, { version })` tokens shared by plugins, served by the defining plugin's facet with `host.serve(R, { compose, accepts? })`; contributions from facets (`defineFacet((host, self) => self.contribute(R, value))`) and host-lifetime objects (`StoreSelf.contribute`), plain values or Chord watchables, in a table sorted by plugin name and composed lazily (once at the end of `load`'s setups, then when read or followed, in the batch that changed the table); readers through `self.registry(R)` (a declared dependency: an object activates once the registry has a value, and fails `transient` with the new reason `unavailable/undeclared` when nobody serves it) and `host.registry(R, ctx)`, a `RegistryValue` with `observe(observer, ctx)`; older contributions and readers adapted by `accepts`; the helpers `keyedEntries` and `orderedEntries`; the `onError` kinds `registry_conflict`, `registry_failed`, `contribution_failed` and `contribution_refused`. `self.writable(initial)` is an in-process value an attempt sets (identity compared); native sources (writables, registry values, `ref.state`) are derive sources. `self.retryOn(...sources)` retries a failed object when a registry value changes. `host.root(fn, context, { principal? })` runs code outside objects with a handler's `ctx`, sealed when `fn` settles, `context` aborts or the host closes. - Chord v2 gains plugins (slice 7c): `src/next-node/plugins` (Node, to be exported as `@earendil-works/chord/next/plugins`) finds plugin directories, builds each facet with esbuild (now an optional peer dependency of `@earendil-works/chord`, imported only there), evaluates it with `vm.Script`, maps plugin stack traces (`mapStack`), installs plugin dependencies, watches plugin and base code, and rolls reloads out across hosts. `host.load` and `host.reload` work per plugin: a plugin whose setup, factory or declarations fail is left out at `load` (`{ failed }`) or kept on its old code at `reload` (`failed` in the report, `reload(facets, ctx, { failed })` for build and evaluation failures), with a closure rule keeping a consumer old when its provider was kept old; an abort after a reload's commit rejects with `details.committed`. +- Chord v2 gains HTTP and workers (slice 10): `src/next-node/http` (to be exported as `@earendil-works/chord/next/http`) builds a Web-standard `fetch` handler from the routes, wraps, sockets, hosts, relays and CSP sources plugins contribute to the registry `Http`, with authenticators from the registry `Authenticators`, responses as descriptions (`json`, `html`, `bytes` with ranges, `redirect`, `error`; `fileResponse`), and a security baseline on by default (provenance by the socket and `Host`, foreign requests anonymous, `Origin` checks, default headers with a per-response CSP nonce, signed cookies, safe downloads); Chord's own Node listener as a host-lifetime object, `nodeRequest` for apps mounting it on their own Node server, the links endpoint and plugin sockets with a heartbeat, closed at once when their contribution leaves. `ws` is an optional peer dependency, imported only there. `src/next-node/workers` (`createWorkers`, `workerMain`) runs a forked worker per address of a host kind, started on demand, stopped when idle, never restarted by itself (a failure stays until `retry` or a rollout), its plugins built by the server and shipped. The plugins component defines one definition per facet kind (`kinds: { name: { platform, hostPackages } }`, modules given at `plugins.load(host, { kind, core, modules })`), splits loading into `createPluginRuntime()` (no esbuild), offers `scripts(kind)`, rolls out with one result per host (`RolloutStep` returns `{ hosts }`), and resolves cross-plugin imports by plugin name (`@app/notes/contract`). Kernel: `host.route` returns a `RouteHandle` (`retry`, `suspend`) and a route has no backoff timer (a failed `resolve` fails its holders `waiting`); `host.root(fn, ctx, { principal: undefined })` runs anonymous; a link reads `origin` only on parent and route links (a protocol error on a child link) and drops a per-origin record once it is idle. - Chord v2, experimental and not exported yet: the next object runtime in `src/next` (contracts, hosts and objects, documents, derives, links and relays, routines and jobs, the store), with its tests and runnable examples in `test/next` and its design in `docs/next` (`design.md`, the decision log `decisions.md`, the slice plan `slices.md`). Its API changes with every slice until it gets an entry point. - `NativeWatchable`: an in-process value that changes and can be watched (`value`, `watch(context)` returning a `NativeWatchHandle`), the in-process counterpart of `Watchable`. - `WatchHandle` and `WatchEnd`, the one watch API, moved from Durable: a serialized observation with `value`, `start(listener)`, `stop()`, and `closed`; the listener gets a `WatchChange` (`{ value, ops, retired }`) and the context. `WatchEnd` reasons are `stopped`, `cancelled`, `ended`, `listener_error`, and `failed`. diff --git a/packages/chord/docs/next/benchmarks.md b/packages/chord/docs/next/benchmarks.md index d425f4154..ea2de1efa 100644 --- a/packages/chord/docs/next/benchmarks.md +++ b/packages/chord/docs/next/benchmarks.md @@ -914,3 +914,46 @@ derive commit 10.34-10.36 → 10.43-10.49 µs (14,899 → 14,901 B), activation 0.026-0.030 → 0.028-0.033 ms (pinned 0.030-0.035 → 0.030-0.032 ms); retained per hold equal (one binary hold 176.2 → 170.1 KiB). `reload.ts`: swap of 50 / 1,000 objects 1.44-1.73 / 30.83-32.71 → 1.52-1.53 / 30.55-33.25 ms, restart 1.01-1.05 / 15.54-16.87 → 0.95-0.99 / 15.90-16.02 ms until the reload resolves. + +## Slice 10: HTTP and workers (2026-10-31) + +`test/next/bench/http.ts` and `test/next/bench/workers.ts`, Node 24.4.1, CPU 6 (the forked workers inherit the pin). + +| Measurement | Result | +| --- | --- | +| keep-alive round trip, a JSON body: plain Node handler / a route (same process as the client) | build: 42.0-42.5 / 46.4-47.0 µs: +4.2-4.9 µs (9.9-11.7%); fix round 1: 42.5-42.9 / 62.8-64.4 µs: +20.3-21.5 µs | +| the same after 30,000 warm-up requests (fix round 1) | 30.9-31.3 / 43.4-44.0 µs: +12.1-12.9 µs; with an authenticator reading `request.headers`: +15.9 µs | +| the same, server and client in two processes (CPU 6 and 8; build) | 36.4-37.0 / 41.7-42.4 µs: +5 µs | +| 100 MiB file: `pipeline` of a read stream / `fileResponse` | build (64 KiB reads): 602-618 / 487-565 MiB/s; fix round 1 (256 KiB, pulled): 477-751 / 560-611 MiB/s | +| a worker's cold start with one shipped script: fork to `started` / to `ready` (median of 7) | 531-532 / 538-539 ms | +| the server's heap per idle worker (20 workers, one reference each) | 49.5 KiB | + +A request's overhead was 16 µs at first: `host.root` cost about 7 µs of it per request (a linked abort signal, a +`Promise.race` and an `awaitWithContext` per scope). Root scopes now take the caller's signal with one listener and +settle one promise (`root.ts`; behaviour and tests unchanged): 4.2-4.9 µs, about the bar of 10% of a round trip. The +rest is the request's scope (its abort controller, `CallContext`), admission and the response's header merge. The cold +start is mostly Node's startup stripping the types of the source tree it imports; the start timer (60 s) is far from +it. + +**Fix round 1** (the `fetch` core, the security baseline, Chord's listener as a native object). A route's overhead in +the standard run grew from +4.5 µs to +20-21 µs (+41 µs before the Node fast path: the listener reads Node's parsed +headers and makes a Web `Request` only when something reads `request.headers` or the body or a relay verifies; whole +bodies are written as strings, in one write with the head). Warmed, +12-13 µs; a CPU profile of the warmed route path +(per request, `test/next/bench/http.ts`'s route): the listener's request function 13.6 µs inclusive against the plain +handler's 8.9 µs, of which writing the answer 8.3 µs (the plain handler's `writeHead` and `end`: 9.1 µs); `serve` 4.2 +µs (the root scope 2.1, the URL parse 0.9, default headers 0.6, admission and routing 1.1, the answer 0.7); the root +scope's seal in a later microtask 0.9 µs. The baseline's extra headers cost 0.4 µs (a plain handler sending them). +The rest, about 5 µs, sits in no function of the component: Node's tick and microtask machinery (33 µs inclusive +against 18 µs for the plain handler). Opus's two trims are in (`lookup` decodes only segments with `%`; one abort +controller per request: the Node side's own is gone). Kernel rows, HEAD `bc9a20f84` → fix round 1, CPU 6, alternating: +binary 1 MiB call median 0.182-0.220 → 0.196 ms; reconnect of 1 / 100 references 1.02-1.10 / 79.1-80.3 → 1.04-1.05 / +81.3-81.5 ms; watched `writable.set` 0.836-0.868 → 0.861-0.863 µs; watched derive commit 10.42-10.58 → 10.36-10.60 µs; +activation churn 58.9-60.5 → 59.1-60.8 µs. + +**Kernel hot paths** (routes without a timer, `RouteHandle`, `Path` events, root scopes), HEAD `bc9a20f84` → build at +paths of equal length, CPU 6, alternating runs. `calls.ts`: local call 2.70-2.83 → 2.70-2.83 µs (6,660 B both), +watched derive commit 10.36-10.60 → 10.34-10.79 µs, activation churn 59.43-61.30 → 59.48-60.72 µs. `links.ts`: commit +to frame and encode/decode alike; reconnect of 1 / 100 references 0.94-1.15 / 80.8-84.0 → 0.94-1.15 / 80.6-86.5 ms; +1,000 parked calls 24.4-25.2 → 24.3-24.6 ms. `reload.ts`: swap of 50 / 1,000 objects 1.44-1.55 / 30.78-31.42 → +1.32-1.56 / 31.20-31.83 ms, restart 0.96-0.98 / 15.75-16.34 → 0.95-1.10 / 15.91-15.94 ms until the reload resolves. +`registries.ts` alike (one of 100 contributions set 1.47-2.45 → 1.48-3.41 µs, run to run). diff --git a/packages/chord/package.json b/packages/chord/package.json index f2be5a564..908005452 100644 --- a/packages/chord/package.json +++ b/packages/chord/package.json @@ -94,11 +94,15 @@ "typebox": "1.3.27" }, "peerDependencies": { - "esbuild": "0.28.2" + "esbuild": "0.28.2", + "ws": "8.21.0" }, "peerDependenciesMeta": { "esbuild": { "optional": true + }, + "ws": { + "optional": true } }, "devDependencies": { diff --git a/packages/chord/src/next-node/http/README.md b/packages/chord/src/next-node/http/README.md new file mode 100644 index 000000000..266ba00d0 --- /dev/null +++ b/packages/chord/src/next-node/http/README.md @@ -0,0 +1,166 @@ +# http + +The HTTP component (slice 10 §2, design §23; decisions "The HTTP core is a fetch handler", "The HTTP server is a native +object", "The HTTP component's security baseline", "Slice 10 fix round, allowed hosts only through contributions", +"Settings are arguments, extensions are registries", "A CSP nonce per response"): the server's HTTP and WebSocket +entry. Plugins contribute routes, wraps, sockets, allowed hosts, framers, relays and +CSP sources to the registry `Http`; routes return response descriptions. The core is one Web-standard handler, +`fetch(Request) → Response`; Chord's own Node server (the listener) is the easy path, an app with a server of its own +mounts the handler. Node-only; `ws` is an optional peer dependency of `@earendil-works/chord`, imported only here. +Exported as `@earendil-works/chord/next/http` once v2 has its entry point. + +## For apps (`index.ts`) + +```ts +// the app's settings are arguments, fixed for the component's life (changing one is a restart) +const http = createHttp(server, { + listen: { host: "127.0.0.1", port: 8788 }, // Chord's own listener; port 0 picks one; none: no listener + links: { path: "/chord", options: (request, principal) => ({ limits, maxPayload }) }, // or a value + closeTimeout: 30_000, + onError: (error) => log(error), // HttpError +}, ctx); +// the app's core facet: who someone is (a registry: plugins may add authenticators too) +const app = defineFacet((_host, self) => { + self.contribute(Authenticators, (request, ctx) => principalFromCookie(readCookie(request, "session", secret))); +}); +const core = [{ name: "chord.http", facet: http.facet }, { name: "@app/core", facet: app }]; +await plugins.load(server, { kind: "server", core, modules }, ctx); +const { port } = await http.listening(ctx); +… +await http.close(ctx); // first: drains running requests (host.close would seal their scopes at once) +await server.close(ctx); +``` + +```ts +// a plugin's server facet +self.contribute(Http, { + routes: { + "GET /notes/:id": async (request, ctx) => ({ kind: "json", value: … }), + "GET /": { page: true, handler: async (request) => ({ kind: "html", body: `` }; + }, + }, + framers: ["https://shell.example.com"], + csp: { "img-src": ["https://cdn.example.com"] }, + }), + }); + const page = await send(server.port, "GET", "/page"); + expect(page.headers).toMatchObject({ + "cache-control": "no-store", + "x-content-type-options": "nosniff", + "referrer-policy": "no-referrer", + }); + expect(page.headers["x-frame-options"]).toBeUndefined(); + expect(page.headers["content-security-policy"]).toBe( + `default-src 'none'; script-src 'self' 'nonce-${nonce}'; connect-src 'self' ws://127.0.0.1:${server.port}; base-uri 'none'; form-action 'self'; frame-ancestors https://shell.example.com; img-src https://cdn.example.com`, + ); + // Fable's recheck: a nonce read from one page must not run a script injected into another. + const first = nonce; + await send(server.port, "GET", "/page"); + expect(nonce).not.toBe(first); + const missing = await send(server.port, "GET", "/nope"); + expect([missing.status, missing.headers["x-content-type-options"]]).toEqual([404, "nosniff"]); + expect(missing.headers["content-security-policy"]).toMatch(/'nonce-[A-Za-z0-9+/]{22}=='/); + expect(missing.headers["content-security-policy"]).not.toContain(nonce); + await server.close(); + const plain = await served({}); + expect((await send(plain.port, "GET", "/")).headers["x-frame-options"]).toBe("DENY"); + await plain.close(); + }); + + it("lets a relay's verified request through with its framer", async () => { + const server = await served({ + "@app/who": whoami, + "@app/relay": contributing({ + relays: [ + { + name: "radius", + verify: (request) => + request.headers.get("x-relay") === "secret" + ? { origin: "https://me.relay.example", framer: "https://relay.example" } + : undefined, + }, + ], + }), + }); + const forwarded = { "x-forwarded-for": "203.0.113.9" }; + expect((await send(server.port, "GET", "/whoami", { headers: forwarded })).status).toBe(403); + const relayed = await send(server.port, "GET", "/whoami", { headers: { ...forwarded, "x-relay": "secret" } }); + expect(relayed.status).toBe(200); + expect(relayed.headers["content-security-policy"]).toContain("frame-ancestors https://relay.example"); + await server.close(); + }); +}); + +describe("routes and wraps failing", () => { + it("answers a throwing route 500 without its message, a throwing wrap 500 naming its plugin, an HttpFailure its status", async () => { + const server = await served({ + "@app/a": contributing({ + routes: { + "GET /throws": async () => { + throw new Error("secret"); + }, + "GET /failure": async () => { + throw new HttpFailure(422, "no"); + }, + }, + }), + "@app/b": contributing({ + routes: { "GET /wrapped": async () => ({ kind: "json", value: 1 }) }, + wraps: [ + { + name: "broken", + phase: "headers", + wrap: async () => { + throw new Error("wrap"); + }, + }, + ], + }), + }); + const thrown = await send(server.port, "GET", "/throws"); + expect([thrown.status, thrown.body.toString()]).toEqual([500, "Internal Server Error"]); + expect((await send(server.port, "GET", "/failure")).status).toBe(422); + expect((await send(server.port, "GET", "/wrapped")).status).toBe(500); + expect(server.errors).toMatchObject([ + { kind: "route_failed", pattern: "GET /throws", plugin: "@app/a", error: { message: "secret" } }, + { kind: "wrap_failed", name: "broken", plugin: "@app/b", error: { message: "wrap" } }, + ]); + await server.close(); + }); + + it("runs wraps outermost first around the route, in its request's scope", async () => { + const order: string[] = []; + const server = await served({ + "@app/a": contributing({ + routes: { + "GET /r": async (_request, ctx) => { + order.push(`route as ${ctx.principal?.kind ?? "anonymous"}`); + return { kind: "json", value: 1 }; + }, + }, + wraps: [ + { + name: "inner", + phase: "transform", + wrap: async (request, next) => { + order.push("transform"); + return next(request); + }, + }, + { + name: "outer", + phase: "observe", + wrap: async (request, next) => { + order.push("observe"); + const response = await next(request); + return { ...response, headers: { "X-Observed": "1" } }; + }, + }, + ], + }), + }); + const answer = await send(server.port, "GET", "/r", { headers: { "x-principal": "owner" } }); + expect([order, answer.headers["x-observed"]]).toEqual([["observe", "transform", "route as owner"], "1"]); + await server.close(); + }); +}); + +describe("reloads and close", () => { + it("a reload swapping a route leaves a running request on the old router; the next uses the new one", async () => { + const gate = Promise.withResolvers(); + const old = contributing({ + routes: { + "GET /v": async () => { + await gate.promise; + return { kind: "html", body: "old" }; + }, + }, + }); + const server = await served({ "@app/v": old }); + const running = send(server.port, "GET", "/v"); + await new Promise((resolve) => setTimeout(resolve, 50)); + const next = contributing({ routes: { "GET /v": async () => ({ kind: "html", body: "new" }) } }); + await server.host.reload(server.facets({ "@app/v": next }), context); + expect((await send(server.port, "GET", "/v")).body.toString()).toBe("new"); + gate.resolve(); + expect((await running).body.toString()).toBe("old"); + await server.close(); + }); + + it("close refuses new requests (503), waits for running ones up to closeTimeout, then seals them (503)", async () => { + let sealed: AbortSignal | undefined; + const server = await served( + { + "@app/slow": contributing({ + routes: { + "GET /quick": async () => { + await new Promise((resolve) => setTimeout(resolve, 30)); + return { kind: "html", body: "done" }; + }, + "GET /stuck": async (_request, ctx: Ctx) => { + sealed = ctx.abortSignal; + return new Promise(() => {}); + }, + }, + }), + }, + { config: { listen: { host: "127.0.0.1", port: 0 }, closeTimeout: 100 } }, + ); + const quick = send(server.port, "GET", "/quick"); + const stuck = send(server.port, "GET", "/stuck"); + await new Promise((resolve) => setTimeout(resolve, 10)); + const closing = server.http.close(context); + expect((await server.http.fetch(loopback("/quick"), LOOPBACK)).status).toBe(503); + expect((await quick).body.toString()).toBe("done"); + await closing; + expect(sealed?.aborted).toBe(true); + await server.host.close(context); + expect((await stuck).status).toBe(503); + await expect(send(server.port, "GET", "/quick")).rejects.toThrow(); + }); + + // Astra's build review: a request authenticating when close began ran its route afterwards. + it("never runs a route for a request still authenticating when close sealed it", async () => { + const entered = Promise.withResolvers(); + const gate = Promise.withResolvers(); + let calls = 0; + const server = await served( + { + "@app/r": contributing({ + routes: { + "GET /": async () => { + calls++; + return { kind: "json", value: 1 }; + }, + }, + }), + }, + { + config: { listen: { host: "127.0.0.1", port: 0 }, closeTimeout: 20 }, + authenticate: async () => { + entered.resolve(); + await gate.promise; + return { kind: "owner" }; + }, + }, + ); + const answer = server.http.fetch(loopback("/"), LOOPBACK); + await entered.promise; + await server.http.close(context); + gate.resolve(); + expect([(await answer).status, calls]).toEqual([503, 0]); + await server.close(); + }); +}); + +describe("mounting", () => { + it("without a connection a request is never local", async () => { + const server = await served({ "@app/who": whoami }, { config: { listen: undefined } }); + expect((await server.http.fetch(loopback("/whoami"))).status).toBe(403); + await server.close(); + }); +}); + +describe("cancellation", () => { + // Sol's second recheck C: after cancellation the chain still started the next authenticator. + it("starts no further authenticator once close sealed the request", async () => { + const entered = Promise.withResolvers(); + const gate = Promise.withResolvers(); + let later = 0; + const server = await served( + { + "@app/r": contributing({ routes: { "GET /": async () => ({ kind: "html", body: "ok" }) } }), + "@app/z-later": defineFacet((_host, self) => + self.contribute(Authenticators, () => { + later++; + return { kind: "owner" }; + }), + ), + }, + { + config: { listen: undefined, closeTimeout: 10 }, + authenticate: async () => { + entered.resolve(); + await gate.promise; + return undefined; + }, + }, + ); + const answer = server.http.fetch(loopback("/"), LOOPBACK); + await entered.promise; + await server.http.close(context); + expect((await answer).status).toBe(503); + gate.resolve(); + await new Promise((resolve) => setTimeout(resolve, 20)); + expect(later).toBe(0); + await server.close(); + }); + + it("asks no authenticator for a request aborted before it arrived", async () => { + let calls = 0; + const server = await served( + { "@app/r": contributing({ routes: { "GET /": async () => ({ kind: "html", body: "ok" }) } }) }, + { + config: { listen: undefined }, + authenticate: () => { + calls++; + return undefined; + }, + }, + ); + await server.http.fetch(loopback("/", { signal: AbortSignal.abort() }), LOOPBACK); + expect(calls).toBe(0); + await server.close(); + }); + + // Sol's third recheck: a request aborted while its route returned the `bytes` description got a stream born sealed, + // which never errored; the request never finished. + it.each(["inside the route", "after the route returned"] as const)( + "ends a request whose bytes Response was born sealed (aborted %s)", + async (when) => { + const abort = new AbortController(); + const entered = Promise.withResolvers(); + let reads = 0; + let bornSealed: boolean | undefined; + const server = await served( + { + "@app/b": contributing({ + routes: { + "GET /": async (_request, ctx) => { + entered.resolve(); + if (when === "inside the route") abort.abort(new Error("client gone")); + bornSealed = ctx.abortSignal?.aborted; + return { + kind: "bytes", + mime: "application/octet-stream", + read: async () => { + reads++; + return new Uint8Array([1]); + }, + }; + }, + }, + }), + }, + { config: { listen: undefined, closeTimeout: 1_000 } }, + ); + const answer = server.http.fetch(loopback("/", { signal: abort.signal }), LOOPBACK); + await entered.promise; + if (when === "after the route returned") abort.abort(new Error("client gone")); + await answer; + await new Promise((resolve) => setImmediate(resolve)); + const started = Date.now(); + await server.http.close(context); + expect([Date.now() - started < 500, reads, bornSealed]).toEqual([true, 0, when === "inside the route"]); + await server.close(); + }, + ); + + // Sol's second recheck B: an aborted `Response` nobody touched again kept its request: close waited its bound. + it("ends a request whose bytes Response was aborted and dropped without another read", async () => { + const server = await served( + { + "@app/b": contributing({ + routes: { + "GET /": async () => ({ + kind: "bytes", + mime: "application/octet-stream", + read: async () => new Uint8Array([1]), + }), + }, + }), + }, + { config: { listen: undefined, closeTimeout: 1_000 } }, + ); + const abort = new AbortController(); + await server.http.fetch(loopback("/", { signal: abort.signal }), LOOPBACK); + abort.abort(new Error("client gone")); + await new Promise((resolve) => setImmediate(resolve)); + const started = Date.now(); + await server.http.close(context); + expect(Date.now() - started).toBeLessThan(500); + await server.close(); + }); + + // Astra's recheck R4, Sol's N1: a close before the first resolution started a stopped watch; its throw ended the process. + it("a close before the publication leaves nothing running: the waiting request answers 503", async () => { + const server = await served({}, { load: false, config: { listen: undefined } }); + const waiting = server.http.fetch(loopback("/x"), LOOPBACK); + await server.http.close(withAbortSignal(AbortSignal.abort(), context)); + await server.load(); + expect((await waiting).status).toBe(503); + await new Promise((resolve) => setTimeout(resolve, 50)); + await server.close(); + }); + + it("a close while the first request resolves the registries starts no follow", async () => { + const server = await served({ "@app/who": whoami }, { config: { listen: undefined, closeTimeout: 10 } }); + const pending = server.http.fetch(loopback("/whoami"), LOOPBACK); + await server.http.close(withAbortSignal(AbortSignal.timeout(20), context)); + expect([200, 503]).toContain((await pending).status); + await new Promise((resolve) => setTimeout(resolve, 50)); + await server.close(); + }); + + // Sol's recheck N9. + it("a close whose context is aborted already does not wait", async () => { + const entered = Promise.withResolvers(); + const gate = Promise.withResolvers(); + const server = await served( + { + "@app/wait": contributing({ + routes: { + "GET /wait": async () => { + entered.resolve(); + await gate.promise; + return { kind: "html", body: "done" }; + }, + }, + }), + }, + { config: { listen: undefined, closeTimeout: 1000 } }, + ); + const answer = server.http.fetch(loopback("/wait"), LOOPBACK); + await entered.promise; + const started = Date.now(); + await server.http.close(withAbortSignal(AbortSignal.abort(), context)); + expect(Date.now() - started).toBeLessThan(500); + gate.resolve(); + await answer; + await server.close(); + }); + + // Sol's recheck N5: an authenticator ignoring its signal kept fetch pending after close returned. + it("close settles a request whose authenticator ignores its signal (503)", async () => { + const entered = Promise.withResolvers(); + const gate = Promise.withResolvers(); + const server = await served( + { "@app/r": contributing({ routes: { "GET /": async () => ({ kind: "html", body: "ok" }) } }) }, + { + config: { listen: undefined, closeTimeout: 10 }, + authenticate: async () => { + entered.resolve(); + await gate.promise; + return { kind: "owner" }; + }, + }, + ); + const answer = server.http.fetch(loopback("/"), LOOPBACK); + await entered.promise; + await server.http.close(context); + const settled = await Promise.race([answer.then((response) => response.status), sleepThen(30, "pending")]); + expect(settled).toBe(503); + gate.resolve(); + await server.close(); + }); + + // Astra's recheck R2, Sol's N3: a held `Response` went on calling `read` after its scope was sealed. + it.each(["the request aborted", "the component closed"] as const)("reads no bytes after %s", async (kind) => { + let reads = 0; + let readAfterSeal = false; + const abort = new AbortController(); + const server = await served( + { + "@app/bytes": contributing({ + routes: { + "GET /stream": async () => ({ + kind: "bytes", + mime: "application/octet-stream", + read: async (_offset, _length, ctx) => { + reads++; + readAfterSeal ||= ctx.abortSignal?.aborted ?? false; + return new Uint8Array([reads]); + }, + }), + }, + }), + }, + { config: { listen: undefined, closeTimeout: 10 } }, + ); + const response = await server.http.fetch(loopback("/stream", { signal: abort.signal }), LOOPBACK); + const reader = response.body!.getReader(); + await reader.read(); + if (kind === "the request aborted") abort.abort(new Error("client gone")); + else await server.http.close(context); + await new Promise((resolve) => setImmediate(resolve)); + await expect(reader.read()).rejects.toThrow(); + expect({ reads, readAfterSeal }).toEqual({ reads: 1, readAfterSeal: false }); + await server.close(); + }); + + // Sol's recheck N4. + it("a body that errors after valid JSON fails the request (400); the route's side effect never runs", async () => { + let writes = 0; + const server = await served( + { + "@app/body": contributing({ + routes: { + "POST /body": async (request) => { + const value = await request.json(); + writes++; + return { kind: "json", value }; + }, + }, + }), + }, + { config: { listen: undefined } }, + ); + let pulls = 0; + const body = new ReadableStream({ + pull(controller) { + if (pulls++ === 0) controller.enqueue(new TextEncoder().encode("{}")); + else controller.error(new Error("incoming body checksum failed")); + }, + }); + const init = { method: "POST", body, duplex: "half" as const }; + const response = await server.http.fetch(loopback("/body", init), LOOPBACK); + expect([response.status, writes]).toEqual([400, 0]); + await server.close(); + }); +}); + +describe("answers", () => { + // Sol's recheck N8: `Response` threw on a body for these statuses. + it.each([204, 205, 304])("answers %s without a body", async (status) => { + const server = await served( + { "@app/s": contributing({ routes: { "GET /": async () => ({ kind: "json", value: null, status }) } }) }, + { config: { listen: undefined } }, + ); + const response = await server.http.fetch(loopback("/"), LOOPBACK); + expect([response.status, response.body, response.headers.has("content-length")]).toEqual([status, null, false]); + await server.close(); + }); + + // Fable's recheck N3: the kernel's `forbidden` reached the client as a reported 500. + it("answers a forbidden escaping a route 403, unreported", async () => { + const server = await served( + { + "@app/r": contributing({ + routes: { + "GET /": async () => { + throw new ChordError({ code: "forbidden", message: "no" }); + }, + }, + }), + }, + { config: { listen: undefined } }, + ); + expect((await server.http.fetch(loopback("/"), LOOPBACK)).status).toBe(403); + expect(server.errors).toEqual([]); + await server.close(); + }); + + // Opus's recheck R2: one value per header name allowed one cookie per response. + it("sends several Set-Cookie headers", async () => { + const server = await served({ + "@app/r": contributing({ + routes: { "GET /": async () => ({ kind: "json", value: 1, headers: { "Set-Cookie": ["a=1", "b=2"] } }) }, + }), + }); + expect((await send(server.port, "GET", "/")).headers["set-cookie"]).toEqual(["a=1", "b=2"]); + const fetched = await server.http.fetch( + new Request(`http://127.0.0.1:${server.port}/`, { headers: { host: `127.0.0.1:${server.port}` } }), + { localAddress: "127.0.0.1", localPort: server.port }, + ); + expect(fetched.headers.getSetCookie()).toEqual(["a=1", "b=2"]); + await server.close(); + }); +}); + +describe("GC", () => { + // Sol's second recheck B, her GC probe: 8 of 8 such requests kept their scopes. + it("collects the scopes of aborted bytes Responses dropped without another read", async () => { + const refs: WeakRef[] = []; + const server = await served( + { + "@app/b": contributing({ + routes: { + "GET /": async (_request, ctx) => { + refs.push(new WeakRef(ctx)); + return { kind: "bytes", mime: "application/octet-stream", read: async () => new Uint8Array([1]) }; + }, + }, + }), + }, + { config: { listen: undefined } }, + ); + for (let i = 0; i < 8; i++) { + const abort = new AbortController(); + await server.http.fetch(loopback("/", { signal: abort.signal }), LOOPBACK); + abort.abort(new Error("client gone")); + } + await expectCollected(...refs); + await server.close(); + }); + + it("collects the scopes of bytes Responses born sealed (aborted while the route returned)", async () => { + const refs: WeakRef[] = []; + const aborts: AbortController[] = []; + const server = await served( + { + "@app/b": contributing({ + routes: { + "GET /": async (_request, ctx) => { + refs.push(new WeakRef(ctx)); + aborts.at(-1)?.abort(new Error("client gone")); + return { kind: "bytes", mime: "application/octet-stream", read: async () => new Uint8Array([1]) }; + }, + }, + }), + }, + { config: { listen: undefined } }, + ); + // In a function of its own: the test's frame keeps no last request. + const fetchAborted = async (): Promise => { + const abort = new AbortController(); + aborts.push(abort); + await server.http.fetch(loopback("/", { signal: abort.signal }), LOOPBACK); + }; + for (let i = 0; i < 8; i++) await fetchAborted(); + aborts.length = 0; + await expectCollected(...refs); + await server.close(); + }); + + it("collects requests' scopes and requests once answered", async () => { + const refs: WeakRef[] = []; + const server = await served({ + "@app/a": contributing({ + routes: { + "GET /r": async (request, ctx) => { + refs.push(new WeakRef(ctx), new WeakRef(request)); + return { kind: "json", value: 1 }; + }, + }, + }), + }); + for (let i = 0; i < 5; i++) await send(server.port, "GET", "/r"); + await expectCollected(...refs); + await server.close(); + }); +}); diff --git a/packages/chord/test/next-node/http/node.test.ts b/packages/chord/test/next-node/http/node.test.ts new file mode 100644 index 000000000..8349bd751 --- /dev/null +++ b/packages/chord/test/next-node/http/node.test.ts @@ -0,0 +1,215 @@ +// Tests of Chord's own Node server (the listener's native): request targets (origin-form only; a malformed or +// double-slash path never crashes or reroutes), raw `Host` headers, Web requests made only when read, the request's scope lasting until its response is +// flushed (and `close` waiting for it), and a client's disconnect aborting the scope. + +import { once } from "node:events"; +import { createServer, request as httpRequest, type IncomingMessage } from "node:http"; +import { connect } from "node:net"; +import { describe, expect, it } from "vitest"; +import { nodeRequest } from "../../../src/next-node/http/index.ts"; +import { context, contributing, served } from "./support.ts"; + +/** What `port` answers a raw request `text` with: its status line and body. */ +async function raw(port: number, text: string): Promise<{ readonly status: string; readonly body: string }> { + const socket = connect(port, "127.0.0.1"); + let out = ""; + socket.on("data", (chunk) => { + out += chunk.toString("latin1"); + }); + socket.on("error", () => {}); + await once(socket, "connect"); + socket.write(text); + await once(socket, "close"); + const [head = "", body = ""] = out.split("\r\n\r\n"); + return { status: head.split("\r\n")[0] ?? "", body }; +} + +/** A route answering its path and params, under `GET /who` and `GET /n/:name`. */ +const echo = contributing({ + routes: { + "GET /who": async (request) => ({ kind: "json", value: request.path }), + "GET /n/:name": async (request) => ({ kind: "json", value: request.params }), + }, +}); + +describe("targets", () => { + // Astra's build review, Fable's security audit C1: `GET //[` threw out of the request handler and ended the process. + it("answers a malformed or double-slash path without crashing or rerouting it (Opus's audit H5)", async () => { + const server = await served({ "@app/echo": echo }); + const host = `Host: 127.0.0.1:${server.port}`; + const get = (target: string) => + raw(server.port, `GET ${target} HTTP/1.1\r\n${host}\r\nConnection: close\r\n\r\n`); + expect((await get("//[")).status).toBe("HTTP/1.1 404 Not Found"); + expect((await get("//evil.example/who")).status).toBe("HTTP/1.1 404 Not Found"); + expect(await get("/a/../who")).toEqual({ status: "HTTP/1.1 200 OK", body: '"/who"' }); + // The URL parser resolves `%2e%2e` as a dot segment: `/n/%2e%2e` is `/`. + expect((await get("/n/%2e%2e")).status).toBe("HTTP/1.1 404 Not Found"); + expect((await get("/n/a%5Cb")).status).toBe("HTTP/1.1 400 Bad Request"); + expect((await get("/n/..%2F..%2Fsecret")).status).toBe("HTTP/1.1 400 Bad Request"); + expect((await get("/who")).status).toBe("HTTP/1.1 200 OK"); + await server.close(); + }); + + it("refuses an absolute-form target, two Host headers and none (403)", async () => { + const server = await served({ "@app/echo": echo }); + const host = `Host: 127.0.0.1:${server.port}`; + const answers = await Promise.all([ + raw(server.port, `GET http://evil.example/who HTTP/1.1\r\n${host}\r\nConnection: close\r\n\r\n`), + raw(server.port, `GET /who HTTP/1.1\r\n${host}\r\nHost: evil.example\r\nConnection: close\r\n\r\n`), + raw(server.port, "GET /who HTTP/1.0\r\n\r\n"), + ]); + expect(answers.map(({ status }) => status)).toEqual([ + "HTTP/1.1 403 Forbidden", + "HTTP/1.1 403 Forbidden", + "HTTP/1.1 403 Forbidden", + ]); + await server.close(); + }); + + it("refuses a Host with userinfo (403) and a method a Web Request cannot carry (501)", async () => { + const server = await served({ "@app/echo": echo }); + const answers = await Promise.all([ + raw(server.port, `GET /who HTTP/1.1\r\nHost: attacker@127.0.0.1:${server.port}\r\nConnection: close\r\n\r\n`), + raw(server.port, `TRACE /who HTTP/1.1\r\nHost: 127.0.0.1:${server.port}\r\nConnection: close\r\n\r\n`), + ]); + // `TRACK` never reaches Chord: Node's parser refuses it (400); `CONNECT` goes to Node's `connect` event. + expect(answers.map(({ status }) => status)).toEqual(["HTTP/1.1 403 Forbidden", "HTTP/1.1 501 Not Implemented"]); + await server.close(); + }); +}); + +describe("mounting on an app's own Node server", () => { + // Sol's recheck N6: a hand-made `Request` threw unhandled on `Host: [` or an absolute-form target (the process + // ended) and kept only the first of two `Host`s. + it("converts with the listener's checks, never throwing: bad targets, userinfo, TRACE refused; duplicate Hosts kept", async () => { + const server = await served({ "@app/echo": echo }, { config: { listen: undefined } }); + const own = createServer(async (message, res) => { + const converted = nodeRequest(message, res); + if (converted.kind === "refused") return void res.writeHead(converted.status).end(); + const response = await server.http.fetch(converted.request, converted.connection); + for (const [name, value] of response.headers) res.appendHeader(name, value); + res.writeHead(response.status).end(Buffer.from(await response.arrayBuffer())); + }); + await new Promise((resolve) => own.listen(0, "127.0.0.1", resolve)); + const address = own.address(); + const port = typeof address === "object" && address !== null ? address.port : 0; + const host = `Host: 127.0.0.1:${port}`; + const get = (text: string) => raw(port, `${text}Connection: close\r\n\r\n`); + expect(await get(`GET /who HTTP/1.1\r\n${host}\r\n`)).toEqual({ status: "HTTP/1.1 200 OK", body: '"/who"' }); + expect((await get("GET /who HTTP/1.1\r\nHost: [\r\n")).status).toBe("HTTP/1.1 400 Bad Request"); + expect((await get(`GET http://evil.example/who HTTP/1.1\r\n${host}\r\n`)).status).toBe("HTTP/1.1 403 Forbidden"); + expect((await get(`GET /who HTTP/1.1\r\n${host}\r\nHost: evil.example\r\n`)).status).toBe( + "HTTP/1.1 403 Forbidden", + ); + // Sol's second recheck A: both threw out of the conversion; the app's callback ended the process. + expect((await get(`TRACE /who HTTP/1.1\r\n${host}\r\n`)).status).toBe("HTTP/1.1 501 Not Implemented"); + expect((await get(`GET /who HTTP/1.1\r\nHost: attacker@127.0.0.1:${port}\r\n`)).status).toBe( + "HTTP/1.1 403 Forbidden", + ); + expect((await get(`GET /who HTTP/1.1\r\n${host}\r\n`)).status).toBe("HTTP/1.1 200 OK"); + await new Promise((resolve) => own.close(resolve)); + await server.close(); + }); +}); + +describe("Web requests", () => { + it("makes a Web Request only when something reads the headers or the body (here: no authenticator does)", async () => { + const server = await served( + { + "@app/r": contributing({ + routes: { + "GET /plain": async (request) => ({ kind: "json", value: request.path }), + "GET /headers": async (request) => ({ kind: "json", value: request.headers.get("x-thing") }), + "POST /body": async (request) => ({ kind: "json", value: await request.json() }), + }, + }), + }, + { authenticate: () => undefined }, + ); + const original = globalThis.Request; + let made = 0; + globalThis.Request = class extends original { + constructor(...args: ConstructorParameters) { + super(...args); + made++; + } + }; + try { + const host = `Host: 127.0.0.1:${server.port}\r\nConnection: close`; + await raw(server.port, `GET /plain HTTP/1.1\r\n${host}\r\n\r\n`); + expect(made).toBe(0); + const read = await raw(server.port, `GET /headers HTTP/1.1\r\n${host}\r\nX-Thing: t\r\n\r\n`); + const posted = await raw(server.port, `POST /body HTTP/1.1\r\n${host}\r\nContent-Length: 3\r\n\r\n"a"`); + expect([read.body, posted.body, made]).toEqual(['"t"', '"a"', 2]); + } finally { + globalThis.Request = original; + } + await server.close(); + }); +}); + +describe("the request's scope", () => { + // Sol's build review: the scope sealed, and `close` stopped waiting, before a large response was flushed. + it("lasts until the response is flushed; close waits for it", async () => { + const size = 32 * 1024 * 1024; + let signal: AbortSignal | undefined; + const server = await served( + { + "@app/large": contributing({ + routes: { + "GET /large": async (_request, ctx) => { + signal = ctx.abortSignal; + return { kind: "html", body: "x".repeat(size) }; + }, + }, + }), + }, + { config: { listen: { host: "127.0.0.1", port: 0 }, closeTimeout: 2000 } }, + ); + const request = httpRequest({ host: "127.0.0.1", port: server.port, path: "/large", agent: false }); + const response = await new Promise((resolve) => { + request.on("response", (arrived: IncomingMessage) => { + arrived.pause(); + resolve(arrived); + }); + request.end(); + }); + let length = 0; + response.on("data", (chunk: Buffer) => { + length += chunk.length; + }); + response.pause(); + await new Promise((resolve) => setTimeout(resolve, 20)); + const sealedBeforeFlush = signal?.aborted; + const closing = server.http.close(context); + await new Promise((resolve) => setTimeout(resolve, 20)); + response.resume(); + await once(response, "end"); + await closing; + expect({ sealedBeforeFlush, length }).toEqual({ sealedBeforeFlush: false, length: size }); + await server.close(); + }); + + it("is aborted when the client disconnects", async () => { + const entered = Promise.withResolvers(); + const server = await served({ + "@app/slow": contributing({ + routes: { + "GET /slow": async (_request, ctx) => { + entered.resolve(ctx.abortSignal!); + return new Promise(() => {}); + }, + }, + }), + }); + const request = httpRequest({ host: "127.0.0.1", port: server.port, path: "/slow", agent: false }); + request.on("error", () => {}); + request.end(); + const signal = await entered.promise; + const aborted = new Promise((resolve) => signal.addEventListener("abort", resolve, { once: true })); + request.destroy(); + await aborted; + expect(signal.aborted).toBe(true); + await server.close(); + }); +}); diff --git a/packages/chord/test/next-node/http/request.test.ts b/packages/chord/test/next-node/http/request.test.ts new file mode 100644 index 000000000..a48a2429e --- /dev/null +++ b/packages/chord/test/next-node/http/request.test.ts @@ -0,0 +1,119 @@ +// Tests of requests (slice 10 §2, D4): the summary a route gets, `json` and `body` with their limits and statuses +// (400, 413, an `HttpFailure` escaping a route answering its status), and a body nobody read (drained or closed). + +import { Agent, request as httpRequest } from "node:http"; +import { describe, expect, it } from "vitest"; +import type { HttpRequest } from "../../../src/next-node/http/index.ts"; +import { contributing, send, served } from "./support.ts"; + +describe("the request a route gets", () => { + it("has its method, path, params, query, headers, origin, provenance and the response's nonce", async () => { + let seen: HttpRequest | undefined; + const server = await served({ + "@app/a": contributing({ + routes: { + "GET /notes/:id": async (request) => { + seen = request; + return { kind: "json", value: null }; + }, + }, + }), + }); + await send(server.port, "GET", "/notes/a%20b?x=1&x=2", { headers: { "X-Thing": "t" } }); + expect(seen).toMatchObject({ + method: "GET", + path: "/notes/a%20b", + params: { id: "a b" }, + origin: `http://127.0.0.1:${server.port}`, + provenance: { kind: "local" }, + }); + expect([seen?.query.getAll("x"), seen?.headers.get("x-thing")]).toEqual([["1", "2"], "t"]); + expect(seen?.nonce).toMatch(/^[A-Za-z0-9+/]{22}==$/); + await server.close(); + }); +}); + +describe("bodies", () => { + /** A server whose `POST /json` answers the JSON it read (limit 16), `POST /body` the bytes it read (limit 16). */ + const reading = () => + served({ + "@app/a": contributing({ + routes: { + "POST /json": async (request) => ({ kind: "json", value: await request.json(16) }), + "POST /body": async (request) => { + let bytes = 0; + for await (const chunk of request.body(16)) bytes += chunk.byteLength; + return { kind: "json", value: bytes }; + }, + }, + }), + }); + + it("parses JSON within its limit; malformed is 400, larger is 413 (declared or streamed)", async () => { + const server = await reading(); + const { port } = server; + expect((await send(port, "POST", "/json", { body: '{"a":1}' })).body.toString()).toBe('{"a":1}'); + expect((await send(port, "POST", "/json", { body: "{nope" })).status).toBe(400); + expect((await send(port, "POST", "/json", { body: JSON.stringify({ long: "x".repeat(40) }) })).status).toBe(413); + const chunked = await send(port, "POST", "/body", { + headers: { "transfer-encoding": "chunked" }, + body: "y".repeat(40), + }); + expect(chunked.status).toBe(413); + expect((await send(port, "POST", "/body", { body: "y".repeat(10) })).body.toString()).toBe("10"); + expect(server.errors).toEqual([]); + await server.close(); + }); + + // Sol's and Astra's build reviews: an unfinished chunked upload past the limit got no answer. + it("answers 413 before a chunked upload past the limit has ended", async () => { + const server = await reading(); + const request = httpRequest({ + host: "127.0.0.1", + port: server.port, + path: "/json", + method: "POST", + headers: { "transfer-encoding": "chunked" }, + agent: false, + }); + request.on("error", () => {}); + const status = new Promise((resolve) => { + request.on("response", (response) => { + response.resume(); + resolve(response.statusCode ?? 0); + }); + }); + request.write(`"${"x".repeat(40)}`); + expect(await status).toBe(413); + request.destroy(); + await server.close(); + }); + + it("drains a small body nobody read and keeps the connection; a large one closes it", async () => { + const server = await served({ + "@app/a": contributing({ routes: { "POST /ignore": async () => ({ kind: "json", value: 1 }) } }), + }); + const agent = new Agent({ keepAlive: true, maxSockets: 1 }); + const post = (body: string) => + new Promise<{ connection: string | undefined; socket: unknown }>((resolve, reject) => { + const request = httpRequest( + { host: "127.0.0.1", port: server.port, method: "POST", path: "/ignore", agent }, + (response) => { + response.resume(); + response.on("end", () => + resolve({ connection: response.headers.connection, socket: request.socket }), + ); + }, + ); + request.on("error", reject); + request.end(body); + }); + const first = await post("x".repeat(1000)); + const second = await post("x".repeat(1000)); + expect(first.connection).toBe("keep-alive"); + expect(second.socket).toBe(first.socket); + expect((await post("x".repeat(100_000))).connection).toBe("close"); + agent.destroy(); + await server.close(); + }); +}); diff --git a/packages/chord/test/next-node/http/response.test.ts b/packages/chord/test/next-node/http/response.test.ts new file mode 100644 index 000000000..3c953ba85 --- /dev/null +++ b/packages/chord/test/next-node/http/response.test.ts @@ -0,0 +1,245 @@ +// Tests of responses (slice 10 §2, D5, R6; security baseline (4), (6)): every kind written by a running server, the +// response's headers over the default ones (an invalid one reported, 500), ranges (206, 416, multi-range), `HEAD`, +// attachments and inline bodies, safe download names, bodies without `size`, a short or throwing read, a disconnect +// stopping the reads. + +import { request as httpRequest } from "node:http"; +import { describe, expect, it } from "vitest"; +import type { Ctx } from "../../../src/next/host/index.ts"; +import type { HttpResponse } from "../../../src/next-node/http/index.ts"; +import { contributing, send, served } from "./support.ts"; + +/** A server answering `GET /r` with `response`. */ +async function answering(response: HttpResponse) { + return served({ "@app/r": contributing({ routes: { "GET /r": async () => response } }) }); +} + +/** A `bytes` response over `data` that records its reads. */ +function bytesOf(data: Buffer, extra: Partial> = {}) { + const reads: [number, number][] = []; + const response: HttpResponse = { + kind: "bytes", + mime: "application/octet-stream", + size: data.length, + read: async (offset, length) => { + reads.push([offset, length]); + return data.subarray(offset, offset + length); + }, + ...extra, + }; + return { response, reads }; +} + +describe("descriptions", () => { + it("writes json, html, redirect and error, the default headers first, the response's replacing same-named ones", async () => { + const cases: [HttpResponse, number, string, string | undefined][] = [ + [{ kind: "json", value: { a: 1 } }, 200, '{"a":1}', "application/json; charset=utf-8"], + [ + { kind: "html", body: "

    x

    ", status: 201, headers: { "content-security-policy": "own" } }, + 201, + "

    x

    ", + "text/html; charset=utf-8", + ], + [{ kind: "redirect", to: "/next", status: 303 }, 303, "", undefined], + [{ kind: "error", status: 418 }, 418, "I'm a Teapot", "text/plain; charset=utf-8"], + [{ kind: "error", status: 409, message: "taken" }, 409, "taken", "text/plain; charset=utf-8"], + ]; + for (const [response, status, body, type] of cases) { + const server = await answering(response); + const answer = await send(server.port, "GET", "/r"); + expect([answer.status, answer.body.toString(), answer.headers["content-type"]]).toEqual([status, body, type]); + expect([answer.headers["cache-control"], answer.headers["x-content-type-options"]]).toEqual([ + "no-store", + "nosniff", + ]); + expect(answer.headers["content-security-policy"]).toMatch( + response.headers === undefined ? /^default-src 'none'/ : /^own$/, + ); + if (response.kind === "redirect") expect(answer.headers.location).toBe("/next"); + await server.close(); + } + }); + + it("answers 500 for a header with a CR or LF and a value that is no JSON, reported response_failed", async () => { + const server = await served({ + "@app/r": contributing({ + routes: { + "GET /header": async () => ({ kind: "json", value: 1, headers: { "X-Split": "a\r\nSet-Cookie: x=1" } }), + "GET /bigint": async () => ({ kind: "json", value: { n: 1n } as never }), + }, + }), + }); + for (const path of ["/header", "/bigint"]) { + const answer = await send(server.port, "GET", path); + expect([answer.status, answer.headers["set-cookie"], answer.headers["x-content-type-options"]]).toEqual([ + 500, + undefined, + "nosniff", + ]); + } + expect(server.errors).toMatchObject([ + { kind: "response_failed", pattern: "GET /header" }, + { kind: "response_failed", pattern: "GET /bigint" }, + ]); + await server.close(); + }); +}); + +describe("bytes", () => { + const data = Buffer.from("0123456789".repeat(60_000)); + + it("sends Content-Length, reads 256 KiB pieces, answers a single range 206 and an unsatisfiable one 416", async () => { + const { response, reads } = bytesOf(data, { ranges: true }); + const server = await answering(response); + const whole = await send(server.port, "GET", "/r"); + expect([whole.status, whole.headers["content-length"], whole.headers["accept-ranges"]]).toEqual([ + 200, + "600000", + "bytes", + ]); + expect(whole.body.equals(data)).toBe(true); + expect(reads).toEqual([ + [0, 262144], + [262144, 262144], + [524288, 75712], + ]); + const range = await send(server.port, "GET", "/r", { headers: { range: "bytes=10-19" } }); + expect([range.status, range.body.toString(), range.headers["content-range"]]).toEqual([ + 206, + "0123456789", + "bytes 10-19/600000", + ]); + const suffix = await send(server.port, "GET", "/r", { headers: { range: "bytes=-3" } }); + expect([suffix.status, suffix.body.toString()]).toEqual([206, "789"]); + const past = await send(server.port, "GET", "/r", { headers: { range: "bytes=600000-" } }); + expect([past.status, past.headers["content-range"]]).toEqual([416, "bytes */600000"]); + const multi = await send(server.port, "GET", "/r", { headers: { range: "bytes=0-1,5-6" } }); + expect([multi.status, multi.body.length]).toEqual([200, 600000]); + await server.close(); + }); + + it("ignores ranges unless asked; HEAD sends headers only and reads nothing", async () => { + const { response, reads } = bytesOf(data); + const server = await answering(response); + const ranged = await send(server.port, "GET", "/r", { headers: { range: "bytes=0-1" } }); + expect([ranged.status, ranged.headers["accept-ranges"]]).toEqual([200, undefined]); + reads.length = 0; + const head = await send(server.port, "HEAD", "/r"); + expect([head.status, head.headers["content-length"], head.body.length, reads]).toEqual([200, "600000", 0, []]); + await server.close(); + }); + + it("names a download with an ASCII fallback and the UTF-8 name, control and bidi characters dropped", async () => { + const server = await answering(bytesOf(Buffer.from("x"), { download: 'Grüße "1"\r\n\u202etxt.exe' }).response); + const answer = await send(server.port, "GET", "/r"); + expect(answer.headers["content-disposition"]).toBe( + `attachment; filename="Gr__e _1_txt.exe"; filename*=UTF-8''Gr%C3%BC%C3%9Fe%20%221%22txt.exe`, + ); + await server.close(); + }); + + it("is an attachment unless inline; an inline one gets a sandbox CSP", async () => { + const attached = await answering(bytesOf(Buffer.from("x")).response); + expect((await send(attached.port, "GET", "/r")).headers["content-disposition"]).toBe("attachment"); + await attached.close(); + const shown = await answering(bytesOf(Buffer.from("x"), { inline: true }).response); + const answer = await send(shown.port, "GET", "/r"); + expect(answer.headers["content-disposition"]).toBe("inline"); + expect(answer.headers["content-security-policy"]).toMatch(/; sandbox$/); + await shown.close(); + }); + + it("refuses a foreign request (another site's) 403, reading nothing", async () => { + const { response, reads } = bytesOf(Buffer.from("secret")); + const server = await answering(response); + const foreign = await send(server.port, "GET", "/r", { headers: { "sec-fetch-site": "cross-site" } }); + expect([foreign.status, reads]).toEqual([403, []]); + expect((await send(server.port, "GET", "/r", { headers: { "sec-fetch-site": "same-origin" } })).status).toBe(200); + await server.close(); + }); + + it("without a size: chunked, read with growing offsets until an empty chunk, no ranges (R6)", async () => { + const pieces = [Buffer.from("ab"), Buffer.from("cde"), Buffer.alloc(0)]; + const offsets: number[] = []; + const server = await answering({ + kind: "bytes", + mime: "text/plain", + ranges: true, + read: async (offset) => { + offsets.push(offset); + return pieces.shift()!; + }, + }); + const answer = await send(server.port, "GET", "/r", { headers: { range: "bytes=0-1" } }); + expect([ + answer.status, + answer.body.toString(), + answer.headers["transfer-encoding"], + answer.headers["accept-ranges"], + ]).toEqual([200, "abcde", "chunked", undefined]); + expect(offsets).toEqual([0, 2, 5]); + await server.close(); + }); + + it("destroys the connection on a short or a throwing read, reported response_failed", async () => { + const server = await served({ + "@app/r": contributing({ + routes: { + "GET /short": async () => ({ + kind: "bytes", + mime: "text/plain", + size: 10, + read: async () => Buffer.from("abc"), + }), + // Astra's build review: a throwing read hung the response, unreported. + "GET /throws": async () => ({ + kind: "bytes", + mime: "text/plain", + size: 10, + read: async () => { + throw new Error("read failed"); + }, + }), + }, + }), + }); + await expect(send(server.port, "GET", "/short")).rejects.toThrow(); + await expect(send(server.port, "GET", "/throws")).rejects.toThrow(); + expect(server.errors).toMatchObject([ + { kind: "response_failed", pattern: "GET /short" }, + { kind: "response_failed", pattern: "GET /throws", error: { message: "read failed" } }, + ]); + await server.close(); + }); + + it("stops reading when the client disconnects: the read's context is aborted", async () => { + let reads = 0; + let signal: AbortSignal | undefined; + const server = await answering({ + kind: "bytes", + mime: "application/octet-stream", + size: 1024 * 1024 * 1024, + read: async (_offset, length, ctx: Ctx) => { + reads++; + signal = ctx.abortSignal; + return Buffer.alloc(length); + }, + }); + await new Promise((resolve) => { + const request = httpRequest({ host: "127.0.0.1", port: server.port, path: "/r", agent: false }, (response) => { + response.once("data", () => { + request.destroy(); + resolve(); + }); + }); + request.end(); + }); + await new Promise((resolve) => setTimeout(resolve, 100)); + const after = reads; + await new Promise((resolve) => setTimeout(resolve, 100)); + expect(signal?.aborted).toBe(true); + expect(reads).toBe(after); + expect(after).toBeLessThan(1000); + await server.close(); + }); +}); diff --git a/packages/chord/test/next-node/http/router.test.ts b/packages/chord/test/next-node/http/router.test.ts new file mode 100644 index 000000000..9854820c6 --- /dev/null +++ b/packages/chord/test/next-node/http/router.test.ts @@ -0,0 +1,228 @@ +// Tests of the router (slice 10 §2, D2, D8): patterns, decoding (one rule for `:name` and rest segments), precedence, +// `HEAD` per path, each route's own param names, route options, conflicts, wraps resolved by phase, contributor order, +// scope and route, allowed hosts, framers, relays, CSP sources, sockets. + +import { describe, expect, it } from "vitest"; +import type { RegistryConflict, RegistryTable } from "../../../src/next/registries/index.ts"; +import { + composeRouter, + type HttpContribution, + type HttpRoute, + type HttpWrap, + type RouteMatch, + type Router, +} from "../../../src/next-node/http/router.ts"; + +/** A route named `name` (its handler answers its name). */ +const route = + (name: string): HttpRoute => + async () => ({ kind: "html", body: name }); + +/** The router of `entries` (plugin, contribution), with the conflicts it reported. */ +function compose(...entries: [string, HttpContribution][]): { router: Router; conflicts: RegistryConflict[] } { + const conflicts: RegistryConflict[] = []; + const table: RegistryTable = entries.map(([plugin, value]) => ({ plugin, version: 1, value })); + return { router: composeRouter(table, { report: (conflict) => conflicts.push(conflict) }), conflicts }; +} + +/** The matched route's pattern and params, or what `route` answered. */ +function matched(router: Router, method: string, path: string) { + const match = router.route(method, path); + if (match === undefined || match === "bad_path") return match; + return { pattern: match.pattern, params: match.params }; +} + +describe("patterns and precedence (D8)", () => { + const { router } = compose([ + "@app/a", + { + routes: { + "GET /notes": route("list"), + "GET /notes/:id": route("one"), + "GET /notes/new": route("new"), + "GET /notes/:id/export": route("export"), + "GET /files/*": route("files"), + "GET /files/special/:name": route("special"), + "GET /": route("root"), + }, + }, + ]); + + it("matches exact segments, :name and a trailing *, the most specific from the left", () => { + expect(matched(router, "GET", "/notes")).toEqual({ pattern: "GET /notes", params: {} }); + expect(matched(router, "GET", "/notes/new")).toEqual({ pattern: "GET /notes/new", params: {} }); + expect(matched(router, "GET", "/notes/7")).toEqual({ pattern: "GET /notes/:id", params: { id: "7" } }); + expect(matched(router, "GET", "/notes/7/export")).toMatchObject({ params: { id: "7" } }); + expect(matched(router, "GET", "/files/special/x")).toEqual({ + pattern: "GET /files/special/:name", + params: { name: "x" }, + }); + // The exact branch fails deeper: the rest takes it. + expect(matched(router, "GET", "/files/special/x/y")).toEqual({ + pattern: "GET /files/*", + params: { "*": "special/x/y" }, + }); + expect(matched(router, "GET", "/")).toEqual({ pattern: "GET /", params: {} }); + }); + + it("a /* matches its prefix with an empty rest; :name needs a non-empty segment", () => { + expect(matched(router, "GET", "/files")).toEqual({ pattern: "GET /files/*", params: { "*": "" } }); + expect(matched(router, "GET", "/notes/")).toBeUndefined(); + }); + + // Opus's build review F4, security audit H5: `%2F` in `:name` reached a file path. + it("percent-decodes :name and rest segments; one decoding to /, \\, NUL, . or .. is a bad path", () => { + expect(matched(router, "GET", "/notes/a%20b")).toMatchObject({ params: { id: "a b" } }); + expect(matched(router, "GET", "/files/a%20b/c")).toMatchObject({ params: { "*": "a b/c" } }); + for (const bad of ["a%2Fb", "..%2F..%2Fsecret", "a%5Cb", "a%00b", "%2e%2e", "%2E", ".", ".."]) { + expect(router.route("GET", `/notes/${bad}`)).toBe("bad_path"); + expect(router.route("GET", `/files/x/${bad}`)).toBe("bad_path"); + } + expect(router.route("GET", "/notes/%E0%A4%A")).toBe("bad_path"); + }); + + it("falls back to the GET route for HEAD; another method or path has none", () => { + expect(matched(router, "HEAD", "/notes")).toEqual({ pattern: "GET /notes", params: {} }); + expect(router.route("POST", "/notes")).toBeUndefined(); + expect(router.route("GET", "/nope")).toBeUndefined(); + }); + + // Opus's build review F2, Astra's: one explicit HEAD route anywhere hid every GET fallback. + it("falls back to GET per path when another path has an explicit HEAD route", () => { + const { router } = compose([ + "@app/a", + { routes: { "HEAD /download": route("head"), "GET /download": route("get"), "GET /notes": route("notes") } }, + ]); + expect(matched(router, "HEAD", "/download")).toMatchObject({ pattern: "HEAD /download" }); + expect(matched(router, "HEAD", "/notes")).toMatchObject({ pattern: "GET /notes" }); + }); + + // Opus's build review F3, Astra's: the first route's param name was used for every route at that position. + it("names each route's params by its own pattern", () => { + const { router } = compose(["@app/a", { routes: { "GET /:id/a": route("a"), "GET /:name/b": route("b") } }]); + expect(matched(router, "GET", "/v/a")).toEqual({ pattern: "GET /:id/a", params: { id: "v" } }); + expect(matched(router, "GET", "/v/b")).toEqual({ pattern: "GET /:name/b", params: { name: "v" } }); + }); + + it("takes a route with options: page and crossSite", () => { + const { router } = compose([ + "@app/a", + { + routes: { + "GET /p": { handler: route("p"), page: true }, + "POST /hook": { handler: route("h"), crossSite: true }, + }, + }, + ]); + expect(router.route("GET", "/p")).toMatchObject({ page: true, crossSite: false }); + expect(router.route("POST", "/hook")).toMatchObject({ page: false, crossSite: true }); + expect(router.route("GET", "/notes")).toBeUndefined(); + }); +}); + +describe("conflicts", () => { + it("reports one pattern from two contributions (also with other param names); the first by plugin name wins", () => { + const { router, conflicts } = compose( + ["@app/a", { routes: { "GET /x/:id": route("a") } }], + ["@app/b", { routes: { "GET /x/:key": route("b"), BAD: route("bad") }, sockets: { nope: () => {} } }], + ); + expect((router.route("GET", "/x/1") as RouteMatch).plugin).toBe("@app/a"); + expect(conflicts).toEqual([ + { message: 'route "GET /x/:key" is contributed twice', plugins: ["@app/a", "@app/b"] }, + { message: 'route "BAD" is no "METHOD /pattern"', plugins: ["@app/b"] }, + { message: 'socket "nope" is no pattern', plugins: ["@app/b"] }, + ]); + }); + + it("refuses a route for a method a Web Request cannot carry, reported", () => { + const { router, conflicts } = compose([ + "@app/a", + { routes: { "TRACE /x": route("t"), "CONNECT /y": route("c") } }, + ]); + expect(router.route("TRACE", "/x")).toBeUndefined(); + expect(conflicts).toEqual([ + { message: 'route "TRACE /x": TRACE is served for no resource (501)', plugins: ["@app/a"] }, + { message: 'route "CONNECT /y": CONNECT is served for no resource (501)', plugins: ["@app/a"] }, + ]); + }); +}); + +describe("wraps (D2)", () => { + const wrap = (name: string, phase: HttpWrap["phase"], extra: Partial = {}): HttpWrap => ({ + name, + phase, + wrap: (request, next) => next(request), + ...extra, + }); + + it("orders by phase (outermost first), then contributor order; own wraps around their plugin's routes only", () => { + const { router } = compose( + [ + "@app/a", + { + routes: { "GET /a": route("a") }, + wraps: [wrap("a-transform", "transform"), wrap("a-observe", "observe")], + }, + ], + [ + "@app/b", + { + routes: { "GET /b": route("b") }, + wraps: [wrap("b-all", "headers", { scope: "all" }), wrap("b-own", "observe")], + }, + ], + ); + const names = (path: string) => (router.route("GET", path) as RouteMatch).wraps.map(({ wrap }) => wrap.name); + expect(names("/a")).toEqual(["a-observe", "b-all", "a-transform"]); + expect(names("/b")).toEqual(["b-own", "b-all"]); + }); + + it("narrows to one route; a route nobody contributes is a reported conflict", () => { + const { router, conflicts } = compose([ + "@app/a", + { + routes: { "GET /a": route("a"), "GET /b": route("b") }, + wraps: [wrap("only-a", "observe", { route: "GET /a" }), wrap("ghost", "observe", { route: "GET /c" })], + }, + ]); + expect((router.route("GET", "/a") as RouteMatch).wraps.map(({ wrap }) => wrap.name)).toEqual(["only-a"]); + expect((router.route("GET", "/b") as RouteMatch).wraps).toEqual([]); + expect(conflicts).toEqual([ + { message: 'wrap "ghost" names route "GET /c", which nobody contributes', plugins: ["@app/a"] }, + ]); + }); +}); + +describe("hosts and sockets", () => { + it("collects hosts lowercased, framers, relays and CSP sources; matches sockets with their largest frame", () => { + const terminal = () => {}; + const relay = { name: "radius", verify: () => undefined }; + const contribution: HttpContribution = { + hosts: ["Notes.Example.com"], + sockets: { "/terminal/:id": terminal }, + maxPayload: 10, + framers: ["https://frame.example.com"], + relays: [relay], + csp: { "img-src": ["https://cdn.example.com"] }, + }; + const { router } = compose( + ["@app/a", contribution], + ["@app/b", { hosts: ["b.example.com"], csp: { "img-src": ["data:"] } }], + ); + expect([...router.hosts]).toEqual(["notes.example.com", "b.example.com"]); + expect([[...router.framers], router.relays, [...router.csp]]).toEqual([ + ["https://frame.example.com"], + [relay], + [["img-src", ["https://cdn.example.com", "data:"]]], + ]); + expect(router.socket("/terminal/7")).toMatchObject({ + pattern: "/terminal/:id", + plugin: "@app/a", + params: { id: "7" }, + maxPayload: 10, + }); + expect(router.socket("/terminal/..")).toBe("bad_path"); + expect(router.socket("/nope")).toBeUndefined(); + expect(router.socketHandlers.has(terminal)).toBe(true); + }); +}); diff --git a/packages/chord/test/next-node/http/support.ts b/packages/chord/test/next-node/http/support.ts new file mode 100644 index 000000000..8a4b82b59 --- /dev/null +++ b/packages/chord/test/next-node/http/support.ts @@ -0,0 +1,204 @@ +// Test support for the HTTP component (slice 10 §2): a server host with the component, the app's configuration and +// authenticator, and plugin facets contributing to `Http`; raw requests (any `Host`, any body) and WebSocket clients +// against its listener. + +import { request as httpRequest, type IncomingHttpHeaders } from "node:http"; +import { WebSocket } from "ws"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import type { Address } from "../../../src/next/contracts/index.ts"; +import { createHost, defineFacet, type Facet, type Host, type HostError } from "../../../src/next/host/index.ts"; +import { + type Authenticator, + Authenticators, + createHttp, + Http, + type HttpComponent, + type HttpContribution, + type HttpError, + type HttpOptions, +} from "../../../src/next-node/http/index.ts"; + +/** The principals of these tests. */ +export type TestPrincipal = + | { readonly kind: "server" } + | { readonly kind: "owner" } + | { readonly kind: "guest"; readonly id: string }; + +/** A running server: its host, its component, its port once loaded, what both reported. */ +export type Served = { + readonly host: Host; + readonly http: HttpComponent; + /** The listener's port (after `load`). */ + readonly port: number; + readonly errors: HttpError[]; + readonly hostErrors: HostError[]; + /** The core facets and the plugins, as loaded. */ + facets(plugins?: Readonly>>): { name: string; facet: Facet }[]; + close(): Promise; +}; + +/** A plugin facet contributing `contribution` to `Http`. */ +export function contributing(contribution: HttpContribution): Facet { + return defineFacet((_host, self) => self.contribute(Http, contribution)); +} + +/** + * Builds the component on a new server host (listening on loopback, a free port), then loads it with the app's + * authenticator (`@app/auth`: the `x-principal` header names one, else anonymous) and `plugins` (by name) unless + * `load` is false; `config` overrides the settings (`listen: undefined`: no listener). + */ +export async function served( + plugins: Readonly>> = {}, + options: { + readonly config?: Omit; + readonly authenticate?: Authenticator; + readonly load?: boolean; + readonly readyTimeout?: number; + readonly address?: Address; + } = {}, +): Promise }> { + const errors: HttpError[] = []; + const hostErrors: HostError[] = []; + const host = createHost({ + principal: { kind: "server" }, + onError: (error) => hostErrors.push(error), + ...(options.readyTimeout === undefined ? {} : { readyTimeout: options.readyTimeout }), + ...(options.address === undefined ? {} : { address: options.address }), + }); + const config: Omit = { listen: { host: "127.0.0.1", port: 0 }, ...options.config }; + const http = createHttp(host, { ...config, onError: (error) => errors.push(error) }, context); + const authenticate = options.authenticate ?? ((request) => principalOf(request.headers)); + const core = [ + { name: "chord.http", facet: http.facet }, + { + name: "@app/auth", + facet: defineFacet((_host, self) => self.contribute(Authenticators, authenticate)), + }, + ]; + const facets = (more = plugins) => [...core, ...Object.entries(more).map(([name, facet]) => ({ name, facet }))]; + let port = 0; + const load = async (): Promise => { + await host.load(facets(), context); + if (config.listen !== undefined) port = (await http.listening(context)).port; + }; + if (options.load !== false) await load(); + return { + host, + http, + get port() { + return port; + }, + errors, + hostErrors, + facets, + load, + close: async () => { + await http.close(context); + await host.close(context); + }, + }; +} + +/** The principal an `x-principal` header names: `owner`, `guest:`; none without it. */ +function principalOf(headers: Headers): TestPrincipal | undefined { + const named = headers.get("x-principal"); + if (named === "owner") return { kind: "owner" }; + if (named?.startsWith("guest:")) return { kind: "guest", id: named.slice(6) }; + return undefined; +} + +/** A response as a test reads it. */ +export type Answer = { readonly status: number; readonly headers: IncomingHttpHeaders; readonly body: Buffer }; + +/** + * A request to `port` with raw control: `Host` defaults to `127.0.0.1:`; `body` is sent as is; `signal` aborts it. + */ +export function send( + port: number, + method: string, + path: string, + options: { + readonly headers?: Readonly>; + readonly body?: string | Buffer; + readonly agent?: false; + } = {}, +): Promise { + return new Promise((resolve, reject) => { + const request = httpRequest( + { + host: "127.0.0.1", + port, + method, + path, + agent: false, + headers: { host: `127.0.0.1:${port}`, ...options.headers }, + }, + (response) => { + const chunks: Buffer[] = []; + response.on("data", (chunk: Buffer) => chunks.push(chunk)); + response.on("end", () => + resolve({ status: response.statusCode ?? 0, headers: response.headers, body: Buffer.concat(chunks) }), + ); + response.on("error", reject); + }, + ); + request.on("error", reject); + request.end(options.body); + }); +} + +/** A WebSocket client to `path` on `port` (`Host` loopback), with `headers`; resolves once open, rejects when refused. */ +export function socketTo( + port: number, + path: string, + headers: Readonly> = {}, + options: { readonly maxPayload?: number } = {}, +): Promise { + return new Promise((resolve, reject) => { + const socket = new WebSocket(`ws://127.0.0.1:${port}${path}`, { headers, ...options }); + // Frames sent at once by the server arrive with the upgrade: queued from the start. + inboxes.set(socket, messagesOf(socket)); + socket.once("open", () => resolve(socket)); + socket.once("unexpected-response", (_request, response) => { + reject(new Error(`refused ${response.statusCode}`)); + socket.terminate(); + }); + socket.once("error", reject); + }); +} + +/** The messages of every socket `socketTo` opened, queued from its start. */ +const inboxes = new WeakMap }>(); + +/** The next text message of a socket `socketTo` opened. */ +export function nextMessage(socket: WebSocket): Promise { + return inboxes.get(socket)!.next(); +} + +/** The text messages `socket` receives, from now on, in order: `next()` waits for the next one. */ +export function messagesOf(socket: WebSocket): { next(): Promise } { + const queued: string[] = []; + const waiting: ((text: string) => void)[] = []; + socket.on("message", (data) => { + const text = String(data); + const waiter = waiting.shift(); + if (waiter === undefined) queued.push(text); + else waiter(text); + }); + return { + next: () => { + const text = queued.shift(); + return text === undefined ? new Promise((resolve) => waiting.push(resolve)) : Promise.resolve(text); + }, + }; +} + +/** The close code and reason of `socket`'s close. */ +export function closeOf(socket: WebSocket): Promise<{ readonly code: number; readonly reason: string }> { + return new Promise((resolve) => { + if (socket.readyState === socket.CLOSED) resolve({ code: 1006, reason: "" }); + socket.once("close", (code, reason) => resolve({ code, reason: reason.toString() })); + }); +} + +export { context }; diff --git a/packages/chord/test/next-node/http/upgrades.test.ts b/packages/chord/test/next-node/http/upgrades.test.ts new file mode 100644 index 000000000..f4de24124 --- /dev/null +++ b/packages/chord/test/next-node/http/upgrades.test.ts @@ -0,0 +1,458 @@ +// Tests of upgraded connections (slice 10 §2, D6, Q6; decisions "allowed hosts only through contributions", "plugin +// sockets get a heartbeat"): the links endpoint (`host.accept`, options per socket, refused sockets, the 1 MiB frame +// default, the output cap), plugin sockets (handed to their handler, an oversized frame closing only that socket, a +// throwing handler), connections terminated at once when their socket handler or host leaves (a raw peer can act no +// more), an upgrade checked again after authenticating, the heartbeat, close bounded for a peer that never answers, +// and nothing kept of a closed socket or its `ws` server. + +import { once } from "node:events"; +import { connect, type Socket } from "node:net"; +import { describe, expect, it, vi } from "vitest"; +import { WebSocket, WebSocketServer } from "ws"; +import { arr, hostKind, local, method, remote, str } from "../../../src/next/contracts/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { webSocketLink } from "../../../src/next/link/index.ts"; +import { Http, type HttpContribution } from "../../../src/next-node/http/index.ts"; +import { limitedSocket } from "../../../src/next-node/http/upgrades.ts"; +import { expectCollected, settledHeap } from "../../next/support.ts"; +import { closeOf, context, contributing, nextMessage, send, served, socketTo, type TestPrincipal } from "./support.ts"; + +const Server = hostKind("server"); + +/** Who the server sees calling: served at the server's address. */ +const Who = remote("test.who", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { main: { methods: { who: method(undefined, str()) } } }, +}); + +const serveWho = defineFacet((host) => + host.serve(Who, () => ({ + variant: "main", + members: { who: async (ctx) => JSON.stringify(ctx.principal ?? null) }, + })), +); + +/** A plugin socket that echoes text, with its handler's request params and principal sent first. */ +const echo = contributing({ + sockets: { + "/echo/:room": (socket, request, ctx) => { + socket.send(`${request.params.room} as ${ctx.principal?.kind ?? "nobody"}`); + socket.on("message", (data) => socket.send(String(data))); + }, + }, + maxPayload: 64, +}); + +describe("the links endpoint", () => { + it("accepts a link as the authenticated principal, with the options links.options gave that socket", async () => { + const options: unknown[] = []; + const server = await served( + { "@app/who": serveWho }, + { + address: Server(), + config: { + links: { + path: "/chord", + options: (request, principal) => { + options.push([request.path, principal]); + return { maxPayload: 1024 * 1024 }; + }, + }, + }, + }, + ); + const tab = createHost({ onError: () => {} }); + await tab.load([], context); + // The link wraps the socket at once: the server's `hello` comes with the upgrade. + const socket = new WebSocket(`ws://127.0.0.1:${server.port}/chord`, { headers: { "x-principal": "owner" } }); + tab.connect(webSocketLink(socket as never), { principal: { kind: "server" } }, context); + const who = await tab.get(Who, {}, context); + expect(await who.who(context)).toBe('{"kind":"owner"}'); + expect(options).toEqual([["/chord", { kind: "owner" }]]); + await tab.close(context); + await server.close(); + }); + + it("refuses a socket without a principal, from another Origin or Host, foreign, or whose links.options throws (403)", async () => { + const server = await served( + { "@app/hosts": contributing({ hosts: ["a.example.com"] }) }, + { + config: { + links: { + path: "/chord", + options: (request) => { + if (request.headers.has("x-fail")) throw new Error("options"); + return {}; + }, + }, + }, + }, + ); + const { port } = server; + const owner = { "x-principal": "owner" }; + const refusals = [ + socketTo(port, "/chord"), + socketTo(port, "/chord", { ...owner, origin: "http://evil.com" }), + socketTo(port, "/chord", { ...owner, origin: "null" }), + socketTo(port, "/chord", { ...owner, origin: `HTTP://127.0.0.1:${port}` }), + socketTo(port, "/chord", { ...owner, host: "evil.com" }), + socketTo(port, "/chord", { ...owner, "sec-fetch-site": "cross-site" }), + socketTo(port, "/chord", { ...owner, "x-fail": "1" }), + ]; + for (const refusal of refusals) await expect(refusal).rejects.toThrow("refused 403"); + const same = await socketTo(port, "/chord", { ...owner, origin: `http://127.0.0.1:${port}` }); + same.close(); + expect(server.errors).toEqual([{ kind: "hook_failed", hook: "links.options", error: new Error("options") }]); + await server.close(); + }); + + it("takes frames of at most 1 MiB unless links.options says otherwise (Opus's audit H2)", async () => { + const server = await served({}, { config: { links: { path: "/chord" } } }); + const socket = await socketTo(server.port, "/chord", { "x-principal": "owner" }); + const closed = closeOf(socket); + socket.send(Buffer.alloc(1024 * 1024 + 1)); + expect((await closed).code).toBe(1009); + await server.close(); + }); + + it("cuts off a client that stopped reading: past 4 MiB unsent beyond the frame being sent", () => { + const sent: number[] = []; + let terminated = 0; + const fake = { + OPEN: 1, + readyState: 1, + bufferedAmount: 0, + binaryType: "nodebuffer", + send: (data: Uint8Array) => void sent.push(data.byteLength), + terminate: () => void terminated++, + }; + const limited = limitedSocket(fake as never); + limited.send(new Uint8Array(10)); + fake.bufferedAmount = 5 * 1024 * 1024; + limited.send(new Uint8Array(10)); + expect([sent, terminated]).toEqual([[10], 1]); + }); +}); + +describe("plugin sockets", () => { + it("hands the socket and its request to the handler in a scope with its principal; no principal is refused", async () => { + const server = await served({ "@app/echo": echo }); + await expect(socketTo(server.port, "/echo/lobby")).rejects.toThrow("refused 403"); + await expect(socketTo(server.port, "/nope", { "x-principal": "owner" })).rejects.toThrow("refused 404"); + const socket = await socketTo(server.port, "/echo/lobby", { "x-principal": "owner" }); + expect(await nextMessage(socket)).toBe("lobby as owner"); + socket.send("hi"); + expect(await nextMessage(socket)).toBe("hi"); + socket.close(); + await server.close(); + }); + + it("an oversized frame closes only that socket (1009), never the server", async () => { + const server = await served({ "@app/echo": echo }); + const [big, other] = await Promise.all([ + socketTo(server.port, "/echo/a", { "x-principal": "owner" }), + socketTo(server.port, "/echo/b", { "x-principal": "owner" }), + ]); + await Promise.all([nextMessage(big), nextMessage(other)]); + const closed = closeOf(big); + big.send("x".repeat(100)); + expect((await closed).code).toBe(1009); + other.send("still here"); + expect(await nextMessage(other)).toBe("still here"); + other.close(); + await server.close(); + }); + + it("a throwing handler is reported socket_failed and closes its socket (1011)", async () => { + const server = await served({ + "@app/bad": contributing({ + sockets: { + "/bad": () => { + throw new Error("handler"); + }, + }, + }), + }); + const socket = await socketTo(server.port, "/bad", { "x-principal": "owner" }); + expect((await closeOf(socket)).code).toBe(1011); + expect(server.errors).toMatchObject([{ kind: "socket_failed", pattern: "/bad", plugin: "@app/bad" }]); + await server.close(); + }); +}); + +describe("revocation", () => { + it("terminates a plugin's sockets at once when a reload removes its contribution, sealing their scopes; others stay (Q6)", async () => { + let signal: AbortSignal | undefined; + const removed = contributing({ + sockets: { + "/removed": (_socket, _request, ctx) => { + signal = ctx.abortSignal; + }, + }, + }); + const other = contributing({ sockets: { "/other": () => {} } }); + const server = await served({ "@app/removed": removed, "@app/other": other }); + const doomed = await socketTo(server.port, "/removed", { "x-principal": "owner" }); + const kept = await socketTo(server.port, "/other", { "x-principal": "owner" }); + const closed = closeOf(doomed); + await server.host.reload(server.facets({ "@app/other": other }), context); + expect(signal?.aborted).toBe(true); + expect((await closed).code).toBe(1006); + expect(kept.readyState).toBe(kept.OPEN); + kept.close(); + await server.close(); + }); + + it("terminates links and sockets through a host no longer contributed; loopback ones stay", async () => { + const server = await served( + { "@app/echo": echo, "@app/hosts": contributing({ hosts: ["a.example.com"] }) }, + { config: { links: { path: "/chord" } } }, + ); + const through = { "x-principal": "owner", host: "a.example.com" }; + const link = await socketTo(server.port, "/chord", through); + const socket = await socketTo(server.port, "/echo/a", through); + const loopback = await socketTo(server.port, "/echo/b", { "x-principal": "owner" }); + const closes = [closeOf(link), closeOf(socket)]; + await server.host.reload(server.facets({ "@app/echo": echo }), context); + expect((await Promise.all(closes)).map(({ code }) => code)).toEqual([1006, 1006]); + expect(loopback.readyState).toBe(loopback.OPEN); + expect((await send(server.port, "GET", "/", { headers: { host: "a.example.com" } })).status).toBe(403); + loopback.close(); + await server.close(); + }); + + it("keeps a writable contribution's own sockets when only its hosts change", async () => { + const Tunnel = local("app.tunnel", { version: 1, params: {}, members: { name: method(arr(str())) } }); + const handler = (): void => {}; + const tunnel = defineFacet((host) => + host.serve(Tunnel, (_, self) => { + const contribution = self.writable({ hosts: [], sockets: { "/own": handler } }); + self.contribute(Http, contribution); + return { + name: async (hosts) => { + contribution.set({ hosts, sockets: { "/own": handler } }); + return null; + }, + }; + }), + ); + const server = await served({ "@app/tunnel": tunnel }); + const handle = await server.host.get(Tunnel, {}, context); + const socket = await socketTo(server.port, "/own", { "x-principal": "owner" }); + await handle.name(["t.example.com"], context); + await new Promise((resolve) => setTimeout(resolve, 50)); + expect(socket.readyState).toBe(socket.OPEN); + socket.close(); + await server.close(); + }); + + // Astra's security audit A1: a policy close left the peer able to invoke the plugin's handlers. + it("a raw peer whose host or contribution left can send nothing more to the handler", async () => { + const calls: string[] = []; + const sockets = contributing({ + sockets: { "/echo": (socket) => void socket.on("message", (data) => calls.push(String(data))) }, + }); + const tunnel = contributing({ hosts: ["t.example"] }); + const server = await served({ "@app/sockets": sockets, "@app/tunnel": tunnel }); + const throughHost = await rawSocket(server.port, "t.example", "/echo"); + const local = await rawSocket(server.port, `localhost:${server.port}`, "/echo"); + await server.host.reload(server.facets({ "@app/sockets": sockets }), context); + throughHost.write(frame("after the host left")); + await new Promise((resolve) => setTimeout(resolve, 50)); + await server.host.reload(server.facets({}), context); + local.write(frame("after the contribution left")); + await new Promise((resolve) => setTimeout(resolve, 50)); + expect(calls).toEqual([]); + throughHost.destroy(); + local.destroy(); + await server.close(); + }); + + // Astra's security audit A2: an upgrade authenticating across the removal of its host was still handed out. + it("checks an upgrade again after authenticating: one whose host left meanwhile is refused", async () => { + const gate = Promise.withResolvers(); + const started = Promise.withResolvers(); + const sockets = contributing({ sockets: { "/echo": () => {} } }); + const server = await served( + { "@app/sockets": sockets, "@app/tunnel": contributing({ hosts: ["t.example"] }) }, + { + authenticate: async () => { + started.resolve(); + await gate.promise; + return { kind: "owner" }; + }, + }, + ); + const opening = socketTo(server.port, "/echo", { host: "t.example" }); + await started.promise; + await server.host.reload(server.facets({ "@app/sockets": sockets }), context); + gate.resolve(); + await expect(opening).rejects.toThrow("refused 403"); + await server.close(); + }); +}); + +describe("cancellation", () => { + // Astra's recheck R1, Sol's N5: a client gone while its upgrade authenticated left that work running. + // Fable's second recheck: only a reset (RST) was seen; a clean disconnect (FIN, a tab closing) was not. + it.each(["ends its connection", "resets its connection"] as const)( + "aborts an upgrade's authentication when its client %s", + async (how) => { + const entered = Promise.withResolvers(); + let signal: AbortSignal | undefined; + const server = await served( + { "@app/s": contributing({ sockets: { "/s": () => {} } }) }, + { + authenticate: async (_request, ctx) => { + signal = ctx.abortSignal; + entered.resolve(); + return new Promise(() => {}); + }, + }, + ); + const socket = connect(server.port, "127.0.0.1"); + socket.on("error", () => {}); + await once(socket, "connect"); + socket.write( + `GET /s HTTP/1.1\r\nHost: 127.0.0.1:${server.port}\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Version: 13\r\nSec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\n\r\n`, + ); + await entered.promise; + const aborted = new Promise((resolve) => signal?.addEventListener("abort", resolve, { once: true })); + if (how === "ends its connection") socket.end(); + else socket.resetAndDestroy(); + await aborted; + expect(signal?.aborted).toBe(true); + socket.destroy(); + await server.close(); + }, + ); +}); + +describe("heartbeat", () => { + it("pings a plugin socket after 15 s of silence and terminates it 15 s later without an answer", async () => { + const server = await served({ + "@app/quiet": contributing({ + sockets: { "/quiet": (socket) => void socket.on("message", (data) => socket.send(data)) }, + }), + }); + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] }); + try { + const silent = new WebSocket(`ws://127.0.0.1:${server.port}/quiet`, { + headers: { "x-principal": "owner" }, + autoPong: false, + }); + const answering = new WebSocket(`ws://127.0.0.1:${server.port}/quiet`, { + headers: { "x-principal": "owner" }, + }); + await Promise.all([once(silent, "open"), once(answering, "open")]); + const pinged = Promise.all([once(silent, "ping"), once(answering, "ping")]); + await vi.advanceTimersByTimeAsync(15_000); + await pinged; + // The answering client's pong travels for real (only timers are fake): an echo after it shows it arrived. + const echoed = once(answering, "message"); + answering.send("after the pong"); + await echoed; + const closed = closeOf(silent); + await vi.advanceTimersByTimeAsync(15_000); + expect((await closed).code).toBe(1006); + expect(answering.readyState).toBe(answering.OPEN); + answering.close(); + } finally { + vi.useRealTimers(); + } + await server.close(); + }); +}); + +describe("close", () => { + // Sol's build review: close waited on a peer that never acknowledged its close frame. + it("terminates a socket that never acknowledges its close at closeTimeout", async () => { + const server = await served( + { "@app/s": contributing({ sockets: { "/s": () => {} } }) }, + { config: { closeTimeout: 20 } }, + ); + const raw = await rawSocket(server.port, `127.0.0.1:${server.port}`, "/s"); + raw.pause(); + const started = Date.now(); + await server.http.close(context); + expect(Date.now() - started).toBeLessThan(500); + raw.destroy(); + await server.close(); + }); +}); + +describe("GC", () => { + // Sol's build review: a `ws` server was kept per frame size for the component's life. + it("keeps no ws server once its socket closed", async () => { + const refs: WeakRef[] = []; + const original = WebSocketServer.prototype.handleUpgrade; + WebSocketServer.prototype.handleUpgrade = function (this: WebSocketServer, ...args: Parameters) { + refs.push(new WeakRef(this)); + return original.apply(this, args); + }; + let maxPayload = 1024; + const server = await served( + {}, + { config: { links: { path: "/chord", options: () => ({ maxPayload: maxPayload++ }) } } }, + ); + try { + for (let i = 0; i < 8; i++) { + const socket = await socketTo(server.port, "/chord", { "x-principal": "owner" }); + const closed = closeOf(socket); + socket.close(); + await closed; + } + await settledHeap(); + await expectCollected(...refs); + } finally { + WebSocketServer.prototype.handleUpgrade = original; + await server.close(); + } + }); + + it("keeps nothing of a closed plugin socket", async () => { + const refs: WeakRef[] = []; + const server = await served({ + "@app/kept": contributing({ + sockets: { + "/kept": (socket, request, ctx) => { + refs.push(new WeakRef(socket), new WeakRef(request), new WeakRef(ctx)); + }, + }, + }), + }); + const socket = await socketTo(server.port, "/kept", { "x-principal": "owner" }); + const closed = closeOf(socket); + socket.close(); + await closed; + await new Promise((resolve) => setTimeout(resolve, 50)); + await expectCollected(...refs); + await server.close(); + }); +}); + +/** A raw TCP client upgraded to the plugin socket `path` through `host` as the owner. */ +async function rawSocket(port: number, host: string, path: string): Promise { + const socket = connect(port, "127.0.0.1"); + await once(socket, "connect"); + socket.on("error", () => {}); + socket.write( + `GET ${path} HTTP/1.1\r\nHost: ${host}\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Key: AAAAAAAAAAAAAAAAAAAAAA==\r\nSec-WebSocket-Version: 13\r\nx-principal: owner\r\n\r\n`, + ); + const [answer] = await once(socket, "data"); + expect(String(answer)).toContain("101 Switching Protocols"); + return socket; +} + +/** A masked client text frame. */ +function frame(text: string): Buffer { + const bytes = Buffer.from(text); + const mask = Buffer.from([1, 2, 3, 4]); + return Buffer.concat([ + Buffer.from([0x81, 0x80 | bytes.length]), + mask, + Buffer.from(bytes.map((byte, index) => byte ^ mask[index % 4]!)), + ]); +} diff --git a/packages/chord/test/next-node/plugins/build.test.ts b/packages/chord/test/next-node/plugins/build.test.ts index 623e85873..8525cfadf 100644 --- a/packages/chord/test/next-node/plugins/build.test.ts +++ b/packages/chord/test/next-node/plugins/build.test.ts @@ -222,7 +222,7 @@ describe("building a facet: the import table", () => { }); await writeFiles(root, { "base.ts": "export const base = 1;" }); await writePlugin(root, "copier", "@test/copier", { - "worker.ts": 'import { y } from "../shared/shared.ts";\nexport default { setup: () => y };', + "worker.ts": 'import { y } from "@test/shared/shared";\nexport default { setup: () => y };', }); await writePlugin(root, "shared", "@test/shared", { "shared.ts": 'import y from "some-package";\nexport { y };', @@ -231,7 +231,7 @@ describe("building a facet: the import table", () => { const out = plugins.find(({ name }) => name === "@test/out")!; expect(errors(await buildFacet(out, "worker", { ...NODE, plugins }, context))).toEqual([ 'worker.ts:1:22: "../base.ts": outside the plugin (base code is reached through a host package, packages through dependencies)', - 'worker.ts:2:15: "left-out": neither a host package nor in the dependencies of plugin "@test/out"', + 'worker.ts:2:15: "left-out": neither a host package, in the dependencies of plugin "@test/out" nor a discovered plugin "left-out"', ]); const copier = plugins.find(({ name }) => name === "@test/copier")!; expect(errors(await buildFacet(copier, "worker", { ...NODE, plugins }, context))).toEqual([ @@ -239,6 +239,43 @@ describe("building a facet: the import table", () => { ]); }); + it("imports another plugin's file by name from any root; refuses relative paths into it, names a missing plugin", async () => { + const first = await tempRoot(); + const second = await tempRoot(); + await writePlugin(second, "elsewhere", "@test/contracts", { + "contract.ts": 'export const name = "contract";', + "other.ts": 'import { name } from "./contract.ts";\nexport const other = name;', + }); + await writePlugin(first, "user", "@test/user", { + "worker.ts": [ + 'import { name } from "@test/contracts/contract";', + 'import { other } from "@test/contracts/other";', + "export default { setup: () => [name, other] };", + ].join("\n"), + }); + await writePlugin(first, "relative", "@test/relative", { + "worker.ts": 'import { name } from "../user/worker.ts";\nexport default { setup: () => name };', + }); + await writePlugin(first, "missing", "@test/missing", { + "worker.ts": 'import { name } from "@test/absent/contract";\nexport default { setup: () => name };', + }); + const { plugins } = await discoverPlugins([first, second], { kinds: KINDS }, context); + const build = (name: string) => + buildFacet(plugins.find((each) => each.name === name)!, "worker", { ...NODE, plugins }, context); + const user = await build("@test/user"); + if (user.kind === "failed") throw new Error(user.errors.join("\n")); + expect(user.copied.map((file) => file.slice(second.length + 1)).sort()).toEqual([ + "elsewhere/contract.ts", + "elsewhere/other.ts", + ]); + expect(errors(await build("@test/relative"))).toEqual([ + 'worker.ts:1:22: "../user/worker.ts": a relative path into plugin "@test/user" (import its files by name: "@test/user/…")', + ]); + expect(errors(await build("@test/missing"))).toEqual([ + 'worker.ts:1:22: "@test/absent/contract": neither a host package, in the dependencies of plugin "@test/missing" nor a discovered plugin "@test/absent"', + ]); + }); + it("returns a syntax error and a missing facet as failed results, never a throw", async () => { const root = await tempRoot(); await writePlugin(root, "syntax", "@test/syntax", { "worker.ts": "export default {" }); diff --git a/packages/chord/test/next-node/plugins/plugins.test.ts b/packages/chord/test/next-node/plugins/plugins.test.ts index b9c435d10..b202b3e82 100644 --- a/packages/chord/test/next-node/plugins/plugins.test.ts +++ b/packages/chord/test/next-node/plugins/plugins.test.ts @@ -12,7 +12,12 @@ import { createHost, defineFacet, type Host, type ReloadReport } from "../../../ import type { ChordHost } from "../../../src/next/host/internal.ts"; import { memoryLinkPair } from "../../../src/next/link/index.ts"; import { constant } from "../../../src/next/state/index.ts"; -import { createPlugins, type Plugins, type RolloutStep } from "../../../src/next-node/plugins/index.ts"; +import { + createPlugins, + type Plugins, + type RolloutHost, + type RolloutStep, +} from "../../../src/next-node/plugins/index.ts"; import { changesOf, delivered, @@ -22,7 +27,7 @@ import { transported, valuesOf, } from "../../next/support.ts"; -import { HOST_PACKAGES, KINDS, tempRoot, writeFiles, writePackage, writePlugin } from "./support.ts"; +import { HOST_PACKAGES, kindsOf, tempRoot, writeFiles, writePackage, writePlugin } from "./support.ts"; /** The fixtures' contracts, as the `contracts` fixture declares them. */ const Worker = hostKind("test.worker"); @@ -46,15 +51,20 @@ async function itemPlugin(root: string, dir: string, label: string): Promise reported.push(e), }, context, @@ -81,7 +91,11 @@ describe("per plugin at startup", () => { const { plugins } = component(root); const host = createHost({ onError: () => {} }); const core = defineFacet((h) => h.serve(Item, () => ({ label: constant("core") }))); - const report = await plugins.load(host, { kind: "worker", core: [{ name: "@app/core", facet: core }] }, context); + const report = await plugins.load( + host, + { kind: "worker", core: [{ name: "@app/core", facet: core }], modules: HOST_PACKAGES }, + context, + ); expect(report.failed.map(({ plugin, error }) => [plugin, (error as Error).message])).toEqual([ [ "@test/syntax", @@ -105,7 +119,7 @@ describe("per plugin at reload", () => { for (const dir of ["a", "b", "c"]) await itemPlugin(root, dir, `${dir}1`); const { plugins } = component(root); const host = createHost({ onError: () => {} }); - await plugins.load(host, { kind: "worker", core: [] }, context); + await plugins.load(host, { kind: "worker", core: [], modules: HOST_PACKAGES }, context); const label = async (dir: string) => (await host.get(local(`test.${dir}`, { version: 1, params: {}, members: { label: str() } }), {}, context)) .label.value; @@ -180,7 +194,7 @@ describe("per plugin at reload", () => { await writePlugin(root, "consumer", "@test/consumer", { "worker.ts": consumer(1) }); const { plugins } = component(root); const host = createHost({ onError: () => {} }); - await plugins.load(host, { kind: "worker", core: [] }, context); + await plugins.load(host, { kind: "worker", core: [], modules: HOST_PACKAGES }, context); const C = local("test.consumer", { version: 1, params: {}, members: { label: str() } }); const shown = await host.get(C, {}, context); await writeFiles(join(root, "provider"), { "worker.ts": `${provider(2)}\nexport const broken = ;` }); @@ -201,14 +215,19 @@ describe("per plugin at reload", () => { }); describe("rollout", () => { - /** A step recording its name into `order`, answering `report`, or rejecting with `error`. */ - const step = (name: string, order: string[], outcome: { report?: ReloadReport; error?: Error }): RolloutStep => ({ + /** A step recording its name into `order`, answering `report` for one host (or `hosts`), or rejecting with `error`. */ + const step = ( + name: string, + order: string[], + outcome: { report?: ReloadReport; hosts?: readonly RolloutHost[]; error?: Error }, + kinds: readonly string[] = ["worker"], + ): RolloutStep => ({ name, - kinds: ["worker"], + kinds, reload: async () => { order.push(name); if (outcome.error !== undefined) throw outcome.error; - return outcome.report!; + return { hosts: outcome.hosts ?? [{ host: name, report: outcome.report! }] }; }, }); const report = (failed: readonly string[]): ReloadReport => ({ @@ -244,6 +263,56 @@ describe("rollout", () => { plugins.close(); }); + it("counts a host's error as the step's built plugins failing there; a host stopped cleanly dirties nothing (D12)", async () => { + const root = await tempRoot(); + await itemPlugin(root, "a", "a1"); + await itemPlugin(root, "b", "b1"); + const { plugins } = component(root); + await plugins.build([], context); + await writeFile(join(root, "a", "worker.ts"), "// a\n"); + await vi.waitFor(() => expect(plugins.state.value).toMatchObject({ dirty: ["@test/a"] }), { timeout: 5_000 }); + const stopped: RolloutHost[] = [{ host: "session|S1", stopped: true }]; + await plugins.rollout( + [step("workers", [], { hosts: stopped })], + await plugins.build(["@test/a"], context), + context, + ); + expect(plugins.state.value).toMatchObject({ dirty: [] }); + await writeFile(join(root, "a", "worker.ts"), "// a again\n"); + await vi.waitFor(() => expect(plugins.state.value).toMatchObject({ dirty: ["@test/a"] }), { timeout: 5_000 }); + const crashed: RolloutHost[] = [ + { host: "session|S1", report: report([]) }, + { host: "session|S2", error: new Error("crashed") }, + ]; + const result = await plugins.rollout( + [step("workers", [], { hosts: crashed })], + await plugins.build(["@test/a"], context), + context, + ); + expect(result.steps).toEqual([{ name: "workers", hosts: crashed }]); + expect(plugins.state.value).toMatchObject({ dirty: ["@test/a"] }); + plugins.close(); + }); + + it("keeps dirty a plugin whose built kind no step shipped", async () => { + const root = await tempRoot(); + await writePlugin(root, "a", "@test/a", { "worker.ts": "export default { setup: () => {} };", "server.ts": "" }); + const { plugins } = component(root); + await plugins.build([], context); + await writeFile(join(root, "a", "worker.ts"), "export default { setup: () => {} }; // edited\n"); + await vi.waitFor(() => expect(plugins.state.value).toMatchObject({ dirty: ["@test/a"] }), { timeout: 5_000 }); + const built = await plugins.build(["@test/a"], context); + await plugins.rollout([step("worker", [], { report: report([]) })], built, context); + expect(plugins.state.value).toMatchObject({ dirty: ["@test/a"] }); + await plugins.rollout( + [step("worker", [], { report: report([]) }), step("server", [], { report: report([]) }, ["server"])], + built, + context, + ); + expect(plugins.state.value).toMatchObject({ dirty: [] }); + plugins.close(); + }); + it("stops at a step that rejects, clearing no mark; a change during the build keeps its mark", async () => { const root = await tempRoot(); await itemPlugin(root, "a", "a1"); @@ -275,6 +344,38 @@ describe("rollout", () => { }); }); +describe("current sets (slice 10 D11)", () => { + it("scripts(kind): built on first use; a build changes no set; a rollout applies it right before a step of its kind", async () => { + const root = await tempRoot(); + await itemPlugin(root, "a", "a1"); + await writePlugin(root, "b", "@test/b", { "server.ts": "export default { setup: () => {} };" }); + const { plugins } = component(root); + const digests = async () => + (await plugins.scripts("worker", context)).map(({ plugin, built }) => [ + plugin.name, + built.kind === "script" ? built.digest.slice(0, 8) : built.kind, + ]); + const first = await digests(); + expect(first.map(([name]) => name)).toEqual(["@test/a"]); + await itemPlugin(root, "a", "a2"); + const built = await plugins.build(["@test/a"], context); + expect(await digests()).toEqual(first); + const rejecting: RolloutStep = { + name: "server", + kinds: ["server"], + reload: async () => { + throw new Error("closed"); + }, + }; + const shipped: RolloutStep = { name: "workers", kinds: ["worker"], reload: async () => ({ hosts: [] }) }; + await plugins.rollout([rejecting, shipped], built, context); + expect(await digests()).toEqual(first); + await plugins.rollout([shipped], built, context); + expect(await digests()).not.toEqual(first); + plugins.close(); + }); +}); + describe("installs through the component", () => { it("clears the plugin's install mark and dirties it", async () => { const root = await tempRoot(); @@ -285,8 +386,7 @@ describe("installs through the component", () => { const plugins = createPlugins( { roots: [root], - kinds: KINDS, - hostPackages: HOST_PACKAGES, + kinds: kindsOf(HOST_PACKAGES), install: { npmCommand: [process.execPath, join(tools, "npm.mjs")] }, onError: (error) => reported.push(error), }, @@ -321,12 +421,10 @@ describe("GC", () => { "export default defineFacet((host) => host.serve(Item, () => ({ label: constant(state.label) })));", ].join("\n"); await writePlugin(root, "gc", "@test/gc", { "worker.ts": source("a") }); - const plugins = createPlugins( - { roots: [root], kinds: KINDS, hostPackages: { ...HOST_PACKAGES, "@test/probe": probe }, onError: () => {} }, - context, - ); + const modules = { ...HOST_PACKAGES, "@test/probe": probe }; + const plugins = createPlugins({ roots: [root], kinds: kindsOf(modules), onError: () => {} }, context); const host = createHost({ onError: () => {} }); - await plugins.load(host, { kind: "worker", core: [] }, context); + await plugins.load(host, { kind: "worker", core: [], modules }, context); const first = await plugins.build(["@test/gc"], context); await writeFiles(join(root, "gc"), { "worker.ts": source("b") }); const second = await plugins.build(["@test/gc"], context); @@ -358,8 +456,8 @@ describe("end to end: a server and a worker", () => { { authorize: () => true, resolve: async () => ({ link: down, principal: { kind: "worker" } }) }, context, ); - await plugins.load(worker, { kind: "worker", core: [] }, context); - await plugins.load(server, { kind: "server", core: [] }, context); + await plugins.load(worker, { kind: "worker", core: [], modules: HOST_PACKAGES }, context); + await plugins.load(server, { kind: "server", core: [], modules: HOST_PACKAGES }, context); const notes = await server.get(Notes, {}, context); const index = await server.get(Index, {}, context); expect([notes.label.value, index.label.value]).toEqual(["v1", "v1"]); @@ -370,7 +468,9 @@ describe("end to end: a server and a worker", () => { await writeFile(join(root, "notes", "label.ts"), 'export const label = "v2";\n'); await vi.waitFor(() => expect(plugins.state.value).toMatchObject({ dirty: ["@test/notes"] }), { timeout: 5_000 }); const rolled = await plugins.rollout(steps, await plugins.build(plugins.state.value.dirty, context), context); - expect(rolled.steps.map(({ name, report }) => [name, report.plugins.changed, report.objects])).toEqual([ + expect( + rolled.steps.map(({ name, hosts }) => [name, reportOf(hosts).plugins.changed, reportOf(hosts).objects]), + ).toEqual([ ["server", ["@test/notes"], [{ object: "test.index", kind: "swapped" }]], ["worker", ["@test/notes"], [{ object: "test.notes", kind: "swapped" }]], ]); @@ -383,7 +483,7 @@ describe("end to end: a server and a worker", () => { await writeFile(join(root, "notes", "label.ts"), "export const label = ;\n"); await vi.waitFor(() => expect(plugins.state.value).toMatchObject({ dirty: ["@test/notes"] }), { timeout: 5_000 }); const broken = await plugins.rollout(steps, await plugins.build(plugins.state.value.dirty, context), context); - expect(broken.steps.map(({ name, report }) => [name, failedNames(report)])).toEqual([ + expect(broken.steps.map(({ name, hosts }) => [name, failedNames(reportOf(hosts))])).toEqual([ ["server", ["@test/notes"]], ["worker", ["@test/notes"]], ]); @@ -403,7 +503,7 @@ describe("fix round 1", () => { await itemPlugin(root, "b", "b1"); const { plugins } = component(root); const host = createHost({ onError: () => {} }); - await plugins.load(host, { kind: "worker", core: [] }, context); + await plugins.load(host, { kind: "worker", core: [], modules: HOST_PACKAGES }, context); await itemPlugin(root, "a", "a2"); const builtA = await plugins.build(["@test/a"], context); await itemPlugin(root, "b", "b2"); @@ -424,7 +524,7 @@ describe("fix round 1", () => { await itemPlugin(root, "b", "b1"); const { plugins } = component(root); const host = createHost({ onError: () => {} }); - await plugins.load(host, { kind: "worker", core: [] }, context); + await plugins.load(host, { kind: "worker", core: [], modules: HOST_PACKAGES }, context); await itemPlugin(root, "a", "a2"); const first = await plugins.build(["@test/a"], context); await rm(join(root, "b"), { recursive: true }); @@ -444,7 +544,7 @@ describe("fix round 1", () => { await itemPlugin(root, "b", "b1"); const { plugins } = component(root); const host = createHost({ onError: () => {} }); - await plugins.load(host, { kind: "worker", core: [] }, context); + await plugins.load(host, { kind: "worker", core: [], modules: HOST_PACKAGES }, context); const Item = local("test.a", { version: 1, params: {}, members: { label: str() } }); const item = await host.get(Item, {}, context); await itemPlugin(root, "a", "a2"); @@ -476,7 +576,7 @@ describe("fix round 1", () => { await itemPlugin(root, "a", "a1"); const { plugins } = component(root); const host = createHost({ onError: () => {} }); - await plugins.load(host, { kind: "worker", core: [] }, context); + await plugins.load(host, { kind: "worker", core: [], modules: HOST_PACKAGES }, context); // The plugin keeps a server facet only. await rm(join(root, "a", "worker.ts")); await writeFiles(join(root, "a"), { @@ -500,7 +600,7 @@ describe("fix round 1", () => { await itemPlugin(root, "a", "a1"); const { plugins } = component(root); const host = createHost({ onError: () => {} }); - await plugins.load(host, { kind: "worker", core: [] }, context); + await plugins.load(host, { kind: "worker", core: [], modules: HOST_PACKAGES }, context); const item = await host.get(local("test.a", { version: 1, params: {}, members: { label: str() } }), {}, context); await itemPlugin(root, "a", "a2"); await vi.waitFor(() => expect(plugins.state.value).toMatchObject({ dirty: ["@test/a"] }), { timeout: 5_000 }); @@ -526,8 +626,7 @@ describe("fix round 1", () => { const plugins = createPlugins( { roots: [root], - kinds: KINDS, - hostPackages: HOST_PACKAGES, + kinds: kindsOf(HOST_PACKAGES), install: { npmCommand: [process.execPath, join(tools, "npm.mjs")] }, onError: () => {}, }, @@ -567,12 +666,10 @@ describe("the reload queues and garbage collection", () => { "});", ].join("\n"); const dir = await writePlugin(root, "failed", "@test/failed", { "worker.ts": source(false) }); - const plugins = createPlugins( - { roots: [root], kinds: KINDS, hostPackages: { ...HOST_PACKAGES, "@test/probe": probe }, onError: () => {} }, - context, - ); + const modules = { ...HOST_PACKAGES, "@test/probe": probe }; + const plugins = createPlugins({ roots: [root], kinds: kindsOf(modules), onError: () => {} }, context); const host = createHost({ onError: () => {} }); - await plugins.load(host, { kind: "worker", core: [] }, context); + await plugins.load(host, { kind: "worker", core: [], modules }, context); await writeFiles(dir, { "worker.ts": source(true) }); const built = await plugins.build(["@test/failed"], context); // The report (its failure's error holds the new facet's closure) is dropped here; the queues must not keep it. diff --git a/packages/chord/test/next-node/plugins/runtime.test.ts b/packages/chord/test/next-node/plugins/runtime.test.ts new file mode 100644 index 000000000..52b850d10 --- /dev/null +++ b/packages/chord/test/next-node/plugins/runtime.test.ts @@ -0,0 +1,126 @@ +// Tests of the plugin runtime (slice 10 §3, D11): loading and reloading hosts with complete sets of shipped scripts +// (unchanged digests kept as the same facet, removed = loaded minus shipped, failed builds and evaluations kept old), +// and that importing it loads no esbuild. + +import { execFileSync } from "node:child_process"; +import { fileURLToPath } from "node:url"; +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { local, str } from "../../../src/next/contracts/index.ts"; +import { createHost } from "../../../src/next/host/index.ts"; +import { + buildFacet, + createPluginRuntime, + discoverPlugins, + type PluginScript, +} from "../../../src/next-node/plugins/index.ts"; +import { HOST_PACKAGES, tempRoot, writePlugin } from "./support.ts"; + +/** Writes plugin `@test/` serving `test.` labelled `label` from its worker facet (or `source` as it is). */ +async function plugin(root: string, dir: string, label: string, source?: string): Promise { + await writePlugin(root, dir, `@test/${dir}`, { + "worker.ts": + source ?? + [ + 'import { constant, defineFacet, local, str } from "@earendil-works/chord/next";', + `const Item = local("test.${dir}", { version: 1, params: {}, members: { label: str() } });`, + `export default defineFacet((host) => host.serve(Item, () => ({ label: constant(${JSON.stringify(label)}) })));`, + ].join("\n"), + }); +} + +/** The worker scripts of every plugin under `root`, built now: what the server would ship. */ +async function shipped(root: string): Promise { + const { plugins } = await discoverPlugins([root], { kinds: ["worker"] }, context); + const options = { hostPackages: Object.keys(HOST_PACKAGES), platform: "node", plugins } as const; + return Promise.all( + plugins.map(async (each) => ({ plugin: each, built: await buildFacet(each, "worker", options, context) })), + ); +} + +/** The label `test.` shows on `host`. */ +async function label(host: ReturnType, dir: string): Promise { + const Item = local(`test.${dir}`, { version: 1, params: {}, members: { label: str() } }); + return (await host.get(Item, {}, context)).label.value; +} + +describe("loading and reloading with complete sets", () => { + it("loads a set and leaves out a plugin whose build or evaluation failed, reported with the host's report", async () => { + const root = await tempRoot(); + await plugin(root, "a", "a1"); + await plugin(root, "broken", "", "export const = ;"); + await plugin(root, "throws", "", 'throw new Error("at import");'); + const runtime = createPluginRuntime(); + const host = createHost({ onError: () => {} }); + const report = await runtime.load( + host, + { kind: "worker", core: [], scripts: await shipped(root), modules: HOST_PACKAGES }, + context, + ); + expect(report.failed.map(({ plugin }) => plugin).sort()).toEqual(["@test/broken", "@test/throws"]); + expect(await label(host, "a")).toBe("a1"); + await host.close(context); + }); + + it("reloads a complete set: same digests unchanged, a changed one applied, a loaded one not shipped removed", async () => { + const root = await tempRoot(); + await plugin(root, "a", "a1"); + await plugin(root, "b", "b1"); + await plugin(root, "c", "c1"); + const runtime = createPluginRuntime(); + const host = createHost({ onError: () => {} }); + await runtime.load( + host, + { kind: "worker", core: [], scripts: await shipped(root), modules: HOST_PACKAGES }, + context, + ); + await plugin(root, "b", "b2"); + const next = (await shipped(root)).filter(({ plugin }) => plugin.name !== "@test/c"); + const report = await runtime.reload(host, next, context); + expect(report.plugins).toEqual({ added: [], changed: ["@test/b"], removed: ["@test/c"] }); + expect([await label(host, "a"), await label(host, "b")]).toEqual(["a1", "b2"]); + await host.close(context); + }); + + it("keeps the loaded facet of a plugin whose new build or evaluation failed; an added failing one is left out", async () => { + const root = await tempRoot(); + await plugin(root, "a", "a1"); + const runtime = createPluginRuntime(); + const host = createHost({ onError: () => {} }); + await runtime.load( + host, + { kind: "worker", core: [], scripts: await shipped(root), modules: HOST_PACKAGES }, + context, + ); + await plugin(root, "a", "", "export const = ;"); + await plugin(root, "new", "", 'throw new Error("at import");'); + const report = await runtime.reload(host, await shipped(root), context); + expect(report.failed.map(({ plugin }) => plugin).sort()).toEqual(["@test/a", "@test/new"]); + expect(report.plugins).toEqual({ added: [], changed: [], removed: [] }); + expect(await label(host, "a")).toBe("a1"); + await host.close(context); + }); +}); + +describe("the runtime's imports", () => { + it("loads no esbuild: a worker that only evaluates never imports it", () => { + const runtime = fileURLToPath(new URL("../../../src/next-node/plugins/runtime.ts", import.meta.url)); + // A hook refusing esbuild, then the runtime imported. + const script = [ + 'import { registerHooks } from "node:module";', + "registerHooks({ resolve(specifier, context, next) {", + ' if (specifier === "esbuild") throw new Error("esbuild imported");', + " return next(specifier, context);", + "} });", + `const { createPluginRuntime } = await import(${JSON.stringify(runtime)});`, + "createPluginRuntime();", + 'console.log("ok");', + ].join("\n"); + const output = execFileSync( + process.execPath, + ["--experimental-strip-types", "--no-warnings", "--input-type=module", "--eval", script], + { encoding: "utf8" }, + ); + expect(output.trim()).toBe("ok"); + }); +}); diff --git a/packages/chord/test/next-node/plugins/support.ts b/packages/chord/test/next-node/plugins/support.ts index 86e444156..f4c1de6fc 100644 --- a/packages/chord/test/next-node/plugins/support.ts +++ b/packages/chord/test/next-node/plugins/support.ts @@ -7,7 +7,7 @@ import { dirname, join } from "node:path"; import { fileURLToPath } from "node:url"; import { afterEach } from "vitest"; import * as chordNext from "../../../src/next/index.ts"; -import type { HostPackages } from "../../../src/next-node/plugins/index.ts"; +import type { HostPackages, KindDefinition } from "../../../src/next-node/plugins/index.ts"; /** The fixture plugins' directory. */ export const FIXTURES = fileURLToPath(new URL("../fixtures/plugins/", import.meta.url)); @@ -18,6 +18,16 @@ export const HOST_PACKAGES: HostPackages = { "@earendil-works/chord/next": chord /** The facet kinds of the fixtures' app. */ export const KINDS = ["server", "worker", "web"] as const; +/** The fixtures' app's kinds as the component takes them: Node servers and workers, a browser web, `modules`' names. */ +export function kindsOf(modules: HostPackages): Readonly> { + const hostPackages = Object.keys(modules); + return { + server: { platform: "node", hostPackages }, + worker: { platform: "node", hostPackages }, + web: { platform: "browser", hostPackages }, + }; +} + const made: string[] = []; afterEach(async () => { diff --git a/packages/chord/test/next-node/workers/control.test.ts b/packages/chord/test/next-node/workers/control.test.ts new file mode 100644 index 000000000..a7054269a --- /dev/null +++ b/packages/chord/test/next-node/workers/control.test.ts @@ -0,0 +1,55 @@ +// Tests of the control channel's crossings (slice 10 §3): errors cross as `{ message, code?, reason? }` and back, a +// reload report with its errors and parked causes. + +import { describe, expect, it } from "vitest"; +import { ChordError, isChordError } from "../../../src/next/data/index.ts"; +import { + controlOf, + crossed, + crossedReport, + envelope, + uncrossed, + uncrossedReport, +} from "../../../src/next-node/workers/control.ts"; + +describe("errors", () => { + it("cross as message, code and reason, never a stack or cause; come back as ChordErrors when they had a code", () => { + const chord = new ChordError({ code: "unavailable", message: "gone" }, { cause: new Error("inner") }); + expect(crossed(chord)).toEqual({ message: "gone", code: "unavailable" }); + expect(crossed(new Error("plain"))).toEqual({ message: "plain" }); + expect(crossed("text")).toEqual({ message: "text" }); + const back = uncrossed({ message: "gone", code: "released", reason: "closed" }); + expect(isChordError(back, "released") && back.reason).toBe("closed"); + expect(uncrossed({ message: "odd", code: "no-such-code" })).toMatchObject({ message: "odd" }); + }); + + it("crosses a report's failures and parked causes and back", () => { + const report = crossedReport({ + plugins: { added: ["@a/x"], changed: [], removed: [] }, + failed: [{ plugin: "@a/y", error: new Error("broken") }], + objects: [ + { object: "a.one", kind: "swapped" }, + { + object: "a.two", + kind: "parked", + cause: { kind: "waiting", error: new ChordError({ code: "unavailable", message: "w" }) }, + }, + ], + }); + expect(JSON.parse(JSON.stringify(report))).toEqual(report); + const back = uncrossedReport(report); + expect(back.failed[0]!.error).toBeInstanceOf(Error); + expect(back.objects[1]).toMatchObject({ + kind: "parked", + cause: { kind: "waiting", error: { code: "unavailable" } }, + }); + }); +}); + +describe("envelopes", () => { + it("tells control messages from anything else on the channel", () => { + expect(controlOf(envelope({ kind: "started" }))).toEqual({ kind: "started" }); + expect(controlOf(new Uint8Array(1))).toBeUndefined(); + expect(controlOf({ kind: "started" })).toBeUndefined(); + }); +}); diff --git a/packages/chord/test/next-node/workers/lock.test.ts b/packages/chord/test/next-node/workers/lock.test.ts new file mode 100644 index 000000000..de1564720 --- /dev/null +++ b/packages/chord/test/next-node/workers/lock.test.ts @@ -0,0 +1,62 @@ +// Tests of process locks (slice 10 §3): one holder at a time, a waiter taking the lock when the holder releases it +// (told by its connection's close, no polling), a waiter's abort. + +import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context, withAbortSignal } from "../../../src/context/index.ts"; +import { processLock } from "../../../src/next-node/workers/index.ts"; +import { lockAt } from "../../../src/next-node/workers/lock.ts"; + +describe("processLock", () => { + it("makes a second taker wait until the holder releases, then takes it", async () => { + const name = `chord-test-lock-${process.pid}-${Date.now()}`; + const first = await processLock(name, context); + let taken = false; + const second = processLock(name, context).then((lock) => { + taken = true; + return lock; + }); + await new Promise((resolve) => setTimeout(resolve, 100)); + expect(taken).toBe(false); + const released = Date.now(); + await first.release(); + await (await second).release(); + expect(taken).toBe(true); + expect(Date.now() - released).toBeLessThan(100); + }); + + it("stops waiting when the waiter's context aborts", async () => { + const name = `chord-test-lock-abort-${process.pid}-${Date.now()}`; + const held = await processLock(name, context); + const abort = new AbortController(); + const waiting = processLock(name, withAbortSignal(abort.signal, context)); + abort.abort(new Error("gave up")); + await expect(waiting).rejects.toThrow("gave up"); + await held.release(); + await (await processLock(name, context)).release(); + }); +}); + +describe("socket file locks (macOS, the BSDs)", () => { + it("fail closed on a stale lock file, naming it: two takers never both take it over (Sol)", async () => { + const dir = await mkdtemp(join(tmpdir(), "chord-lock-file-")); + const address = join(dir, "lock.sock"); + // What a crashed holder leaves: a path nobody accepts on. + await writeFile(address, ""); + const takers = [lockAt({ kind: "file", address }, context), lockAt({ kind: "file", address }, context)]; + for (const taker of takers) await expect(taker).rejects.toThrow(`lock: ${address} is a stale lock file`); + await rm(dir, { recursive: true, force: true }); + }); + + it("are taken, waited for and released like the others", async () => { + const dir = await mkdtemp(join(tmpdir(), "chord-lock-file-")); + const address = join(dir, "lock.sock"); + const first = await lockAt({ kind: "file", address }, context); + const second = lockAt({ kind: "file", address }, context); + await first.release(); + await (await second).release(); + await rm(dir, { recursive: true, force: true }); + }); +}); diff --git a/packages/chord/test/next-node/workers/main.test.ts b/packages/chord/test/next-node/workers/main.test.ts new file mode 100644 index 000000000..017c9df04 --- /dev/null +++ b/packages/chord/test/next-node/workers/main.test.ts @@ -0,0 +1,87 @@ +// Tests of the worker's side (slice 10 §3): the start message, the lock, `started` on `connect`, `ready`; shutdown +// during startup, the parent's disconnect and SIGTERM each run the closers and exit; `fatal` redacted. + +import { mkdtemp, readFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { describe, expect, it } from "vitest"; +import { hostKind, str } from "../../../src/next/contracts/index.ts"; +import type { FromWorker } from "../../../src/next-node/workers/control.ts"; +import { type Exit, WorkerProcess } from "../../../src/next-node/workers/process.ts"; +import type { WorkerBehaviour } from "../fixtures/workers/control.ts"; +import { until } from "./support.ts"; + +const ENTRY = fileURLToPath(new URL("../fixtures/workers/worker.ts", import.meta.url)); + +const Session = hostKind("session", { id: str() }); + +/** The fixture worker forked and started with `behaviour`; its control messages and exit collected. */ +function started(behaviour: WorkerBehaviour) { + const control: FromWorker[] = []; + let exit: Exit | undefined; + const child = new WorkerProcess( + { + entry: ENTRY, + execArgv: ["--conditions=source", "--experimental-strip-types", "--no-warnings"], + onOutput: () => {}, + }, + { + control: (message) => control.push(message), + linkLost: () => {}, + linkClosed: () => {}, + exited: (value) => { + exit = value; + }, + }, + ); + child.send({ + kind: "start", + address: Session({ id: "S1" }), + start: behaviour, + scripts: [], + lock: `chord-test-main:${process.pid}:${Date.now()}:${Math.random()}`, + }); + return { child, control, exited: () => exit }; +} + +describe("the worker's side", () => { + it("sends started when the app connects, then ready with its load report and idle state", async () => { + const { child, control } = started({}); + await until(() => control.some(({ kind }) => kind === "ready")); + expect(control.map(({ kind }) => kind)).toEqual(["started", "ready"]); + expect(control[1]).toMatchObject({ idle: true, report: { plugins: { added: [] }, failed: [] } }); + child.kill(); + }); + + it("shuts down during its startup: aborts it, runs the closers registered so far in reverse order, exits 0", async () => { + const dir = await mkdtemp(join(tmpdir(), "chord-main-")); + const { child, control, exited } = started({ hang: true, log: join(dir, "log") }); + await until(() => control.some(({ kind }) => kind === "started")); + child.send({ kind: "shutdown" }); + await until(() => exited() !== undefined); + expect(exited()).toEqual({ code: 0, signal: null }); + expect(await readFile(join(dir, "log"), "utf8")).toBe("store\nhost\n"); + await rm(dir, { recursive: true, force: true }); + }); + + it("shuts down on SIGTERM and on the parent's disconnect", async () => { + const term = started({}); + await until(() => term.control.some(({ kind }) => kind === "ready")); + process.kill(term.child.pid, "SIGTERM"); + await until(() => term.exited() !== undefined); + expect(term.exited()).toEqual({ code: 0, signal: null }); + const gone = started({}); + await until(() => gone.control.some(({ kind }) => kind === "ready")); + gone.child.link.close(); + await until(() => gone.exited() !== undefined); + expect(gone.exited()).toEqual({ code: 0, signal: null }); + }); + + it("sends fatal with the redacted error and exits 1", async () => { + const { control, exited } = started({ fatal: "no secret-key-42 here" }); + await until(() => exited() !== undefined); + expect(control).toEqual([{ kind: "fatal", error: { message: "no [redacted] here" } }]); + expect(exited()?.code).toBe(1); + }); +}); diff --git a/packages/chord/test/next-node/workers/process.test.ts b/packages/chord/test/next-node/workers/process.test.ts new file mode 100644 index 000000000..b84589f6e --- /dev/null +++ b/packages/chord/test/next-node/workers/process.test.ts @@ -0,0 +1,46 @@ +// Tests of worker processes (slice 10 §3): forked with their output piped (the stderr tail kept), the link made at the +// fork (an early frame queued), control messages told, and the whole tree killed. + +import { fileURLToPath } from "node:url"; +import { describe, expect, it } from "vitest"; +import type { FromWorker } from "../../../src/next-node/workers/control.ts"; +import { WorkerProcess } from "../../../src/next-node/workers/process.ts"; +import { until } from "./support.ts"; + +const CHILD = fileURLToPath(new URL("../fixtures/workers/child.ts", import.meta.url)); + +/** Whether process `pid` lives. */ +function alive(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch { + return false; + } +} + +describe("a worker process", () => { + it("pipes its output, keeps its stderr tail, queues an early frame on its link, tells control messages", async () => { + const output: string[] = []; + const control: FromWorker[] = []; + const child = new WorkerProcess( + { + entry: CHILD, + execArgv: ["--conditions=source", "--experimental-strip-types", "--no-warnings"], + onOutput: (stream, chunk) => output.push(`${stream}: ${Buffer.from(chunk).toString().trim()}`), + }, + { control: (message) => control.push(message), linkLost: () => {}, linkClosed: () => {}, exited: () => {} }, + ); + await until(() => control.length > 0 && output.length >= 2); + const frames: number[][] = []; + child.link.onMessage((message) => frames.push([...message])); + expect(frames).toEqual([[1, 2, 3]]); + expect(control).toEqual([{ kind: "started" }]); + expect(child.stderrTail).toBe("an error line\n"); + const grandchild = Number(/grandchild (\d+)/.exec(output.join("\n"))![1]); + expect(alive(grandchild)).toBe(true); + child.kill(); + expect(await child.exited).toEqual({ code: null, signal: "SIGKILL" }); + await until(() => !alive(grandchild), 2_000); + }); +}); diff --git a/packages/chord/test/next-node/workers/support.ts b/packages/chord/test/next-node/workers/support.ts new file mode 100644 index 000000000..4931905d5 --- /dev/null +++ b/packages/chord/test/next-node/workers/support.ts @@ -0,0 +1,151 @@ +// Test support for the workers component (slice 10 §3, §4): a server routing `Session` to forked fixture workers, the +// fixture plugins copied into a temporary root, each address's start data (its behaviour) chosen by the test. + +import { cp, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { afterEach } from "vitest"; +import { type Context, BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import type { JsonValue } from "../../../src/json.ts"; +import { hostKind, remote, str } from "../../../src/next/contracts/index.ts"; +import { createHost, type Host, type HostError } from "../../../src/next/host/index.ts"; +import * as chordNext from "../../../src/next/index.ts"; +import { createPlugins, type Plugins } from "../../../src/next-node/plugins/index.ts"; +import { createWorkers, type Workers, type WorkersError } from "../../../src/next-node/workers/index.ts"; +import { Session, type WorkerBehaviour } from "../fixtures/workers/control.ts"; + +/** The server's own kind. */ +export const Server = hostKind("server"); + +/** The fixture plugin's notes at a worker. */ +export const Notes = remote("test.notes", { + version: 1, + params: { id: str() }, + hostAddress: ({ id }) => Session({ id }), + variants: { main: { state: { label: str() } } }, +}); + +/** The principals of these tests. */ +export type TestPrincipal = + | { readonly kind: "server" } + | { readonly kind: "worker"; readonly id: string } + | { readonly kind: "guest"; readonly id: string }; + +const ENTRY = fileURLToPath(new URL("../fixtures/workers/worker.ts", import.meta.url)); + +const PLUGINS = fileURLToPath(new URL("../fixtures/workers/plugins/", import.meta.url)); + +const made: { readonly root: string; readonly close: () => Promise }[] = []; + +afterEach(async () => { + for (const { root, close } of made.splice(0)) { + await close(); + await rm(root, { recursive: true, force: true }); + } +}); + +/** A server with workers; closed after the test. */ +export type Fleet = { + readonly server: Host; + readonly workers: Workers; + readonly plugins: Plugins; + /** The copy of the fixture plugins. */ + readonly root: string; + readonly errors: WorkersError[]; + readonly hostErrors: HostError[]; + /** The addresses `start` ran for, in order. */ + readonly starts: string[]; + /** The workers' stdout lines, each with its address key. */ + readonly output: string[]; + /** Each address's behaviour (its start data); none: a plain worker. */ + readonly behaviours: Map; + /** Writes `label` into the copy's notes plugin. */ + label(label: string): Promise; +}; + +/** Starts a server routing `Session` to fixture workers; `options` override the workers' defaults. */ +export async function fleet( + options: { + readonly idleMs?: number; + readonly heartbeat?: { readonly ping?: number; readonly close?: number }; + readonly start?: (address: { readonly key: string }, context: Context) => Promise; + readonly principal?: () => TestPrincipal; + readonly authorize?: () => boolean; + } = {}, +): Promise { + const root = await mkdtemp(join(tmpdir(), "chord-workers-")); + await cp(PLUGINS, root, { recursive: true }); + const errors: WorkersError[] = []; + const hostErrors: HostError[] = []; + const starts: string[] = []; + const behaviours = new Map(); + const output: string[] = []; + const modules = { "@earendil-works/chord/next": chordNext }; + const plugins = createPlugins( + { roots: [root], kinds: { worker: { platform: "node", hostPackages: Object.keys(modules) } }, onError: () => {} }, + context, + ); + const server = createHost({ + address: Server(), + principal: { kind: "server" }, + onError: (error) => hostErrors.push(error), + ...(options.heartbeat === undefined ? {} : { heartbeat: options.heartbeat }), + }); + const workers = createWorkers( + server, + Session, + { + kind: "worker", + entry: ENTRY, + execArgv: ["--conditions=source", "--experimental-strip-types", "--no-warnings"], + authorize: options.authorize ?? (() => true), + principal: options.principal ?? ((address) => ({ kind: "worker", id: address.params.id }) as const), + start: async (address, startContext) => { + starts.push(address.key); + if (options.start !== undefined) return options.start(address, startContext); + return (behaviours.get(address.params.id) ?? {}) as JsonValue; + }, + lock: (address) => `chord-test:${root}:${address.key}`, + idleMs: () => options.idleMs ?? 60_000, + plugins, + onOutput: (address, stream, chunk) => { + if (stream === "stdout") output.push(`${address.key} ${Buffer.from(chunk).toString().trim()}`); + }, + onError: (error) => errors.push(error), + }, + context, + ); + await server.load([], context); + // The worker kind's set made now (discovery, builds): the watcher marks edits from then on. + await plugins.scripts("worker", context); + const close = async () => { + await workers.close(context); + await server.close(context); + plugins.close(); + }; + made.push({ root, close }); + return { + server, + workers, + plugins, + root, + errors, + hostErrors, + starts, + output, + behaviours, + label: (label) => writeFile(join(root, "notes", "label.ts"), `export const label = ${JSON.stringify(label)};\n`), + }; +} + +/** Waits until `ready()` holds, polling every 10 ms, for at most `ms`. */ +export async function until(ready: () => boolean, ms = 10_000): Promise { + const deadline = Date.now() + ms; + while (!ready()) { + if (Date.now() > deadline) throw new Error("until: timed out"); + await new Promise((resolve) => setTimeout(resolve, 10)); + } +} + +export { context, Session }; diff --git a/packages/chord/test/next-node/workers/workers.test.ts b/packages/chord/test/next-node/workers/workers.test.ts new file mode 100644 index 000000000..d7c980784 --- /dev/null +++ b/packages/chord/test/next-node/workers/workers.test.ts @@ -0,0 +1,605 @@ +// Tests of the workers component (slice 10 §3, §4): starts on demand (the link made at the fork, `started` on +// `connect`, the start timer covering only the start), failures that stay (a crash, a `fatal`, a lost link, a failed +// start: no fork for a new hold, holders `failed { waiting }`) until `retry` or a rollout, idle stops and `busy`, `stop` +// and its suspension, the per-address queue, `close`, the rollout step, and nothing kept of 1,000 addresses. + +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it, vi } from "vitest"; +import { withAbortSignal } from "../../../src/context/index.ts"; +import { createHost } from "../../../src/next/host/index.ts"; +import { memoryLinkPair } from "../../../src/next/link/index.ts"; +import type { RolloutHost } from "../../../src/next-node/plugins/index.ts"; +import { expectCollected } from "../../next/support.ts"; +import { Control } from "../fixtures/workers/control.ts"; +import { context, fleet, Notes, Session, type TestPrincipal, until } from "./support.ts"; + +/** A step's hosts as `[host, what]`. */ +function listed(hosts: readonly RolloutHost[]) { + return hosts.map((entry) => [ + entry.host, + "report" in entry ? entry.report.plugins : "stopped" in entry ? "stopped" : (entry.error as Error).message, + ]); +} + +/** The state entry of `id`. */ +const stateOf = (state: unknown, id: string) => + (state as Record)[`session|${id}`]; + +describe("starts on demand", () => { + it("starts a worker for the first hold: starting, then running; its link binds the holder", async () => { + const { server, workers, starts } = await fleet(); + const notes = server.hold(Notes, { id: "S1" }, context); + await until(() => stateOf(workers.state.value, "S1") !== undefined); + expect(stateOf(workers.state.value, "S1")?.status).toBe("starting"); + await until(() => notes.state.value.kind === "live"); + expect(notes.label.value).toBe("S1 v1"); + await until(() => stateOf(workers.state.value, "S1")?.status === "running"); + expect(starts).toEqual(["session|S1"]); + notes.release(); + }); + + it("binds through a load longer than the heartbeat: the link is made at the fork, the worker connects first", async () => { + const { server, behaviours } = await fleet({ heartbeat: { ping: 50, close: 200 } }); + behaviours.set("S1", { loadMs: 600 }); + const notes = await server.get(Notes, { id: "S1" }, context); + expect(notes.label.value).toBe("S1 v1"); + notes.release(); + }); +}); + +describe("failures stay until retry or a rollout (R2)", () => { + it("a start that throws fails the address: holders fail waiting with it, a new hold forks nothing; retry starts it", async () => { + let fail = true; + const { server, workers, errors, starts } = await fleet({ + start: async () => { + if (fail) throw new Error("no data dir"); + return {}; + }, + }); + const notes = server.hold(Notes, { id: "S1" }, context); + await until(() => notes.state.value.kind === "failed"); + expect(notes.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "waiting", error: { code: "unavailable", message: "worker failed to start" } }, + }); + expect(stateOf(workers.state.value, "S1")).toMatchObject({ + status: "failed", + error: { message: "worker failed to start" }, + }); + const again = server.hold(Notes, { id: "S1" }, context); + await until(() => again.state.value.kind === "failed"); + expect(starts).toEqual(["session|S1"]); + // The full error only for the app's logs (decision "Slice 10 fix round, worker errors in public"). + expect(errors).toMatchObject([ + { kind: "worker_failed", address: "session|S1", error: { message: "no data dir" } }, + ]); + fail = false; + await workers.retry(Session({ id: "S1" }), context); + await until(() => notes.state.value.kind === "live" && again.state.value.kind === "live"); + expect(stateOf(workers.state.value, "S1")?.status).not.toBe("failed"); + notes.release(); + again.release(); + }); + + it("a crash after ready stays failed (worker_exited, worker_failed), nothing restarts", async () => { + const { server, workers, errors, starts } = await fleet(); + const notes = await server.get(Notes, { id: "S1" }, context); + const control = await server.get(Control, { id: "S1" }, context); + await control.crash(7, context); + await until(() => stateOf(workers.state.value, "S1")?.status === "failed"); + expect(stateOf(workers.state.value, "S1")?.error?.message).toBe("worker exited (code 7)"); + expect(errors.map(({ kind }) => kind)).toEqual(["worker_exited", "worker_failed"]); + await until(() => notes.state.value.kind === "failed"); + await new Promise((resolve) => setTimeout(resolve, 200)); + expect(starts).toEqual(["session|S1"]); + notes.release(); + control.release(); + }); + + it("a fatal crosses redacted: the secret never reaches the public state", async () => { + const { server, workers, errors, behaviours } = await fleet(); + behaviours.set("S1", { fatal: "boom secret-token-123" }); + const notes = server.hold(Notes, { id: "S1" }, context); + await until(() => stateOf(workers.state.value, "S1")?.status === "failed"); + expect(stateOf(workers.state.value, "S1")?.error?.message).toBe("boom [redacted]"); + // Public: the state; the report's stderr tail is the app's to redact (a later test). + expect(JSON.stringify(workers.state.value)).not.toContain("secret"); + expect(errors.find(({ kind }) => kind === "worker_failed")).toMatchObject({ + error: { message: "boom [redacted]" }, + }); + notes.release(); + }); + + it("a link lost while its child lives (a blocked event loop past the heartbeat): killed after the grace, failed", async () => { + const { server, workers, starts, errors } = await fleet({ heartbeat: { ping: 100, close: 300 } }); + const notes = await server.get(Notes, { id: "S1" }, context); + const control = await server.get(Control, { id: "S1" }, context); + await control.block(20_000, context); + await until(() => stateOf(workers.state.value, "S1")?.status === "stopping", 5_000); + await until(() => stateOf(workers.state.value, "S1")?.status === "failed", 10_000); + expect(stateOf(workers.state.value, "S1")?.error?.message).toBe("worker exited (SIGKILL)"); + expect(errors.find(({ kind }) => kind === "worker_failed")).toMatchObject({ + error: { message: "worker: link lost" }, + }); + expect(starts).toEqual(["session|S1"]); + notes.release(); + control.release(); + }, 20_000); + + it("an exit before started fails the start; a worker that never connects is killed at the start timer", async () => { + const { server, workers, behaviours, errors } = await fleet(); + behaviours.set("S1", { exit: 3 }); + const exited = server.hold(Notes, { id: "S1" }, context); + await until(() => stateOf(workers.state.value, "S1")?.status === "failed"); + expect(stateOf(workers.state.value, "S1")?.error?.message).toBe("worker failed to start"); + expect(errors.find(({ kind }) => kind === "worker_failed")).toMatchObject({ + error: { message: expect.stringContaining("exited before it was ready (code 3)") }, + }); + exited.release(); + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] }); + try { + behaviours.set("S2", { connect: false }); + const silent = server.hold(Notes, { id: "S2" }, context); + await vi.waitFor(() => expect(stateOf(workers.state.value, "S2")?.status).toBe("starting"), { + timeout: 5_000, + }); + await vi.advanceTimersByTimeAsync(60_000); + await vi.waitFor(() => expect(stateOf(workers.state.value, "S2")?.status).toBe("failed"), { timeout: 5_000 }); + expect(stateOf(workers.state.value, "S2")?.error?.message).toBe("worker failed to start"); + expect(errors.findLast(({ kind }) => kind === "worker_failed")).toMatchObject({ + error: { message: expect.stringContaining("did not start within 60 s") }, + }); + silent.release(); + } finally { + vi.useRealTimers(); + } + }); + + it("the start timer covers only the start: a slow load after connecting is not killed (Q4)", async () => { + const { server, workers, behaviours } = await fleet(); + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] }); + try { + behaviours.set("S1", { hang: true }); + const notes = server.hold(Notes, { id: "S1" }, context); + await vi.waitFor(() => expect(stateOf(workers.state.value, "S1")?.status).toBe("starting"), { + timeout: 5_000, + }); + await new Promise((resolve) => setImmediate(resolve)); + await vi.advanceTimersByTimeAsync(120_000); + expect(stateOf(workers.state.value, "S1")?.status).toBe("starting"); + expect(notes.state.value.kind).toBe("loading"); + notes.release(); + } finally { + vi.useRealTimers(); + } + }); +}); + +describe("keep-alive (D9, R3, Q5)", () => { + it("stops a worker idle since idleMs; a hold keeps one running; busy keeps it too", async () => { + const { server, workers } = await fleet({ idleMs: 200 }); + const held = await server.get(Notes, { id: "held" }, context); + const released = await server.get(Notes, { id: "free" }, context); + const control = await server.get(Control, { id: "busy" }, context); + await control.busy(1_500, context); + control.release(); + released.release(); + await until(() => stateOf(workers.state.value, "free") === undefined, 5_000); + await new Promise((resolve) => setTimeout(resolve, 600)); + expect(stateOf(workers.state.value, "held")?.status).toBe("running"); + expect(stateOf(workers.state.value, "busy")?.status).toBe("running"); + await until(() => stateOf(workers.state.value, "busy") === undefined, 5_000); + held.release(); + }, 20_000); + + it("1,000 sequential local calls in a worker nobody holds send no message (Sol)", async () => { + const { server, behaviours, output } = await fleet(); + behaviours.set("S1", { localCalls: 1_000 }); + // Released at once: the calls run 300 ms after the load, with no hold arriving (no reference keeps it busy). + (await server.get(Notes, { id: "S1" }, context)).release(); + await until(() => output.some((line) => line.startsWith("session|S1 local calls sent"))); + expect(output).toContain("session|S1 local calls sent 0"); + }); +}); + +describe("stop", () => { + it("suspends while held: holders fail waiting suspended, nothing starts; forgotten with the last holder (Q3)", async () => { + const { server, workers, starts } = await fleet(); + const notes = await server.get(Notes, { id: "S1" }, context); + await workers.stop(Session({ id: "S1" }), context); + expect(stateOf(workers.state.value, "S1")).toBeUndefined(); + await until(() => notes.state.value.kind === "failed"); + expect(notes.state.value).toMatchObject({ + cause: { kind: "waiting", error: { message: "session|S1: suspended" } }, + }); + expect(starts).toEqual(["session|S1"]); + notes.release(); + await new Promise((resolve) => setTimeout(resolve, 20)); + const again = await server.get(Notes, { id: "S1" }, context); + expect(starts).toEqual(["session|S1", "session|S1"]); + again.release(); + }); + + it("a stop during a start waits for it, then stops; a start during a stop waits for its exit", async () => { + const { server, workers, behaviours, starts } = await fleet(); + behaviours.set("S1", { loadMs: 300 }); + const notes = server.hold(Notes, { id: "S1" }, context); + await until(() => stateOf(workers.state.value, "S1") !== undefined); + await workers.stop(Session({ id: "S1" }), context); + expect(stateOf(workers.state.value, "S1")).toBeUndefined(); + notes.release(); + await new Promise((resolve) => setTimeout(resolve, 20)); + behaviours.set("S1", { log: undefined }); + const again = await server.get(Notes, { id: "S1" }, context); + const stopping = workers.stop(Session({ id: "S1" }), context); + again.release(); + await stopping; + const third = await server.get(Notes, { id: "S1" }, context); + expect(starts).toHaveLength(3); + third.release(); + }); + + it("runs the worker's closers in reverse order at its stop", async () => { + const dir = await mkdtemp(join(tmpdir(), "chord-workers-log-")); + const { server, workers, behaviours } = await fleet(); + behaviours.set("S1", { log: join(dir, "log"), closeMs: 50 }); + const notes = await server.get(Notes, { id: "S1" }, context); + notes.release(); + await workers.stop(Session({ id: "S1" }), context); + expect(await readFile(join(dir, "log"), "utf8")).toBe("store\nhost\n"); + await rm(dir, { recursive: true, force: true }); + }); +}); + +describe("close", () => { + it("cancels a start waiting in start(address): nothing forks; new starts reject; no worker_failed at shutdown", async () => { + let waiting = false; + const { server, workers, errors } = await fleet({ + start: async (address, startContext) => { + if (address.key === "session|wait") { + waiting = true; + await new Promise((_, reject) => + startContext.abortSignal?.addEventListener("abort", () => reject(startContext.abortSignal?.reason)), + ); + } + return {}; + }, + }); + const running = await server.get(Notes, { id: "S1" }, context); + const waits = server.hold(Notes, { id: "wait" }, context); + await until(() => waiting); + await workers.close(context); + expect(workers.state.value).toEqual({}); + expect(errors).toEqual([]); + const late = server.hold(Notes, { id: "late" }, context); + await until(() => late.state.value.kind === "failed"); + expect(late.state.value).toMatchObject({ cause: { error: { code: "released", reason: "closed" } } }); + for (const ref of [running, waits, late]) ref.release(); + }); +}); + +describe("the rollout step (§4)", () => { + it("ships the current set to running workers, retries held failed ones, lists each", async () => { + const { server, workers, plugins, label } = await fleet(); + const one = await server.get(Notes, { id: "S1" }, context); + const two = await server.get(Notes, { id: "S2" }, context); + const control = await server.get(Control, { id: "S3" }, context); + const failed = await server.get(Notes, { id: "S3" }, context); + await control.crash(1, context); + await until(() => stateOf(workers.state.value, "S3")?.status === "failed"); + await label("v2"); + await until(() => plugins.state.value.dirty.includes("@test/notes")); + const result = await plugins.rollout( + [workers.step("workers")], + await plugins.build(plugins.state.value.dirty, context), + context, + ); + const hosts = result.steps[0]!.hosts; + expect(listed(hosts).sort()).toEqual([ + ["session|S1", { added: [], changed: ["@test/notes"], removed: [] }], + ["session|S2", { added: [], changed: ["@test/notes"], removed: [] }], + ["session|S3", { added: ["@test/notes"], changed: [], removed: [] }], + ]); + await until(() => one.label.value === "S1 v2" && two.label.value === "S2 v2" && failed.label.value === "S3 v2"); + expect(plugins.state.value.dirty).toEqual([]); + for (const ref of [one, two, failed, control]) ref.release(); + }); + + it("lists a worker stopped cleanly during it as stopped and one that crashed with its error (it stays dirty)", async () => { + const { server, workers, plugins, label } = await fleet(); + const one = await server.get(Notes, { id: "S1" }, context); + const control = await server.get(Control, { id: "S2" }, context); + await label("v2"); + await until(() => plugins.state.value.dirty.includes("@test/notes")); + const built = await plugins.build(plugins.state.value.dirty, context); + // Both workers block their event loops: their reloads wait; S1 is stopped meanwhile, S2 crashes before reading it. + const first = await server.get(Control, { id: "S1" }, context); + await first.block(400, context); + await control.blockThenCrash(400, context); + // Their event loops are blocked once their timers ran. + await new Promise((resolve) => setTimeout(resolve, 50)); + const rolling = plugins.rollout([workers.step("workers")], built, context); + await new Promise((resolve) => setTimeout(resolve, 50)); + const stopped = workers.stop(Session({ id: "S1" }), context); + const result = await rolling; + await stopped; + expect(listed(result.steps[0]!.hosts).sort()).toEqual([ + ["session|S1", "stopped"], + ["session|S2", "worker: its link is gone"], + ]); + expect(plugins.state.value.dirty).toEqual(["@test/notes"]); + for (const ref of [one, first, control]) ref.release(); + }); + + it("a worker starting during it gets the new set: a start after the rollout loads it", async () => { + const { server, workers, plugins, label } = await fleet(); + await label("v2"); + await until(() => plugins.state.value.dirty.includes("@test/notes")); + await plugins.rollout( + [workers.step("workers")], + await plugins.build(plugins.state.value.dirty, context), + context, + ); + const notes = await server.get(Notes, { id: "S9" }, context); + expect(notes.label.value).toBe("S9 v2"); + notes.release(); + }); + + it("an aborted wait lists the worker aborted; its late reply keeps the child's slot until it arrives", async () => { + const { server, workers, plugins, label } = await fleet(); + const control = await server.get(Control, { id: "S1" }, context); + const notes = await server.get(Notes, { id: "S1" }, context); + await label("v2"); + await until(() => plugins.state.value.dirty.includes("@test/notes")); + const built = await plugins.build(plugins.state.value.dirty, context); + await control.block(300, context); + await new Promise((resolve) => setTimeout(resolve, 50)); + const abort = new AbortController(); + const step = workers.step("workers"); + const waited = plugins.rollout([step], built, withAbortSignal(abort.signal, context)); + setTimeout(() => abort.abort(new Error("stop waiting")), 50); + const first = await waited; + expect(listed(first.steps[0]!.hosts)).toEqual([["session|S1", "stop waiting"]]); + expect(plugins.state.value.dirty).toEqual(["@test/notes"]); + await label("v3"); + await until(() => plugins.state.value.dirty.includes("@test/notes")); + const next = await plugins.rollout([step], await plugins.build(plugins.state.value.dirty, context), context); + expect(listed(next.steps[0]!.hosts)).toEqual([ + ["session|S1", { added: [], changed: ["@test/notes"], removed: [] }], + ]); + await until(() => notes.label.value === "S1 v3"); + control.release(); + notes.release(); + }); +}); + +describe("GC", () => { + // Real forks are costly (100 took the suite from 103 s to 700 s on one CPU at 1,000): 100 here; the 1,000 failed and + // retried addresses below cover the per-address records and suspensions at scale. + it("100 addresses started and stopped leave no state, no record, no suspension (Astra, Sol)", async () => { + const refs: WeakRef[] = []; + const { server, workers } = await fleet({ + start: async (address) => { + refs.push(new WeakRef(address)); + return {}; + }, + }); + // Twenty at a time: each a real fork, start, hold, stop and exit. + for (let batch = 0; batch < 5; batch++) { + await Promise.all( + Array.from({ length: 20 }, async (_, index) => { + const id = `R${batch * 20 + index}`; + const notes = await server.get(Notes, { id }, context); + notes.release(); + await workers.stop(Session({ id }), context); + }), + ); + } + expect(workers.state.value).toEqual({}); + await expectCollected(...refs); + }, 120_000); + + it("1,000 failed and retried addresses leave no state, no record and no suspension", async () => { + const refs: WeakRef[] = []; + const { server, workers } = await fleet({ + start: async (address) => { + refs.push(new WeakRef(address)); + throw new Error("no worker"); + }, + }); + for (let i = 0; i < 1_000; i++) { + const id = `A${i}`; + const ref = server.hold(Notes, { id }, context); + await until(() => ref.state.value.kind === "failed"); + ref.release(); + await workers.stop(Session({ id }), context); + await workers.retry(Session({ id }), context); + } + expect(workers.state.value).toEqual({}); + await expectCollected(...refs.slice(0, 50)); + }, 120_000); +}); + +describe("fix round 2", () => { + // Astra's recheck R3: a throwing `principal(address)` reached the guest with its message, the child left starting. + it("a throwing principal hook fails the start with the public message; the error stays private", async () => { + const { server, workers, errors } = await fleet({ + principal: () => { + throw new Error("secret token=sk-1 at /home/owner/private"); + }, + }); + const notes = server.hold(Notes, { id: "S1" }, context); + await until(() => notes.state.value.kind === "failed"); + expect(JSON.stringify(notes.state.value)).not.toContain("secret"); + expect(notes.state.value).toMatchObject({ + kind: "failed", + cause: { error: { message: "worker failed to start" } }, + }); + await until(() => stateOf(workers.state.value, "S1")?.status === "failed"); + expect(stateOf(workers.state.value, "S1")).toMatchObject({ error: { message: "worker failed to start" } }); + // Opus's second recheck: the component's own kill of a failed start is no `worker_exited`. + await until(() => errors.length > 0); + await new Promise((resolve) => setTimeout(resolve, 50)); + expect(errors).toMatchObject([ + { kind: "worker_failed", address: "session|S1", error: { message: expect.stringContaining("secret") } }, + ]); + notes.release(); + }); + + it("a throwing authorize refuses the hold (forbidden); the error stays private, reported authorize_failed", async () => { + const { server, errors, starts } = await fleet({ + authorize: () => { + throw new Error("secret lookup"); + }, + }); + const remote = createHost({ onError: () => {} }); + const [up, down] = memoryLinkPair(); + server.accept(up, { principal: { kind: "guest", id: "g" } }, context); + remote.connect(down, { principal: { kind: "server" } }, context); + await remote.load([], context); + const notes = remote.hold(Notes, { id: "S1" }, context); + await until(() => notes.state.value.kind !== "loading"); + expect(JSON.stringify(notes.state.value)).not.toContain("secret"); + expect(notes.state.value).toMatchObject({ kind: "failed", cause: { error: { code: "forbidden" } } }); + expect([errors, starts]).toMatchObject([[{ kind: "authorize_failed", address: "session|S1" }], []]); + notes.release(); + await remote.close(context); + }); +}); + +describe("fix round 1", () => { + it("close during a forked start that never connects returns at once, no failure (Opus F1)", async () => { + const { server, workers, behaviours, errors } = await fleet(); + behaviours.set("S1", { connect: false }); + const notes = server.hold(Notes, { id: "S1" }, context); + await until(() => stateOf(workers.state.value, "S1")?.status === "starting"); + const started = Date.now(); + await workers.close(context); + expect(Date.now() - started).toBeLessThan(5_000); + expect(errors).toEqual([]); + notes.release(); + }); + + it("close during an app's start that ignores its signal returns at once; nothing forks after it (Sol)", async () => { + let forked = false; + const { server, workers, errors } = await fleet({ start: () => new Promise(() => {}) }); + const notes = server.hold(Notes, { id: "S1" }, context); + await new Promise((resolve) => setTimeout(resolve, 50)); + await workers.close(context); + forked = stateOf(workers.state.value, "S1") !== undefined; + expect([forked, errors]).toEqual([false, []]); + notes.release(); + }); + + it("a stop during the worker's startup is a stop: its closers run, nothing fails (Astra)", async () => { + const dir = await mkdtemp(join(tmpdir(), "chord-workers-log-")); + const { server, workers, behaviours, errors } = await fleet(); + behaviours.set("S1", { loadMs: 5_000, closeMs: 200, log: join(dir, "log") }); + const notes = server.hold(Notes, { id: "S1" }, context); + await until(() => stateOf(workers.state.value, "S1") !== undefined); + await new Promise((resolve) => setTimeout(resolve, 300)); + await workers.stop(Session({ id: "S1" }), context); + expect(errors).toEqual([]); + expect(await readFile(join(dir, "log"), "utf8")).toBe("store\nhost\n"); + notes.release(); + await rm(dir, { recursive: true, force: true }); + }); + + it("a reload queued behind one the child never answers settles when the child exits (Astra)", async () => { + const { server, workers, plugins, label } = await fleet(); + const control = await server.get(Control, { id: "S1" }, context); + await label("v2"); + await until(() => plugins.state.value.dirty.length > 0); + const built = await plugins.build(plugins.state.value.dirty, context); + await control.blockThenCrash(600, context); + await new Promise((resolve) => setTimeout(resolve, 50)); + const first = plugins.rollout([workers.step("one")], built, context); + await new Promise((resolve) => setTimeout(resolve, 50)); + const second = plugins.rollout([workers.step("two")], built, context); + const outcomes = await Promise.all([first, second]); + expect(outcomes.map((result) => listed(result.steps[0]!.hosts))).toEqual([ + [["session|S1", "worker: its link is gone"]], + [["session|S1", "worker: its link is gone"]], + ]); + control.release(); + }); + + it("a plugin whose evaluation failed is shipped again by the next rollout and stays dirty (Astra)", async () => { + const { server, workers, plugins, root } = await fleet(); + const notes = await server.get(Notes, { id: "S1" }, context); + await writeFile( + join(root, "notes", "label.ts"), + 'throw new Error("evaluation failed");\nexport const label = "v2";\n', + ); + await until(() => plugins.state.value.dirty.length > 0); + const built = await plugins.build(plugins.state.value.dirty, context); + const failed = (result: { steps: readonly { hosts: readonly RolloutHost[] }[] }) => { + const [entry] = result.steps[0]!.hosts; + return entry !== undefined && "report" in entry ? entry.report.failed.map(({ plugin }) => plugin) : entry; + }; + expect(failed(await plugins.rollout([workers.step("one")], built, context))).toEqual(["@test/notes"]); + expect(failed(await plugins.rollout([workers.step("two")], built, context))).toEqual(["@test/notes"]); + expect(plugins.state.value.dirty).toEqual(["@test/notes"]); + expect(notes.label.value).toBe("S1 v1"); + notes.release(); + }); + + it("two rollouts started together each ship their own build; the later one stays (Astra)", async () => { + const { server, workers, plugins, label } = await fleet(); + const notes = await server.get(Notes, { id: "S1" }, context); + await label("v2"); + const v2 = await plugins.build(["@test/notes"], context); + await label("v3"); + const v3 = await plugins.build(["@test/notes"], context); + const results = await Promise.all([ + plugins.rollout([workers.step("v2")], v2, context), + plugins.rollout([workers.step("v3")], v3, context), + ]); + for (const result of results) { + expect(listed(result.steps[0]!.hosts)).toEqual([ + ["session|S1", { added: [], changed: ["@test/notes"], removed: [] }], + ]); + } + await until(() => notes.label.value === "S1 v3"); + notes.release(); + }); + + it("reaps a worker that closed its own IPC channel and lives on: killed after the grace, failed (Sol)", async () => { + const { server, workers, behaviours } = await fleet(); + behaviours.set("S1", { disconnectAndBlock: true }); + const notes = await server.get(Notes, { id: "S1" }, context); + await until(() => stateOf(workers.state.value, "S1")?.status === "failed", 10_000); + expect(stateOf(workers.state.value, "S1")?.error?.message).toBe("worker exited (SIGKILL)"); + await until(() => notes.state.value.kind === "failed"); + notes.release(); + }, 15_000); + + it("keeps no reload report once consumed: the reload queue settles to nothing (Sol)", async () => { + const env = await fleet(); + const notes = await env.server.get(Notes, { id: "S1" }, context); + const report = await (async () => { + await env.label("v2"); + const built = await env.plugins.build(["@test/notes"], context); + const result = await env.plugins.rollout([env.workers.step("workers")], built, context); + const [entry] = result.steps[0]!.hosts; + if (entry === undefined || !("report" in entry)) throw new Error("no report"); + return new WeakRef(entry.report); + })(); + await expectCollected(report); + notes.release(); + }); + + it("keeps a worker's stderr out of the public state: only worker_failed carries its tail (Astra)", async () => { + const { server, workers, behaviours, errors } = await fleet(); + behaviours.set("S1", { fatal: "boom secret-token-123" }); + const notes = server.hold(Notes, { id: "S1" }, context); + await until(() => stateOf(workers.state.value, "S1")?.status === "failed"); + expect(JSON.stringify(workers.state.value)).not.toContain("secret"); + expect(JSON.stringify(notes.state.value)).not.toContain("secret"); + const failure = errors.find(({ kind }) => kind === "worker_failed"); + expect(failure).toMatchObject({ stderrTail: expect.stringContaining("secret-token-123") }); + notes.release(); + }); +}); diff --git a/packages/chord/test/next/bench/http.ts b/packages/chord/test/next/bench/http.ts new file mode 100644 index 000000000..21f3b2274 --- /dev/null +++ b/packages/chord/test/next/bench/http.ts @@ -0,0 +1,120 @@ +// Benchmark of the HTTP component (slice 10 §7): a route's overhead per request against a plain Node handler (keep-alive +// round trips, median µs per request over bursts), and `bytes` throughput for a 100 MiB file (`fileResponse`) against +// `pipeline` of a read stream. +// Run by hand from packages/chord (not in CI), pinned: +// taskset -c 6 node --conditions=source --experimental-strip-types --expose-gc test/next/bench/http.ts +import { createReadStream } from "node:fs"; +import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import { Agent, createServer, request } from "node:http"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { performance } from "node:perf_hooks"; +import { pipeline } from "node:stream/promises"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { createHttp, fileResponse, Http } from "../../../src/next-node/http/index.ts"; + +const BURSTS = 7; +const REQUESTS = 2_000; +const FILE = 100 * 1024 * 1024; + +/** GETs `path` on `port` over `agent`, reading the whole body; resolves with its length. */ +function get(port: number, path: string, agent: Agent): Promise { + return new Promise((resolve, reject) => { + const req = request({ host: "127.0.0.1", port, path, agent }, (res) => { + let length = 0; + res.on("data", (chunk: Buffer) => { + length += chunk.length; + }); + res.on("end", () => resolve(length)); + res.on("error", reject); + }); + req.on("error", reject); + req.end(); + }); +} + +/** The median µs per request of `REQUESTS` sequential keep-alive requests, over bursts. */ +async function perRequest(port: number, path: string): Promise { + const agent = new Agent({ keepAlive: true, maxSockets: 1 }); + for (let i = 0; i < 200; i++) await get(port, path, agent); + const times: number[] = []; + for (let burst = 0; burst < BURSTS; burst++) { + const start = performance.now(); + for (let i = 0; i < REQUESTS; i++) await get(port, path, agent); + times.push(((performance.now() - start) * 1000) / REQUESTS); + } + agent.destroy(); + return median(times); +} + +/** MiB/s of downloading `path` once (after one warm-up). */ +async function throughput(port: number, path: string): Promise { + const agent = new Agent({ keepAlive: true, maxSockets: 1 }); + await get(port, path, agent); + const times: number[] = []; + for (let burst = 0; burst < 3; burst++) { + const start = performance.now(); + await get(port, path, agent); + times.push(performance.now() - start); + } + agent.destroy(); + return FILE / 1024 / 1024 / (median(times) / 1000); +} + +function median(values: readonly number[]): number { + const sorted = [...values].sort((a, b) => a - b); + return sorted[Math.floor(sorted.length / 2)]!; +} + +const dir = await mkdtemp(join(tmpdir(), "chord-bench-http-")); +const file = join(dir, "big.bin"); +await writeFile(file, Buffer.alloc(FILE, 1)); + +// A plain Node server: the same JSON body, the same file through `pipeline`. +const plain = createServer((req, res) => { + if (req.url === "/file") { + res.writeHead(200, { "Content-Type": "application/octet-stream", "Content-Length": String(FILE) }); + void pipeline(createReadStream(file), res).catch(() => {}); + return; + } + const body = JSON.stringify({ ok: true }); + res.writeHead(200, { "Content-Type": "application/json; charset=utf-8", "Content-Length": String(body.length) }); + res.end(body); +}); +await new Promise((resolve) => plain.listen(0, "127.0.0.1", resolve)); +const plainPort = (plain.address() as { port: number }).port; + +// The component with Chord's own listener: one plugin with the same route and the file. +const host = createHost({ principal: { kind: "server" }, onError: () => {} }); +const http = createHttp(host, { listen: { host: "127.0.0.1", port: 0 }, onError: () => {} }, context); +const plugin = defineFacet((_host, self) => { + self.contribute(Http, { + routes: { + "GET /json": async () => ({ kind: "json", value: { ok: true } }), + "GET /file": async () => fileResponse(file), + }, + }); +}); +await host.load( + [ + { name: "chord.http", facet: http.facet }, + { name: "@bench/routes", facet: plugin }, + ], + context, +); +const { port } = await http.listening(context); + +const plainRequest = await perRequest(plainPort, "/json"); +const routeRequest = await perRequest(port, "/json"); +console.log( + `request round trip: plain ${plainRequest.toFixed(1)} µs, route ${routeRequest.toFixed(1)} µs (overhead ${(routeRequest - plainRequest).toFixed(1)} µs, ${(((routeRequest - plainRequest) / plainRequest) * 100).toFixed(1)}%)`, +); +const plainBytes = await throughput(plainPort, "/file"); +const routeBytes = await throughput(port, "/file"); +console.log(`100 MiB file: pipeline ${plainBytes.toFixed(0)} MiB/s, fileResponse ${routeBytes.toFixed(0)} MiB/s`); + +await http.close(context); +await host.close(context); +plain.close(); +await rm(dir, { recursive: true, force: true }); diff --git a/packages/chord/test/next/bench/plugins.ts b/packages/chord/test/next/bench/plugins.ts index 3cd231ebb..b80a56ee9 100644 --- a/packages/chord/test/next/bench/plugins.ts +++ b/packages/chord/test/next/bench/plugins.ts @@ -25,6 +25,7 @@ const RELOADS = 100; const gc = (globalThis as { gc?: () => void }).gc!; const HOST_PACKAGES = { "@earendil-works/chord/next": chordNext }; const KINDS = ["server", "worker"]; +const NODE = { platform: "node", hostPackages: Object.keys(HOST_PACKAGES) } as const; /** A plugin's worker facet: a store over a document, a derive, a method, a helper module and the shared contract. */ function worker(index: number, label: string): string { @@ -111,12 +112,9 @@ for (let run = 0; run < RUNS; run++) { console.log(`evaluation per facet: ${median(evaluations).toFixed(2)} ms (median of ${RUNS})`); // The heap after 100 reloads of one plugin on a host, against after a warm-up (two builds alternating). -const component = createPlugins( - { roots: [root], kinds: KINDS, hostPackages: HOST_PACKAGES, onError: () => {} }, - context, -); +const component = createPlugins({ roots: [root], kinds: { server: NODE, worker: NODE }, onError: () => {} }, context); const host = createHost({ onError: () => {} }); -await component.load(host, { kind: "worker", core: [] }, context); +await component.load(host, { kind: "worker", core: [], modules: HOST_PACKAGES }, context); await writeFile(join(root, "p0", "worker.ts"), worker(0, "a")); const first = await component.build(["@bench/p0"], context); await writeFile(join(root, "p0", "worker.ts"), worker(0, "b")); diff --git a/packages/chord/test/next/bench/workers.ts b/packages/chord/test/next/bench/workers.ts new file mode 100644 index 000000000..077b6df10 --- /dev/null +++ b/packages/chord/test/next/bench/workers.ts @@ -0,0 +1,107 @@ +// Benchmark of the workers component (slice 10 §7): a worker's cold start with shipped scripts (fork to `started`, fork +// to `ready`, medians over runs) and the server's memory per idle worker (heap after a GC, 20 workers running idle). +// Run by hand from packages/chord (not in CI), pinned: +// taskset -c 6 node --conditions=source --experimental-strip-types --expose-gc test/next/bench/workers.ts +import { fileURLToPath } from "node:url"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { hostKind, str } from "../../../src/next/contracts/index.ts"; +import { createHost } from "../../../src/next/host/index.ts"; +import * as chordNext from "../../../src/next/index.ts"; +import { createPlugins } from "../../../src/next-node/plugins/index.ts"; +import type { FromWorker } from "../../../src/next-node/workers/control.ts"; +import { createWorkers } from "../../../src/next-node/workers/index.ts"; +import { WorkerProcess } from "../../../src/next-node/workers/process.ts"; +import { Control } from "../../next-node/fixtures/workers/control.ts"; + +const RUNS = 7; +const WORKERS = 20; +const ENTRY = fileURLToPath(new URL("../../next-node/fixtures/workers/worker.ts", import.meta.url)); +const ROOT = fileURLToPath(new URL("../../next-node/fixtures/workers/plugins/", import.meta.url)); +const EXEC_ARGV = ["--conditions=source", "--experimental-strip-types", "--no-warnings"]; + +const gc = (globalThis as { gc?: () => void }).gc!; +const Session = hostKind("session", { id: str() }); +const modules = { "@earendil-works/chord/next": chordNext }; +const plugins = createPlugins( + { roots: [ROOT], kinds: { worker: { platform: "node", hostPackages: Object.keys(modules) } }, onError: () => {} }, + context, +); +const scripts = await plugins.scripts("worker", context); + +function median(values: readonly number[]): number { + const sorted = [...values].sort((a, b) => a - b); + return sorted[Math.floor(sorted.length / 2)]!; +} + +// Cold starts: a fork with the shipped scripts, timed to `started` and to `ready`. +const started: number[] = []; +const ready: number[] = []; +for (let run = 0; run < RUNS; run++) { + const forked = performance.now(); + const times: { started?: number; ready?: number } = {}; + const done = Promise.withResolvers(); + const child = new WorkerProcess( + { entry: ENTRY, execArgv: EXEC_ARGV, onOutput: () => {} }, + { + control: (message: FromWorker) => { + if (message.kind === "started") times.started = performance.now() - forked; + if (message.kind === "ready") { + times.ready = performance.now() - forked; + done.resolve(); + } + }, + linkLost: () => {}, + linkClosed: () => {}, + exited: () => {}, + }, + ); + child.send({ + kind: "start", + address: Session({ id: `R${run}` }), + start: {}, + scripts, + lock: `bench:${process.pid}:${run}`, + }); + await done.promise; + started.push(times.started!); + ready.push(times.ready!); + child.kill(); + await child.exited; +} +console.log( + `cold start (${scripts.length} shipped script): fork to started ${median(started).toFixed(0)} ms, to ready ${median(ready).toFixed(0)} ms (median of ${RUNS})`, +); + +// Memory per idle worker on the server: its records, process handle, link and remote references. +const server = createHost({ address: hostKind("server")(), principal: { kind: "server" }, onError: () => {} }); +const workers = createWorkers( + server, + Session, + { + kind: "worker", + entry: ENTRY, + execArgv: EXEC_ARGV, + authorize: () => true, + principal: () => ({ kind: "worker" }), + start: () => ({}), + lock: (address) => `bench:${process.pid}:${address.key}`, + plugins, + onOutput: () => {}, + onError: () => {}, + }, + context, +); +await server.load([], context); +gc(); +const before = process.memoryUsage().heapUsed; +const held = []; +for (let i = 0; i < WORKERS; i++) held.push(await server.get(Control, { id: `W${i}` }, context)); +gc(); +const after = process.memoryUsage().heapUsed; +console.log( + `server heap per idle worker: ${((after - before) / WORKERS / 1024).toFixed(1)} KiB (${WORKERS} workers, one reference each)`, +); +for (const ref of held) ref.release(); +await workers.close(context); +await server.close(context); +plugins.close(); diff --git a/packages/chord/test/next/examples/12-routes.ts b/packages/chord/test/next/examples/12-routes.ts index 7f6516887..e9dedbbb5 100644 --- a/packages/chord/test/next/examples/12-routes.ts +++ b/packages/chord/test/next/examples/12-routes.ts @@ -144,8 +144,8 @@ await guest.get(Notes, { session: "S2" }, context).catch((error) => { const lobby = await guest.get(Lobby, {}, context); console.log("guest via the lobby:", await lobby.peek({ session: "S1" }, context)); -// 5. The worker drops: references go loading, the route resolves the address again after its backoff (1 s), and -// the new worker (a fresh run, empty notes here) binds them again. +// 5. The worker drops: references go loading, the route resolves the address again at once (its link had completed +// its handshake), and the new worker (a fresh run, empty notes here) binds them again. const states: string[] = []; const watch = await notes.state.watch(context); watch.start(async (call) => { diff --git a/packages/chord/test/next/examples/35-plugins/main.ts b/packages/chord/test/next/examples/35-plugins/main.ts index bdfaa89d9..e5fee1544 100644 --- a/packages/chord/test/next/examples/35-plugins/main.ts +++ b/packages/chord/test/next/examples/35-plugins/main.ts @@ -29,13 +29,15 @@ async function until(ready: () => boolean): Promise { while (!ready()) await new Promise((resolve) => setTimeout(resolve, 10)); } -// 1. The component: the roots, the facet kinds, and the host packages every script gets (this process's Chord: a -// plugin's `import { defineFacet } from "@earendil-works/chord/next"` is the app's own instance). +// 1. The component: the roots, and the facet kinds, each built for Node with Chord as a host package (kept external in +// the builds). The hosts give the modules themselves when they load (this process's Chord: a plugin's +// `import { defineFacet } from "@earendil-works/chord/next"` is the app's own instance). +const modules = { "@earendil-works/chord/next": chordNext }; +const node = { platform: "node", hostPackages: Object.keys(modules) } as const; const plugins = createPlugins( { roots: [root], - kinds: ["server", "worker"], - hostPackages: { "@earendil-works/chord/next": chordNext }, + kinds: { server: node, worker: node }, onError: (error) => console.log("plugins:", error.kind), }, context, @@ -52,8 +54,8 @@ server.route( { authorize: () => true, resolve: async () => ({ link: down, principal: { kind: "worker" } }) }, context, ); -const workerLoad = await plugins.load(worker, { kind: "worker", core: [] }, context); -const serverLoad = await plugins.load(server, { kind: "server", core: [] }, context); +const workerLoad = await plugins.load(worker, { kind: "worker", core: [], modules }, context); +const serverLoad = await plugins.load(server, { kind: "server", core: [], modules }, context); console.log( "1. discovered:", plugins.discovery.plugins.map(({ name, facets }) => `${name} [${Object.keys(facets)}]`).join(", "), @@ -84,8 +86,17 @@ console.log( built.facets.map(({ plugin, facet, kind }) => `${plugin}/${facet}: ${kind}`), ); const rolled = await plugins.rollout(steps, built, context); -for (const { name, report } of rolled.steps) { - console.log(` ${name}:`, JSON.stringify(report.plugins), "objects", JSON.stringify(report.objects)); +for (const { name, hosts } of rolled.steps) { + for (const host of hosts) { + if ("report" in host) { + console.log( + ` ${name}:`, + JSON.stringify(host.report.plugins), + "objects", + JSON.stringify(host.report.objects), + ); + } + } } await until(() => notes.label.value === "v2"); console.log("4. notes:", notes.label.value, "count", notes.count.value, "| index:", index.label.value); diff --git a/packages/chord/test/next/examples/35-plugins/plugins/clock/server.ts b/packages/chord/test/next/examples/35-plugins/plugins/clock/server.ts index 5bc4b6ee3..29f0d3d98 100644 --- a/packages/chord/test/next/examples/35-plugins/plugins/clock/server.ts +++ b/packages/chord/test/next/examples/35-plugins/plugins/clock/server.ts @@ -2,7 +2,7 @@ // this plugin, so a reload leaves it as it is: same script, same facet, same object. import { constant, defineFacet } from "@earendil-works/chord/next"; -import { Clock } from "../contracts/contract.ts"; +import { Clock } from "@example/contracts/contract"; const evaluatedAt = new Date().toISOString(); diff --git a/packages/chord/test/next/examples/35-plugins/plugins/notes/server.ts b/packages/chord/test/next/examples/35-plugins/plugins/notes/server.ts index ed3ded889..141e104d7 100644 --- a/packages/chord/test/next/examples/35-plugins/plugins/notes/server.ts +++ b/packages/chord/test/next/examples/35-plugins/plugins/notes/server.ts @@ -1,7 +1,7 @@ // The notes plugin's server facet: serves `Index` with the same label. import { constant, defineFacet } from "@earendil-works/chord/next"; -import { Index } from "../contracts/contract.ts"; +import { Index } from "@example/contracts/contract"; import { label } from "./label.ts"; export default defineFacet((host) => host.serve(Index, () => ({ label: constant(label) }))); diff --git a/packages/chord/test/next/examples/35-plugins/plugins/notes/worker.ts b/packages/chord/test/next/examples/35-plugins/plugins/notes/worker.ts index b4f4c8ae4..012486c94 100644 --- a/packages/chord/test/next/examples/35-plugins/plugins/notes/worker.ts +++ b/packages/chord/test/next/examples/35-plugins/plugins/notes/worker.ts @@ -1,7 +1,7 @@ // The notes plugin's worker facet: serves `Notes`, its count kept in a document. import { constant, defineFacet, defineState, int, obj } from "@earendil-works/chord/next"; -import { Notes } from "../contracts/contract.ts"; +import { Notes } from "@example/contracts/contract"; import { label } from "./label.ts"; const Count = defineState("count", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); diff --git a/packages/chord/test/next/examples/36-plugin-failures/main.ts b/packages/chord/test/next/examples/36-plugin-failures/main.ts index 1fe16092c..103b1b3fb 100644 --- a/packages/chord/test/next/examples/36-plugin-failures/main.ts +++ b/packages/chord/test/next/examples/36-plugin-failures/main.ts @@ -63,11 +63,12 @@ const Counter = local("example.counter", { version: 1, params: {}, members: { st const Summary = local("example.summary", { version: 1, params: {}, members: { text: str() } }); const Status = local("example.status", { version: 1, params: {}, members: { text: str() } }); +// The host packages: this process's Chord. +const modules = { "@earendil-works/chord/next": chordNext }; const plugins = createPlugins( { roots: [root], - kinds: ["server"], - hostPackages: { "@earendil-works/chord/next": chordNext }, + kinds: { server: { platform: "node", hostPackages: Object.keys(modules) } }, onError: (error) => console.log("plugins:", error.kind), }, context, @@ -76,7 +77,7 @@ const host = createHost({ onError: () => {} }); // 1. Startup: the weather plugin's setup throws; it is left out and reported, the other plugins run. A reference to // what it would serve waits (`failed`, cause `waiting`: not served here), and so does its consumer's object. -const loaded = await plugins.load(host, { kind: "server", core: [] }, context); +const loaded = await plugins.load(host, { kind: "server", core: [], modules }, context); for (const { plugin, error } of loaded.failed) console.log(`1. left out ${plugin}: ${(error as Error).message}`); const weather = host.hold(Weather, {}, context); const forecast = host.hold(Forecast, {}, context); diff --git a/packages/chord/test/next/examples/36-plugin-failures/plugins/forecast/server.ts b/packages/chord/test/next/examples/36-plugin-failures/plugins/forecast/server.ts index 54a92d46f..c7a04a1d8 100644 --- a/packages/chord/test/next/examples/36-plugin-failures/plugins/forecast/server.ts +++ b/packages/chord/test/next/examples/36-plugin-failures/plugins/forecast/server.ts @@ -1,7 +1,7 @@ // The forecast plugin's server facet: a consumer of the weather plugin's `Weather`. import { defineFacet, local, str } from "@earendil-works/chord/next"; -import { Weather } from "../weather/contract.ts"; +import { Weather } from "@example/weather/contract"; const Forecast = local("example.forecast", { version: 1, params: {}, members: { text: str() } }); diff --git a/packages/chord/test/next/examples/36-plugin-failures/plugins/summary/server.ts b/packages/chord/test/next/examples/36-plugin-failures/plugins/summary/server.ts index 3aad5f3e1..6db7ff40f 100644 --- a/packages/chord/test/next/examples/36-plugin-failures/plugins/summary/server.ts +++ b/packages/chord/test/next/examples/36-plugin-failures/plugins/summary/server.ts @@ -2,7 +2,7 @@ // contract declares. import { defineFacet, local, str } from "@earendil-works/chord/next"; -import { Price } from "../prices/contract.ts"; +import { Price } from "@example/prices/contract"; const Summary = local("example.summary", { version: 1, params: {}, members: { text: str() } }); diff --git a/packages/chord/test/next/examples/37-plugin-imports/main.ts b/packages/chord/test/next/examples/37-plugin-imports/main.ts index 8ed51b7d2..7ec709c46 100644 --- a/packages/chord/test/next/examples/37-plugin-imports/main.ts +++ b/packages/chord/test/next/examples/37-plugin-imports/main.ts @@ -29,17 +29,18 @@ for (const dir of ["plugins", "app", "deps"]) { } const root = join(copy, "plugins"); -// 1. The component. Its host packages: Chord, the app's ES module and a CommonJS module of the app. Its npm command -// (here a stand-in): what `installDependencies` runs in a plugin's directory. +// 1. The component. Its host packages: Chord, the app's ES module and a CommonJS module of the app (their names kept +// external in the server kind's builds, the modules given at load). Its npm command (here a stand-in): what +// `installDependencies` runs in a plugin's directory. +const modules = { + "@earendil-works/chord/next": chordNext, + "@example/app": app, + "@example/legacy": createRequire(import.meta.url)("./app/legacy.cjs"), +}; const plugins = createPlugins( { roots: [root], - kinds: ["server"], - hostPackages: { - "@earendil-works/chord/next": chordNext, - "@example/app": app, - "@example/legacy": createRequire(import.meta.url)("./app/legacy.cjs"), - }, + kinds: { server: { platform: "node", hostPackages: Object.keys(modules) } }, install: { npmCommand: [process.execPath, fileURLToPath(new URL("./fake-npm.mjs", import.meta.url))] }, onError: (error) => console.log("plugins:", error.kind), }, @@ -49,7 +50,7 @@ const plugins = createPlugins( // 2. Startup: every plugin whose build fails is left out with its errors (one per line: file, line, column, why). The // report plugin's dependency is not installed yet. const host = createHost({ onError: () => {} }); -const loaded = await plugins.load(host, { kind: "server", core: [] }, context); +const loaded = await plugins.load(host, { kind: "server", core: [], modules }, context); console.log("1. left out at startup:"); for (const { plugin, error } of loaded.failed) { const [first, ...errors] = (error as Error).message.split("\n"); @@ -67,7 +68,9 @@ for (const facet of built.facets) { console.log(` built ${facet.plugin}/${facet.facet}: read ${files.join(", ")}; copied ${facet.copied.length} file`); } const report = await plugins.reload(host, built, context); -console.log(" reload:", JSON.stringify(report.plugins), "failed", report.failed.length); +// The reload passes the complete set: the refused plugins fail again and stay out. +const stillRefused = report.failed.map(({ plugin }) => plugin).join(", "); +console.log(" reload:", JSON.stringify(report.plugins), "| still refused:", stillRefused); const text = (await host.get(Report, {}, context)).text.value; console.log("3. the report:", text); diff --git a/packages/chord/test/next/examples/37-plugin-imports/plugins/report/server.ts b/packages/chord/test/next/examples/37-plugin-imports/plugins/report/server.ts index 41201eeea..601072036 100644 --- a/packages/chord/test/next/examples/37-plugin-imports/plugins/report/server.ts +++ b/packages/chord/test/next/examples/37-plugin-imports/plugins/report/server.ts @@ -6,12 +6,12 @@ import { pluginFile } from "@earendil-works/chord/next/plugins"; // Host packages: an ES module (default and named imports) and a CommonJS one, the app's own instances. import app, { currency } from "@example/app"; import legacy from "@example/legacy"; +// A contract of another plugin: a copy built into this script. +import { Report } from "@example/shapes/contract"; // An npm dependency: inlined into this script by the build (installed first). import { greet } from "example-greeting"; // A `chord.external` package: left external, loaded from this plugin's node_modules when the script runs. import { where } from "example-native"; -// A contract of another plugin: a copy built into this script. -import { Report } from "../shapes/contract.ts"; // A file of this plugin, read at run time by its absolute path. const rates = JSON.parse(readFileSync(pluginFile("data/rates.json"), "utf8")) as { readonly vat: number }; diff --git a/packages/chord/test/next/examples/37-plugin-imports/plugins/ruler/server.ts b/packages/chord/test/next/examples/37-plugin-imports/plugins/ruler/server.ts index ec9a0dd74..a67049cac 100644 --- a/packages/chord/test/next/examples/37-plugin-imports/plugins/ruler/server.ts +++ b/packages/chord/test/next/examples/37-plugin-imports/plugins/ruler/server.ts @@ -2,7 +2,7 @@ // copied file may import only host packages and files of its own plugin. import { defineFacet } from "@earendil-works/chord/next"; -import { Millimetres } from "../shapes/units.ts"; +import { Millimetres } from "@example/shapes/units"; export default defineFacet(() => { console.log(Millimetres); diff --git a/packages/chord/test/next/examples/38-watch-and-rollout/main.ts b/packages/chord/test/next/examples/38-watch-and-rollout/main.ts index 7eeb92f1e..53313cf75 100644 --- a/packages/chord/test/next/examples/38-watch-and-rollout/main.ts +++ b/packages/chord/test/next/examples/38-watch-and-rollout/main.ts @@ -48,17 +48,21 @@ const line = (report: ReloadReport) => /** A rollout, step by step. */ function print(result: RolloutResult): void { - for (const { name, report } of result.steps) console.log(` ${name}:`, line(report)); + for (const { name, hosts } of result.steps) { + for (const host of hosts) if ("report" in host) console.log(` ${name}:`, line(host.report)); + } if (result.stopped !== undefined) { console.log(` stopped at ${result.stopped.step}: ${(result.stopped.error as Error).message}`); } } +// The host packages: this process's Chord, kept external in Node builds. +const modules = { "@earendil-works/chord/next": chordNext }; +const node = { platform: "node", hostPackages: Object.keys(modules) } as const; const plugins = createPlugins( { roots: [root], - kinds: ["server", "worker"], - hostPackages: { "@earendil-works/chord/next": chordNext }, + kinds: { server: node, worker: node }, // The app's own code: a change marks `base`. base: [join(copy, "app")], // A stand-in for npm (no network; these plugins have no dependencies). @@ -69,8 +73,8 @@ const plugins = createPlugins( ); const server = createHost({ onError: () => {} }); const worker = createHost({ onError: () => {} }); -await plugins.load(server, { kind: "server", core: [] }, context); -await plugins.load(worker, { kind: "worker", core: [] }, context); +await plugins.load(server, { kind: "server", core: [], modules }, context); +await plugins.load(worker, { kind: "worker", core: [], modules }, context); const steps = [plugins.step("server", server as Host), plugins.step("worker", worker as Host)]; console.log("1. loaded:", plugins.discovery.plugins.map(({ name }) => name).join(", "), "| marks:", marks()); @@ -79,7 +83,7 @@ await edit("plugins/notes/label.ts", "notes v1", "notes v2"); await dirty("@example/notes"); console.log("2. a file of notes edited:", marks()); await edit("plugins/contracts/contract.ts", "one type per plugin", "one type per plugin (edited)"); -await dirty("@example/contracts", "@example/tags"); +await dirty("@example/shared", "@example/tags"); console.log(" the shared contract edited:", marks()); await edit("plugins/tags/package.json", '"private": true', '"private": true,\n\t"description": "tags"'); await until(() => plugins.state.value.install.includes("@example/tags")); diff --git a/packages/chord/test/next/examples/38-watch-and-rollout/plugins/contracts/package.json b/packages/chord/test/next/examples/38-watch-and-rollout/plugins/contracts/package.json index d1c133527..dfc4cefdc 100644 --- a/packages/chord/test/next/examples/38-watch-and-rollout/plugins/contracts/package.json +++ b/packages/chord/test/next/examples/38-watch-and-rollout/plugins/contracts/package.json @@ -1,5 +1,5 @@ { - "name": "@example/contracts", + "name": "@example/shared", "private": true, "type": "module" } diff --git a/packages/chord/test/next/examples/38-watch-and-rollout/plugins/notes/server.ts b/packages/chord/test/next/examples/38-watch-and-rollout/plugins/notes/server.ts index 23a936014..5670a4766 100644 --- a/packages/chord/test/next/examples/38-watch-and-rollout/plugins/notes/server.ts +++ b/packages/chord/test/next/examples/38-watch-and-rollout/plugins/notes/server.ts @@ -1,7 +1,7 @@ // The notes plugin's server facet: its label. import { constant, defineFacet } from "@earendil-works/chord/next"; -import { labelOf } from "../contracts/contract.ts"; +import { labelOf } from "@example/shared/contract"; import { label } from "./label.ts"; export default defineFacet((host) => diff --git a/packages/chord/test/next/examples/38-watch-and-rollout/plugins/notes/worker.ts b/packages/chord/test/next/examples/38-watch-and-rollout/plugins/notes/worker.ts index e01fc1159..58d814c27 100644 --- a/packages/chord/test/next/examples/38-watch-and-rollout/plugins/notes/worker.ts +++ b/packages/chord/test/next/examples/38-watch-and-rollout/plugins/notes/worker.ts @@ -1,7 +1,7 @@ // The notes plugin's worker facet: its label. import { constant, defineFacet } from "@earendil-works/chord/next"; -import { labelOf } from "../contracts/contract.ts"; +import { labelOf } from "@example/shared/contract"; import { label } from "./label.ts"; export default defineFacet((host) => diff --git a/packages/chord/test/next/examples/38-watch-and-rollout/plugins/old/server.ts b/packages/chord/test/next/examples/38-watch-and-rollout/plugins/old/server.ts index afb0165f9..3d87897be 100644 --- a/packages/chord/test/next/examples/38-watch-and-rollout/plugins/old/server.ts +++ b/packages/chord/test/next/examples/38-watch-and-rollout/plugins/old/server.ts @@ -1,6 +1,6 @@ // The old plugin's server facet; the example deletes the plugin's directory. import { constant, defineFacet } from "@earendil-works/chord/next"; -import { labelOf } from "../contracts/contract.ts"; +import { labelOf } from "@example/shared/contract"; export default defineFacet((host) => host.serve(labelOf("old"), () => ({ text: constant("old on the server") }))); diff --git a/packages/chord/test/next/examples/38-watch-and-rollout/plugins/tags/server.ts b/packages/chord/test/next/examples/38-watch-and-rollout/plugins/tags/server.ts index 8b8ce9d68..64210746e 100644 --- a/packages/chord/test/next/examples/38-watch-and-rollout/plugins/tags/server.ts +++ b/packages/chord/test/next/examples/38-watch-and-rollout/plugins/tags/server.ts @@ -1,6 +1,6 @@ // The tags plugin's server facet. import { constant, defineFacet } from "@earendil-works/chord/next"; -import { labelOf } from "../contracts/contract.ts"; +import { labelOf } from "@example/shared/contract"; export default defineFacet((host) => host.serve(labelOf("tags"), () => ({ text: constant("tags on the server") }))); diff --git a/packages/chord/test/next/examples/38-watch-and-rollout/plugins/tags/worker.ts b/packages/chord/test/next/examples/38-watch-and-rollout/plugins/tags/worker.ts index 7cbeb3ae4..796e053a3 100644 --- a/packages/chord/test/next/examples/38-watch-and-rollout/plugins/tags/worker.ts +++ b/packages/chord/test/next/examples/38-watch-and-rollout/plugins/tags/worker.ts @@ -1,6 +1,6 @@ // The tags plugin's worker facet. import { constant, defineFacet } from "@earendil-works/chord/next"; -import { labelOf } from "../contracts/contract.ts"; +import { labelOf } from "@example/shared/contract"; export default defineFacet((host) => host.serve(labelOf("tags"), () => ({ text: constant("tags on the worker") }))); diff --git a/packages/chord/test/next/examples/39-browser-build/main.ts b/packages/chord/test/next/examples/39-browser-build/main.ts index 484951d89..f7a05af4d 100644 --- a/packages/chord/test/next/examples/39-browser-build/main.ts +++ b/packages/chord/test/next/examples/39-browser-build/main.ts @@ -1,4 +1,4 @@ -// A browser build (slice 7c §2): a facet of a browser kind (`browserKinds: ["web"]`) is built for the browser. Its +// A browser build (slice 7c §2): a facet of a browser kind (`web: { platform: "browser", … }`) is built for the browser. Its // stylesheets become one CSS file (`css`), an asset imported by URL (`./logo.svg?url`) a content-hashed file whose // import evaluates to its URL under the app's `publicPath`. The build is plain data: script, stylesheet and assets, // what the app serves to its pages (slice 11 ships it); a page evaluates the script with its host packages, as Node @@ -20,10 +20,8 @@ const hostPackages = { "@earendil-works/chord/next": chordNext }; const plugins = createPlugins( { roots: [root], - kinds: ["web"], - browserKinds: ["web"], + kinds: { web: { platform: "browser", hostPackages: Object.keys(hostPackages) } }, publicPath: "/plugin-assets/", - hostPackages, onError: (error) => console.log("plugins:", error.kind), }, context, diff --git a/packages/chord/test/next/examples/40-http/main.ts b/packages/chord/test/next/examples/40-http/main.ts new file mode 100644 index 000000000..78dbca808 --- /dev/null +++ b/packages/chord/test/next/examples/40-http/main.ts @@ -0,0 +1,56 @@ +// HTTP (slice 10): a server answers HTTP requests with routes that plugins contribute. The notes plugin (`plugins/`) +// contributes a page, a JSON route and a file download; the server's own code only builds the component, says where +// Chord's own listener listens, and loads the plugins. Every answer carries the security baseline's headers. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/40-http/main.ts +import { fileURLToPath } from "node:url"; +import { BACKGROUND_CONTEXT as context } from "../../../../src/context/index.ts"; +import { createHost } from "../../../../src/next/host/index.ts"; +import * as chordNext from "../../../../src/next/index.ts"; +import * as chordHttp from "../../../../src/next-node/http/index.ts"; +import { createPlugins } from "../../../../src/next-node/plugins/index.ts"; + +// 1. A server host and its HTTP component; its settings are arguments: Chord's own listener listens on a free +// loopback port. Nobody signs in here: no authenticator, every request is anonymous. +const server = createHost({ principal: { kind: "server" }, onError: console.error }); +const http = chordHttp.createHttp(server, { listen: { host: "127.0.0.1", port: 0 }, onError: console.error }, context); + +// 2. The plugins: the component's facet (its registries and the listener) goes first, then every plugin's server +// facet. The listener opens once they are published. +const modules = { "@earendil-works/chord/next": chordNext, "@earendil-works/chord/next/http": chordHttp }; +const plugins = createPlugins( + { + roots: [fileURLToPath(new URL("./plugins/", import.meta.url))], + kinds: { server: { platform: "node", hostPackages: Object.keys(modules) } }, + onError: console.error, + }, + context, +); +await plugins.load(server, { kind: "server", core: [{ name: "chord.http", facet: http.facet }], modules }, context); +const { port } = await http.listening(context); + +// 3. Some requests, printed like curl would. +async function curl(path: string, headers: Record = {}): Promise { + const response = await fetch(`http://127.0.0.1:${port}${path}`, { headers }); + const range = headers.Range === undefined ? "" : ` (Range: ${headers.Range})`; + console.log(`GET ${path}${range} -> ${response.status} ${response.headers.get("content-type")}`); + const download = response.headers.get("content-disposition"); + if (download !== null) console.log(` ${download}`); + console.log(` ${JSON.stringify((await response.text()).replace(/nonce="[^"]+"/, 'nonce="…"'))}`); +} + +await curl("/"); +await curl("/notes/hello"); +await curl("/notes/hello/download", { Range: "bytes=0-6" }); +await curl("/notes/nope/download"); +await curl("/nowhere"); +const baseline = await fetch(`http://127.0.0.1:${port}/notes/hello`); +for (const name of ["cache-control", "x-content-type-options", "x-frame-options"]) { + console.log(` ${name}: ${baseline.headers.get(name)}`); +} + +// 4. Close the component first: it refuses new requests and waits for running ones (the host's close would seal them +// at once). Then the host. +await http.close(context); +await server.close(context); +plugins.close(); diff --git a/packages/chord/test/next/examples/40-http/plugins/notes/files/hello.md b/packages/chord/test/next/examples/40-http/plugins/notes/files/hello.md new file mode 100644 index 000000000..13cbb3050 --- /dev/null +++ b/packages/chord/test/next/examples/40-http/plugins/notes/files/hello.md @@ -0,0 +1,3 @@ +# Hello + +A note served as a download. diff --git a/packages/chord/test/next/examples/40-http/plugins/notes/package.json b/packages/chord/test/next/examples/40-http/plugins/notes/package.json new file mode 100644 index 000000000..d59f5a427 --- /dev/null +++ b/packages/chord/test/next/examples/40-http/plugins/notes/package.json @@ -0,0 +1,5 @@ +{ + "name": "@example/notes", + "private": true, + "type": "module" +} diff --git a/packages/chord/test/next/examples/40-http/plugins/notes/server.ts b/packages/chord/test/next/examples/40-http/plugins/notes/server.ts new file mode 100644 index 000000000..48e6b9388 --- /dev/null +++ b/packages/chord/test/next/examples/40-http/plugins/notes/server.ts @@ -0,0 +1,29 @@ +// The notes plugin's server facet: a page, a JSON route and a file download, contributed to the HTTP component's +// registry. + +import { defineFacet } from "@earendil-works/chord/next"; +import { fileResponse, Http } from "@earendil-works/chord/next/http"; +import { pluginFile } from "@earendil-works/chord/next/plugins"; + +export default defineFacet((_host, self) => + self.contribute(Http, { + routes: { + // A page: a link from another site may open it. Its inline script carries the server's nonce (the CSP). + "GET /": { + page: true, + handler: async (request) => ({ + kind: "html", + body: `

    Notes

    `, + }), + }, + // `:name` is one path segment, decoded. + "GET /notes/:name": async (request) => ({ kind: "json", value: { note: request.params.name } }), + // A file of this plugin, saved under its name, with ranges. + "GET /notes/:name/download": async (request) => + fileResponse(pluginFile(`files/${request.params.name}.md`), { + download: `${request.params.name}.md`, + ranges: true, + }), + }, + }), +); diff --git a/packages/chord/test/next/examples/41-workers/main.ts b/packages/chord/test/next/examples/41-workers/main.ts new file mode 100644 index 000000000..35460577b --- /dev/null +++ b/packages/chord/test/next/examples/41-workers/main.ts @@ -0,0 +1,68 @@ +// Workers (slice 10): the server routes a session's address to its own worker process. The first reference to a +// session starts its worker; once nobody uses it for `idleMs`, the worker stops. The worker's code is the sessions +// plugin (`plugins/`), built by the server and shipped to the worker. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/41-workers/main.ts +import { fileURLToPath } from "node:url"; +import { BACKGROUND_CONTEXT as context } from "../../../../src/context/index.ts"; +import { hostKind } from "../../../../src/next/contracts/index.ts"; +import { createHost } from "../../../../src/next/host/index.ts"; +import { createPlugins } from "../../../../src/next-node/plugins/index.ts"; +import { createWorkers } from "../../../../src/next-node/workers/index.ts"; +import { Greeting, Session } from "./plugins/sessions/contract.ts"; + +const plugins = createPlugins( + { + roots: [fileURLToPath(new URL("./plugins/", import.meta.url))], + kinds: { worker: { platform: "node", hostPackages: ["@earendil-works/chord/next"] } }, + onError: console.error, + }, + context, +); + +// 1. The server, and its workers: one forked process per session, stopped after 300 ms without use. +const server = createHost({ address: hostKind("server")(), principal: { kind: "server" }, onError: console.error }); +const workers = createWorkers( + server, + Session, + { + kind: "worker", + entry: fileURLToPath(new URL("./worker.ts", import.meta.url)), + execArgv: ["--conditions=source", "--experimental-strip-types", "--no-warnings"], + authorize: () => true, + principal: (address) => ({ kind: "worker", session: address.params.id }), + start: () => ({}), + lock: (address) => `example-41:${process.pid}:${address.key}`, + idleMs: () => 300, + plugins, + onError: console.error, + }, + context, +); +await server.load([], context); + +// 2. Every change of the workers' state, printed. +const watch = await workers.state.watch(context); +watch.start(async (call) => { + if (call.kind !== "value") return; + const entries = Object.entries(call.value).map(([key, state]) => `${key} ${state.status}`); + console.log(` workers: ${entries.join(", ") || "none"}`); +}); + +// 3. The first reference starts the worker; the greeting comes from that process. +console.log("1. getting S1's greeting"); +const greeting = await server.get(Greeting, { id: "S1" }, context); +console.log(`2. "${greeting.text.value}" (worker pid ${greeting.pid.value}, server pid ${process.pid})`); + +// 4. Releasing it leaves the worker idle: 300 ms later it stops. +greeting.release(); +console.log("3. released; waiting for the idle stop"); +while (Object.keys(workers.state.value).length > 0) { + await new Promise((resolve) => setTimeout(resolve, 50)); +} +console.log("4. stopped"); + +await watch.stop(); +await workers.close(context); +await server.close(context); +plugins.close(); diff --git a/packages/chord/test/next/examples/41-workers/plugins/sessions/contract.ts b/packages/chord/test/next/examples/41-workers/plugins/sessions/contract.ts new file mode 100644 index 000000000..0ef89bfff --- /dev/null +++ b/packages/chord/test/next/examples/41-workers/plugins/sessions/contract.ts @@ -0,0 +1,13 @@ +// A session lives in its own worker process: `Session({ id })` is its address. + +import { hostKind, int, remote, str } from "@earendil-works/chord/next"; + +export const Session = hostKind("session", { id: str() }); + +/** A session's greeting, served by the worker at the session's address. */ +export const Greeting = remote("example.greeting", { + version: 1, + params: { id: str() }, + hostAddress: ({ id }) => Session({ id }), + variants: { main: { state: { text: str(), pid: int() } } }, +}); diff --git a/packages/chord/test/next/examples/41-workers/plugins/sessions/package.json b/packages/chord/test/next/examples/41-workers/plugins/sessions/package.json new file mode 100644 index 000000000..3819a6f05 --- /dev/null +++ b/packages/chord/test/next/examples/41-workers/plugins/sessions/package.json @@ -0,0 +1,5 @@ +{ + "name": "@example/sessions", + "private": true, + "type": "module" +} diff --git a/packages/chord/test/next/examples/41-workers/plugins/sessions/worker.ts b/packages/chord/test/next/examples/41-workers/plugins/sessions/worker.ts new file mode 100644 index 000000000..79ee9efca --- /dev/null +++ b/packages/chord/test/next/examples/41-workers/plugins/sessions/worker.ts @@ -0,0 +1,11 @@ +// The sessions plugin's worker facet: each worker greets with its process id. + +import { constant, defineFacet } from "@earendil-works/chord/next"; +import { Greeting } from "./contract.ts"; + +export default defineFacet((host) => + host.serve(Greeting, ({ id }) => ({ + variant: "main", + members: { text: constant(`hello from session ${id}`), pid: constant(process.pid) }, + })), +); diff --git a/packages/chord/test/next/examples/41-workers/worker.ts b/packages/chord/test/next/examples/41-workers/worker.ts new file mode 100644 index 000000000..2fb4fccaa --- /dev/null +++ b/packages/chord/test/next/examples/41-workers/worker.ts @@ -0,0 +1,22 @@ +// The worker process: a host at the session's address, connected to the server, loading the plugins it was shipped. + +import { BACKGROUND_CONTEXT } from "../../../../src/context/index.ts"; +import { createHost } from "../../../../src/next/host/index.ts"; +import * as chordNext from "../../../../src/next/index.ts"; +import { workerMain } from "../../../../src/next-node/workers/index.ts"; + +workerMain( + async ({ address, parent, plugins, onClose }, context) => { + const host = createHost({ address, principal: { kind: "worker" }, onError: console.error }); + onClose(() => host.close(BACKGROUND_CONTEXT)); + // Connect first: the server sees the worker started; then load. + host.connect(parent, { principal: { kind: "server" }, trusted: true }, context); + await plugins.load(host, { kind: "worker", core: [] }, context); + return {}; + }, + { + hostPackages: { "@earendil-works/chord/next": chordNext }, + redact: (error) => ({ message: String(error) }), + onError: console.error, + }, +); diff --git a/packages/chord/test/next/examples/42-worker-crash/main.ts b/packages/chord/test/next/examples/42-worker-crash/main.ts new file mode 100644 index 000000000..37297f924 --- /dev/null +++ b/packages/chord/test/next/examples/42-worker-crash/main.ts @@ -0,0 +1,67 @@ +// A worker crash (slice 10): workers never restart by themselves. A crashed worker's address stays failed, its +// holders see `failed` with the error, and nothing forks until the app retries (a Retry button) or a rollout ships new +// code. Here: a session crashes, stays failed for a while, then Retry brings it back. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/42-worker-crash/main.ts +import { fileURLToPath } from "node:url"; +import { BACKGROUND_CONTEXT as context } from "../../../../src/context/index.ts"; +import { hostKind } from "../../../../src/next/contracts/index.ts"; +import { createHost } from "../../../../src/next/host/index.ts"; +import { createPlugins } from "../../../../src/next-node/plugins/index.ts"; +import { createWorkers } from "../../../../src/next-node/workers/index.ts"; +import { Process, Session } from "./plugins/sessions/contract.ts"; + +const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); + +const plugins = createPlugins( + { + roots: [fileURLToPath(new URL("./plugins/", import.meta.url))], + kinds: { worker: { platform: "node", hostPackages: ["@earendil-works/chord/next"] } }, + onError: console.error, + }, + context, +); +const server = createHost({ address: hostKind("server")(), principal: { kind: "server" }, onError: () => {} }); +const workers = createWorkers( + server, + Session, + { + kind: "worker", + entry: fileURLToPath(new URL("./worker.ts", import.meta.url)), + execArgv: ["--conditions=source", "--experimental-strip-types", "--no-warnings"], + authorize: () => true, + principal: (address) => ({ kind: "worker", session: address.params.id }), + start: () => ({}), + lock: (address) => `example-42:${process.pid}:${address.key}`, + plugins, + onError: (error) => console.log(` reported: ${error.kind}`), + }, + context, +); +await server.load([], context); + +// 1. A held session (a tab showing it): its worker runs. +const session = server.hold(Process, { id: "S1" }, context); +const state = () => session.state.value.kind; +while (state() !== "live") await sleep(10); +console.log(`1. S1 runs in process ${session.pid.value}`); + +// 2. It crashes. The address is failed, and the holder sees why; it stays on the address's path. +await session.crash(context); +while (state() !== "failed") await sleep(10); +const failed = workers.state.value["session|S1"]; +console.log(`2. crashed: ${failed?.status === "failed" ? failed.error.message : failed?.status}`); + +// 3. No restart loop: a second later it is still failed, and no other worker was started. +await sleep(1_000); +console.log(`3. a second later: ${workers.state.value["session|S1"]?.status}, the holder: ${state()}`); + +// 4. Retry (the app's button): the worker starts again and the same holder binds to it. +await workers.retry(Session({ id: "S1" }), context); +while (state() !== "live") await sleep(10); +console.log(`4. after Retry: S1 runs in process ${session.pid.value}`); + +session.release(); +await workers.close(context); +await server.close(context); +plugins.close(); diff --git a/packages/chord/test/next/examples/42-worker-crash/plugins/sessions/contract.ts b/packages/chord/test/next/examples/42-worker-crash/plugins/sessions/contract.ts new file mode 100644 index 000000000..0a995c248 --- /dev/null +++ b/packages/chord/test/next/examples/42-worker-crash/plugins/sessions/contract.ts @@ -0,0 +1,13 @@ +// A session lives in its own worker process: `Session({ id })` is its address. + +import { hostKind, int, method, remote, str } from "@earendil-works/chord/next"; + +export const Session = hostKind("session", { id: str() }); + +/** A session's process: its pid, and a way to make it crash. */ +export const Process = remote("example.process", { + version: 1, + params: { id: str() }, + hostAddress: ({ id }) => Session({ id }), + variants: { main: { state: { pid: int() }, methods: { crash: method() } } }, +}); diff --git a/packages/chord/test/next/examples/42-worker-crash/plugins/sessions/package.json b/packages/chord/test/next/examples/42-worker-crash/plugins/sessions/package.json new file mode 100644 index 000000000..3819a6f05 --- /dev/null +++ b/packages/chord/test/next/examples/42-worker-crash/plugins/sessions/package.json @@ -0,0 +1,5 @@ +{ + "name": "@example/sessions", + "private": true, + "type": "module" +} diff --git a/packages/chord/test/next/examples/42-worker-crash/plugins/sessions/worker.ts b/packages/chord/test/next/examples/42-worker-crash/plugins/sessions/worker.ts new file mode 100644 index 000000000..cb8cae63c --- /dev/null +++ b/packages/chord/test/next/examples/42-worker-crash/plugins/sessions/worker.ts @@ -0,0 +1,17 @@ +// The sessions plugin's worker facet: its process id, and `crash`, which ends the process (a bug, an out-of-memory). + +import { constant, defineFacet } from "@earendil-works/chord/next"; +import { Process } from "./contract.ts"; + +export default defineFacet((host) => + host.serve(Process, () => ({ + variant: "main", + members: { + pid: constant(process.pid), + crash: async () => { + setTimeout(() => process.exit(13), 10); + return null; + }, + }, + })), +); diff --git a/packages/chord/test/next/examples/42-worker-crash/worker.ts b/packages/chord/test/next/examples/42-worker-crash/worker.ts new file mode 100644 index 000000000..2fb4fccaa --- /dev/null +++ b/packages/chord/test/next/examples/42-worker-crash/worker.ts @@ -0,0 +1,22 @@ +// The worker process: a host at the session's address, connected to the server, loading the plugins it was shipped. + +import { BACKGROUND_CONTEXT } from "../../../../src/context/index.ts"; +import { createHost } from "../../../../src/next/host/index.ts"; +import * as chordNext from "../../../../src/next/index.ts"; +import { workerMain } from "../../../../src/next-node/workers/index.ts"; + +workerMain( + async ({ address, parent, plugins, onClose }, context) => { + const host = createHost({ address, principal: { kind: "worker" }, onError: console.error }); + onClose(() => host.close(BACKGROUND_CONTEXT)); + // Connect first: the server sees the worker started; then load. + host.connect(parent, { principal: { kind: "server" }, trusted: true }, context); + await plugins.load(host, { kind: "worker", core: [] }, context); + return {}; + }, + { + hostPackages: { "@earendil-works/chord/next": chordNext }, + redact: (error) => ({ message: String(error) }), + onError: console.error, + }, +); diff --git a/packages/chord/test/next/examples/43-rollout/main.ts b/packages/chord/test/next/examples/43-rollout/main.ts new file mode 100644 index 000000000..5fd443756 --- /dev/null +++ b/packages/chord/test/next/examples/43-rollout/main.ts @@ -0,0 +1,85 @@ +// A rollout across the server and its workers (slice 10): an edit of a plugin is built once by the server and rolled +// out step by step, the server first, then every running worker; each host reports what changed. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/43-rollout/main.ts +import { cp, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { BACKGROUND_CONTEXT as context } from "../../../../src/context/index.ts"; +import { hostKind } from "../../../../src/next/contracts/index.ts"; +import { createHost } from "../../../../src/next/host/index.ts"; +import * as chordNext from "../../../../src/next/index.ts"; +import { createPlugins, type RolloutHost } from "../../../../src/next-node/plugins/index.ts"; +import { createWorkers } from "../../../../src/next-node/workers/index.ts"; +import { Banner, Label, Session } from "./plugins/notes/contract.ts"; + +/** Waits until `done()` holds (a watcher's mark, a frame from a worker), checking every 10 ms. */ +async function waitUntil(done: () => boolean): Promise { + while (!done()) await new Promise((resolve) => setTimeout(resolve, 10)); +} + +/** One line for what a rollout step did on one of its hosts. */ +function describe(host: RolloutHost): string { + if ("report" in host) return `changed ${host.report.plugins.changed.join(", ")}`; + if ("stopped" in host) return "stopped"; + return "failed"; +} + +// The example edits a plugin: it works on a copy of `plugins/`. +const root = await mkdtemp(join(tmpdir(), "chord-example-43-")); +await cp(fileURLToPath(new URL("./plugins/", import.meta.url)), root, { recursive: true }); + +const node = { platform: "node", hostPackages: ["@earendil-works/chord/next"] } as const; +const plugins = createPlugins( + { roots: [root], kinds: { server: node, worker: node }, onError: console.error }, + context, +); +const server = createHost({ address: hostKind("server")(), principal: { kind: "server" }, onError: console.error }); +const workers = createWorkers( + server, + Session, + { + kind: "worker", + entry: fileURLToPath(new URL("./worker.ts", import.meta.url)), + execArgv: ["--conditions=source", "--experimental-strip-types", "--no-warnings"], + authorize: () => true, + principal: (address) => ({ kind: "worker", session: address.params.id }), + start: () => ({}), + lock: (address) => `example-43:${process.pid}:${address.key}`, + plugins, + onError: console.error, + }, + context, +); +await plugins.load(server, { kind: "server", core: [], modules: { "@earendil-works/chord/next": chordNext } }, context); + +// 1. The server's banner and two running sessions. +const banner = await server.get(Banner, {}, context); +const s1 = await server.get(Label, { id: "S1" }, context); +const s2 = await server.get(Label, { id: "S2" }, context); +console.log(`1. ${banner.text.value} | ${s1.text.value} | ${s2.text.value}`); + +// 2. An edit: the watcher marks the plugin; one build serves both kinds. +await writeFile(join(root, "notes", "label.ts"), 'export const label = "v2";\n'); +await waitUntil(() => plugins.state.value.dirty.includes("@example/notes")); +const built = await plugins.build(plugins.state.value.dirty, context); + +// 3. The rollout: the server, then the workers (each running one reloads in parallel). +const result = await plugins.rollout([plugins.step("server", server), workers.step("workers")], built, context); +for (const step of result.steps) { + for (const host of step.hosts) console.log(`2. ${step.name}: ${host.host} ${describe(host)}`); +} + +// 4. Every host shows the new label (the workers' frames arrive over their links). +await waitUntil(() => s2.text.value === "S2: v2"); +console.log(`3. ${banner.text.value} | ${s1.text.value} | ${s2.text.value}`); +console.log(`4. still dirty: ${JSON.stringify(plugins.state.value.dirty)}`); + +banner.release(); +s1.release(); +s2.release(); +await workers.close(context); +await server.close(context); +plugins.close(); +await rm(root, { recursive: true, force: true }); diff --git a/packages/chord/test/next/examples/43-rollout/plugins/notes/contract.ts b/packages/chord/test/next/examples/43-rollout/plugins/notes/contract.ts new file mode 100644 index 000000000..86d192d7e --- /dev/null +++ b/packages/chord/test/next/examples/43-rollout/plugins/notes/contract.ts @@ -0,0 +1,16 @@ +// The notes' contracts: a label on the server, and one in each session's worker. + +import { hostKind, local, remote, str } from "@earendil-works/chord/next"; + +export const Session = hostKind("session", { id: str() }); + +/** The label as the server shows it. */ +export const Banner = local("example.banner", { version: 1, params: {}, members: { text: str() } }); + +/** The label as a session's worker shows it. */ +export const Label = remote("example.label", { + version: 1, + params: { id: str() }, + hostAddress: ({ id }) => Session({ id }), + variants: { main: { state: { text: str() } } }, +}); diff --git a/packages/chord/test/next/examples/43-rollout/plugins/notes/label.ts b/packages/chord/test/next/examples/43-rollout/plugins/notes/label.ts new file mode 100644 index 000000000..c8c560a6e --- /dev/null +++ b/packages/chord/test/next/examples/43-rollout/plugins/notes/label.ts @@ -0,0 +1,3 @@ +// The label every host shows; the example edits it. + +export const label = "v1"; diff --git a/packages/chord/test/next/examples/43-rollout/plugins/notes/package.json b/packages/chord/test/next/examples/43-rollout/plugins/notes/package.json new file mode 100644 index 000000000..d59f5a427 --- /dev/null +++ b/packages/chord/test/next/examples/43-rollout/plugins/notes/package.json @@ -0,0 +1,5 @@ +{ + "name": "@example/notes", + "private": true, + "type": "module" +} diff --git a/packages/chord/test/next/examples/43-rollout/plugins/notes/server.ts b/packages/chord/test/next/examples/43-rollout/plugins/notes/server.ts new file mode 100644 index 000000000..ca8c22e00 --- /dev/null +++ b/packages/chord/test/next/examples/43-rollout/plugins/notes/server.ts @@ -0,0 +1,7 @@ +// The notes plugin's server facet. + +import { constant, defineFacet } from "@earendil-works/chord/next"; +import { Banner } from "./contract.ts"; +import { label } from "./label.ts"; + +export default defineFacet((host) => host.serve(Banner, () => ({ text: constant(`server: ${label}`) }))); diff --git a/packages/chord/test/next/examples/43-rollout/plugins/notes/worker.ts b/packages/chord/test/next/examples/43-rollout/plugins/notes/worker.ts new file mode 100644 index 000000000..972e91925 --- /dev/null +++ b/packages/chord/test/next/examples/43-rollout/plugins/notes/worker.ts @@ -0,0 +1,9 @@ +// The notes plugin's worker facet. + +import { constant, defineFacet } from "@earendil-works/chord/next"; +import { Label } from "./contract.ts"; +import { label } from "./label.ts"; + +export default defineFacet((host) => + host.serve(Label, ({ id }) => ({ variant: "main", members: { text: constant(`${id}: ${label}`) } })), +); diff --git a/packages/chord/test/next/examples/43-rollout/worker.ts b/packages/chord/test/next/examples/43-rollout/worker.ts new file mode 100644 index 000000000..2fb4fccaa --- /dev/null +++ b/packages/chord/test/next/examples/43-rollout/worker.ts @@ -0,0 +1,22 @@ +// The worker process: a host at the session's address, connected to the server, loading the plugins it was shipped. + +import { BACKGROUND_CONTEXT } from "../../../../src/context/index.ts"; +import { createHost } from "../../../../src/next/host/index.ts"; +import * as chordNext from "../../../../src/next/index.ts"; +import { workerMain } from "../../../../src/next-node/workers/index.ts"; + +workerMain( + async ({ address, parent, plugins, onClose }, context) => { + const host = createHost({ address, principal: { kind: "worker" }, onError: console.error }); + onClose(() => host.close(BACKGROUND_CONTEXT)); + // Connect first: the server sees the worker started; then load. + host.connect(parent, { principal: { kind: "server" }, trusted: true }, context); + await plugins.load(host, { kind: "worker", core: [] }, context); + return {}; + }, + { + hostPackages: { "@earendil-works/chord/next": chordNext }, + redact: (error) => ({ message: String(error) }), + onError: console.error, + }, +); diff --git a/packages/chord/test/next/examples/44-http-wraps/main.ts b/packages/chord/test/next/examples/44-http-wraps/main.ts new file mode 100644 index 000000000..af970636c --- /dev/null +++ b/packages/chord/test/next/examples/44-http-wraps/main.ts @@ -0,0 +1,37 @@ +// Wraps (slice 10): plugins wrap routes, their own or everyone's, in phases (`observe`, `headers`, `transform`, +// outermost first). Here the notes plugin wraps its own route to add a header, and an audit plugin wraps every route +// to log it; the audit plugin needs no route of its own. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/44-http-wraps/main.ts +import { fileURLToPath } from "node:url"; +import { BACKGROUND_CONTEXT as context } from "../../../../src/context/index.ts"; +import { createHost } from "../../../../src/next/host/index.ts"; +import * as chordNext from "../../../../src/next/index.ts"; +import * as chordHttp from "../../../../src/next-node/http/index.ts"; +import { createPlugins } from "../../../../src/next-node/plugins/index.ts"; + +const server = createHost({ principal: { kind: "server" }, onError: console.error }); +const http = chordHttp.createHttp(server, { listen: { host: "127.0.0.1", port: 0 }, onError: console.error }, context); +const modules = { "@earendil-works/chord/next": chordNext, "@earendil-works/chord/next/http": chordHttp }; +const plugins = createPlugins( + { + roots: [fileURLToPath(new URL("./plugins/", import.meta.url))], + kinds: { server: { platform: "node", hostPackages: Object.keys(modules) } }, + onError: console.error, + }, + context, +); +await plugins.load(server, { kind: "server", core: [{ name: "chord.http", facet: http.facet }], modules }, context); +const { port } = await http.listening(context); + +for (const path of ["/notes", "/missing"]) { + console.log(`GET ${path}`); + const response = await fetch(`http://127.0.0.1:${port}${path}`); + console.log(` -> ${response.status}, Cache-Control: ${response.headers.get("cache-control")}`); +} +// The 404 has no route, so no wrap ran for it: the audit log shows only /notes, and it keeps the default +// `Cache-Control: no-store` that the notes wrap replaced on its own route. + +await http.close(context); +await server.close(context); +plugins.close(); diff --git a/packages/chord/test/next/examples/44-http-wraps/plugins/audit/package.json b/packages/chord/test/next/examples/44-http-wraps/plugins/audit/package.json new file mode 100644 index 000000000..10a68118f --- /dev/null +++ b/packages/chord/test/next/examples/44-http-wraps/plugins/audit/package.json @@ -0,0 +1,5 @@ +{ + "name": "@example/audit", + "private": true, + "type": "module" +} diff --git a/packages/chord/test/next/examples/44-http-wraps/plugins/audit/server.ts b/packages/chord/test/next/examples/44-http-wraps/plugins/audit/server.ts new file mode 100644 index 000000000..e4259d470 --- /dev/null +++ b/packages/chord/test/next/examples/44-http-wraps/plugins/audit/server.ts @@ -0,0 +1,24 @@ +// The audit plugin: no routes of its own, one wrap around every route (`scope: "all"`), outermost (`observe`). + +import { defineFacet } from "@earendil-works/chord/next"; +import { Http } from "@earendil-works/chord/next/http"; + +export default defineFacet((_host, self) => + self.contribute(Http, { + wraps: [ + { + name: "log", + phase: "observe", + scope: "all", + wrap: async (request, next, ctx) => { + const response = await next(request); + const status = "status" in response ? response.status : 200; + console.log( + ` audit: ${ctx.principal === undefined ? "anonymous" : "someone"} ${request.method} ${request.path} ${status}`, + ); + return response; + }, + }, + ], + }), +); diff --git a/packages/chord/test/next/examples/44-http-wraps/plugins/notes/package.json b/packages/chord/test/next/examples/44-http-wraps/plugins/notes/package.json new file mode 100644 index 000000000..d59f5a427 --- /dev/null +++ b/packages/chord/test/next/examples/44-http-wraps/plugins/notes/package.json @@ -0,0 +1,5 @@ +{ + "name": "@example/notes", + "private": true, + "type": "module" +} diff --git a/packages/chord/test/next/examples/44-http-wraps/plugins/notes/server.ts b/packages/chord/test/next/examples/44-http-wraps/plugins/notes/server.ts new file mode 100644 index 000000000..0c2a83103 --- /dev/null +++ b/packages/chord/test/next/examples/44-http-wraps/plugins/notes/server.ts @@ -0,0 +1,20 @@ +// The notes plugin: a route, and its own wrap (around its own routes only) adding a header. + +import { defineFacet } from "@earendil-works/chord/next"; +import { Http } from "@earendil-works/chord/next/http"; + +export default defineFacet((_host, self) => + self.contribute(Http, { + routes: { "GET /notes": async () => ({ kind: "json", value: ["buy milk", "call mom"] }) }, + wraps: [ + { + name: "cache", + phase: "headers", + wrap: async (request, next) => { + const response = await next(request); + return { ...response, headers: { ...response.headers, "Cache-Control": "max-age=60" } }; + }, + }, + ], + }), +); diff --git a/packages/chord/test/next/examples/45-plugin-socket/main.ts b/packages/chord/test/next/examples/45-plugin-socket/main.ts new file mode 100644 index 000000000..9f10cd8b8 --- /dev/null +++ b/packages/chord/test/next/examples/45-plugin-socket/main.ts @@ -0,0 +1,64 @@ +// A plugin socket (slice 10): a plugin contributes a WebSocket endpoint for its own protocol (a terminal, voice). The +// server checks the origin and authenticates first: without a principal the upgrade is refused before plugin code. +// Here the app's authenticator reads a principal from a header (a real app: a signed cookie, `readCookie`). +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/45-plugin-socket/main.ts +import { once } from "node:events"; +import { fileURLToPath } from "node:url"; +import { WebSocket } from "ws"; +import { BACKGROUND_CONTEXT as context } from "../../../../src/context/index.ts"; +import { createHost, defineFacet } from "../../../../src/next/host/index.ts"; +import * as chordNext from "../../../../src/next/index.ts"; +import * as chordHttp from "../../../../src/next-node/http/index.ts"; +import { createPlugins } from "../../../../src/next-node/plugins/index.ts"; + +type Principal = { readonly kind: "server" } | { readonly kind: "user"; readonly name: string }; +const server = createHost({ principal: { kind: "server" }, onError: console.error }); +const http = chordHttp.createHttp(server, { listen: { host: "127.0.0.1", port: 0 }, onError: console.error }, context); +// The app's core facet: who someone is. +const app = defineFacet((_host, self) => { + self.contribute(chordHttp.Authenticators, (request) => { + const name = request.headers.get("x-user"); + return name === null ? undefined : { kind: "user", name }; + }); +}); +const modules = { "@earendil-works/chord/next": chordNext, "@earendil-works/chord/next/http": chordHttp }; +const plugins = createPlugins( + { + roots: [fileURLToPath(new URL("./plugins/", import.meta.url))], + kinds: { server: { platform: "node", hostPackages: Object.keys(modules) } }, + onError: console.error, + }, + context, +); +const core = [ + { name: "chord.http", facet: http.facet }, + { name: "@example/app", facet: app }, +]; +await plugins.load(server, { kind: "server", core, modules }, context); +const { port } = await http.listening(context); + +// 1. Without a principal: refused (403), the plugin never sees it. +const refused = new WebSocket(`ws://127.0.0.1:${port}/echo/lobby`); +const [, response] = await once(refused, "unexpected-response"); +console.log(`1. anonymous: refused ${response.statusCode}`); + +// 2. As ada: the plugin's handler greets, then echoes. +const socket = new WebSocket(`ws://127.0.0.1:${port}/echo/lobby`, { headers: { "x-user": "ada" } }); +const messages: string[] = []; +socket.on("message", (data) => messages.push(String(data))); +await once(socket, "open"); +socket.send("hello"); +// The greeting and the echo. +while (messages.length < 2) { + await new Promise((resolve) => setTimeout(resolve, 10)); +} +for (const message of messages) console.log(`2. ${message}`); + +// 3. Closing the component closes its sockets (1001, going away); then the host. +const closed = once(socket, "close"); +await http.close(context); +await server.close(context); +const [code] = await closed; +console.log(`3. closed with ${code}`); +plugins.close(); diff --git a/packages/chord/test/next/examples/45-plugin-socket/plugins/echo/package.json b/packages/chord/test/next/examples/45-plugin-socket/plugins/echo/package.json new file mode 100644 index 000000000..ed606991b --- /dev/null +++ b/packages/chord/test/next/examples/45-plugin-socket/plugins/echo/package.json @@ -0,0 +1,5 @@ +{ + "name": "@example/echo", + "private": true, + "type": "module" +} diff --git a/packages/chord/test/next/examples/45-plugin-socket/plugins/echo/server.ts b/packages/chord/test/next/examples/45-plugin-socket/plugins/echo/server.ts new file mode 100644 index 000000000..abf014eb3 --- /dev/null +++ b/packages/chord/test/next/examples/45-plugin-socket/plugins/echo/server.ts @@ -0,0 +1,16 @@ +// The echo plugin: a WebSocket of its own protocol at `/echo/:room`, after the server checked who is connecting. + +import { defineFacet } from "@earendil-works/chord/next"; +import { Http } from "@earendil-works/chord/next/http"; + +export default defineFacet((_host, self) => + self.contribute(Http, { + sockets: { + "/echo/:room": (socket, request, ctx) => { + const who = JSON.stringify(ctx.principal); + socket.send(`welcome to ${request.params.room}, ${who}`); + socket.on("message", (data) => socket.send(`echo: ${String(data)}`)); + }, + }, + }), +); diff --git a/packages/chord/test/next/examples/46-allowed-hosts/main.ts b/packages/chord/test/next/examples/46-allowed-hosts/main.ts new file mode 100644 index 000000000..67c346516 --- /dev/null +++ b/packages/chord/test/next/examples/46-allowed-hosts/main.ts @@ -0,0 +1,61 @@ +// Allowed hosts (slice 10): the server answers only requests on loopback or through a host a plugin contributes. A +// plugin can allow a host at runtime, a tunnel naming its public host, and take it back; links and sockets through a +// host no longer allowed are terminated at once. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/46-allowed-hosts/main.ts +import { request } from "node:http"; +import { fileURLToPath } from "node:url"; +import { BACKGROUND_CONTEXT as context } from "../../../../src/context/index.ts"; +import { local, method, str } from "../../../../src/next/contracts/index.ts"; +import { createHost } from "../../../../src/next/host/index.ts"; +import * as chordNext from "../../../../src/next/index.ts"; +import * as chordHttp from "../../../../src/next-node/http/index.ts"; +import { createPlugins } from "../../../../src/next-node/plugins/index.ts"; + +/** The tunnel plugin's object, as the app's code declares it (the same contract). */ +const Tunnel = local("example.tunnel", { version: 1, params: {}, members: { open: method(str()), close: method() } }); + +const server = createHost({ principal: { kind: "server" }, onError: console.error }); +const http = chordHttp.createHttp(server, { listen: { host: "127.0.0.1", port: 0 }, onError: console.error }, context); +const modules = { "@earendil-works/chord/next": chordNext, "@earendil-works/chord/next/http": chordHttp }; +const plugins = createPlugins( + { + roots: [fileURLToPath(new URL("./plugins/", import.meta.url))], + kinds: { server: { platform: "node", hostPackages: Object.keys(modules) } }, + onError: console.error, + }, + context, +); +await plugins.load(server, { kind: "server", core: [{ name: "chord.http", facet: http.facet }], modules }, context); +const { port } = await http.listening(context); + +/** GET /hello through `host` (as a tunnel would forward it), printed. */ +async function hello(host: string): Promise { + const answer = await new Promise((resolve) => { + const req = request({ host: "127.0.0.1", port, path: "/hello", headers: { host } }, (res) => { + let body = ""; + res.on("data", (chunk) => { + body += chunk; + }); + res.on("end", () => resolve(`${res.statusCode} ${body}`)); + }); + req.end(); + }); + console.log(` GET /hello via ${host} -> ${answer}`); +} + +const tunnel = await server.get(Tunnel, {}, context); +console.log("1. no tunnel yet"); +await hello(`localhost:${port}`); +await hello("notes.example.net"); +console.log("2. the tunnel opens"); +await tunnel.open("notes.example.net", context); +await hello("notes.example.net"); +console.log("3. the tunnel closes"); +await tunnel.close(context); +await hello("notes.example.net"); + +tunnel.release(); +await http.close(context); +await server.close(context); +plugins.close(); diff --git a/packages/chord/test/next/examples/46-allowed-hosts/plugins/tunnel/package.json b/packages/chord/test/next/examples/46-allowed-hosts/plugins/tunnel/package.json new file mode 100644 index 000000000..7681eec3f --- /dev/null +++ b/packages/chord/test/next/examples/46-allowed-hosts/plugins/tunnel/package.json @@ -0,0 +1,5 @@ +{ + "name": "@example/tunnel", + "private": true, + "type": "module" +} diff --git a/packages/chord/test/next/examples/46-allowed-hosts/plugins/tunnel/server.ts b/packages/chord/test/next/examples/46-allowed-hosts/plugins/tunnel/server.ts new file mode 100644 index 000000000..2e1239c10 --- /dev/null +++ b/packages/chord/test/next/examples/46-allowed-hosts/plugins/tunnel/server.ts @@ -0,0 +1,31 @@ +// The tunnel plugin: when its tunnel opens, it allows the tunnel's public host (requests through it are https); when +// it closes, it takes the host back. Its contribution is a writable, so the router follows each change. + +import { defineFacet, local, method, str } from "@earendil-works/chord/next"; +import { Http, type HttpContribution } from "@earendil-works/chord/next/http"; + +export const Tunnel = local("example.tunnel", { + version: 1, + params: {}, + members: { open: method(str()), close: method() }, +}); + +export default defineFacet((host, self) => { + self.contribute(Http, { + routes: { "GET /hello": async (request) => ({ kind: "json", value: { origin: request.origin } }) }, + }); + host.serve(Tunnel, (_params, tunnel) => { + const allowed = tunnel.writable({ hosts: [] }); + tunnel.contribute(Http, allowed); + return { + open: async (name) => { + allowed.set({ hosts: [name] }); + return null; + }, + close: async () => { + allowed.set({ hosts: [] }); + return null; + }, + }; + }); +}); diff --git a/packages/chord/test/next/examples/47-http-own-server/main.ts b/packages/chord/test/next/examples/47-http-own-server/main.ts new file mode 100644 index 000000000..6595b2517 --- /dev/null +++ b/packages/chord/test/next/examples/47-http-own-server/main.ts @@ -0,0 +1,70 @@ +// Your own server (slice 10): the HTTP component is a Web-standard `fetch` handler, so an app with a server of its own +// mounts it instead of Chord's listener. Here a plain Node server converts every request with `nodeRequest` (the +// listener's checks: a malformed target refused, a second `Host` kept for admission to refuse; and the connection: +// loopback requests count as local only with it), passes it to `http.fetch`, and every upgrade to `http.upgrade`. +// With Hono the same is one line: `app.all("/*", (c) => http.fetch(c.req.raw, connection))`. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/47-http-own-server/main.ts +import { createServer, request } from "node:http"; +import { fileURLToPath } from "node:url"; +import { BACKGROUND_CONTEXT as context } from "../../../../src/context/index.ts"; +import { createHost } from "../../../../src/next/host/index.ts"; +import * as chordNext from "../../../../src/next/index.ts"; +import * as chordHttp from "../../../../src/next-node/http/index.ts"; +import { createPlugins } from "../../../../src/next-node/plugins/index.ts"; + +// 1. The component, without a listener: no `listen` setting. +const server = createHost({ principal: { kind: "server" }, onError: console.error }); +const http = chordHttp.createHttp(server, { onError: console.error }, context); +const modules = { "@earendil-works/chord/next": chordNext, "@earendil-works/chord/next/http": chordHttp }; +const plugins = createPlugins( + { + roots: [fileURLToPath(new URL("./plugins/", import.meta.url))], + kinds: { server: { platform: "node", hostPackages: Object.keys(modules) } }, + onError: console.error, + }, + context, +); +await plugins.load(server, { kind: "server", core: [{ name: "chord.http", facet: http.facet }], modules }, context); + +// 2. The app's own server: a Web `Request` in, a Web `Response` out (each header line kept: several `Set-Cookie`). A +// real app adds its own routes around it. +const own = createServer(async (message, res) => { + const converted = chordHttp.nodeRequest(message, res); + if (converted.kind === "refused") return void res.writeHead(converted.status).end(); + const response = await http.fetch(converted.request, converted.connection); + for (const [name, value] of response.headers) res.appendHeader(name, value); + res.writeHead(response.status).end(Buffer.from(await response.arrayBuffer())); +}); +own.on("upgrade", http.upgrade); +await new Promise((resolve) => own.listen(0, "127.0.0.1", resolve)); +const address = own.address(); +const port = typeof address === "object" && address !== null ? address.port : 0; + +// 3. Requests through the app's server get the same routes, checks and headers. +for (const path of ["/notes", "/nowhere"]) { + const response = await fetch(`http://127.0.0.1:${port}${path}`); + console.log(`GET ${path} -> ${response.status} ${await response.text()}`); + console.log(` x-content-type-options: ${response.headers.get("x-content-type-options")}`); +} + +// 4. What a Web `Request` cannot carry is refused at the conversion, as Chord's listener refuses it; the app's server +// lives on (`TRACE`: 501; a `Host` with userinfo: 403). +for (const [method, host] of [ + ["TRACE", `127.0.0.1:${port}`], + ["GET", `attacker@127.0.0.1:${port}`], +] as const) { + const status = await new Promise((resolve) => { + request({ host: "127.0.0.1", port, method, path: "/notes", headers: { host } }, (response) => { + response.resume(); + resolve(response.statusCode); + }).end(); + }); + console.log(`${method} /notes (Host: ${host}) -> ${status}`); +} + +// 5. The app closes its server and the component (it refuses, then waits for running requests), then the host. +await new Promise((resolve) => own.close(resolve)); +await http.close(context); +await server.close(context); +plugins.close(); diff --git a/packages/chord/test/next/examples/47-http-own-server/plugins/notes/package.json b/packages/chord/test/next/examples/47-http-own-server/plugins/notes/package.json new file mode 100644 index 000000000..d59f5a427 --- /dev/null +++ b/packages/chord/test/next/examples/47-http-own-server/plugins/notes/package.json @@ -0,0 +1,5 @@ +{ + "name": "@example/notes", + "private": true, + "type": "module" +} diff --git a/packages/chord/test/next/examples/47-http-own-server/plugins/notes/server.ts b/packages/chord/test/next/examples/47-http-own-server/plugins/notes/server.ts new file mode 100644 index 000000000..2b7a0abd9 --- /dev/null +++ b/packages/chord/test/next/examples/47-http-own-server/plugins/notes/server.ts @@ -0,0 +1,12 @@ +// The notes plugin's server facet: one route, the same whichever server runs it. + +import { defineFacet } from "@earendil-works/chord/next"; +import { Http } from "@earendil-works/chord/next/http"; + +export default defineFacet((_host, self) => + self.contribute(Http, { + routes: { + "GET /notes": async (request) => ({ kind: "json", value: { notes: ["buy milk"], via: request.origin } }), + }, + }), +); diff --git a/packages/chord/test/next/examples/48-http-baseline/main.ts b/packages/chord/test/next/examples/48-http-baseline/main.ts new file mode 100644 index 000000000..764fcce21 --- /dev/null +++ b/packages/chord/test/next/examples/48-http-baseline/main.ts @@ -0,0 +1,84 @@ +// The security baseline (slice 10): on by default, the app decides only who someone is. Here the app signs people in +// with a signed cookie (`cookie`, `readCookie`) and the notes plugin contributes a page, routes and a webhook. Requests +// another site's page makes in a browser (Fetch Metadata says so) are never authenticated, a foreign `Origin` is +// refused, but a link to a page still opens it signed in, and a cross-site webhook runs anonymous. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/48-http-baseline/main.ts +import { request } from "node:http"; +import { fileURLToPath } from "node:url"; +import { BACKGROUND_CONTEXT as context } from "../../../../src/context/index.ts"; +import { createHost, defineFacet } from "../../../../src/next/host/index.ts"; +import * as chordNext from "../../../../src/next/index.ts"; +import * as chordHttp from "../../../../src/next-node/http/index.ts"; +import { createPlugins } from "../../../../src/next-node/plugins/index.ts"; + +type Principal = { readonly kind: "server" } | { readonly kind: "user"; readonly name: string }; +const secret = "a long random secret"; +const server = createHost({ principal: { kind: "server" }, onError: console.error }); +const http = chordHttp.createHttp(server, { listen: { host: "127.0.0.1", port: 0 }, onError: console.error }, context); + +// 1. The app's core facet: a sign-in route setting the cookie, and an authenticator reading it back. +const app = defineFacet((_host, self) => { + self.contribute(chordHttp.Http, { + routes: { + "POST /sign-in": async (request) => { + const set = chordHttp.cookie("user", String(await request.json()), { secret, request, sameSite: "Lax" }); + return { kind: "json", value: "signed in", headers: { "Set-Cookie": set } }; + }, + }, + }); + self.contribute(chordHttp.Authenticators, (request) => { + const name = chordHttp.readCookie(request, "user", secret); + return name === undefined ? undefined : { kind: "user", name }; + }); +}); +const modules = { "@earendil-works/chord/next": chordNext, "@earendil-works/chord/next/http": chordHttp }; +const plugins = createPlugins( + { + roots: [fileURLToPath(new URL("./plugins/", import.meta.url))], + kinds: { server: { platform: "node", hostPackages: Object.keys(modules) } }, + onError: console.error, + }, + context, +); +const core = [ + { name: "chord.http", facet: http.facet }, + { name: "@example/app", facet: app }, +]; +await plugins.load(server, { kind: "server", core, modules }, context); +const { port } = await http.listening(context); +const base = `http://127.0.0.1:${port}`; + +// 2. Ada signs in; her browser keeps the cookie. +const signIn = await fetch(`${base}/sign-in`, { method: "POST", body: '"ada"' }); +const cookie = signIn.headers.get("set-cookie")!.split(";")[0]!; +console.log(`1. signed in: ${cookie.slice(0, 12)}…`); + +/** A request from Ada's browser: her cookie, and what Fetch Metadata says about who made it (`fetch` sets its own). */ +async function browser(label: string, method: string, path: string, headers: Record = {}, body = "") { + const answer = await new Promise((resolve) => { + const sent = request({ host: "127.0.0.1", port, method, path, headers: { cookie, ...headers } }, (response) => { + let text = ""; + response.on("data", (chunk) => { + text += chunk; + }); + response.on("end", () => resolve(`${response.statusCode} ${text}`)); + }); + sent.end(body); + }); + console.log(`${label}: ${answer}`); +} + +const ours = { "sec-fetch-site": "same-origin", origin: base }; +const evil = { "sec-fetch-site": "cross-site", origin: "https://evil.example" }; +await browser("2. our page reads the notes", "GET", "/notes", ours); +await browser("3. our page adds a note", "POST", "/notes", ours, '"call mom"'); +await browser("4. evil.example posts a note", "POST", "/notes", evil, '"pwned"'); +await browser("5. evil.example reads the notes", "GET", "/notes", { "sec-fetch-site": "cross-site" }); +const link = { "sec-fetch-site": "cross-site", "sec-fetch-mode": "navigate", "sec-fetch-dest": "document" }; +await browser("6. a link on evil.example to the page", "GET", "/", link); +await browser("7. a build server's webhook", "POST", "/hooks/build", { origin: "https://ci.example" }); + +await http.close(context); +await server.close(context); +plugins.close(); diff --git a/packages/chord/test/next/examples/48-http-baseline/plugins/notes/package.json b/packages/chord/test/next/examples/48-http-baseline/plugins/notes/package.json new file mode 100644 index 000000000..d59f5a427 --- /dev/null +++ b/packages/chord/test/next/examples/48-http-baseline/plugins/notes/package.json @@ -0,0 +1,5 @@ +{ + "name": "@example/notes", + "private": true, + "type": "module" +} diff --git a/packages/chord/test/next/examples/48-http-baseline/plugins/notes/server.ts b/packages/chord/test/next/examples/48-http-baseline/plugins/notes/server.ts new file mode 100644 index 000000000..0980b62f1 --- /dev/null +++ b/packages/chord/test/next/examples/48-http-baseline/plugins/notes/server.ts @@ -0,0 +1,33 @@ +// The notes plugin's server facet: the notes page (a link from another site may open it), the notes as JSON, adding +// a note, and a webhook another site calls (cross-site: it always runs anonymous). + +import { defineFacet } from "@earendil-works/chord/next"; +import { Http } from "@earendil-works/chord/next/http"; + +const notes = ["buy milk"]; + +export default defineFacet((_host, self) => + self.contribute(Http, { + routes: { + "GET /": { + page: true, + handler: async (_request, ctx) => ({ kind: "html", body: `hi ${who(ctx.principal)}` }), + }, + "GET /notes": async (_request, ctx) => ({ kind: "json", value: { notes, as: who(ctx.principal) } }), + "POST /notes": async (request, ctx) => { + if (ctx.principal === undefined) return { kind: "error", status: 401 }; + notes.push(String(await request.json())); + return { kind: "json", value: notes }; + }, + "POST /hooks/build": { + crossSite: true, + handler: async (_request, ctx) => ({ kind: "json", value: who(ctx.principal) }), + }, + }, + }), +); + +/** Who a principal is, for the answers. */ +function who(principal: unknown): string { + return principal === undefined ? "anonymous" : JSON.stringify(principal); +} diff --git a/packages/chord/test/next/host/revocation.test.ts b/packages/chord/test/next/host/revocation.test.ts index 22bc7a9de..a73b222f8 100644 --- a/packages/chord/test/next/host/revocation.test.ts +++ b/packages/chord/test/next/host/revocation.test.ts @@ -536,6 +536,11 @@ describe("revocation of this host's own requests over its paths (fix round 1)", it("ends a guest's reference over a route link that dropped, and nothing holds it on the next one", async () => { const routed: RawPeer[] = []; + // The drop resolves again at once (slice 10 R2); the next link comes only after the revocation. + let next: () => void = () => {}; + const revoked = new Promise((resolve) => { + next = resolve; + }); const host = createHost({ principal: { kind: "owner" }, revokedBy: guestRevokedBy, @@ -546,6 +551,7 @@ describe("revocation of this host's own requests over its paths (fix round 1)", { authorize: () => true, resolve: async () => { + if (routed.length > 0) await revoked; const [a, b] = memoryLinkPair(); routed.push(new RawPeer(b)); return { link: a, principal: { kind: "owner" }, codec: testCodec() }; @@ -569,6 +575,7 @@ describe("revocation of this host's own requests over its paths (fix round 1)", first.close(); await transported(); host.revoke("g", context); + next(); await transported(); expect(states).toEqual(["app.remote: revoked"]); expect(routed.flatMap((peer, i) => (i === 0 ? [] : peer.received.filter(({ kind }) => kind === "hold")))).toEqual( diff --git a/packages/chord/test/next/host/root.test.ts b/packages/chord/test/next/host/root.test.ts index 76c1c2b7e..0847d0477 100644 --- a/packages/chord/test/next/host/root.test.ts +++ b/packages/chord/test/next/host/root.test.ts @@ -1,7 +1,7 @@ import { getEventListeners } from "node:events"; import { describe, expect, it } from "vitest"; import { BACKGROUND_CONTEXT as context, withAbortSignal, withCancel } from "../../../src/context/index.ts"; -import { int, local, method } from "../../../src/next/contracts/index.ts"; +import { hostKind, int, local, method, remote, str } from "../../../src/next/contracts/index.ts"; import { type Ctx, createHost, defineFacet, sleep } from "../../../src/next/host/index.ts"; import { constant } from "../../../src/next/state/index.ts"; import { expectCollected, flushed, loaded, rejection, type TestPrincipal } from "../support.ts"; @@ -33,6 +33,39 @@ describe("host.root", () => { await host.close(context); }); + it("runs anonymous with an explicit undefined principal; only an omitted one takes the host's (slice 10 §6)", async () => { + const host = createHost({ principal: { kind: "server" }, onError: () => {} }); + await host.load([], context); + expect(await host.root(async (ctx) => ctx.principal, context, { principal: undefined })).toBeUndefined(); + expect(await host.root(async (ctx) => ctx.principal, context, {})).toEqual({ kind: "server" }); + await host.close(context); + }); + + // Opus's build review F5: an anonymous scope's `ctx.get` of a remote type served here failed on a WeakMap key. + it("refuses an anonymous scope a remote type served here (forbidden); a local one it gets", async () => { + const Server = hostKind("server"); + const Secrets = remote("app.secrets", { + version: 1, + params: {}, + hostAddress: () => Server(), + variants: { main: { methods: { read: method(undefined, str()) } } }, + }); + const host = createHost({ address: Server(), principal: { kind: "server" }, onError: () => {} }); + const facet = defineFacet((h) => { + serveInfo(h); + h.serve(Secrets, () => ({ variant: "main", members: { read: async () => "secret" } })); + }); + await host.load([{ name: "@app/secrets", facet }], context); + const anonymous = { principal: undefined }; + const refused = await rejection(host.root(async (ctx) => ctx.get(Secrets, {}), context, anonymous)); + expect(refused).toMatchObject({ + code: "forbidden", + message: "app.secrets: an anonymous caller gets no remote object", + }); + expect(await host.root(async (ctx) => (await ctx.get(Info, { id: 3 })).id.value, context, anonymous)).toBe(3); + await host.close(context); + }); + it("seals the context when fn settles: its references released, its signal aborted, `ctx.get` refused `released`", async () => { const { host } = await loaded(serveInfo); let kept: Ctx | undefined; diff --git a/packages/chord/test/next/host/route.test.ts b/packages/chord/test/next/host/route.test.ts index 756ae9429..3d9fbf1f8 100644 --- a/packages/chord/test/next/host/route.test.ts +++ b/packages/chord/test/next/host/route.test.ts @@ -119,39 +119,71 @@ forEachCodec(() => { }); }); - describe("resolve failures and re-resolves", () => { + describe("resolve failures and re-resolves (no timer, slice 10 R2)", () => { fakeTimers(); - it("a rejected resolve leaves the holds loading, is reported, and resolves again after the address's backoff (race 2)", async () => { - let failures = 2; - const { tab, resolves, errors } = await relayed({ + it("a rejected resolve fails its holders waiting with its error, is reported, and runs no timer (race 2)", async () => { + let attempts = 0; + const { tab, server, errors } = await relayed({ + resolve: async () => { + attempts++; + throw new Error("no worker"); + }, + }); + // One relayed from the tab, one the server's own. + const notes = tab.hold(PlacedNotes, { session: "S1" }, context); + const own = server.hold(PlacedNotes, { session: "S1" }, context); + await transported(); + for (const ref of [notes, own]) { + expect(ref.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "waiting", error: { code: "unavailable", message: "session|S1: no worker" } }, + }); + } + expect(errors.server).toMatchObject([{ kind: "resolve_failed", address: "session|S1" }]); + await elapse(120_000); + expect(attempts).toBe(1); + }); + + it("a new hold resolves a failed address again; the earlier holders bind on its link without a re-hold (Q1)", async () => { + let fail = true; + const { tab, resolves } = await relayed({ resolve: async (_address, worker) => { - if (failures-- > 0) throw new Error("no worker yet"); + if (fail) throw new Error("no worker"); + return worker(); + }, + }); + const first = tab.hold(PlacedNotes, { session: "S1" }, context); + await transported(); + expect(first.state.value).toMatchObject({ kind: "failed", cause: { kind: "waiting" } }); + fail = false; + const second = tab.hold(PlacedNotes, { session: "S1" }, context); + await transported(); + expect(resolves).toEqual(["session|S1"]); + expect([first.state.value, second.state.value]).toEqual([{ kind: "live" }, { kind: "live" }]); + }); + + it("RouteHandle.retry resolves a failed address now; a retry of an address nobody holds keeps nothing", async () => { + let fail = true; + const { tab, route, resolves } = await relayed({ + resolve: async (_address, worker) => { + if (fail) throw new Error("no worker"); return worker(); }, }); const notes = tab.hold(PlacedNotes, { session: "S1" }, context); await transported(); - expect(notes.state.value).toEqual({ kind: "loading" }); - expect(errors.server).toMatchObject([{ kind: "resolve_failed", address: "session|S1" }]); - await elapse(999); - expect(errors.server).toHaveLength(1); - await elapse(1); - // The second failure doubles the window. - expect(errors.server).toHaveLength(2); - await elapse(1999); - expect(notes.state.value).toEqual({ kind: "loading" }); - await elapse(1); - expect(resolves).toEqual(["session|S1"]); + fail = false; + route.retry(Session({ id: "S1" })); + route.retry(Session({ id: "S9" })); + await transported(); expect(notes.state.value).toEqual({ kind: "live" }); + expect(resolves).toEqual(["session|S1"]); }); - it("a returned link that is closed already starts the backoff", async () => { - let closed = true; - const { tab, resolves } = await relayed({ - resolve: async (address, worker) => { - if (!closed) return worker(); - closed = false; + it("a returned link closed already fails the holders", async () => { + const { tab } = await relayed({ + resolve: async (address) => { const [a, b] = memoryLinkPair(); b.close(); await flushed(); @@ -160,20 +192,35 @@ forEachCodec(() => { }); const notes = tab.hold(PlacedNotes, { session: "S1" }, context); await transported(); - expect(notes.state.value).toEqual({ kind: "loading" }); - await elapse(1000); - expect(resolves).toEqual(["session|S1"]); - expect(notes.state.value).toEqual({ kind: "live" }); + expect(notes.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "waiting", error: { message: "session|S1: the link closed before its handshake" } }, + }); }); - it("a dropped link resolves again while held: references go loading, then live on a fresh worker (race 6)", async () => { + it("a link closing before its handshake fails its holders: no re-resolve loop over a worker dying at start", async () => { + let attempts = 0; + const { tab } = await relayed({ + resolve: async (address) => { + attempts++; + const [a, b] = memoryLinkPair(); + setTimeout(() => b.close(), 10); + return { link: a, principal: { kind: "worker", session: address.params.id } }; + }, + }); + const notes = tab.hold(PlacedNotes, { session: "S1" }, context); + await elapse(10); + expect(notes.state.value).toMatchObject({ kind: "failed", cause: { kind: "waiting" } }); + await elapse(120_000); + expect(attempts).toBe(1); + }); + + it("a dropped link that completed its handshake resolves again at once, once (race 6)", async () => { const { tab, resolves, routeLinks, workers } = await relayed(); const notes = await tab.get(PlacedNotes, { session: "S1" }, context); const first = workers.get("session|S1")!; routeLinks[0]!.close(); await transported(); - expect(notes.state.value).toEqual({ kind: "loading" }); - await elapse(1000); expect(resolves).toEqual(["session|S1", "session|S1"]); expect(workers.get("session|S1")).not.toBe(first); expect(notes.state.value).toEqual({ kind: "live" }); @@ -182,42 +229,13 @@ forEachCodec(() => { expect(notes.notes.value).toEqual([{ text: "fresh" }]); }); - it("only a link that completed its handshake resets the backoff", async () => { - let silent = 0; - const { tab, resolves, routeLinks } = await relayed({ - resolve: async (address, worker) => { - if (silent-- <= 0) return worker(); - // A link whose other end never greets and closes. - const [a, b] = memoryLinkPair(); - setTimeout(() => b.close(), 10); - return { link: a, principal: { kind: "worker", session: address.params.id } }; - }, - }); - const notes = await tab.get(PlacedNotes, { session: "S1" }, context); - silent = 1; - routeLinks[0]!.close(); - await elapse(1000); - // The silent link closed before its handshake: the window doubles. - await elapse(10); - await elapse(1999); - expect(resolves).toEqual(["session|S1"]); - await elapse(1); - expect(resolves).toEqual(["session|S1", "session|S1"]); - expect(notes.state.value).toEqual({ kind: "live" }); - // That link greeted: the next drop starts again at 1 s. - routeLinks.at(-1)!.close(); - await elapse(1000); - expect(resolves).toHaveLength(3); - }); - - it("a link whose peer never greets closes after 30 s of silence; the address backs off, then resolves again while held", async () => { + it("a link whose peer never greets closes after 30 s of silence and fails its holders", async () => { // The heartbeat's timer too, before the hosts start it. vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout", "setInterval", "clearInterval", "Date"], now: 0 }); - let silent = true; - const { tab, resolves } = await relayed({ - resolve: async (address, worker) => { - if (!silent) return worker(); - silent = false; + let attempts = 0; + const { tab } = await relayed({ + resolve: async (address) => { + attempts++; // Its other end is never read: no hello, no close. const [a] = memoryLinkPair(); return { link: a, principal: { kind: "worker", session: address.params.id } }; @@ -225,14 +243,12 @@ forEachCodec(() => { }); const notes = tab.hold(PlacedNotes, { session: "S1" }, context); await elapse(29_000); - expect(resolves).toEqual([]); expect(notes.state.value).toEqual({ kind: "loading" }); - // The heartbeat checks every 1.5 s: the silent link closes within it, then its address waits 1 s. + // The heartbeat checks every 1.5 s: the silent link closes within it. await elapse(2_500); - await elapse(1_000); - expect(resolves).toEqual(["session|S1"]); - await transported(); - expect(notes.state.value).toEqual({ kind: "live" }); + expect(notes.state.value).toMatchObject({ kind: "failed", cause: { kind: "waiting" } }); + await elapse(120_000); + expect(attempts).toBe(1); }); it("an address without holds is not resolved again after its link drops", async () => { @@ -276,6 +292,42 @@ forEachCodec(() => { }); }); + describe("RouteHandle.suspend (R1, Q3)", () => { + it("resolves nothing while held: the holders fail waiting suspended once the link is gone; retry lifts it", async () => { + const { tab, route, resolves, routeLinks } = await relayed(); + const notes = await tab.get(PlacedNotes, { session: "S1" }, context); + route.suspend(Session({ id: "S1" })); + expect(notes.state.value).toEqual({ kind: "live" }); + routeLinks[0]!.close(); + await transported(); + expect(notes.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "waiting", error: { code: "unavailable", message: "session|S1: suspended" } }, + }); + const another = tab.hold(PlacedNotes, { session: "S1" }, context); + await transported(); + expect(another.state.value).toMatchObject({ kind: "failed", cause: { kind: "waiting" } }); + expect(resolves).toEqual(["session|S1"]); + route.retry(Session({ id: "S1" })); + await transported(); + expect(resolves).toEqual(["session|S1", "session|S1"]); + expect([notes.state.value, another.state.value]).toEqual([{ kind: "live" }, { kind: "live" }]); + }); + + it("goes with the last holder: a later hold resolves on demand; an unheld address is never suspended", async () => { + const { tab, route, resolves, routeLinks } = await relayed(); + const notes = await tab.get(PlacedNotes, { session: "S1" }, context); + route.suspend(Session({ id: "S1" })); + routeLinks[0]!.close(); + await transported(); + notes.release(); + await transported(); + route.suspend(Session({ id: "S1" })); + await tab.get(PlacedNotes, { session: "S1" }, context); + expect(resolves).toEqual(["session|S1", "session|S1"]); + }); + }); + describe("host.close during a resolve (race 7)", () => { it("aborts the resolve's context and closes a link returned afterwards", async () => { let routing: AbortSignal | undefined; @@ -375,6 +427,37 @@ forEachCodec(() => { }); describe("GC", () => { + it("a failed or suspended address keeps no route state once its holders are released", async () => { + const { refs, server } = await (async () => { + const refs: WeakRef[] = []; + const { tab, server, route, routeLinks, workers, errors } = await relayed({ + authorize: (_principal, address) => { + refs.push(new WeakRef(address)); + return true; + }, + resolve: async (address, worker) => { + if (address.params.id === "failed") throw new Error("no worker"); + return worker(); + }, + }); + const failed = tab.hold(PlacedNotes, { session: "failed" }, context); + const held = await tab.get(PlacedNotes, { session: "held" }, context); + route.suspend(Session({ id: "held" })); + routeLinks[0]!.close(); + await transported(); + for (const ref of [failed, held]) ref.release(); + await transported(); + // The reported error's stack keeps the test's `resolve` frame (and its address) alive, the worker host its + // own address: not the route's. + errors.server.length = 0; + routeLinks.length = 0; + workers.clear(); + return { refs, server }; + })(); + await expectCollected(...refs); + expect(server).toBeDefined(); + }); + it("a closed route link and its released references keep no route state", async () => { const { refs, tab } = await (async () => { const { tab, routeLinks, workers } = await relayed(); diff --git a/packages/chord/test/next/remote/host-link.test.ts b/packages/chord/test/next/remote/host-link.test.ts index c5205ea12..a3da2f05d 100644 --- a/packages/chord/test/next/remote/host-link.test.ts +++ b/packages/chord/test/next/remote/host-link.test.ts @@ -3,7 +3,7 @@ import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; import { int, local, method, remote, str } from "../../../src/next/contracts/index.ts"; import { createHost, defineFacet, type FacetHost, type HostError } from "../../../src/next/host/index.ts"; import { type Compression, memoryLinkPair, type PartLimits } from "../../../src/next/link/index.ts"; -import { jsonPart } from "../../../src/next/link/internal.ts"; +import { jsonPart, type Message } from "../../../src/next/link/internal.ts"; import { constant } from "../../../src/next/state/index.ts"; import { nodeCompression } from "../../../src/next-node/compression.ts"; import { @@ -19,12 +19,15 @@ import { partValue, RAW_PEER, RawPeer, + rawLinked, rawWorkers, rejection, relayed, + SERVER_ADDRESS, Server, Session, serveNotes, + settledHeap, type TestPrincipal, testCodec, transported, @@ -431,3 +434,76 @@ forEachCodec(() => { }); }); }); + +describe("origins only from relays (decision, slice 10 security audit K1/K2)", () => { + const { name: type, version, fingerprint } = Notes; + const hold = { + kind: "hold", + hold: 0, + type, + version, + fingerprint, + params: jsonPart({ session: "S1" }), + address: SERVER_ADDRESS, + }; + const withOrigin: Record = { + hold: { ...hold, origin: 1 } as Message, + call: { kind: "call", origin: 1, call: 0, hold: 0, method: EDITOR.methods.wait }, + release: { kind: "release", origin: 1, hold: 0 }, + abort: { kind: "abort", origin: 1, call: 0 }, + retry: { kind: "retry", origin: 1, hold: 0, hops: 1 } as Message, + ack: { kind: "ack", origin: 1 }, + closed: { kind: "closed", origin: 1 }, + }; + + for (const [kind, message] of Object.entries(withOrigin)) { + it(`closes a child link sending ${kind} with an origin (a protocol error)`, async () => { + const { peer, errors } = await rawLinked((h) => void serveNotes(h)); + peer.send(message); + await transported(); + expect(peer.closed).toBe(true); + expect(errors).toMatchObject([ + { kind: "protocol_error", error: { message: `${kind}: an origin on a child link` } }, + ]); + }); + } + + it("honours origins on a parent link (a relay's): one window of calls per origin", async () => { + let probe!: ReturnType; + const { peer } = await rawLinked( + (h) => { + probe = serveNotes(h); + }, + { role: "parent" }, + ); + peer.send({ ...hold, origin: 1 } as Message); + peer.send({ ...hold, origin: 2 } as Message); + await transported(); + for (const origin of [1, 2]) peer.send({ kind: "call", origin, call: 0, hold: 0, method: EDITOR.methods.wait }); + await transported(); + expect([peer.closed, probe.waits.length]).toEqual([false, 2]); + }); + + it("keeps no record of an origin that holds nothing: 20,000 fresh origins held and released", async () => { + const { peer } = await rawLinked((h) => void serveNotes(h), { role: "parent" }); + const churn = async (from: number, to: number) => { + for (let origin = from; origin <= to; origin++) { + peer.send({ ...hold, origin } as Message); + peer.send({ kind: "release", origin, hold: 0 }); + if (origin % 1_000 === 0) { + await transported(); + peer.received.length = 0; + } + } + await transported(); + peer.received.length = 0; + }; + // A warm-up first: what a first run allocates for good (compiled code, caches) is no record. + await churn(1, 2_000); + const before = await settledHeap(); + await churn(2_001, 22_000); + // About 1 KiB per origin before: 20 MiB. + expect((await settledHeap()) - before).toBeLessThan(512 * 1024); + expect(peer.closed).toBe(false); + }, 60_000); +}); diff --git a/packages/chord/test/next/remote/path.test.ts b/packages/chord/test/next/remote/path.test.ts index c3efb197d..ff58b786c 100644 --- a/packages/chord/test/next/remote/path.test.ts +++ b/packages/chord/test/next/remote/path.test.ts @@ -1,4 +1,5 @@ import { describe, expect, it } from "vitest"; +import { ChordError } from "../../../src/next/data/index.ts"; import type { HostLink } from "../../../src/next/remote/host-link.ts"; import { Path, type Upstream } from "../../../src/next/remote/path.ts"; @@ -67,4 +68,23 @@ describe("a path", () => { path.connected(linkOf("2")); expect(log).toEqual(["a@1", "b@1"]); }); + + it("says what changed: a holder attached without a link, a drop, the last release; fails its holders waiting", () => { + const events: string[] = []; + const failed: string[] = []; + const path = new Path((event) => events.push(event.kind)); + const a = { + attach: () => {}, + dropped: () => {}, + unreachable: (error: { readonly message: string }) => void failed.push(error.message), + } as unknown as Upstream; + path.attach(a); + path.unreachable(new ChordError({ code: "unavailable", message: "no worker" })); + const link = linkOf("1"); + path.connected(link); + path.disconnected(link); + path.detach(a); + expect(events).toEqual(["attached", "dropped", "released"]); + expect(failed).toEqual(["no worker"]); + }); }); diff --git a/packages/chord/test/next/remote/provider.test.ts b/packages/chord/test/next/remote/provider.test.ts index dc9f30aed..5abb539e3 100644 --- a/packages/chord/test/next/remote/provider.test.ts +++ b/packages/chord/test/next/remote/provider.test.ts @@ -74,13 +74,13 @@ const READER = wireNames(Notes, "reader"); const { add, fail, wait } = EDITOR.methods as { add: number; fail: number; wait: number }; /** A server serving `Notes` and a raw consumer peer stamped `child`. */ -async function notesServer(child: TestPrincipal = { kind: "owner" }) { +async function notesServer(child: TestPrincipal = { kind: "owner" }, role: "child" | "parent" = "child") { let probe: NotesProbe | undefined; const linked = await rawLinked( (h) => { probe = serveNotes(h); }, - { child }, + { child, role }, ); return { ...linked, probe: probe! }; } @@ -551,7 +551,7 @@ function serveGrid(h: FacetHost): void { } /** A server serving `Notes` and `Grid` to a raw peer that acks only when told, the link's context counting. */ -async function slowConsumer() { +async function slowConsumer(role: "child" | "parent" = "child") { const counting = await metered(); const server = createHost({ address: Server(), principal: { kind: "server" }, onError: () => {} }); await server.load( @@ -567,7 +567,9 @@ async function slowConsumer() { context, ); const [a, b] = memoryLinkPair(); - server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, counting.context); + const options = { principal: { kind: "owner" }, codec: testCodec() } as const; + if (role === "parent") server.connect(a, options, counting.context); + else server.accept(a, options, counting.context); const peer = new RawPeer(b, { ack: false }); await transported(); return { server, peer, counts: counting.counts }; @@ -667,9 +669,9 @@ forEachCodec(() => { }); }); - describe("origins at the provider", () => { + describe("origins at the provider (a relay's: on its parent link)", () => { it("key holds and calls by (origin, id): the same ids on two origins are two holds; answers carry the origin", async () => { - const { peer } = await slowConsumer(); + const { peer } = await slowConsumer("parent"); hold(peer, 0, Notes, { session: "S1" }); hold(peer, 0, Notes, { session: "S2" }, {}, 3); const [own, relayed] = [await peer.next("bound"), await peer.next("bound")]; @@ -685,7 +687,7 @@ forEachCodec(() => { }); it("keep a window per origin: a stalled origin folds its commits, another keeps receiving each", async () => { - const { server, peer } = await slowConsumer(); + const { server, peer } = await slowConsumer("parent"); hold(peer, 0, Notes, { session: "S1" }); hold(peer, 0, Notes, { session: "S1" }, {}, 1); await peer.next("bound"); @@ -710,7 +712,7 @@ forEachCodec(() => { }); it("count calls per origin against N: 256 on each of two origins run, one more on one closes the link", async () => { - const { peer, errors } = await notesServer(); + const { peer, errors } = await notesServer({ kind: "owner" }, "parent"); hold(peer, 0, Notes, { session: "S1" }); hold(peer, 0, Notes, { session: "S1" }, {}, 1); await peer.next("bound"); @@ -730,7 +732,7 @@ forEachCodec(() => { }); it("ignore a call on a hold they do not know before counting it (Q1)", async () => { - const { peer, errors } = await notesServer(); + const { peer, errors } = await notesServer({ kind: "owner" }, "parent"); hold(peer, 0, Notes, { session: "S1" }); await peer.next("bound"); for (let id = 0; id < 256; id++) call(peer, id, 0, wait); @@ -742,7 +744,7 @@ forEachCodec(() => { }); it("closed releases the origin's holds and aborts its calls; another origin keeps its own", async () => { - const { peer, probe, server } = await notesServer(); + const { peer, probe, server } = await notesServer({ kind: "owner" }, "parent"); hold(peer, 0, Notes, { session: "S1" }); hold(peer, 0, Notes, { session: "S2" }, {}, 2); await peer.next("bound"); @@ -824,7 +826,7 @@ forEachCodec(() => { push.call(this, ops); }; try { - const { server, peer } = await slowConsumer(); + const { server, peer } = await slowConsumer("parent"); hold(peer, 0, Notes, { session: "S1" }, {}, 1); await peer.next("bound"); for (const text of ["a", "b", "c"]) await addNote(server, "S1", text); diff --git a/packages/chord/test/next/remote/relay.test.ts b/packages/chord/test/next/remote/relay.test.ts index 94148dccd..6f08a6f14 100644 --- a/packages/chord/test/next/remote/relay.test.ts +++ b/packages/chord/test/next/remote/relay.test.ts @@ -22,6 +22,7 @@ import { rejection, relayed, Server, + Session, serveNotes, type TestPrincipal, testCodec, @@ -352,8 +353,8 @@ forEachCodec(() => { first.close(); expect(await rejection(call)).toMatchObject({ code: "unavailable" }); await transported(); + // The link had completed its handshake: its address resolves again at once (no timer, slice 10 R2). expect(notes.state.value).toEqual({ kind: "loading" }); - await vi.advanceTimersByTimeAsync(1000); const second = await workerAt(workers, 1); const again = await second.next("hold"); expect(again).toMatchObject({ hold: held.hold, origin: held.origin }); @@ -366,7 +367,13 @@ forEachCodec(() => { }); it("answers a call that crossed the loading notice unavailable (race 13)", async () => { - const { raw, workers } = await rawRelay(); + // The address's next resolve never settles: the relayed hold has no upstream link when the call arrives. + const workers: RawPeer[] = []; + const resolve = rawWorkers(workers); + const relay = await relayed({ + resolve: (address) => (workers.length === 0 ? resolve(address) : new Promise(() => {})), + }); + const raw = rawTab(relay.server); holdPlaced(raw, 0); const worker = await workerAt(workers); bindPlaced(worker, await worker.next("hold")); @@ -381,27 +388,48 @@ forEachCodec(() => { describe("a parent link that is down (audit)", () => { it("holds a hold arriving meanwhile on the parent path: it waits loading and binds on the next parent link", async () => { + // The middle host relays its tab's holds to its parent, which reaches it over a route: origins arrive only on + // parent and route links (decision "Origins only from relays"). + const middle = createHost({ + address: Session({ id: "M" }), + principal: { kind: "worker", session: "M" }, + onError: () => {}, + }); + await middle.load([], context); const server = createHost({ address: Server(), principal: { kind: "server" }, onError: () => {}, }); + const links: Link[] = []; + let gate = Promise.resolve(); + server.route( + Session, + { + authorize: () => true, + resolve: async () => { + await gate; + const [a, b] = memoryLinkPair(); + middle.connect(b, { principal: { kind: "server" }, codec: testCodec() }, context); + links.push(a); + return { link: a, principal: { kind: "worker", session: "M" }, codec: testCodec() }; + }, + }, + context, + ); await server.load( [{ name: "@test/app", facet: defineFacet((h) => void serveNotes(h)) }], context, ); - const middle = createHost({ principal: { kind: "server" }, onError: () => {} }); - await middle.load([], context); - /** Connects the middle host to the server over a new link; returns the server's end. */ - const up = (): Link => { - const [a, b] = memoryLinkPair(); - server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); - middle.connect(b, { principal: { kind: "server" }, codec: testCodec() }, context); - return a; - }; - const first = up(); + // The server holds something at the middle's address: its route link exists while it does. + const anchor = server.hold(PlacedNotes, { session: "M" }, context); await transported(); - first.close(); + // The next resolve waits: the middle's parent path is down meanwhile. + let admit: () => void = () => {}; + gate = new Promise((resolve) => { + admit = resolve; + }); + links[0]!.close(); await transported(); const tab = createHost({ onError: () => {} }); await tab.load([], context); @@ -411,9 +439,11 @@ forEachCodec(() => { const notes = tab.hold(Notes, { session: "S1" }, context); await transported(); expect(notes.state.value).toEqual({ kind: "loading" }); - up(); + admit(); + await transported(); await transported(); expect(notes.state.value).toEqual({ kind: "live" }); + anchor.release(); }); }); @@ -447,9 +477,12 @@ forEachCodec(() => { bindPlaced(worker, sent); worker.send(countState([sent], 1)); await transported(); - // The relay no longer relays the hold: dropped there (Q1), each message acknowledged to the worker at once. + // The origin held nothing more: its record went and the worker was told `closed` (decision "Origins only from + // relays"); what crossed it is dropped (Q1). expect(raw.kinds()).toEqual(["hello"]); - expect(worker.received.filter(({ kind }) => kind === "ack")).toHaveLength(2); + expect(worker.received.filter(({ kind }) => kind === "closed")).toEqual([ + { kind: "closed", origin: sent.origin }, + ]); }); }); diff --git a/packages/chord/test/next/support.ts b/packages/chord/test/next/support.ts index 7037ab438..20732c04f 100644 --- a/packages/chord/test/next/support.ts +++ b/packages/chord/test/next/support.ts @@ -40,6 +40,7 @@ import { type HostError, type PluginFailure, type RefState, + type RouteHandle, type Self, type StoreSelf, } from "../../src/next/host/index.ts"; @@ -770,6 +771,8 @@ export type Resolved = { export type Relayed = { readonly tab: Host; readonly server: Host; + /** The server's route to `Session`'s addresses. */ + readonly route: RouteHandle; /** The worker of each resolved session address, the latest one per key. */ readonly workers: Map>; /** The server's end of each resolved route link, in order. */ @@ -851,7 +854,7 @@ export async function relayed( return { link: a, principal, trusted: options.trusted, codec }; }; const resolve = options.resolve ?? (async (_address, made) => made()); - server.route( + const route = server.route( Session, { authorize: options.authorize ?? (() => true), @@ -870,7 +873,7 @@ export async function relayed( const accepted = { principal: options.tab ?? { kind: "owner" }, codec: tabCodec, limits: options.limits } as const; server.accept(a, accepted, context); tab.connect(b, { principal: { kind: "server" }, codec: tabCodec }, context); - return { tab, server, workers, routeLinks, tabLink: a, resolves, errors }; + return { tab, server, route, workers, routeLinks, tabLink: a, resolves, errors }; } /** diff --git a/tsconfig.json b/tsconfig.json index b49e773ff..7a9cef85c 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -10,6 +10,15 @@ "@earendil-works/chord/delta": ["./packages/chord/src/delta/index.ts"], "@earendil-works/chord/next": ["./packages/chord/src/next/index.ts"], "@earendil-works/chord/next/plugins": ["./packages/chord/src/next-node/plugins/index.ts"], + "@earendil-works/chord/next/plugins/runtime": ["./packages/chord/src/next-node/plugins/runtime.ts"], + "@test/contracts/*": ["./packages/chord/test/next-node/fixtures/plugins/contracts/*.ts"], + "@example/contracts/*": ["./packages/chord/test/next/examples/35-plugins/plugins/contracts/*.ts"], + "@example/weather/*": ["./packages/chord/test/next/examples/36-plugin-failures/plugins/weather/*.ts"], + "@example/prices/*": ["./packages/chord/test/next/examples/36-plugin-failures/plugins/prices/*.ts"], + "@example/shapes/*": ["./packages/chord/test/next/examples/37-plugin-imports/plugins/shapes/*.ts"], + "@example/shared/*": ["./packages/chord/test/next/examples/38-watch-and-rollout/plugins/contracts/*.ts"], + "@earendil-works/chord/next/http": ["./packages/chord/src/next-node/http/index.ts"], + "@earendil-works/chord/next/workers": ["./packages/chord/src/next-node/workers/index.ts"], "@earendil-works/chord/node": ["./packages/chord/src/node.ts"], "@earendil-works/chord/output": ["./packages/chord/src/output.ts"], "@earendil-works/chord/telemetry-sqlite": ["./packages/chord/src/telemetry-sqlite/index.ts"], From b22c5f6941b3472cc071461f933b6b7bd072823f Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 20:43:29 +0000 Subject: [PATCH 105/175] docs(chord): slice 10 as built (design 23, spec, explainer) --- packages/chord/docs/next/decisions.md | 6 +- packages/chord/docs/next/design.md | 135 +++++++++++----- packages/chord/docs/next/http-workers/book.md | 63 ++++---- .../chord/docs/next/http-workers/index.html | 72 +++++---- packages/chord/docs/next/slice-10.md | 152 +++++------------- packages/chord/docs/next/slices.md | 5 + 6 files changed, 221 insertions(+), 212 deletions(-) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 941b1dee7..e7aed524b 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -1877,7 +1877,7 @@ validation and copies the v2 kernel added around Delta. reload, swapping the committed plugin back silently. Rejected: resolving with the report after the commit despite the abort (it would change the rule that an abort after the commit rejects). Examples for 7c (owner): full coverage, each example a folder with the host code and real plugin directories (35-39). -- **Slice 10's points (owner, 2026-10-30; `slice-10.md`):** (D1, "a") the HTTP component defines the registry `Http` +- **Slice 10's points (owner, 2026-10-30; `slice-10.md`):** (D1, "a"; `httpServer` **superseded** by `createHttp`, decisions "The HTTP core is a fetch handler" and "Settings are arguments") the HTTP component defines the registry `Http` (`"chord.http"`) and serves it through a facet of its own (`http.facet`, plugin name `chord.http`) that the app loads with its plugins; its `compose` builds the router; the server reads it with `host.registry(Http, ctx)` per request. (D2, "a") Wraps are built in slice 10 as designed (decision "HTTP routes and wraps", design §23): a contribution's @@ -1885,7 +1885,7 @@ validation and copies the v2 kernel added around Delta. phase in contributor order, by default around their own plugin's routes (`scope: "own"`; `"all"` explicit; `route` narrows to one, a missing one a reported conflict). The server's `headers` option stays for every response, also those no route makes (a 404, a refused request). - (D3, "a") Authentication is one app function, `authenticate(request, ctx) => P | undefined`, called once per request + (D3, "a"; **superseded** by "The HTTP server is a native object": the `Authenticators` registry) Authentication is one app function, `authenticate(request, ctx) => P | undefined`, called once per request and per upgrade; no Chord registry of resolvers (p3 composes its own from sharing, devices and the relay, with its own precedence rules). (D4, "k": a) Request bodies: `request.json(limit?)` (parsed JSON, 1 MiB by default, malformed: 400) and @@ -1948,7 +1948,7 @@ validation and copies the v2 kernel added around Delta. (R3, "a"; refines D9) The child reports only the flip: `{ idle: true }` when every count (`host.activity` and `busy`) becomes zero, `{ idle: false }` when one becomes nonzero; the server's idle clock starts at `true` (a status message costs about 6 µs on each side, a local call 2.9 µs: no report per call). - (R4, "a if configurable at runtime so allowed domains can be added and removed by a plugin") Before `authenticate`, the + (R4; **superseded in part** by "allowed hosts only through contributions" and "The HTTP component's security baseline": no `allowedHosts` option, provenance by the socket; "a if configurable at runtime so allowed domains can be added and removed by a plugin") Before `authenticate`, the component refuses (403) a `Host` that is neither loopback nor allowed, and an upgrade (`/chord`, a plugin socket) whose `Origin` is not the request's own origin; `request.origin` comes only from an accepted `Host` (`http://` on loopback, `https://` for an allowed host); `headers(request, principal)` gets the principal. Allowed hosts are the diff --git a/packages/chord/docs/next/design.md b/packages/chord/docs/next/design.md index 966776b47..d40d82677 100644 --- a/packages/chord/docs/next/design.md +++ b/packages/chord/docs/next/design.md @@ -243,7 +243,7 @@ host). The workers component (§23) does this. | `hold(Type, params, context, { readyTimeout? })` | a reference at once that follows its object until released (section 7) | | `close(context)` | closes the links (children, routes, then the parent), then ends every object at once and tears them down, the last to go live first (reason `close`); rejects with an `AggregateError` of the native closes that threw | | `revoke(r, context)` | revocation (section 4; slice 4b) | -| `root(fn, context, { principal? })` | a call scope for code outside objects (a Durable hook, a CLI): `fn` gets a handler's `ctx`, sealed when `fn` settles, `context` aborts or the host closes (slice 8 D3, section 8) | +| `root(fn, context, { principal? })` | a call scope for code outside objects (a Durable hook, a CLI, an HTTP request): `fn` gets a handler's `ctx`, sealed when `fn` settles, `context` aborts or the host closes (slice 8 D3, section 8); an omitted `principal` acts as the host, an explicit `undefined` runs anonymous (slice 10) | | `registry(R, context, { readyTimeout? })` | registry `R`'s composed value for code outside objects once it is usable; waits for the publication, then while it has none up to its `readyTimeout`, then rejects with the last error (slice 8, 7a, section 17) | **Options of `createHost`:** `address`, `principal`, `principalKey` (section 4; the same function on every host), @@ -2113,32 +2113,70 @@ public API; nothing in the kernel depends on them. ### HTTP (`@earendil-works/chord/next/http`) -`httpServer(host, { listen, allowedHosts?, authenticate, headers?, links?, closeTimeout?, onError }, ctx)` listens -first; requests and upgrades wait only for the router's publication (`host.registry(Http)`, resolved once), never for -the start barrier. `Host` (loopback or allowed: the option plus contributed `hosts`) and an upgrade's `Origin` are -checked before anything else (R4); connections through a host no longer allowed close. `authenticate(request, ctx) => P | undefined` is the app's one hook (D3: who a -request is, and which credential wins, is app policy; no Chord registry of resolvers); `headers` are added to every -response, also those no route makes; `links.path` (`/chord`) upgrades to `host.accept(webSocketLink(ws), { principal, -limits })`, a socket without a principal refused. `ws` is an optional peer dependency (D7). - -The registry `Http` (`"chord.http"`) is defined by the component and served by its own facet (`http.facet`, loaded -with the plugins, D1); plugins contribute `routes`, `wraps` and `sockets`. A route is `(req, ctx) => -Promise`, run in a `host.root` scope with the request's principal (`ctx.principal`) and a signal aborted -when the client disconnects; `req` has `method`, `path`, `params`, `query`, `headers`, `origin`, `json(limit?)` and -`body(limit?)` (D4: nothing read until asked; 1 MiB by default; past the limit 413). Patterns (D8): exact segments, -`:name`, a trailing `/*`; the most specific match wins, segment by segment from the left; one pattern from two -contributions is a reported conflict, the first by plugin name wins; `HEAD` falls back to `GET`. A request uses the -router as it was when it started. +The core is one Web-standard handler built from the registry `Http` (decision "The HTTP core is a fetch handler"): ```ts -type HttpResponse = { readonly headers?: Readonly> } & ( // D5 +const http = createHttp(server, { // settings are arguments, fixed for the component's life + listen: { host: "127.0.0.1", port: 8788 }, // Chord's own Node listener; none: the app mounts `fetch` + links: { path: "/chord", options: (request, principal) => ({ limits, maxPayload }) }, + closeTimeout: 30_000, + onError, // HttpError +}, ctx); +await plugins.load(server, { kind: "server", core: [{ name: "chord.http", facet: http.facet }, appCore], modules }, ctx); +// an app with its own server: hono.all("/*", (c) => http.fetch(c.req.raw, { localAddress, localPort, remoteAddress })) +// plain Node: nodeRequest(message, res) converts with the listener's checks; server.on("upgrade", http.upgrade) +await http.close(ctx); // before host.close: drains running requests +``` + +- **The facet** (plugin `chord.http`, passed as the same object at every reload) serves the registries `Http` + (`compose` builds the router) and `Authenticators`, and with `listen` the listener: a host-lifetime object whose + native is Chord's Node server, opened right after `load`'s publication, so a reload never closes it (decisions "The + HTTP server is a native object", "Settings are arguments, extensions are registries"). `fetch` resolves the + registries once; requests before the publication wait (503 past `readyTimeout`). +- **Contributions to `Http`:** `{ routes?, wraps?, sockets?, maxPayload?, hosts?, framers?, relays?, csp? }`, a value + or a writable (a change recomposes the router). A request uses the router as it was when it started. +- **The security baseline** (decision "The HTTP component's security baseline", on for every request and upgrade + before any plugin code): (1) **provenance**: local only when the connection's local address is loopback, `Host` is + a loopback name with the port and no `Forwarded`/`X-Forwarded-*` header; a forwarded request on loopback only + through a contributed relay whose `verify` names its origin (`{ kind: "relayed", relay, origin }`); otherwise only a + contributed host (https); duplicate or missing `Host`, userinfo in it, absolute-form targets: 403; an unparsable + target 400; `TRACE`, `TRACK`, `CONNECT` 501 (a route for them is refused at compose). A mounted handler without + `localAddress` and `localPort` is never local. (2) **Foreign requests** (Fetch Metadata) never reach the + authenticators: anonymous, except a top-level `GET` navigation to a `page: true` route. (3) A foreign `Origin` is + 403; an unsafe method from a browser needs the app's own `Origin`; a `crossSite: true` route opts out and always + runs anonymous. (4) **Default headers** on every answer, the component's own included: `no-store`, `nosniff`, + `no-referrer`, `X-Frame-Options: DENY` (without framers), a CSP (`default-src 'none'`, scripts `'self'` and a + per-response nonce `request.nonce`, `connect-src` self and its WebSocket, `frame-ancestors` the framers or + `'none'`, plus contributed `csp` sources); a response's own headers replace same-named ones. (5) `cookie(name, + value, { secret, request, sameSite?, maxAge? })` and `readCookie(request, name, secret)`: `HttpOnly`, `Path=/`, + `SameSite=Strict` or `Lax`, `Secure` on a remote origin, value and expiry signed. (6) `bytes` refuse foreign + requests, are attachments unless `inline` (then with a `sandbox` CSP), download names stripped of control and + bidirectional characters. +- **Who:** the `Authenticators` registry, in contributor order (an app needing precedence contributes one that + decides); the first principal wins, a throwing one is reported and skipped, none: anonymous. They run raced against + the request's signal and stop once it aborts. Who someone is stays app policy (D3); the registry replaces D3's + single function so plugins (a device pairing, a share) can add credentials. +- **A request's scope:** one `host.root` with its principal (an explicit `undefined`: anonymous; an anonymous scope + gets no remote object served here, `forbidden` → 403) spans matching, wraps, the route and the response until its + body is written or abandoned; its signal aborts on disconnect. `HttpRequest`: `method`, `path`, `params`, `query`, + `headers`, `origin`, `provenance`, `nonce`, `json(limit?)` (1 MiB; 400, 413), `body(limit?)`. +- **Patterns** (D8): exact segments, `:name`, a trailing `/*`; decoded segments with `/`, `\`, NUL, `.` or `..` are + 400; the most specific match wins segment by segment; a pattern from two contributions is a reported conflict, the + first by plugin name wins; `HEAD` falls back to `GET`; no route: 404. A route is a function or `{ handler, page?, + crossSite? }`. Wraps (D2) as below. + +```ts +type HttpResponse = { readonly headers?: Readonly> } & ( // D5 | { kind: "json"; value: JsonValue; status?: number } | { kind: "html"; body: string; status?: number } - | { kind: "bytes"; mime: string; size?: number; read(offset, length, ctx): Promise; download?: string; ranges?: boolean } // no size: chunked (R6) + | { kind: "bytes"; mime: string; size?: number; read(offset, length, ctx): Promise; download?: string; inline?: true; ranges?: boolean } | { kind: "redirect"; to: string; status?: 302 | 303 | 307 } | { kind: "error"; status: number; message?: string }); ``` -The component does ranges (206/416), `HEAD`, safe download headers, 64 KiB reads with backpressure stopped on -disconnect, and a 500 without the message for a throwing route; `fileResponse(path, options)` serves a file. +`bytes` are pulled in 256 KiB reads as the consumer reads (ranges 206/416, `HEAD`, chunked without `size`, R6); no +read starts once the request's scope is sealed; a short or throwing read destroys the connection. 204, 205 and 304 +carry no body. `fileResponse(path, { download?, inline?, ranges?, mime?, root? })` stats at once (missing: 404; with +`root`, a real path outside it: 403), opens its handle at the first read, closes it when the scope seals. A throwing +route or wrap is a 500 without its message. ```ts type Wrap = { @@ -2152,26 +2190,47 @@ type Wrap = { Wraps (D2) run within a phase in contributor order (no numeric priorities: plugins have no single registration order; Fastify's lifecycle phases and encapsulation are the model). Wraps may add a security boundary on top of a -route's checks, never instead of them. `sockets` (D6): `{ "/terminal": (socket, req, ctx) => … }`, a `ws` WebSocket -after the origin check and `authenticate` (no principal: 403 before plugin code), for a plugin's own protocol; wraps do -not apply; closed (1012) when the contribution that handed them out leaves the table (§17). +route's checks, never instead of them. + +- **Sockets** (D6): `sockets: { "/terminal/:id": (socket, request, ctx) => … }`, a `ws` WebSocket after admission + and the authenticators (no principal: 403 before plugin code), `maxPayload` per contribution (1 MiB), pinged after + 15 s of silence and terminated 15 s later. **Links:** `links.path` upgrades to `host.accept(webSocketLink(socket), { + principal, limits })`; a client that stops reading is cut off past 4 MiB waiting. **What is removed is closed at + once** (decisions Q6, "allowed hosts only through contributions"): a link or socket whose host, relay or socket + handler leaves the router is terminated and its scope sealed; an upgrade is checked again against the current router + after its authenticators answered. +- **`close(ctx)`:** new requests and upgrades refused (503), sockets and links closed (1001, terminated at + `closeTimeout`), running requests waited for up to `closeTimeout`, then sealed; before `host.close`. +- **Cost** (decision "a route's overhead accepted"): +20-21 µs per request over a plain Node handler (standard bench), + +12-13 µs warmed. `ws` is an optional peer dependency (D7). ### Workers (`@earendil-works/chord/next/workers`) -`createWorkers(host, Kind, { kind, entry, start, plugins, lock, idleMs?, onOutput?, onError }, ctx)` routes `Kind` -itself (a forked worker per address over IPC, the link made at the fork, a start message with the app's start data and -the worker kind's complete scripts; the 60 s start timer covers fork, lock and connect only, Q4), with a `state` -watchable per address, `stop`, `retry` and `close`; `workerMain(fn, { hostPackages, redact, onError })` is the child's -side. One live worker per address (a kernel lock the child takes, waiting for a draining old worker without polling). -Keep-alive (D9, R3, Q5): the child reports when `host.activity`'s references, jobs and lingering objects and the app's -`busy` counts flip between all zero and not (local calls do not count; the first state with `ready`); the server stops -a worker idle (all zero) for `idleMs` (5 min). Workers start on demand and never restart by themselves (R2): a crash, -a `fatal`, a link lost while its child lives (killed after the grace) or a failed start leaves the address failed with -its error; its holders see `failed { waiting }` and stay on the path (Q1); `workers.retry(address)` or a rollout -reaching workers clears it and starts the held ones (Q2). `workers.stop` suspends the address while it has holders -(R1, Q3). Workers never build (D11): the server's plugins component builds every plugin; workers evaluate shipped -complete sets (`createPluginRuntime`, no esbuild), and `workers.step(name)` ships a rollout's set to every worker (D12: -one result per host; a worker stopped cleanly during it is listed `stopped`). +`createWorkers(host, Kind, { kind, entry, execArgv?, env?, authorize, principal, start, lock, idleMs?, plugins, +onOutput?, onError }, ctx)` routes `Kind` itself (`host.route`, keeping its `RouteHandle`): a forked worker per +address over IPC, the link made at the fork, a start message `{ address, start, scripts, lock }` with the app's start +data and the kind's complete current scripts (`plugins.scripts(kind)`); the 60 s start timer covers fork, lock and +connect only (Q4). `workers.state` (a JSON watchable per address key), `stop`, `retry`, `step(name)`, `close`; +`workerMain(fn, { hostPackages, redact, onError })` is the child's side (`fn({ address, start, parent, plugins, +onClose }, ctx)`, `plugins` a `WorkerPlugins` evaluating the shipped scripts, closers in reverse order). One live +worker per address (`processLock`: an abstract socket on Linux, a named pipe on Windows, a socket file elsewhere that +fails closed when stale; a start waits for a draining old holder without polling). + +- **Keep-alive** (D9, R3, Q5): the child reports when `host.activity`'s references, jobs and lingering objects and the + app's `busy` counts flip between all zero and not (local calls do not count; the first state with `ready`, which + also carries the load report); the server stops a worker idle for `idleMs` (5 min). +- **Never restarted by itself** (R2): a crash, a `fatal`, a link lost while its child lives (killed after a 5 s grace) + or a failed start leaves the address failed; its holders see `failed { waiting }` and stay on the path (Q1); + `retry(address)` or a rollout reaching workers clears it and starts the held ones (Q2). `stop` suspends the address + while it has holders (R1, Q3). +- **Errors in public and private** (decision "worker errors in public"): `state` and holders get a fixed message + (`worker failed to start`, `worker exited (SIGSEGV)`) or the child's redacted `fatal`; the full error and the stderr + tail go only to `onError` (`worker_failed`). Every app hook of a start (`start`, `principal`, `lock`, the fork) is + inside that boundary; a throwing `authorize` refuses the hold and is reported `authorize_failed`. `worker_exited` is + reported only for an exit nobody asked for. +- **Workers never build** (D11): the server's plugins component builds every plugin; workers evaluate shipped + complete sets (`createPluginRuntime`, no esbuild); `workers.step(name)` ships a rollout's set to every worker (D12: + `{ host, report } | { host, stopped: true } | { host, error }` per worker). ### Channels diff --git a/packages/chord/docs/next/http-workers/book.md b/packages/chord/docs/next/http-workers/book.md index 9b9992337..f340004ef 100644 --- a/packages/chord/docs/next/http-workers/book.md +++ b/packages/chord/docs/next/http-workers/book.md @@ -11,8 +11,7 @@ stats: note: | Written by the redesign session from `slice-10.md`, the decision entries "Slice 10's points", "Origins only from relays", "The HTTP component's security baseline" and their neighbours in `decisions.md`, design §3 and §23, and the - build in `/home/badlogic/pi-7b`. The build exists; its fix round (the security baseline among it) is next. Where the - build and the decisions differ, this book says what is decided. + build after its four fix rounds (security baseline, `fetch` core, error boundaries). colophon: Serving and Spawning · first edition · 2026-10-31 · Chord v2 slice 10 --- @@ -75,13 +74,14 @@ const plugins = createPlugins({ }, onError, }, ctx); -const http = await httpServer(server, { listen: { host: "127.0.0.1", port: 8788 }, authenticate, onError }, ctx); +const http = createHttp(server, { listen: { host: "127.0.0.1", port: 8788 }, onError }, ctx); // settings are arguments const workers = createWorkers(server, Session, { kind: "worker", entry: "./worker.ts", plugins, principal: (a) => ({ kind: "worker", session: a.params.id }), start: (address, ctx) => ({ id: address.params.id, dir: sessionDir(address) }), lock: (address) => `chat:${dataDir}:${address.key}`, authorize: mayReach, onError, }, ctx); await plugins.load(server, { kind: "server", core: [core, { name: "chord.http", facet: http.facet }], modules }, ctx); +// `core` contributes the app's authenticator: self.contribute(Authenticators, authenticate) ``` ```ts @@ -198,27 +198,29 @@ A route that throws is a 500 without its message: error text often holds paths o # Who Is Asking {#who lede="The app's one security function, and everything the component does so the app does not have to."} -The app writes exactly one function about identity: +The app writes one function about identity and contributes it to the registry `Authenticators` (plugins may add +more, a device pairing for instance; the first principal wins): ```ts -async function authenticate(request: AuthRequest, ctx: Context): Promise

    { - const device = verifyCookie(request.cookies.device, secret); - if (device) return { kind: "owner", access: "device", device: device.id }; - const guest = verifyCookie(request.cookies.guest, secret); - if (guest) return { kind: "guest", session: guest.session, grant: guest.id }; +const authenticate: Authenticator = async (request, ctx) => { + const device = readCookie(request, "device", secret); // signature and expiry checked + if (device) return { kind: "owner", access: "device", device }; + const guest = readCookie(request, "guest", secret); + if (guest) return { kind: "guest", grant: guest }; if (request.provenance.kind === "local") return { kind: "owner", access: "local" }; - return undefined; // anonymous -} + return undefined; // anonymous +}; +self.contribute(Authenticators, authenticate); // in the app's core facet ``` It never parses `Host`, never checks `Origin` and never sees a foreign request: the component already did all of it. It may read `request.provenance` (`local`, `remote` with the host, or `relayed` through a verified relay such as -Radius). Cookies are written with the component's helper, which sets `HttpOnly`, `SameSite`, `Secure` on a remote -origin and a signature. +Radius). Cookies are written with `cookie(name, value, { secret, request, maxAge })`, which sets `HttpOnly`, +`SameSite`, `Secure` on a remote origin, and signs the value with its expiry. Every response, the component's own 404s and 403s included, carries the same headers: no caching, no MIME sniffing, no referrer, no framing (`X-Frame-Options: DENY` and `frame-ancestors 'none'`, except for framers the app allowed), and a -Content Security Policy with a per-server script nonce, so an injected ` + From 7ef8680abd32a4804f080a7228f47373b426cc79 Mon Sep 17 00:00:00 2001 From: Armin Ronacher Date: Sat, 31 Oct 2026 22:25:03 +0100 Subject: [PATCH 109/175] fix: sync esbuild platform package dev flags in lockfile esbuild is a non-dev package through chord's optional peer dependency, so its optional @esbuild/* platform packages are non-dev as well. npm install regenerates this; the committed lockfile had stale dev flags. --- package-lock.json | 26 -------------------------- 1 file changed, 26 deletions(-) diff --git a/package-lock.json b/package-lock.json index 02e687f30..f02d0d520 100644 --- a/package-lock.json +++ b/package-lock.json @@ -854,7 +854,6 @@ "cpu": [ "ppc64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -871,7 +870,6 @@ "cpu": [ "arm" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -888,7 +886,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -905,7 +902,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -922,7 +918,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -939,7 +934,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -956,7 +950,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -973,7 +966,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -990,7 +982,6 @@ "cpu": [ "arm" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1007,7 +998,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1024,7 +1014,6 @@ "cpu": [ "ia32" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1041,7 +1030,6 @@ "cpu": [ "loong64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1058,7 +1046,6 @@ "cpu": [ "mips64el" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1075,7 +1062,6 @@ "cpu": [ "ppc64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1092,7 +1078,6 @@ "cpu": [ "riscv64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1109,7 +1094,6 @@ "cpu": [ "s390x" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1126,7 +1110,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1143,7 +1126,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1160,7 +1142,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1177,7 +1158,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1194,7 +1174,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1211,7 +1190,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1228,7 +1206,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1245,7 +1222,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1262,7 +1238,6 @@ "cpu": [ "ia32" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1279,7 +1254,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ From fab2c5353ea4adcbb0d35701897892941f185235 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 21:39:27 +0000 Subject: [PATCH 110/175] feat(schema): add pi-schema, JSON Schema builders and validation without eval @earendil-works/pi-schema compiles JSON Schema (drafts 2020-12 and 7) once into plain closures: no eval or new Function, so it runs under a strict CSP. It passes 2,222 of the 2,223 core tests of the official JSON-Schema-Test-Suite, checks Chord-style argument schemas within 1.2x of TypeBox's JIT, and keeps 3,200 compiled schemas in 7.9 MB (TypeBox: 54.7 MB). - compile(schema, options) returns a Validator: check() is the boolean check (a type guard for builder schemas), errors() reports each failure with a JSON Pointer path, missing and unknown fields at the field. - Builders (str, int, obj, opt, union, ...) return plain JSON Schema typed with the values it accepts (Static). - Formats are an optional entry point (./formats), adapted from TypeBox (MIT). - Tests run the vendored suite (drafts 2020-12 and 7, with optional and format tests); npm run bench compares conformance, speed, setup cost, memory and bundle size with TypeBox, @cfworker/json-schema and omptype. --- package-lock.json | 36 + package.json | 4 +- packages/schema/CHANGELOG.md | 9 + packages/schema/LICENSES/typebox-LICENSE | 23 + packages/schema/README.md | 178 ++ packages/schema/package.json | 56 + packages/schema/src/annotations.ts | 208 ++ packages/schema/src/builders.ts | 187 ++ packages/schema/src/checks.ts | 446 +++++ packages/schema/src/compiler.ts | 618 ++++++ packages/schema/src/explain.ts | 327 ++++ packages/schema/src/formats/index.ts | 46 + packages/schema/src/formats/typebox/date.ts | 47 + .../schema/src/formats/typebox/date_time.ts | 40 + .../schema/src/formats/typebox/duration.ts | 38 + packages/schema/src/formats/typebox/email.ts | 38 + .../schema/src/formats/typebox/hostname.ts | 39 + .../schema/src/formats/typebox/idn_email.ts | 38 + .../src/formats/typebox/idn_hostname.ts | 40 + .../src/formats/typebox/idna/format/bidi.ts | 127 ++ .../src/formats/typebox/idna/format/puny.ts | 167 ++ .../src/formats/typebox/idna/hostname.ts | 42 + .../src/formats/typebox/idna/idn-hostname.ts | 72 + .../schema/src/formats/typebox/idna/index.ts | 30 + .../src/formats/typebox/idna/label/ascii.ts | 40 + .../src/formats/typebox/idna/label/puny.ts | 53 + .../src/formats/typebox/idna/label/unicode.ts | 122 ++ .../formats/typebox/idna/pattern/pattern.ts | 116 ++ packages/schema/src/formats/typebox/ipv4.ts | 37 + packages/schema/src/formats/typebox/ipv6.ts | 38 + packages/schema/src/formats/typebox/iri.ts | 63 + .../src/formats/typebox/iri_reference.ts | 42 + .../src/formats/typebox/json_pointer.ts | 38 + packages/schema/src/formats/typebox/regex.ts | 40 + .../formats/typebox/relative_json_pointer.ts | 38 + packages/schema/src/formats/typebox/time.ts | 56 + packages/schema/src/formats/typebox/uri.ts | 38 + .../src/formats/typebox/uri_reference.ts | 38 + .../src/formats/typebox/uri_template.ts | 40 + packages/schema/src/formats/typebox/uuid.ts | 38 + packages/schema/src/index.ts | 31 + packages/schema/src/resolver.ts | 152 ++ packages/schema/src/shared.ts | 50 + packages/schema/src/types.ts | 36 + packages/schema/src/validator.ts | 58 + packages/schema/test/bench/all.ts | 36 + packages/schema/test/bench/candidates.ts | 101 + packages/schema/test/bench/cases.ts | 261 +++ packages/schema/test/bench/conformance.ts | 54 + packages/schema/test/bench/corpus.ts | 122 ++ packages/schema/test/bench/memory.ts | 73 + packages/schema/test/bench/setup.ts | 51 + packages/schema/test/bench/size.ts | 43 + packages/schema/test/bench/speed.ts | 62 + packages/schema/test/builders.test.ts | 104 + packages/schema/test/conformance.test.ts | 51 + .../fixtures/json-schema-test-suite/LICENSE | 19 + .../fixtures/json-schema-test-suite/README.md | 9 + .../remotes/baseUriChange/folderInteger.json | 3 + .../baseUriChangeFolder/folderInteger.json | 3 + .../folderInteger.json | 3 + .../draft2019-09/dependentRequired.json | 7 + .../draft2019-09/ignore-prefixItems.json | 7 + .../baseUriChange/folderInteger.json | 4 + .../baseUriChangeFolder/folderInteger.json | 4 + .../folderInteger.json | 4 + .../draft2020-12/detached-dynamicref.json | 13 + .../remotes/draft2020-12/detached-ref.json | 13 + .../draft2020-12/different-id-ref-string.json | 5 + .../draft2020-12/extendible-dynamic-ref.json | 21 + .../draft2020-12/format-assertion-false.json | 13 + .../draft2020-12/format-assertion-true.json | 13 + .../remotes/draft2020-12/integer.json | 4 + .../locationIndependentIdentifier.json | 12 + .../metaschema-no-validation.json | 13 + .../metaschema-optional-vocabulary.json | 14 + .../remotes/draft2020-12/name-defs.json | 16 + .../nested-absolute-ref-to-string.json | 9 + .../draft2020-12/nested/foo-ref-string.json | 7 + .../remotes/draft2020-12/nested/string.json | 4 + .../remotes/draft2020-12/prefixItems.json | 7 + .../remotes/draft2020-12/ref-and-defs.json | 12 + .../remotes/draft2020-12/subSchemas.json | 11 + .../remotes/draft2020-12/tree.json | 17 + .../remotes/draft2020-12/urn-ref-string.json | 5 + .../remotes/draft7/detached-ref.json | 13 + .../draft7/ignore-dependentRequired.json | 7 + .../draft7/locationIndependentIdentifier.json | 11 + .../remotes/draft7/name.json | 15 + .../remotes/draft7/ref-and-definitions.json | 11 + .../remotes/draft7/subSchemas.json | 10 + .../remotes/integer.json | 3 + .../remotes/nested/foo-ref-string.json | 6 + .../remotes/nested/string.json | 3 + .../draft2020-12/additionalProperties.json | 219 +++ .../tests/draft2020-12/allOf.json | 312 +++ .../tests/draft2020-12/anchor.json | 120 ++ .../tests/draft2020-12/anyOf.json | 203 ++ .../tests/draft2020-12/boolean_schema.json | 104 + .../tests/draft2020-12/const.json | 431 +++++ .../tests/draft2020-12/contains.json | 176 ++ .../tests/draft2020-12/content.json | 131 ++ .../tests/draft2020-12/default.json | 82 + .../tests/draft2020-12/defs.json | 21 + .../tests/draft2020-12/dependentRequired.json | 152 ++ .../tests/draft2020-12/dependentSchemas.json | 171 ++ .../tests/draft2020-12/dynamicRef.json | 874 +++++++++ .../tests/draft2020-12/enum.json | 419 ++++ .../tests/draft2020-12/exclusiveMaximum.json | 31 + .../tests/draft2020-12/exclusiveMinimum.json | 31 + .../tests/draft2020-12/format.json | 838 ++++++++ .../tests/draft2020-12/if-then-else.json | 308 +++ .../draft2020-12/infinite-loop-detection.json | 37 + .../tests/draft2020-12/items.json | 304 +++ .../tests/draft2020-12/maxContains.json | 123 ++ .../tests/draft2020-12/maxItems.json | 50 + .../tests/draft2020-12/maxLength.json | 55 + .../tests/draft2020-12/maxProperties.json | 94 + .../tests/draft2020-12/maximum.json | 60 + .../tests/draft2020-12/minContains.json | 224 +++ .../tests/draft2020-12/minItems.json | 50 + .../tests/draft2020-12/minLength.json | 55 + .../tests/draft2020-12/minProperties.json | 70 + .../tests/draft2020-12/minimum.json | 75 + .../tests/draft2020-12/multipleOf.json | 102 + .../tests/draft2020-12/not.json | 301 +++ .../tests/draft2020-12/oneOf.json | 293 +++ .../tests/draft2020-12/optional/anchor.json | 60 + .../tests/draft2020-12/optional/bignum.json | 110 ++ .../draft2020-12/optional/cross-draft.json | 18 + .../optional/dependencies-compatibility.json | 282 +++ .../draft2020-12/optional/dynamicRef.json | 56 + .../optional/ecmascript-regex.json | 582 ++++++ .../draft2020-12/optional/float-overflow.json | 17 + .../optional/format-assertion.json | 44 + .../optional/format/date-time.json | 250 +++ .../draft2020-12/optional/format/date.json | 423 +++++ .../optional/format/duration.json | 268 +++ .../optional/format/ecmascript-regex.json | 116 ++ .../draft2020-12/optional/format/email.json | 368 ++++ .../optional/format/hostname.json | 386 ++++ .../optional/format/idn-email.json | 140 ++ .../optional/format/idn-hostname.json | 588 ++++++ .../draft2020-12/optional/format/ipv4.json | 217 +++ .../draft2020-12/optional/format/ipv6.json | 266 +++ .../optional/format/iri-reference.json | 116 ++ .../draft2020-12/optional/format/iri.json | 141 ++ .../optional/format/json-pointer.json | 211 +++ .../draft2020-12/optional/format/regex.json | 103 + .../format/relative-json-pointer.json | 141 ++ .../draft2020-12/optional/format/time.json | 303 +++ .../draft2020-12/optional/format/unknown.json | 46 + .../optional/format/uri-reference.json | 182 ++ .../optional/format/uri-template.json | 251 +++ .../draft2020-12/optional/format/uri.json | 253 +++ .../draft2020-12/optional/format/uuid.json | 156 ++ .../tests/draft2020-12/optional/id.json | 53 + .../draft2020-12/optional/no-schema.json | 26 + .../draft2020-12/optional/non-bmp-regex.json | 86 + .../optional/refOfUnknownKeyword.json | 112 ++ .../draft2020-12/optional/unknownKeyword.json | 57 + .../tests/draft2020-12/pattern.json | 90 + .../tests/draft2020-12/patternProperties.json | 211 +++ .../tests/draft2020-12/prefixItems.json | 104 + .../tests/draft2020-12/properties.json | 242 +++ .../tests/draft2020-12/propertyNames.json | 178 ++ .../tests/draft2020-12/ref.json | 1085 +++++++++++ .../tests/draft2020-12/refRemote.json | 342 ++++ .../tests/draft2020-12/required.json | 169 ++ .../tests/draft2020-12/type.json | 501 +++++ .../tests/draft2020-12/unevaluatedItems.json | 854 +++++++++ .../draft2020-12/unevaluatedProperties.json | 1681 +++++++++++++++++ .../tests/draft2020-12/uniqueItems.json | 399 ++++ .../tests/draft2020-12/vocabulary.json | 57 + .../tests/draft7/additionalItems.json | 190 ++ .../tests/draft7/additionalProperties.json | 147 ++ .../tests/draft7/allOf.json | 294 +++ .../tests/draft7/anyOf.json | 189 ++ .../tests/draft7/boolean_schema.json | 104 + .../tests/draft7/const.json | 384 ++++ .../tests/draft7/contains.json | 165 ++ .../tests/draft7/default.json | 79 + .../tests/draft7/definitions.json | 26 + .../tests/draft7/dependencies.json | 286 +++ .../tests/draft7/enum.json | 341 ++++ .../tests/draft7/exclusiveMaximum.json | 30 + .../tests/draft7/exclusiveMinimum.json | 30 + .../tests/draft7/format.json | 614 ++++++ .../tests/draft7/if-then-else.json | 296 +++ .../tests/draft7/infinite-loop-detection.json | 36 + .../tests/draft7/items.json | 282 +++ .../tests/draft7/maxItems.json | 44 + .../tests/draft7/maxLength.json | 49 + .../tests/draft7/maxProperties.json | 70 + .../tests/draft7/maximum.json | 54 + .../tests/draft7/minItems.json | 44 + .../tests/draft7/minLength.json | 49 + .../tests/draft7/minProperties.json | 64 + .../tests/draft7/minimum.json | 69 + .../tests/draft7/multipleOf.json | 87 + .../tests/draft7/not.json | 259 +++ .../tests/draft7/oneOf.json | 274 +++ .../tests/draft7/optional/bignum.json | 93 + .../tests/draft7/optional/content.json | 77 + .../tests/draft7/optional/cross-draft.json | 25 + .../draft7/optional/ecmascript-regex.json | 552 ++++++ .../tests/draft7/optional/float-overflow.json | 13 + .../draft7/optional/format/date-time.json | 247 +++ .../tests/draft7/optional/format/date.json | 420 ++++ .../optional/format/ecmascript-regex.json | 98 + .../tests/draft7/optional/format/email.json | 365 ++++ .../draft7/optional/format/hostname.json | 380 ++++ .../draft7/optional/format/idn-email.json | 137 ++ .../draft7/optional/format/idn-hostname.json | 582 ++++++ .../tests/draft7/optional/format/ipv4.json | 214 +++ .../tests/draft7/optional/format/ipv6.json | 263 +++ .../draft7/optional/format/iri-reference.json | 113 ++ .../tests/draft7/optional/format/iri.json | 138 ++ .../draft7/optional/format/json-pointer.json | 208 ++ .../tests/draft7/optional/format/regex.json | 100 + .../format/relative-json-pointer.json | 138 ++ .../tests/draft7/optional/format/time.json | 300 +++ .../tests/draft7/optional/format/unknown.json | 43 + .../draft7/optional/format/uri-reference.json | 179 ++ .../draft7/optional/format/uri-template.json | 248 +++ .../tests/draft7/optional/format/uri.json | 250 +++ .../tests/draft7/optional/id.json | 114 ++ .../tests/draft7/optional/non-bmp-regex.json | 82 + .../tests/draft7/optional/unknownKeyword.json | 56 + .../tests/draft7/pattern.json | 59 + .../tests/draft7/patternProperties.json | 171 ++ .../tests/draft7/properties.json | 236 +++ .../tests/draft7/propertyNames.json | 164 ++ .../tests/draft7/ref.json | 1043 ++++++++++ .../tests/draft7/refRemote.json | 257 +++ .../tests/draft7/required.json | 161 ++ .../tests/draft7/type.json | 474 +++++ .../tests/draft7/uniqueItems.json | 389 ++++ .../test/fixtures/meta-schemas/README.md | 10 + .../meta-schemas/draft-07-schema.json | 172 ++ .../draft2020-12-meta-applicator.json | 45 + .../draft2020-12-meta-content.json | 14 + .../meta-schemas/draft2020-12-meta-core.json | 48 + .../draft2020-12-meta-format-annotation.json | 11 + .../draft2020-12-meta-meta-data.json | 34 + .../draft2020-12-meta-unevaluated.json | 12 + .../draft2020-12-meta-validation.json | 95 + .../meta-schemas/draft2020-12-schema.json | 58 + packages/schema/test/shared.test.ts | 41 + packages/schema/test/suite.ts | 77 + packages/schema/test/validator.test.ts | 190 ++ packages/schema/tsconfig.build.json | 9 + packages/schema/vitest.config.ts | 11 + scripts/check-entry-graphs.mjs | 5 + tsconfig.json | 2 + 255 files changed, 37963 insertions(+), 2 deletions(-) create mode 100644 packages/schema/CHANGELOG.md create mode 100644 packages/schema/LICENSES/typebox-LICENSE create mode 100644 packages/schema/README.md create mode 100644 packages/schema/package.json create mode 100644 packages/schema/src/annotations.ts create mode 100644 packages/schema/src/builders.ts create mode 100644 packages/schema/src/checks.ts create mode 100644 packages/schema/src/compiler.ts create mode 100644 packages/schema/src/explain.ts create mode 100644 packages/schema/src/formats/index.ts create mode 100644 packages/schema/src/formats/typebox/date.ts create mode 100644 packages/schema/src/formats/typebox/date_time.ts create mode 100644 packages/schema/src/formats/typebox/duration.ts create mode 100644 packages/schema/src/formats/typebox/email.ts create mode 100644 packages/schema/src/formats/typebox/hostname.ts create mode 100644 packages/schema/src/formats/typebox/idn_email.ts create mode 100644 packages/schema/src/formats/typebox/idn_hostname.ts create mode 100644 packages/schema/src/formats/typebox/idna/format/bidi.ts create mode 100644 packages/schema/src/formats/typebox/idna/format/puny.ts create mode 100644 packages/schema/src/formats/typebox/idna/hostname.ts create mode 100644 packages/schema/src/formats/typebox/idna/idn-hostname.ts create mode 100644 packages/schema/src/formats/typebox/idna/index.ts create mode 100644 packages/schema/src/formats/typebox/idna/label/ascii.ts create mode 100644 packages/schema/src/formats/typebox/idna/label/puny.ts create mode 100644 packages/schema/src/formats/typebox/idna/label/unicode.ts create mode 100644 packages/schema/src/formats/typebox/idna/pattern/pattern.ts create mode 100644 packages/schema/src/formats/typebox/ipv4.ts create mode 100644 packages/schema/src/formats/typebox/ipv6.ts create mode 100644 packages/schema/src/formats/typebox/iri.ts create mode 100644 packages/schema/src/formats/typebox/iri_reference.ts create mode 100644 packages/schema/src/formats/typebox/json_pointer.ts create mode 100644 packages/schema/src/formats/typebox/regex.ts create mode 100644 packages/schema/src/formats/typebox/relative_json_pointer.ts create mode 100644 packages/schema/src/formats/typebox/time.ts create mode 100644 packages/schema/src/formats/typebox/uri.ts create mode 100644 packages/schema/src/formats/typebox/uri_reference.ts create mode 100644 packages/schema/src/formats/typebox/uri_template.ts create mode 100644 packages/schema/src/formats/typebox/uuid.ts create mode 100644 packages/schema/src/index.ts create mode 100644 packages/schema/src/resolver.ts create mode 100644 packages/schema/src/shared.ts create mode 100644 packages/schema/src/types.ts create mode 100644 packages/schema/src/validator.ts create mode 100644 packages/schema/test/bench/all.ts create mode 100644 packages/schema/test/bench/candidates.ts create mode 100644 packages/schema/test/bench/cases.ts create mode 100644 packages/schema/test/bench/conformance.ts create mode 100644 packages/schema/test/bench/corpus.ts create mode 100644 packages/schema/test/bench/memory.ts create mode 100644 packages/schema/test/bench/setup.ts create mode 100644 packages/schema/test/bench/size.ts create mode 100644 packages/schema/test/bench/speed.ts create mode 100644 packages/schema/test/builders.test.ts create mode 100644 packages/schema/test/conformance.test.ts create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/LICENSE create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/README.md create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/baseUriChange/folderInteger.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/baseUriChangeFolder/folderInteger.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/baseUriChangeFolderInSubschema/folderInteger.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2019-09/dependentRequired.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2019-09/ignore-prefixItems.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/baseUriChange/folderInteger.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/baseUriChangeFolder/folderInteger.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/baseUriChangeFolderInSubschema/folderInteger.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/detached-dynamicref.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/detached-ref.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/different-id-ref-string.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/extendible-dynamic-ref.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/format-assertion-false.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/format-assertion-true.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/integer.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/locationIndependentIdentifier.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/metaschema-no-validation.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/metaschema-optional-vocabulary.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/name-defs.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/nested-absolute-ref-to-string.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/nested/foo-ref-string.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/nested/string.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/prefixItems.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/ref-and-defs.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/subSchemas.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/tree.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/urn-ref-string.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/detached-ref.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/ignore-dependentRequired.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/locationIndependentIdentifier.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/name.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/ref-and-definitions.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/subSchemas.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/integer.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/nested/foo-ref-string.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/remotes/nested/string.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/additionalProperties.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/allOf.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/anchor.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/anyOf.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/boolean_schema.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/const.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/contains.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/content.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/default.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/defs.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/dependentRequired.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/dependentSchemas.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/dynamicRef.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/enum.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/exclusiveMaximum.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/exclusiveMinimum.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/format.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/if-then-else.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/infinite-loop-detection.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/items.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maxContains.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maxItems.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maxLength.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maxProperties.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maximum.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minContains.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minItems.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minLength.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minProperties.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minimum.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/multipleOf.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/not.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/oneOf.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/anchor.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/bignum.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/cross-draft.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/dependencies-compatibility.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/dynamicRef.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/ecmascript-regex.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/float-overflow.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format-assertion.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/date-time.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/date.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/duration.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/ecmascript-regex.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/email.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/hostname.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/idn-email.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/idn-hostname.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/ipv4.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/ipv6.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/iri-reference.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/iri.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/json-pointer.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/regex.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/relative-json-pointer.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/time.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/unknown.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/uri-reference.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/uri-template.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/uri.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/uuid.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/id.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/no-schema.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/non-bmp-regex.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/refOfUnknownKeyword.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/unknownKeyword.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/pattern.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/patternProperties.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/prefixItems.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/properties.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/propertyNames.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/ref.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/refRemote.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/required.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/type.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/unevaluatedItems.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/unevaluatedProperties.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/uniqueItems.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/vocabulary.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/additionalItems.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/additionalProperties.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/allOf.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/anyOf.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/boolean_schema.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/const.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/contains.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/default.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/definitions.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/dependencies.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/enum.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/exclusiveMaximum.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/exclusiveMinimum.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/format.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/if-then-else.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/infinite-loop-detection.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/items.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/maxItems.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/maxLength.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/maxProperties.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/maximum.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/minItems.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/minLength.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/minProperties.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/minimum.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/multipleOf.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/not.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/oneOf.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/bignum.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/content.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/cross-draft.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/ecmascript-regex.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/float-overflow.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/date-time.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/date.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/ecmascript-regex.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/email.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/hostname.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/idn-email.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/idn-hostname.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/ipv4.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/ipv6.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/iri-reference.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/iri.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/json-pointer.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/regex.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/relative-json-pointer.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/time.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/unknown.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/uri-reference.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/uri-template.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/uri.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/id.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/non-bmp-regex.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/unknownKeyword.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/pattern.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/patternProperties.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/properties.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/propertyNames.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/ref.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/refRemote.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/required.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/type.json create mode 100644 packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/uniqueItems.json create mode 100644 packages/schema/test/fixtures/meta-schemas/README.md create mode 100644 packages/schema/test/fixtures/meta-schemas/draft-07-schema.json create mode 100644 packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-applicator.json create mode 100644 packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-content.json create mode 100644 packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-core.json create mode 100644 packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-format-annotation.json create mode 100644 packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-meta-data.json create mode 100644 packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-unevaluated.json create mode 100644 packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-validation.json create mode 100644 packages/schema/test/fixtures/meta-schemas/draft2020-12-schema.json create mode 100644 packages/schema/test/shared.test.ts create mode 100644 packages/schema/test/suite.ts create mode 100644 packages/schema/test/validator.test.ts create mode 100644 packages/schema/tsconfig.build.json create mode 100644 packages/schema/vitest.config.ts diff --git a/package-lock.json b/package-lock.json index f02d0d520..d26f79648 100644 --- a/package-lock.json +++ b/package-lock.json @@ -570,6 +570,13 @@ "dev": true, "license": "MIT OR Apache-2.0" }, + "node_modules/@cfworker/json-schema": { + "version": "4.1.1", + "resolved": "https://registry.npmjs.org/@cfworker/json-schema/-/json-schema-4.1.1.tgz", + "integrity": "sha512-gAmrUZSGtKc3AiBL71iNWxDsyUC5uMaKKGdvzYsBoTW/xi42JQHl7eKV2OYzCUqvc+D2RCcf7EXY2iCyFIk6og==", + "dev": true, + "license": "MIT" + }, "node_modules/@cloudflare/kv-asset-handler": { "version": "0.5.0", "resolved": "https://registry.npmjs.org/@cloudflare/kv-asset-handler/-/kv-asset-handler-0.5.0.tgz", @@ -828,6 +835,10 @@ "resolved": "packages/mcp", "link": true }, + "node_modules/@earendil-works/pi-schema": { + "resolved": "packages/schema", + "link": true + }, "node_modules/@earendil-works/pi-telemetry": { "resolved": "packages/telemetry", "link": true @@ -1896,6 +1907,17 @@ "@jridgewell/sourcemap-codec": "^1.4.10" } }, + "node_modules/@oh-my-pi/omptype": { + "version": "18.13.3", + "resolved": "https://registry.npmjs.org/@oh-my-pi/omptype/-/omptype-18.13.3.tgz", + "integrity": "sha512-ZLi+kb3ff554n2ndH4iEviVRWHI1t78LZu7qf/NSXkYf4STcl6vEnJ5A2FBhfUaTLVk/HRs09LzyFnySOkwpvQ==", + "dev": true, + "license": "MIT", + "engines": { + "bun": ">=1.4.4", + "node": ">=20" + } + }, "node_modules/@oxc-project/types": { "version": "0.154.0", "resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.154.0.tgz", @@ -5748,6 +5770,20 @@ "node": ">=22.19.0" } }, + "packages/schema": { + "name": "@earendil-works/pi-schema", + "version": "1.4.0", + "license": "MIT", + "devDependencies": { + "@cfworker/json-schema": "4.1.1", + "@oh-my-pi/omptype": "18.13.3", + "typebox": "1.3.27", + "vitest": "4.1.12" + }, + "engines": { + "node": ">=22.19.0" + } + }, "packages/telemetry": { "name": "@earendil-works/pi-telemetry", "version": "1.4.0", diff --git a/package.json b/package.json index b158f2f00..352767d97 100644 --- a/package.json +++ b/package.json @@ -12,8 +12,8 @@ ], "scripts": { "clean": "npm run clean --workspaces", - "build": "cd packages/telemetry && npm run build && cd ../chord && npm run build && cd ../tui && npm run build && cd ../codemode && npm run build && cd ../mcp && npm run build && cd ../ai && npm run build && cd ../durable && npm run build && cd ../env && npm run build && cd ../agent && npm run build && cd ../coding-agent && npm run build", - "build:offline": "cd packages/telemetry && npm run build && cd ../chord && npm run build && cd ../tui && npm run build && cd ../codemode && npm run build && cd ../mcp && npm run build && cd ../ai && npm run build:offline && cd ../durable && npm run build && cd ../env && npm run build && cd ../agent && npm run build && cd ../coding-agent && npm run build", + "build": "cd packages/telemetry && npm run build && cd ../schema && npm run build && cd ../chord && npm run build && cd ../tui && npm run build && cd ../codemode && npm run build && cd ../mcp && npm run build && cd ../ai && npm run build && cd ../durable && npm run build && cd ../env && npm run build && cd ../agent && npm run build && cd ../coding-agent && npm run build", + "build:offline": "cd packages/telemetry && npm run build && cd ../schema && npm run build && cd ../chord && npm run build && cd ../tui && npm run build && cd ../codemode && npm run build && cd ../mcp && npm run build && cd ../ai && npm run build:offline && cd ../durable && npm run build && cd ../env && npm run build && cd ../agent && npm run build && cd ../coding-agent && npm run build", "build:native:darwin": "npm --prefix packages/tui run build:native:darwin", "build:native:linux": "npm --prefix packages/tui run build:native:linux", "build:native:win32": "npm --prefix packages/tui run build:native:win32", diff --git a/packages/schema/CHANGELOG.md b/packages/schema/CHANGELOG.md new file mode 100644 index 000000000..d23ca137c --- /dev/null +++ b/packages/schema/CHANGELOG.md @@ -0,0 +1,9 @@ +# Changelog + +## [Unreleased] + +### Added + +- Initial release: `compile(schema, options)` validates JSON Schema drafts 2020-12 and 7 without `eval` or `new Function`, compiling a schema once into plain closures. `Validator.check` is the boolean check (a type guard for schemas built with the builders), `Validator.errors` reports each failure with a JSON Pointer path. Format checks are in the optional `@earendil-works/pi-schema/formats` entry point. +- Schema builders (`str`, `num`, `int`, `bool`, `nul`, `unknown`, `lit`, `oneOf`, `obj`, `opt`, `arr`, `tuple`, `map`, `union`, `nullable`, `bytes`) that return plain JSON Schema typed with the values it accepts (`Static`). +- `npm run bench` compares conformance, speed, setup cost, memory and bundle size with TypeBox, @cfworker/json-schema and omptype. diff --git a/packages/schema/LICENSES/typebox-LICENSE b/packages/schema/LICENSES/typebox-LICENSE new file mode 100644 index 000000000..0330da432 --- /dev/null +++ b/packages/schema/LICENSES/typebox-LICENSE @@ -0,0 +1,23 @@ +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. \ No newline at end of file diff --git a/packages/schema/README.md b/packages/schema/README.md new file mode 100644 index 000000000..d31caa521 --- /dev/null +++ b/packages/schema/README.md @@ -0,0 +1,178 @@ +# @earendil-works/pi-schema + +JSON Schema builders and validation without `eval`. A schema compiles once into plain closures, so it checks values +about as fast as validators that generate code, also under a strict Content Security Policy, and needs no +dependencies. + +- Drafts 2020-12 and 7. Passes 2,222 of the 2,223 core tests of the official + [JSON-Schema-Test-Suite](https://github.com/json-schema-org/JSON-Schema-Test-Suite). +- Builders return plain JSON Schema typed with the values it accepts; a compiled check narrows to that type. +- Errors as JSON Pointer paths. A missing or unknown field is reported at the field. +- Never modifies a schema; frozen schemas work. +- 8.5 KB minified and gzipped. Format checks are an optional import. + +## Installation + +```bash +npm install @earendil-works/pi-schema +``` + +## Usage + +```typescript +import { compile, int, obj, oneOf, opt, type Static, str } from "@earendil-works/pi-schema"; + +const Read = obj({ + path: str({ description: "Path to the file", minLength: 1 }), + offset: opt(int({ minimum: 1 })), + mode: oneOf(["text", "binary"]), +}); +type Read = Static; // { path: string; mode: "text" | "binary"; offset?: number } + +const read = compile(Read); +if (read.check(value)) value.path; // value: Read + +read.errors({ offset: 0, extra: true }); +// [ +// { path: "/path", keyword: "required", message: "missing" }, +// { path: "/mode", keyword: "required", message: "missing" }, +// { path: "/offset", keyword: "minimum", message: "expected >= 1" }, +// { path: "/extra", keyword: "additionalProperties", message: "unknown field" }, +// ] +``` + +`Read` is plain JSON Schema (`{ type: "object", properties: {...}, required: ["path", "mode"], additionalProperties: +false }`), ready for a tool definition or `JSON.stringify`. `compile` takes any JSON Schema, built with the builders +or not. + +`check(value)` is the fast path: a boolean, no allocations for most schemas, and a plain function you can pass +around. `errors(value)` explains a failure and returns `[]` for a valid value. It is built on the first failure and +kept, so a schema whose values always pass never pays for it. + +### Builders + +| builder | schema | type | +|---|---|---| +| `str({ minLength, maxLength, pattern, format })` | `{ type: "string", ... }` | `string` | +| `num({ minimum, maximum, exclusiveMinimum, exclusiveMaximum, multipleOf })`, `int(...)` | `{ type: "number" }`, `{ type: "integer" }` | `number` | +| `bool()`, `nul()`, `unknown()` | `{ type: "boolean" }`, `{ type: "null" }`, `{}` | `boolean`, `null`, `unknown` | +| `lit(value)` | `{ const: value }` | the literal | +| `oneOf(["a", "b"])` | `{ enum: [...] }` | `"a" \| "b"` | +| `obj({ a: str(), b: opt(num()) }, { open })` | closed object; `open: true` allows other properties | `{ a: string; b?: number }` | +| `arr(items, { minItems, maxItems, uniqueItems })` | `{ type: "array", items }` | `T[]` | +| `tuple([str(), num()])` | `prefixItems`, no more items | `[string, number]` | +| `map(value)` | `{ type: "object", additionalProperties: value }` | `Record` | +| `union([...])` | `{ anyOf: [...] }` | the union | +| `nullable(schema)` | `{ anyOf: [schema, { type: "null" }] }` | `T \| null` | +| `bytes({ minLength, maxLength })` | `{ type: "bytes", ... }` | `Uint8Array` | + +Every builder also takes `title`, `description`, `examples`, `deprecated` and `default`. A `default` is an annotation; +validation never inserts it. + +A union of objects tagged by a `lit()` property checks and reports by the tag: only the branch the tag selects runs. + +```typescript +const Block = union([obj({ kind: lit("text"), text: str() }), obj({ kind: lit("image"), width: int({ minimum: 0 }) })]); +compile(Block).errors({ kind: "video" }); +// [{ path: "/kind", keyword: "anyOf", message: 'expected one of ["text","image"]' }] +``` + +### Options + +```typescript +import { formats } from "@earendil-works/pi-schema/formats"; + +const validator = compile(schema, { + draft: "7", // "2020-12" (default) or "7" + formats, // check `format`; without it `format` is an annotation, as 2020-12 specifies + schemas: { "https://example.com/point.json": point }, // documents for $refs to other documents, by URI +}); +``` + +`formats` has every format 2020-12 defines (`date-time`, `email`, `uri`, `uuid`, `ipv6`, ...). Pass your own map to +check fewer formats, or custom ones. A `$ref` that names nothing throws when the schema compiles. + +### TypeBox schemas + +TypeBox 1.x schemas are JSON Schema, so they compile as they are: + +```typescript +import Type from "typebox"; + +const user = compile(Type.Object({ name: Type.String(), tags: Type.Array(Type.String()) })); +``` + +TypeBox's JavaScript-only types (`BigInt`, `Undefined`, `Function`, ...), refinements and codecs are not JSON Schema +and are not supported. + +### Values + +- A property whose value is `undefined` counts as absent, as in TypeBox. +- `{ type: "bytes", minLength, maxLength }` matches a `Uint8Array` of that length, an extension for values that never + pass through JSON. + +## Supported keywords + +- Every validation keyword of drafts 2020-12 and 7, including draft 7's `dependencies` and `items` arrays with + `additionalItems`. +- `$id`, `$anchor`, `$ref` (local, by anchor, relative and remote), `$dynamicRef` and `$dynamicAnchor`. +- `unevaluatedProperties` and `unevaluatedItems`. +- `pattern` and `patternProperties` are ECMAScript regular expressions with the `u` flag. +- Not supported: meta-schemas that switch vocabularies on or off (`$vocabulary`), `contentMediaType` and + `contentEncoding` assertions (optional in the spec), and draft 2019-09's `$recursiveRef`. + +## Benchmarks + +Node 24.4.1 on one core of an i9-13900, against TypeBox 1.3.27, @cfworker/json-schema 4.1.1 and @oh-my-pi/omptype +18.13.3 (`fromJsonSchema`). + +| test suite (tests passed) | pi-schema | TypeBox | cfworker | omptype | +|---|---:|---:|---:|---:| +| draft 2020-12 core (1,298) | 1,297 | 1,276 | 1,249 | 863 | +| draft 7 core (925) | 925 | 924 | 918 | 659 | +| draft 2020-12 format (895), with `formats` | 895 | 895 | 740 | 412 | + +| check, valid value | pi-schema | TypeBox | cfworker | omptype | +|---|---:|---:|---:|---:| +| nested object with tagged unions | 535 ns | 464 ns | 43 µs | 713 ns | +| union of 5 messages | 96 ns | 94 ns | 10 µs | 94 ns | +| tool schema (read) | 34 ns | 9 ns | 2.0 µs | 8 ns | +| MCP tool schema with `$ref` | 207 ns | 88 ns | 13 µs | 1.7 µs | +| recursive `$ref` schema | 1.1 µs | 1.2 µs | 41 µs | 7.4 µs | + +| same, without eval (strict CSP) | pi-schema | TypeBox | cfworker | omptype | +|---|---:|---:|---:|---:| +| nested object with tagged unions | 527 ns | 42 µs | 44 µs | 2.7 µs | +| MCP tool schema with `$ref` | 209 ns | 38 µs | 13 µs | 2.1 µs | +| recursive `$ref` schema | 1.1 µs | 134 µs | 43 µs | 8.9 µs | + +| cost | pi-schema | TypeBox | cfworker | omptype | +|---|---:|---:|---:|---:| +| compile a new schema | 43 µs | 210 µs | 23 µs | 19 µs + 114 µs on its first check | +| memory, 3,200 distinct schemas | 7.9 MB | 54.7 MB | 7.6 MB | 20.0 MB | +| bundle, minified + gzip | 8.5 KB | 35.2 KB | 5.8 KB | 38.8 KB | + +Reproduce with `npm run bench` in `packages/schema` (about 5 minutes). It runs every library through its fastest +boolean path and keeps the object each returns. The scripts are in `test/bench/`. + +- **conformance:** every library on the vendored test suite, with the remote documents and meta-schemas the tests + reference. +- **speed:** realistic schemas, in ns per check after warm-up; the median of 15 rounds, with the libraries' + order rotated in each round. +- **speed without eval:** the same, run with `--disallow-code-generation-from-strings`. +- **setup:** compiling 1,000 schemas a library has not seen, and their first checks. +- **memory:** 3,200 distinct tool-like schemas, each library in its own process. +- **size:** esbuild bundles of what a validator needs, minified and gzipped. + +Where code generation is unavailable, TypeBox falls back to its interpreter, and omptype throws `EvalError` once per +schema, then validates through its interpreter. + +Flat objects of simple types check in 15–90 ns, against 8–15 ns for TypeBox. Without code generation, reading a +property whose name comes from a list costs a few nanoseconds. + +## Development + +`npm test` runs the official test suite (drafts 2020-12 and 7, including the optional and format tests, vendored in +`test/fixtures/`) and the package's own tests. + +The format checks are adapted from TypeBox 1.3.27 (MIT); its license is in `LICENSES/`. diff --git a/packages/schema/package.json b/packages/schema/package.json new file mode 100644 index 000000000..f6e94e1d2 --- /dev/null +++ b/packages/schema/package.json @@ -0,0 +1,56 @@ +{ + "name": "@earendil-works/pi-schema", + "version": "1.4.0", + "description": "JSON Schema validation without eval: drafts 2020-12 and 7, compiled into plain closures", + "type": "module", + "main": "./dist/index.js", + "types": "./dist/index.d.ts", + "exports": { + ".": { + "source": "./src/index.ts", + "types": "./dist/index.d.ts", + "import": "./dist/index.js" + }, + "./formats": { + "source": "./src/formats/index.ts", + "types": "./dist/formats/index.d.ts", + "import": "./dist/formats/index.js" + } + }, + "sideEffects": false, + "files": [ + "dist", + "README.md", + "CHANGELOG.md", + "LICENSES" + ], + "scripts": { + "clean": "node ../../scripts/package-files.mjs clean dist", + "build": "tsc -p tsconfig.build.json", + "test": "vitest --run", + "bench": "node --expose-gc test/bench/all.ts", + "prepublishOnly": "npm run clean && npm run build" + }, + "keywords": [ + "json-schema", + "validation", + "validator", + "csp" + ], + "author": "Earendil Works", + "license": "MIT", + "repository": { + "type": "git", + "url": "git+https://github.com/earendil-works/pi.git", + "directory": "packages/schema" + }, + "engines": { + "node": ">=22.19.0" + }, + "devDependencies": { + "@cfworker/json-schema": "4.1.1", + "@oh-my-pi/omptype": "18.13.3", + "typebox": "1.3.27", + "vitest": "4.1.12" + } +} diff --git a/packages/schema/src/annotations.ts b/packages/schema/src/annotations.ts new file mode 100644 index 000000000..5bc9cc4f5 --- /dev/null +++ b/packages/schema/src/annotations.ts @@ -0,0 +1,208 @@ +// Annotation tracking for `unevaluatedProperties` and `unevaluatedItems`: annotating checks record which +// properties and items a schema evaluated. Only schemas that use those keywords compile into these closures. + +import { isObject } from "./checks.ts"; +import type { Check } from "./types.ts"; + +/** What a schema evaluated: property names (or all of them), the evaluated item prefix, items matched by `contains`. */ +export interface Annotations { + props: Set | null; + allProps: boolean; + items: number; + allItems: boolean; + indexes: Set | null; +} + +/** A check that records what it evaluated into `a` when it passes. */ +export type AnnotatingCheck = (value: unknown, a: Annotations) => boolean; + +/** Empty annotations. */ +export function newAnnotations(): Annotations { + return { props: null, allProps: false, items: 0, allItems: false, indexes: null }; +} + +function merge(a: Annotations, b: Annotations): void { + if (b.props !== null) for (const k of b.props) addProp(a, k); + a.allProps ||= b.allProps; + if (b.items > a.items) a.items = b.items; + a.allItems ||= b.allItems; + if (b.indexes !== null) for (const i of b.indexes) addIndex(a, i); +} + +function addProp(a: Annotations, key: string): void { + if (a.props === null) a.props = new Set(); + a.props.add(key); +} + +function addIndex(a: Annotations, index: number): void { + if (a.indexes === null) a.indexes = new Set(); + a.indexes.add(index); +} + +export const ANNOTATING_TRUE: AnnotatingCheck = () => true; +export const ANNOTATING_FALSE: AnnotatingCheck = () => false; + +/** The indirection a recursive reference goes through. */ +export function indirectAnnotating(slot: { check: AnnotatingCheck | undefined }): AnnotatingCheck { + return (v, a) => (slot.check as AnnotatingCheck)(v, a); +} + +/** A plain check in annotating position: it evaluates nothing that unevaluated* sees. */ +export function ignoreAnnotations(check: Check): AnnotatingCheck { + return (v) => check(v); +} + +/** An annotating check run as a plain check, with fresh annotations. */ +export function withFreshAnnotations(check: AnnotatingCheck): Check { + return (v) => check(v, newAnnotations()); +} + +/** `check` with `uri` pushed onto the dynamic scope while it runs. */ +export function scopedAnnotating(scope: string[], uri: string, check: AnnotatingCheck): AnnotatingCheck { + return (v, a) => { + scope.push(uri); + const ok = check(v, a); + scope.pop(); + return ok; + }; +} + +/** unevaluated* sees only the annotations of its own schema, never those of its siblings in the parent. */ +export function isolateAnnotations(check: AnnotatingCheck): AnnotatingCheck { + return (v, a) => { + const b = newAnnotations(); + if (!check(v, b)) return false; + merge(a, b); + return true; + }; +} + +/** The non-annotating keywords, then each annotating part in order. */ +export function annotateAll(plain: Check, parts: readonly AnnotatingCheck[]): AnnotatingCheck { + return (v, a) => { + if (!plain(v)) return false; + for (let i = 0; i < parts.length; i++) if (!parts[i](v, a)) return false; + return true; + }; +} + +/** `properties`, `patternProperties` and `additionalProperties`. */ +export function annotateObject( + props: readonly (readonly [string, Check])[], + patterns: readonly (readonly [RegExp, Check])[], + additional: Check | undefined, +): AnnotatingCheck { + const declared = new Set(props.map(([k]) => k)); + return (v, a) => { + if (!isObject(v)) return true; + const seen: string[] = []; + for (const [k, check] of props) { + if (!Object.hasOwn(v, k)) continue; + if (!check(v[k])) return false; + seen.push(k); + } + for (const k of Object.keys(v)) { + let matched = declared.has(k); + for (const [re, check] of patterns) { + if (re.test(k)) { + if (!check(v[k])) return false; + matched = true; + seen.push(k); + } + } + if (!matched && additional !== undefined) { + if (!additional(v[k])) return false; + seen.push(k); + } + } + for (const k of seen) addProp(a, k); + return true; + }; +} + +/** `prefixItems`/`items`/`additionalItems` and `contains` with its bounds. */ +export function annotateArray( + prefix: readonly Check[], + rest: Check | undefined, + contains: Check | undefined, + minContains: number, + maxContains: number, +): AnnotatingCheck { + return (v, a) => { + if (!Array.isArray(v)) return true; + const m = Math.min(v.length, prefix.length); + for (let i = 0; i < m; i++) if (!prefix[i](v[i])) return false; + if (rest !== undefined) { + for (let i = prefix.length; i < v.length; i++) if (!rest(v[i])) return false; + a.allItems = true; + } + if (m > a.items) a.items = m; + if (contains !== undefined) { + let n = 0; + for (let i = 0; i < v.length; i++) { + if (contains(v[i])) { + n++; + addIndex(a, i); + } + } + if (n < minContains || n > maxContains) return false; + } + return true; + }; +} + +/** `anyOf` (or `oneOf` when `exactlyOne`): every passing branch contributes its annotations. */ +export function annotateAny(branches: readonly AnnotatingCheck[], exactlyOne: boolean): AnnotatingCheck { + return (v, a) => { + let n = 0; + for (const branch of branches) { + const b = newAnnotations(); + if (branch(v, b)) { + n++; + merge(a, b); + } + } + return exactlyOne ? n === 1 : n > 0; + }; +} + +/** `if`, `then` and `else`: a passing `if` contributes its annotations even without `then`. */ +export function annotateIf( + cond: AnnotatingCheck, + then: AnnotatingCheck | undefined, + otherwise: AnnotatingCheck | undefined, +): AnnotatingCheck { + return (v, a) => { + const b = newAnnotations(); + if (cond(v, b)) { + merge(a, b); + return then === undefined || then(v, a); + } + return otherwise === undefined || otherwise(v, a); + }; +} + +/** One `dependentSchemas` entry. */ +export function annotateDependent(key: string, check: AnnotatingCheck): AnnotatingCheck { + return (v, a) => !isObject(v) || !Object.hasOwn(v, key) || check(v, a); +} + +/** `unevaluatedProperties`: every property no other keyword evaluated passes `check`. */ +export function unevaluatedProperties(check: Check): AnnotatingCheck { + return (v, a) => { + if (!isObject(v) || a.allProps) return true; + for (const k of Object.keys(v)) if (!(a.props?.has(k) ?? false) && !check(v[k])) return false; + a.allProps = true; + return true; + }; +} + +/** `unevaluatedItems`: every item no other keyword evaluated passes `check`. */ +export function unevaluatedItems(check: Check): AnnotatingCheck { + return (v, a) => { + if (!Array.isArray(v) || a.allItems) return true; + for (let i = a.items; i < v.length; i++) if (!(a.indexes?.has(i) ?? false) && !check(v[i])) return false; + a.allItems = true; + return true; + }; +} diff --git a/packages/schema/src/builders.ts b/packages/schema/src/builders.ts new file mode 100644 index 000000000..1f4e21a13 --- /dev/null +++ b/packages/schema/src/builders.ts @@ -0,0 +1,187 @@ +// Schema builders: short functions that return plain JSON Schema objects and carry the static type of the values +// they accept, so `Static` is the TypeScript type a valid value has. The objects are ordinary data: +// they serialize with JSON.stringify and compile with compile() like any other schema. + +declare const STATIC: unique symbol; +declare const OPTIONAL_TYPE: unique symbol; + +/** A JSON Schema object whose valid values have type `T` (a type-level brand; nothing extra exists at run time). */ +export type Schema = { readonly [keyword: string]: unknown } & { readonly [STATIC]?: T }; + +/** The type of the values `S` accepts; `unknown` for schemas not built with these functions. */ +export type Static = S extends { readonly [STATIC]?: infer T } ? (unknown extends T ? unknown : T) : unknown; + +/** Annotations every builder accepts: they describe the schema and never change what it accepts. */ +export interface Annotations { + readonly title?: string; + readonly description?: string; + readonly examples?: readonly T[]; + readonly deprecated?: boolean; + /** A default value, an annotation for tools and forms; validation never inserts it. */ + readonly default?: T; +} + +/** Length bounds of a string, an array or bytes. */ +export interface LengthBounds { + readonly minLength?: number; + readonly maxLength?: number; +} + +/** A string, with optional length bounds (in code points), pattern and format. */ +export function str( + options: Annotations & LengthBounds & { readonly pattern?: string; readonly format?: string } = {}, +): Schema { + return { type: "string", ...options }; +} + +/** Bounds of a number. */ +export interface NumberBounds { + readonly minimum?: number; + readonly maximum?: number; + readonly exclusiveMinimum?: number; + readonly exclusiveMaximum?: number; + readonly multipleOf?: number; +} + +/** A number within optional bounds. */ +export function num(options: Annotations & NumberBounds = {}): Schema { + return { type: "number", ...options }; +} + +/** An integer within optional bounds. */ +export function int(options: Annotations & NumberBounds = {}): Schema { + return { type: "integer", ...options }; +} + +/** A boolean. */ +export function bool(options: Annotations = {}): Schema { + return { type: "boolean", ...options }; +} + +/** `null`. */ +export function nul(options: Annotations = {}): Schema { + return { type: "null", ...options }; +} + +/** Any value. */ +export function unknown(options: Annotations = {}): Schema { + return { ...options }; +} + +/** Exactly `value` (a string, number, boolean or null). */ +export function lit( + value: V, + options: Annotations = {}, +): Schema { + return { const: value, ...options }; +} + +/** One of the given values. */ +export function oneOf( + values: V, + options: Annotations = {}, +): Schema { + return { enum: values, ...options }; +} + +/** + * Any of the given schemas. A union of objects tagged by a `lit()` property (`{ kind: lit("text"), ... }`) is checked + * by the tag: only the branch it selects runs, and its errors are the ones reported. + */ +export function union( + schemas: S, + options: Annotations> = {}, +): Schema> { + return { anyOf: schemas, ...options }; +} + +/** `schema` or `null`. */ +export function nullable(schema: Schema, options: Annotations = {}): Schema { + return { anyOf: [schema, { type: "null" }], ...options }; +} + +/** An array of `items`, with optional length bounds and unique items. */ +export function arr( + items: Schema, + options: Annotations & { + readonly minItems?: number; + readonly maxItems?: number; + readonly uniqueItems?: boolean; + } = {}, +): Schema { + return { type: "array", items, ...options }; +} + +/** A fixed-length array whose items match `items` in order. */ +export function tuple( + items: S, + options: Annotations<{ -readonly [K in keyof S]: Static }> = {}, +): Schema<{ -readonly [K in keyof S]: Static }> { + return { type: "array", prefixItems: items, items: false, minItems: items.length, ...options }; +} + +/** An object with string keys and values of `value` (a record). */ +export function map(value: Schema, options: Annotations> = {}): Schema> { + return { type: "object", additionalProperties: value, ...options }; +} + +/** A Uint8Array with optional length bounds (an extension of JSON Schema for values that never pass through JSON). */ +export function bytes(options: Annotations & LengthBounds = {}): Schema { + return { type: "bytes", ...options }; +} + +/** An optional property of obj(): `{ name: opt(str()) }` may omit `name` or set it to undefined. */ +export class Optional { + declare readonly [OPTIONAL_TYPE]?: S; + readonly schema: S; + + constructor(schema: S) { + this.schema = schema; + } +} + +/** Marks an obj() property as optional. */ +export function opt(schema: S): Optional { + return new Optional(schema); +} + +/** The properties of obj(): a schema or an opt() schema per name. */ +export type Properties = { readonly [name: string]: Schema | Optional }; + +/** The value type of an obj() schema. */ +export type StaticObject

    = Simplify< + { -readonly [K in keyof P as P[K] extends Optional ? never : K]: Static } & { + -readonly [K in keyof P as P[K] extends Optional ? K : never]?: P[K] extends Optional + ? Static + : never; + } +>; + +type Simplify = { [K in keyof T]: T[K] } & {}; + +/** + * A closed object: the given properties (required unless opt()) and no others. `open: true` allows other + * properties, as tool schemas from other systems often do. + */ +export function obj( + properties: P, + options: Annotations> & { readonly open?: boolean } = {}, +): Schema> { + const { open, ...annotations } = options; + const props: Record = {}; + const required: string[] = []; + for (const [name, property] of Object.entries(properties)) { + if (property instanceof Optional) props[name] = property.schema; + else { + props[name] = property; + required.push(name); + } + } + return { + type: "object", + properties: props, + ...(required.length > 0 ? { required } : {}), + ...(open === true ? {} : { additionalProperties: false }), + ...annotations, + }; +} diff --git a/packages/schema/src/checks.ts b/packages/schema/src/checks.ts new file mode 100644 index 000000000..b686c4c27 --- /dev/null +++ b/packages/schema/src/checks.ts @@ -0,0 +1,446 @@ +// Check factories: each returns a closure that captures only its arguments, never the compiler, so a compiled +// schema keeps no compile-time state alive. Leaf checks are module constants that object and array loops inline. + +import type { Check } from "./types.ts"; + +export const ALWAYS: Check = () => true; +export const NEVER: Check = () => false; +const IS_STRING: Check = (v) => typeof v === "string"; +const IS_NUMBER: Check = (v) => typeof v === "number"; +const IS_INTEGER: Check = (v) => Number.isInteger(v); +const IS_BOOLEAN: Check = (v) => typeof v === "boolean"; +const IS_NULL: Check = (v) => v === null; +export const IS_ARRAY: Check = (v) => Array.isArray(v); +const IS_BYTES: Check = (v) => v instanceof Uint8Array; +export const IS_OBJECT: Check = (v) => isObject(v); + +/** Whether `v` is a JSON object: not null, an array or bytes. */ +export function isObject(v: unknown): v is Record { + return typeof v === "object" && v !== null && !Array.isArray(v) && !(v instanceof Uint8Array); +} + +/** The check of a `type` name; `bytes` (a Uint8Array) extends JSON Schema for values that never pass through JSON. */ +export const TYPE_CHECKS: ReadonlyMap = new Map([ + ["string", IS_STRING], + ["number", IS_NUMBER], + ["integer", IS_INTEGER], + ["boolean", IS_BOOLEAN], + ["null", IS_NULL], + ["array", IS_ARRAY], + ["object", IS_OBJECT], + ["bytes", IS_BYTES], +]); + +/** Leaf checks that object and array loops run inline by code instead of calling them. */ +export const LEAF_CODES: ReadonlyMap = new Map([ + [IS_STRING, 1], + [IS_NUMBER, 2], + [IS_INTEGER, 3], + [IS_BOOLEAN, 4], + [IS_NULL, 5], + [ALWAYS, 6], +]); + +function leafOk(code: number, x: unknown): boolean { + switch (code) { + case 1: + return typeof x === "string"; + case 2: + return typeof x === "number"; + case 3: + return Number.isInteger(x); + case 4: + return typeof x === "boolean"; + case 5: + return x === null; + default: + return true; + } +} + +/** Whether a combined object or array check also checks the type: not at all, strictly, or passing other types. */ +export const GUARD_NONE = 0; +export const GUARD_STRICT = 1; +export const GUARD_LOOSE = 2; +export type Guard = typeof GUARD_NONE | typeof GUARD_STRICT | typeof GUARD_LOOSE; + +/** Flags of a declared property in an object check's entries: its leaf code and two bits. */ +export const CODE = 7; +export const REQUIRED = 8; +export const INHERITED = 16; + +/** All of `parts`; two and three parts get closures without a loop. */ +export function and(parts: readonly Check[]): Check { + switch (parts.length) { + case 0: + return ALWAYS; + case 1: + return parts[0]; + case 2: { + const [a, b] = parts; + return (v) => a(v) && b(v); + } + case 3: { + const [a, b, c] = parts; + return (v) => a(v) && b(v) && c(v); + } + default: + return (v) => { + for (let i = 0; i < parts.length; i++) if (!parts[i](v)) return false; + return true; + }; + } +} + +/** A check whose implementation is set later: the indirection a recursive reference goes through. */ +export function indirect(slot: { check: Check | undefined }): Check { + return (v) => (slot.check as Check)(v); +} + +/** `check` with `uri` pushed onto the dynamic scope while it runs (only for schemas with `$dynamicRef`). */ +export function scopedCheck(scope: string[], uri: string, check: Check): Check { + return (v) => { + scope.push(uri); + const ok = check(v); + scope.pop(); + return ok; + }; +} + +/** Any of the named types. */ +export function typeCheck(types: readonly unknown[]): Check { + const checks = types.map((t) => (typeof t === "string" ? TYPE_CHECKS.get(t) : undefined) ?? NEVER); + if (checks.length === 1) return checks[0]; + return (v) => { + for (let i = 0; i < checks.length; i++) if (checks[i](v)) return true; + return false; + }; +} + +/** A Uint8Array within the length bounds. */ +export function bytesCheck(min: number, max: number): Check { + return (v) => v instanceof Uint8Array && v.length >= min && v.length <= max; +} + +/** The number of Unicode code points of `s`: JSON Schema string lengths count code points, not UTF-16 units. */ +export function codePoints(s: string): number { + let n = 0; + for (let i = 0; i < s.length; i++, n++) { + const c = s.charCodeAt(i); + if (c >= 0xd800 && c <= 0xdbff) i++; + } + return n; +} + +// UTF-16 units bound code points from both sides (units / 2 <= points <= units), so most strings need no count. +function lengthOk(s: string, min: number, max: number): boolean { + const units = s.length; + if (units < min) return false; + if (units < 2 * min && codePoints(s) < min) return false; + return units <= max || codePoints(s) <= max; +} + +/** The string keywords: length bounds, pattern and format. */ +export function stringCheck( + strict: boolean, + min: number, + max: number, + re: RegExp | undefined, + format: ((value: string) => boolean) | undefined, +): Check { + if (re !== undefined && min === 0 && max === Number.POSITIVE_INFINITY && format === undefined) { + return strict ? (v) => typeof v === "string" && re.test(v) : (v) => typeof v !== "string" || re.test(v); + } + if (re === undefined && format === undefined) { + return strict + ? (v) => typeof v === "string" && lengthOk(v, min, max) + : (v) => typeof v !== "string" || lengthOk(v, min, max); + } + return (v) => { + if (typeof v !== "string") return !strict; + return lengthOk(v, min, max) && (re === undefined || re.test(v)) && (format === undefined || format(v)); + }; +} + +/** The number keywords. `mode`: 0 passes other types, 1 requires a number, 2 an integer. */ +export function numberCheck( + mode: 0 | 1 | 2, + min: number, + max: number, + xmin: number, + xmax: number, + multipleOf: number | undefined, +): Check { + return (v) => { + if (typeof v !== "number") return mode === 0; + if (mode === 2 && !Number.isInteger(v)) return false; + return v >= min && v <= max && v > xmin && v < xmax && (multipleOf === undefined || isMultiple(v, multipleOf)); + }; +} + +/** Whether `v` is a multiple of `d`, exactly for decimals: 0.0075 / 0.0001 is 74.99999999999999 in floating point. */ +export function isMultiple(v: number, d: number): boolean { + const q = v / d; + if (Number.isInteger(q)) return true; + if (!Number.isFinite(q)) return v % d === 0; + const [vc, ve] = decimal(v); + const [dc, de] = decimal(d); + return ve >= de ? (vc * 10n ** BigInt(ve - de)) % dc === 0n : vc % (dc * 10n ** BigInt(de - ve)) === 0n; +} + +function decimal(x: number): [bigint, number] { + const [mantissa, exponent = "0"] = Math.abs(x).toString().split("e"); + const point = mantissa.indexOf("."); + const fraction = point < 0 ? 0 : mantissa.length - point - 1; + return [BigInt(mantissa.replace(".", "")), Number(exponent) - fraction]; +} + +/** `inner` for objects; other values pass unless `strict`. */ +export function guardObject(inner: Check, strict: boolean): Check { + return strict ? (v) => isObject(v) && inner(v) : (v) => !isObject(v) || inner(v); +} + +/** `inner` for arrays; other values pass unless `strict`. */ +export function guardArray(inner: Check, strict: boolean): Check { + return strict ? (v) => Array.isArray(v) && inner(v) : (v) => !Array.isArray(v) || inner(v); +} + +/** + * An object with only declared properties (`additionalProperties: false`). `entries` is `[key, check, flags]` per + * property. Declared fields are checked, then the own keys are counted, so unknown keys are found without + * allocating. A property whose value is `undefined` counts as absent. + */ +export function closedObject(entries: readonly unknown[], guard: Guard): Check { + return (v) => { + if (guard !== GUARD_NONE && !isObject(v)) return guard === GUARD_LOOSE; + // Guarded: by the check above, or by the caller for GUARD_NONE. + const o = v as Record; + let present = 0; + for (let i = 0; i < entries.length; i += 3) { + const k = entries[i] as string; + const x = o[k]; + const f = entries[i + 2] as number; + if (x === undefined) { + if (f & REQUIRED) return false; + if (Object.hasOwn(o, k)) present++; + continue; + } + if (f & INHERITED && !Object.hasOwn(o, k)) { + if (f & REQUIRED) return false; + continue; + } + present++; + const c = f & CODE; + if (c !== 0 ? !leafOk(c, x) : !(entries[i + 1] as Check)(x)) return false; + } + let count = 0; + for (const _ in o) if (++count > present) return false; + return true; + }; +} + +/** An object with declared properties, required keys not declared, pattern properties and additional properties. */ +export function openObject( + entries: readonly unknown[], + keys: readonly string[], + extraRequired: readonly string[], + patterns: readonly (readonly [RegExp, Check])[], + additional: Check | undefined, + guard: Guard, +): Check { + const known = patterns.length > 0 || additional !== undefined ? new Set(keys) : undefined; + return (v) => { + if (guard !== GUARD_NONE && !isObject(v)) return guard === GUARD_LOOSE; + const o = v as Record; + for (let i = 0; i < entries.length; i += 3) { + const k = entries[i] as string; + const x = o[k]; + const f = entries[i + 2] as number; + if (x === undefined || (f & INHERITED && !Object.hasOwn(o, k))) { + if (f & REQUIRED) return false; + continue; + } + const c = f & CODE; + if (c !== 0 ? !leafOk(c, x) : !(entries[i + 1] as Check)(x)) return false; + } + for (let i = 0; i < extraRequired.length; i++) if (!Object.hasOwn(o, extraRequired[i])) return false; + if (known === undefined) return true; + for (const k of Object.keys(o)) { + let matched = known.has(k); + for (let i = 0; i < patterns.length; i++) { + if (patterns[i][0].test(k)) { + matched = true; + if (!patterns[i][1](o[k])) return false; + } + } + if (!matched && additional !== undefined && !additional(o[k])) return false; + } + return true; + }; +} + +/** Every property name passes `check` (`v` is an object). */ +export function propertyNamesCheck(check: Check): Check { + return (v) => { + for (const k in v as object) if (!check(k)) return false; + return true; + }; +} + +/** `minProperties` and `maxProperties` (`v` is an object). */ +export function propertyCountCheck(min: number, max: number): Check { + return (v) => { + let n = 0; + for (const _ in v as object) n++; + return n >= min && n <= max; + }; +} + +/** `dependentRequired` (`v` is an object). */ +export function dependentRequiredCheck(entries: readonly (readonly [string, readonly string[]])[]): Check { + return (v) => { + const o = v as object; + for (const [k, list] of entries) + if (Object.hasOwn(o, k)) for (const r of list) if (!Object.hasOwn(o, r)) return false; + return true; + }; +} + +/** `dependentSchemas` (`v` is an object). */ +export function dependentSchemasCheck(entries: readonly (readonly [string, Check])[]): Check { + return (v) => { + for (const [k, check] of entries) if (Object.hasOwn(v as object, k) && !check(v)) return false; + return true; + }; +} + +/** The common array: guard, length bounds, prefix items and the items after them, in one closure. */ +export function listCheck( + guard: Guard, + min: number, + max: number, + prefix: readonly Check[], + rest: Check | undefined, +): Check { + const n = prefix.length; + const code = rest === undefined ? 0 : (LEAF_CODES.get(rest) ?? 0); + return (v) => { + if (!Array.isArray(v)) return guard === GUARD_LOOSE; + const length = v.length; + if (length < min || length > max) return false; + const m = length < n ? length : n; + for (let i = 0; i < m; i++) if (!prefix[i](v[i])) return false; + if (rest === undefined) return true; + if (code !== 0) { + for (let i = n; i < length; i++) if (!leafOk(code, v[i])) return false; + return true; + } + for (let i = n; i < length; i++) if (!rest(v[i])) return false; + return true; + }; +} + +/** Prefix items and the items after them (`v` is an array). */ +export function itemsCheck(prefix: readonly Check[], rest: Check | undefined): Check { + return listCheck(GUARD_NONE, 0, Number.POSITIVE_INFINITY, prefix, rest); +} + +/** `minItems` and `maxItems` (`v` is an array). */ +export function lengthCheck(min: number, max: number): Check { + return (v) => (v as unknown[]).length >= min && (v as unknown[]).length <= max; +} + +/** `contains` with its bounds (`v` is an array). */ +export function containsCheck(check: Check, min: number, max: number): Check { + return (v) => { + const list = v as unknown[]; + let n = 0; + for (let i = 0; i < list.length; i++) if (check(list[i]) && ++n > max) return false; + return n >= min; + }; +} + +/** `uniqueItems` (`v` is an array). */ +export function uniqueCheck(v: unknown): boolean { + const list = v as unknown[]; + for (let i = 1; i < list.length; i++) for (let j = 0; j < i; j++) if (deepEqual(list[i], list[j])) return false; + return true; +} + +/** `enum`: up to 8 primitives by a scan, more by a Set, objects and arrays by deep equality. */ +export function enumCheck(values: readonly unknown[]): Check { + const primitives = values.filter((x) => typeof x !== "object" || x === null); + if (primitives.length === values.length && primitives.length <= 8) { + return (v) => { + for (let i = 0; i < primitives.length; i++) if (primitives[i] === v) return true; + return false; + }; + } + const set = new Set(primitives); + const objects = values.filter((x) => typeof x === "object" && x !== null); + return (v) => set.has(v) || (typeof v === "object" && objects.some((o) => deepEqual(v, o))); +} + +/** `const`. */ +export function constCheck(value: unknown): Check { + if (typeof value !== "object" || value === null) return (v) => v === value; + return (v) => deepEqual(v, value); +} + +/** `anyOf`. */ +export function anyCheck(checks: readonly Check[]): Check { + return (v) => { + for (let i = 0; i < checks.length; i++) if (checks[i](v)) return true; + return false; + }; +} + +/** `oneOf`. */ +export function oneCheck(checks: readonly Check[]): Check { + return (v) => { + let n = 0; + for (let i = 0; i < checks.length; i++) if (checks[i](v) && ++n > 1) return false; + return n === 1; + }; +} + +/** `not`. */ +export function notCheck(check: Check): Check { + return (v) => !check(v); +} + +/** `if`, `then` and `else`. */ +export function ifCheck(cond: Check, then: Check, otherwise: Check): Check { + return (v) => (cond(v) ? then(v) : otherwise(v)); +} + +/** A union of objects tagged by a const on `key`: the tag selects the one branch that can match. */ +export function taggedCheck(key: string, tags: readonly unknown[], checks: readonly Check[]): Check { + return (v) => { + if (!isObject(v)) return false; + const tag = v[key]; + for (let i = 0; i < tags.length; i++) if (tags[i] === tag) return checks[i](v); + return false; + }; +} + +/** JSON equality: same primitives, arrays with equal items in order, objects with the same keys and equal values. */ +export function deepEqual(a: unknown, b: unknown): boolean { + if (a === b) return true; + if (typeof a !== "object" || typeof b !== "object" || a === null || b === null) return false; + const array = Array.isArray(a); + if (array !== Array.isArray(b)) return false; + if (array) { + const x = a as unknown[]; + const y = b as unknown[]; + if (x.length !== y.length) return false; + for (let i = 0; i < x.length; i++) if (!deepEqual(x[i], y[i])) return false; + return true; + } + const x = a as Record; + const y = b as Record; + const keys = Object.keys(x); + if (keys.length !== Object.keys(y).length) return false; + for (const k of keys) if (!Object.hasOwn(y, k) || !deepEqual(x[k], y[k])) return false; + return true; +} diff --git a/packages/schema/src/compiler.ts b/packages/schema/src/compiler.ts new file mode 100644 index 000000000..01ec52bbf --- /dev/null +++ b/packages/schema/src/compiler.ts @@ -0,0 +1,618 @@ +// The compiler: a schema to one check closure, built once. Keywords group by the type they constrain, so a value +// is classified once per schema, and the common shapes (closed objects, arrays of one item schema, tagged unions, +// leaf schemas) get one specialized closure each. + +import { + ANNOTATING_FALSE, + ANNOTATING_TRUE, + type AnnotatingCheck, + annotateAll, + annotateAny, + annotateArray, + annotateDependent, + annotateIf, + annotateObject, + ignoreAnnotations, + indirectAnnotating, + isolateAnnotations, + scopedAnnotating, + unevaluatedItems, + unevaluatedProperties, + withFreshAnnotations, +} from "./annotations.ts"; +import { + ALWAYS, + and, + anyCheck, + bytesCheck, + closedObject, + constCheck, + containsCheck, + dependentRequiredCheck, + dependentSchemasCheck, + enumCheck, + GUARD_LOOSE, + GUARD_NONE, + GUARD_STRICT, + type Guard, + guardArray, + guardObject, + INHERITED, + IS_ARRAY, + IS_OBJECT, + ifCheck, + indirect, + itemsCheck, + LEAF_CODES, + lengthCheck, + listCheck, + NEVER, + notCheck, + numberCheck, + oneCheck, + openObject, + propertyCountCheck, + propertyNamesCheck, + REQUIRED, + scopedCheck, + stringCheck, + taggedCheck, + typeCheck, + uniqueCheck, +} from "./checks.ts"; +import { documentOf, isSchemaObject, Resolver, resolveUri } from "./resolver.ts"; +import { sharedCheck, sharedRegex } from "./shared.ts"; +import type { Check, CompileOptions, FormatCheck, JsonSchema, SchemaObject } from "./types.ts"; + +/** Thrown when a compile without dynamic scope tracking meets a bookended `$dynamicRef`: compile again with it. */ +export const RESTART = Symbol("restart"); + +/** One compile of a schema. Its caches and indexes are garbage once the root check is built. */ +export class Compiler { + readonly r: Resolver; + readonly draft: "2020-12" | "7"; + readonly formats: Readonly> | undefined; + /** Whether checks track the dynamic scope (the schema has a bookended `$dynamicRef`). */ + readonly dynamic: boolean; + private readonly schema: JsonSchema; + private readonly scope: string[]; + private readonly cache = new Map(); + private readonly annotatingCache = new Map(); + + constructor(schema: JsonSchema, options: CompileOptions, dynamic: boolean) { + this.schema = schema; + this.r = new Resolver(schema, options); + this.draft = this.r.draft; + this.formats = options.formats; + this.dynamic = dynamic; + this.scope = []; + } + + /** The check of the root schema. */ + root(): Check { + return this.check(this.schema, this.r.rootBase); + } + + /** The check of a schema (cached by schema object; a recursive reference goes through an indirection). */ + check(node: unknown, base: string): Check { + if (node === true) return ALWAYS; + if (node === false) return NEVER; + if (!isSchemaObject(node)) + throw new Error(`A schema must be an object or a boolean, got ${JSON.stringify(node)}`); + const cached = this.cache.get(node); + if (cached !== undefined) return cached; + const key = this.leafKey(node); + if (key !== undefined) { + const shared = sharedCheck(key, () => this.plain(node, base)); + this.cache.set(node, shared); + return shared; + } + const slot: { check: Check | undefined } = { check: undefined }; + this.cache.set(node, indirect(slot)); + const check = this.build(node, this.r.baseOf.get(node) ?? base); + slot.check = check; + this.cache.set(node, check); + return check; + } + + private build(node: SchemaObject, base: string): Check { + if (this.draft === "7" && typeof node.$ref === "string") return this.ref(node.$ref, base); + if (this.draft !== "7" && (node.unevaluatedProperties !== undefined || node.unevaluatedItems !== undefined)) { + return withFreshAnnotations(this.annotating(node, base)); + } + const check = this.plain(node, base); + return this.dynamic && this.r.isResource(node, base) ? scopedCheck(this.scope, base, check) : check; + } + + /** + * The sharing key of a leaf schema (only assertion keywords; no subschemas, references or identifiers): equal leaves + * in all compiled schemas share one check. Undefined for every other schema. + */ + private leafKey(node: SchemaObject): string | undefined { + const picked: Record = {}; + for (const k of Object.keys(node)) { + if (LEAF_KEYWORDS.has(k)) picked[k] = node[k]; + else if (!NEUTRAL_KEYWORDS.has(k)) return undefined; + } + // A format checks through this compile's format map, so a leaf with one is shared only while formats are off. + if (picked.format !== undefined && this.formats !== undefined) return undefined; + return this.draft + JSON.stringify(picked); + } + + /** Every keyword of `node`, not tracking annotations. */ + private plain(node: SchemaObject, base: string): Check { + const parts: Check[] = []; + const types = node.type === undefined ? undefined : Array.isArray(node.type) ? node.type : [node.type]; + const single = types?.length === 1 ? types[0] : undefined; + let typeInGroup = false; + if (single === "bytes") { + parts.push(bytesCheck(number(node.minLength) ?? 0, number(node.maxLength) ?? Number.POSITIVE_INFINITY)); + typeInGroup = true; + } else { + const groups: [Check | undefined, boolean][] = [ + [this.stringGroup(node, single === "string"), single === "string"], + [ + this.numberGroup(node, single === "number" ? 1 : single === "integer" ? 2 : 0), + single === "number" || single === "integer", + ], + [this.objectGroup(node, base, single === "object"), single === "object"], + [this.arrayGroup(node, base, single === "array"), single === "array"], + ]; + for (const [group, strict] of groups) { + if (group === undefined) continue; + parts.push(group); + if (strict) typeInGroup = true; + } + } + if (types !== undefined && !typeInGroup) parts.unshift(typeCheck(types)); + this.inPlace(node, base, parts); + return and(parts); + } + + // The keywords that apply to the value itself, whatever its type. + private inPlace(node: SchemaObject, base: string, parts: Check[]): void { + if (Array.isArray(node.enum)) parts.push(enumCheck(node.enum)); + if (Object.hasOwn(node, "const")) parts.push(constCheck(node.const)); + if (Array.isArray(node.allOf)) parts.push(and(node.allOf.map((s) => this.check(s, base)))); + if (Array.isArray(node.anyOf)) parts.push(this.union(node.anyOf, base, false)); + if (Array.isArray(node.oneOf)) parts.push(this.union(node.oneOf, base, true)); + if (node.not !== undefined) parts.push(notCheck(this.check(node.not, base))); + if (node.if !== undefined && (node.then !== undefined || node.else !== undefined)) { + const then = node.then === undefined ? ALWAYS : this.check(node.then, base); + const otherwise = node.else === undefined ? ALWAYS : this.check(node.else, base); + parts.push(ifCheck(this.check(node.if, base), then, otherwise)); + } + if (typeof node.$ref === "string") parts.push(this.ref(node.$ref, base)); + if (this.draft !== "7" && typeof node.$dynamicRef === "string") { + parts.push(this.dynamicRef(node.$dynamicRef, base)); + } + } + + private union(branches: readonly unknown[], base: string, exactlyOne: boolean): Check { + const checks = () => branches.map((s) => this.check(s, base)); + return ( + constUnion(branches, exactlyOne) ?? + this.tagged(branches, base) ?? + (exactlyOne ? oneCheck(checks()) : anyCheck(checks())) + ); + } + + private ref(ref: string, base: string): Check { + const target = this.r.resolve(ref, base); + const check = this.check(target.node, target.base); + return this.dynamic && target.base !== base ? scopedCheck(this.scope, target.base, check) : check; + } + + private dynamicRef(ref: string, base: string): Check { + const resolved = this.bookended(ref, base); + if (resolved === undefined) return this.ref(ref, base); + const { initial, name } = resolved; + const fallback = this.check(initial.node, initial.base); + const lookup = this.dynamicLookup(name, (node, b) => this.check(node, b)); + const scope = this.scope; + return (v) => { + for (let i = 0; i < scope.length; i++) { + const check = lookup(scope[i]); + if (check !== null) return check(v); + } + return fallback(v); + }; + } + + /** The initial target and anchor name of a `$dynamicRef` that resolves dynamically, or undefined for a plain ref. */ + private bookended( + ref: string, + base: string, + ): { initial: { node: unknown; base: string }; name: string } | undefined { + const initial = this.r.resolve(ref, base); + const uri = resolveUri(ref, base); + const name = uri.slice(documentOf(uri).length + 1); + const anchored = isSchemaObject(initial.node) && initial.node.$dynamicAnchor === name; + if (name === "" || name.startsWith("/") || !anchored) return undefined; + if (!this.dynamic) throw RESTART; + return { initial, name }; + } + + // Resolved at run time against the dynamic scope, so a schema with a bookended $dynamicRef keeps its compiler. + private dynamicLookup(name: string, compile: (node: SchemaObject, base: string) => T): (uri: string) => T | null { + const found = new Map(); + return (uri) => { + const key = `${uri}#${name}`; + let check = found.get(key); + if (check === undefined) { + const target = this.r.dynamicAnchors.get(key); + check = target === undefined ? null : compile(target, this.r.baseOf.get(target) ?? uri); + found.set(key, check); + } + return check; + }; + } + + /** anyOf/oneOf whose branches are all objects with a distinct required const on one key: the tag picks the branch. */ + private tagged(branches: readonly unknown[], base: string): Check | undefined { + if (branches.length < 2) return undefined; + const resolved: SchemaObject[] = []; + for (const branch of branches) { + if (!isSchemaObject(branch)) return undefined; + const ref = typeof branch.$ref === "string" && Object.keys(branch).length === 1 ? branch.$ref : undefined; + const node = ref === undefined ? branch : this.r.resolve(ref, this.r.baseOf.get(branch) ?? base).node; + if (!isSchemaObject(node)) return undefined; + resolved.push(node); + } + const first = resolved[0].properties; + if (!isSchemaObject(first)) return undefined; + for (const key of Object.keys(first)) { + const tags = resolved.map((s) => tagOf(s, key)); + if (tags.some((t) => t === undefined) || new Set(tags).size !== tags.length) continue; + return taggedCheck( + key, + tags, + branches.map((s) => this.check(s, base)), + ); + } + return undefined; + } + + private stringGroup(node: SchemaObject, strict: boolean): Check | undefined { + const min = number(node.minLength) ?? 0; + const max = number(node.maxLength) ?? Number.POSITIVE_INFINITY; + const re = typeof node.pattern === "string" ? sharedRegex(node.pattern) : undefined; + const format = + this.formats !== undefined && typeof node.format === "string" && Object.hasOwn(this.formats, node.format) + ? this.formats[node.format] + : undefined; + if (min === 0 && max === Number.POSITIVE_INFINITY && re === undefined && format === undefined) return undefined; + return stringCheck(strict, min, max, re, format); + } + + private numberGroup(node: SchemaObject, mode: 0 | 1 | 2): Check | undefined { + const min = number(node.minimum); + const max = number(node.maximum); + const xmin = number(node.exclusiveMinimum); + const xmax = number(node.exclusiveMaximum); + const multipleOf = number(node.multipleOf); + if ([min, max, xmin, xmax, multipleOf].every((x) => x === undefined)) return undefined; + const inf = Number.POSITIVE_INFINITY; + return numberCheck(mode, min ?? -inf, max ?? inf, xmin ?? -inf, xmax ?? inf, multipleOf); + } + + private objectGroup(node: SchemaObject, base: string, strict: boolean): Check | undefined { + const onlyCore = OBJECT_EXTRAS.every((k) => node[k] === undefined); + // A schema with only the core object keywords gets one closure that also does the type guard. + const core = this.objectCore(node, base, onlyCore ? (strict ? GUARD_STRICT : GUARD_LOOSE) : GUARD_NONE); + if (core !== undefined && onlyCore) return core; + const parts: Check[] = core === undefined ? [] : [core]; + if (node.propertyNames !== undefined) parts.push(propertyNamesCheck(this.check(node.propertyNames, base))); + const minProperties = number(node.minProperties); + const maxProperties = number(node.maxProperties); + if (minProperties !== undefined || maxProperties !== undefined) { + parts.push(propertyCountCheck(minProperties ?? 0, maxProperties ?? Number.POSITIVE_INFINITY)); + } + const dependentRequired: [string, string[]][] = []; + const dependentSchemas: [string, Check][] = []; + if (this.draft !== "7") { + for (const [k, list] of entries(node.dependentRequired)) + if (Array.isArray(list)) dependentRequired.push([k, list]); + for (const [k, s] of entries(node.dependentSchemas)) dependentSchemas.push([k, this.check(s, base)]); + } + for (const [k, d] of entries(node.dependencies)) { + if (Array.isArray(d)) dependentRequired.push([k, d]); + else dependentSchemas.push([k, this.check(d, base)]); + } + if (dependentRequired.length > 0) parts.push(dependentRequiredCheck(dependentRequired)); + if (dependentSchemas.length > 0) parts.push(dependentSchemasCheck(dependentSchemas)); + if (parts.length === 0) return strict ? IS_OBJECT : undefined; + return guardObject(and(parts), strict); + } + + private objectCore(node: SchemaObject, base: string, guard: Guard): Check | undefined { + const properties = isSchemaObject(node.properties) ? node.properties : {}; + const required = Array.isArray(node.required) ? node.required.filter((k) => typeof k === "string") : []; + const patterns = isSchemaObject(node.patternProperties) ? node.patternProperties : {}; + const additional = node.additionalProperties; + const keys = Object.keys(properties); + const patternKeys = Object.keys(patterns); + if ( + keys.length + required.length + patternKeys.length === 0 && + (additional === undefined || additional === true) + ) { + return undefined; + } + // One array per object: key, check, flags (leaf code, required, key also names an Object.prototype member). + const entries: unknown[] = []; + for (const k of keys) { + const check = this.check(properties[k], base); + const flags = + (LEAF_CODES.get(check) ?? 0) | + (required.includes(k) ? REQUIRED : 0) | + (k in Object.prototype ? INHERITED : 0); + entries.push(k, check, flags); + } + const extraRequired = required.filter((k) => !Object.hasOwn(properties, k)); + const pats = patternKeys.map((p) => [sharedRegex(p), this.check(patterns[p], base)] as const); + if (additional === false && pats.length === 0 && extraRequired.length === 0) return closedObject(entries, guard); + const extra = additional === undefined || additional === true ? undefined : this.check(additional, base); + return openObject(entries, keys, extraRequired, pats, extra, guard); + } + + private arrayGroup(node: SchemaObject, base: string, strict: boolean): Check | undefined { + let prefix: readonly unknown[] = []; + let rest: unknown; + if (Array.isArray(node.items)) { + prefix = node.items; + rest = node.additionalItems; + } else { + if (this.draft !== "7" && Array.isArray(node.prefixItems)) prefix = node.prefixItems; + rest = node.items; + } + const prefixChecks = prefix.map((s) => this.check(s, base)); + const restCheck = rest === undefined || rest === true ? undefined : this.check(rest, base); + const minItems = number(node.minItems); + const maxItems = number(node.maxItems); + if ( + node.contains === undefined && + node.uniqueItems !== true && + (prefixChecks.length > 0 || restCheck !== undefined) + ) { + // The common array: one closure for the guard, the length bounds and the items. + const guard = strict ? GUARD_STRICT : GUARD_LOOSE; + return listCheck(guard, minItems ?? 0, maxItems ?? Number.POSITIVE_INFINITY, prefixChecks, restCheck); + } + const parts: Check[] = []; + if (prefixChecks.length > 0 || restCheck !== undefined) parts.push(itemsCheck(prefixChecks, restCheck)); + if (minItems !== undefined || maxItems !== undefined) { + parts.push(lengthCheck(minItems ?? 0, maxItems ?? Number.POSITIVE_INFINITY)); + } + if (node.contains !== undefined) { + const min = this.draft === "7" ? 1 : (number(node.minContains) ?? 1); + const max = + this.draft === "7" ? Number.POSITIVE_INFINITY : (number(node.maxContains) ?? Number.POSITIVE_INFINITY); + parts.push(containsCheck(this.check(node.contains, base), min, max)); + } + if (node.uniqueItems === true) parts.push(uniqueCheck); + if (parts.length === 0) return strict ? IS_ARRAY : undefined; + return guardArray(and(parts), strict); + } + + /** The annotating check of a schema evaluated in place (only below `unevaluatedProperties`/`unevaluatedItems`). */ + annotating(node: unknown, base: string): AnnotatingCheck { + if (node === true) return ANNOTATING_TRUE; + if (node === false) return ANNOTATING_FALSE; + if (!isSchemaObject(node)) + throw new Error(`A schema must be an object or a boolean, got ${JSON.stringify(node)}`); + const cached = this.annotatingCache.get(node); + if (cached !== undefined) return cached; + const slot: { check: AnnotatingCheck | undefined } = { check: undefined }; + this.annotatingCache.set(node, indirectAnnotating(slot)); + const b = this.r.baseOf.get(node) ?? base; + const check = this.buildAnnotating(node, b); + slot.check = check; + this.annotatingCache.set(node, check); + return check; + } + + private buildAnnotating(node: SchemaObject, base: string): AnnotatingCheck { + const rest: Record = {}; + let annotates = false; + for (const k of Object.keys(node)) { + if (ANNOTATING.has(k)) annotates = true; + else rest[k] = node[k]; + } + const plain = this.plain(rest, base); + const parts: AnnotatingCheck[] = []; + if (annotates) this.annotatingParts(node, base, parts); + const all = parts.length === 0 ? ignoreAnnotations(plain) : annotateAll(plain, parts); + const own = + node.unevaluatedProperties !== undefined || node.unevaluatedItems !== undefined + ? isolateAnnotations(all) + : all; + return this.dynamic && this.r.isResource(node, base) ? scopedAnnotating(this.scope, base, own) : own; + } + + private annotatingParts(node: SchemaObject, base: string, parts: AnnotatingCheck[]): void { + if ( + node.properties !== undefined || + node.patternProperties !== undefined || + node.additionalProperties !== undefined + ) { + const props = entries(node.properties).map(([k, s]) => [k, this.check(s, base)] as const); + const pats = entries(node.patternProperties).map(([p, s]) => [sharedRegex(p), this.check(s, base)] as const); + const extra = + node.additionalProperties === undefined ? undefined : this.check(node.additionalProperties, base); + parts.push(annotateObject(props, pats, extra)); + } + if (node.prefixItems !== undefined || node.items !== undefined || node.contains !== undefined) { + const prefix = Array.isArray(node.prefixItems) + ? node.prefixItems + : Array.isArray(node.items) + ? node.items + : []; + const rest = Array.isArray(node.items) ? node.additionalItems : node.items; + parts.push( + annotateArray( + prefix.map((s) => this.check(s, base)), + rest === undefined ? undefined : this.check(rest, base), + node.contains === undefined ? undefined : this.check(node.contains, base), + number(node.minContains) ?? 1, + number(node.maxContains) ?? Number.POSITIVE_INFINITY, + ), + ); + } + if (Array.isArray(node.allOf)) for (const s of node.allOf) parts.push(this.annotating(s, base)); + if (Array.isArray(node.anyOf)) + parts.push( + annotateAny( + node.anyOf.map((s) => this.annotating(s, base)), + false, + ), + ); + if (Array.isArray(node.oneOf)) + parts.push( + annotateAny( + node.oneOf.map((s) => this.annotating(s, base)), + true, + ), + ); + if (node.if !== undefined) { + const then = node.then === undefined ? undefined : this.annotating(node.then, base); + const otherwise = node.else === undefined ? undefined : this.annotating(node.else, base); + parts.push(annotateIf(this.annotating(node.if, base), then, otherwise)); + } + if (typeof node.$ref === "string") parts.push(this.annotatingRef(node.$ref, base)); + if (typeof node.$dynamicRef === "string") parts.push(this.annotatingDynamicRef(node.$dynamicRef, base)); + for (const [k, s] of entries(node.dependentSchemas)) parts.push(annotateDependent(k, this.annotating(s, base))); + if (node.unevaluatedProperties !== undefined) { + parts.push(unevaluatedProperties(this.check(node.unevaluatedProperties, base))); + } + if (node.unevaluatedItems !== undefined) parts.push(unevaluatedItems(this.check(node.unevaluatedItems, base))); + } + + private annotatingRef(ref: string, base: string): AnnotatingCheck { + const target = this.r.resolve(ref, base); + const check = this.annotating(target.node, target.base); + return this.dynamic && target.base !== base ? scopedAnnotating(this.scope, target.base, check) : check; + } + + private annotatingDynamicRef(ref: string, base: string): AnnotatingCheck { + const resolved = this.bookended(ref, base); + if (resolved === undefined) return this.annotatingRef(ref, base); + const { initial, name } = resolved; + const fallback = this.annotating(initial.node, initial.base); + const lookup = this.dynamicLookup(name, (node, b) => this.annotating(node, b)); + const scope = this.scope; + return (v, a) => { + for (let i = 0; i < scope.length; i++) { + const check = lookup(scope[i]); + if (check !== null) return check(v, a); + } + return fallback(v, a); + }; + } +} + +// Keywords whose check depends only on their own value, and keywords that never affect a check. +const LEAF_KEYWORDS = new Set([ + "type", + "enum", + "const", + "minLength", + "maxLength", + "pattern", + "format", + "minimum", + "maximum", + "exclusiveMinimum", + "exclusiveMaximum", + "multipleOf", + "minItems", + "maxItems", + "uniqueItems", + "minProperties", + "maxProperties", + "required", + "dependentRequired", +]); + +/** Keywords that never affect a check. */ +export const NEUTRAL_KEYWORDS = new Set([ + "$schema", + "$comment", + "title", + "description", + "default", + "examples", + "deprecated", + "readOnly", + "writeOnly", + "$defs", + "definitions", + "contentEncoding", + "contentMediaType", + "minContains", + "maxContains", + "$vocabulary", +]); + +// The object keywords besides the core (properties, required, patternProperties, additionalProperties). +const OBJECT_EXTRAS = [ + "propertyNames", + "minProperties", + "maxProperties", + "dependentRequired", + "dependentSchemas", + "dependencies", +]; + +// The keywords an annotating check handles itself; every other keyword compiles into its plain part. +const ANNOTATING = new Set([ + "properties", + "patternProperties", + "additionalProperties", + "prefixItems", + "items", + "additionalItems", + "contains", + "minContains", + "maxContains", + "allOf", + "anyOf", + "oneOf", + "if", + "then", + "else", + "$ref", + "$dynamicRef", + "dependentSchemas", + "unevaluatedProperties", + "unevaluatedItems", +]); + +/** anyOf/oneOf whose branches are all `{ const: primitive }`: one shared enum (distinct values make oneOf the same). */ +function constUnion(branches: readonly unknown[], exactlyOne: boolean): Check | undefined { + const values: unknown[] = []; + for (const branch of branches) { + if (!isSchemaObject(branch) || !Object.hasOwn(branch, "const")) return undefined; + for (const k of Object.keys(branch)) if (k !== "const" && !NEUTRAL_KEYWORDS.has(k)) return undefined; + if (typeof branch.const === "object" && branch.const !== null) return undefined; + if (exactlyOne && values.includes(branch.const)) return undefined; + values.push(branch.const); + } + return sharedCheck(`enum${JSON.stringify(values)}`, () => enumCheck(values)); +} + +// The tag of a tagged-union branch: a primitive const on a required property of an object schema. +function tagOf(branch: SchemaObject, key: string): unknown { + const properties = branch.properties; + if (branch.type !== "object" || !Array.isArray(branch.required) || !branch.required.includes(key)) return undefined; + const tag = isSchemaObject(properties) ? properties[key] : undefined; + if (!isSchemaObject(tag) || !Object.hasOwn(tag, "const")) return undefined; + return typeof tag.const === "object" && tag.const !== null ? undefined : tag.const; +} + +function number(value: unknown): number | undefined { + return typeof value === "number" ? value : undefined; +} + +function entries(value: unknown): [string, unknown][] { + return isSchemaObject(value) ? Object.entries(value) : []; +} diff --git a/packages/schema/src/explain.ts b/packages/schema/src/explain.ts new file mode 100644 index 000000000..fc8068a26 --- /dev/null +++ b/packages/schema/src/explain.ts @@ -0,0 +1,327 @@ +// Errors of a failed value: a walk over the schema that reports each failing keyword at the value it failed on. +// It runs only for values that failed the check, using the compiler's checks for subschemas. + +import { newAnnotations } from "./annotations.ts"; +import { + codePoints, + constCheck, + containsCheck, + enumCheck, + isMultiple, + isObject, + NEVER, + TYPE_CHECKS, + uniqueCheck, +} from "./checks.ts"; +import type { Compiler } from "./compiler.ts"; +import { isSchemaObject } from "./resolver.ts"; +import { sharedRegex } from "./shared.ts"; +import type { SchemaObject, ValidationError } from "./types.ts"; + +/** Explains failures of one compiled schema. */ +export class Explainer { + private readonly c: Compiler; + private readonly schema: unknown; + + constructor(compiler: Compiler, schema: unknown) { + this.c = compiler; + this.schema = schema; + compiler.root(); + } + + /** The errors of `value`, at least one (`value` failed the check). */ + errors(value: unknown): ValidationError[] { + const out: ValidationError[] = []; + this.walk(this.schema, this.c.r.rootBase, value, "", out); + if (out.length === 0) out.push({ path: "", keyword: "schema", message: "invalid" }); + return out; + } + + private ok(node: unknown, base: string, v: unknown): boolean { + return this.c.check(node, base)(v); + } + + // For $dynamicRef schemas this check runs outside the original dynamic scope, so their errors are approximate. + private walk(node: unknown, base: string, v: unknown, path: string, out: ValidationError[]): void { + if (node === true) return; + if (node === false) { + out.push({ path, keyword: "false", message: "no value is allowed here" }); + return; + } + if (!isSchemaObject(node)) return; + const b = this.c.r.baseOf.get(node) ?? base; + if (this.ok(node, b, v)) return; + const start = out.length; + const push = (keyword: string, message: string, at = path) => { + out.push({ path: at, keyword, message }); + }; + if (this.c.draft === "7" && typeof node.$ref === "string") { + const target = this.c.r.resolve(node.$ref, b); + this.walk(target.node, target.base, v, path, out); + return; + } + if (node.type !== undefined) { + const types = Array.isArray(node.type) ? node.type : [node.type]; + const checks = types.map((t) => (typeof t === "string" ? TYPE_CHECKS.get(t) : undefined) ?? NEVER); + if (!checks.some((check) => check(v))) { + push("type", `expected ${types.join(" or ")}, got ${typeName(v)}`); + return; + } + } + if (Array.isArray(node.enum) && !enumCheck(node.enum)(v)) + push("enum", `expected one of ${JSON.stringify(node.enum)}`); + if (Object.hasOwn(node, "const") && !constCheck(node.const)(v)) + push("const", `expected ${JSON.stringify(node.const)}`); + if (typeof v === "string" && node.type !== "bytes") this.walkString(node, v, push); + if (v instanceof Uint8Array && node.type === "bytes") this.walkBytes(node, v, push); + if (typeof v === "number") this.walkNumber(node, v, push); + if (isObject(v)) this.walkObject(node, b, v, path, out, push); + if (Array.isArray(v)) this.walkArray(node, b, v, path, out, push); + if (Array.isArray(node.allOf)) for (const s of node.allOf) this.walk(s, b, v, path, out); + if (Array.isArray(node.anyOf)) this.walkUnion(node.anyOf, b, v, path, out, false); + if (Array.isArray(node.oneOf)) this.walkUnion(node.oneOf, b, v, path, out, true); + if (node.not !== undefined && this.ok(node.not, b, v)) push("not", "must not match the schema in not"); + if (node.if !== undefined) { + const branch = this.ok(node.if, b, v) ? node.then : node.else; + if (branch !== undefined) this.walk(branch, b, v, path, out); + } + if (typeof node.$ref === "string") { + const target = this.c.r.resolve(node.$ref, b); + this.walk(target.node, target.base, v, path, out); + } + if (out.length === start && (node.unevaluatedProperties !== undefined || node.unevaluatedItems !== undefined)) { + this.walkUnevaluated(node, b, v, path, out); + } + if (out.length === start && typeof node.$dynamicRef === "string") { + push("$dynamicRef", "does not match the dynamically referenced schema"); + } + } + + private walkString(node: SchemaObject, v: string, push: Push): void { + const n = codePoints(v); + if (typeof node.minLength === "number" && n < node.minLength) { + push("minLength", `expected at least ${node.minLength} characters`); + } + if (typeof node.maxLength === "number" && n > node.maxLength) { + push("maxLength", `expected at most ${node.maxLength} characters`); + } + if (typeof node.pattern === "string" && !sharedRegex(node.pattern).test(v)) { + push("pattern", `expected to match ${node.pattern}`); + } + const formats = this.c.formats; + const format = + formats !== undefined && typeof node.format === "string" && Object.hasOwn(formats, node.format) + ? formats[node.format] + : undefined; + if (format !== undefined && !format(v)) push("format", `expected format ${node.format}`); + } + + private walkBytes(node: SchemaObject, v: Uint8Array, push: Push): void { + if (typeof node.minLength === "number" && v.length < node.minLength) { + push("minLength", `expected at least ${node.minLength} bytes`); + } + if (typeof node.maxLength === "number" && v.length > node.maxLength) { + push("maxLength", `expected at most ${node.maxLength} bytes`); + } + } + + private walkNumber(node: SchemaObject, v: number, push: Push): void { + if (typeof node.minimum === "number" && v < node.minimum) push("minimum", `expected >= ${node.minimum}`); + if (typeof node.maximum === "number" && v > node.maximum) push("maximum", `expected <= ${node.maximum}`); + if (typeof node.exclusiveMinimum === "number" && v <= node.exclusiveMinimum) { + push("exclusiveMinimum", `expected > ${node.exclusiveMinimum}`); + } + if (typeof node.exclusiveMaximum === "number" && v >= node.exclusiveMaximum) { + push("exclusiveMaximum", `expected < ${node.exclusiveMaximum}`); + } + if (typeof node.multipleOf === "number" && !isMultiple(v, node.multipleOf)) { + push("multipleOf", `expected a multiple of ${node.multipleOf}`); + } + } + + private walkObject( + node: SchemaObject, + b: string, + v: Record, + path: string, + out: ValidationError[], + push: Push, + ): void { + const properties = isSchemaObject(node.properties) ? node.properties : {}; + const required = Array.isArray(node.required) ? node.required : []; + for (const k of required) { + if (typeof k === "string" && (!Object.hasOwn(v, k) || v[k] === undefined)) + push("required", "missing", pointer(path, k)); + } + for (const k of Object.keys(properties)) { + if (Object.hasOwn(v, k) && v[k] !== undefined) this.walk(properties[k], b, v[k], pointer(path, k), out); + } + const patterns = isSchemaObject(node.patternProperties) ? Object.entries(node.patternProperties) : []; + for (const k of Object.keys(v)) { + let matched = Object.hasOwn(properties, k); + for (const [p, s] of patterns) { + if (sharedRegex(p).test(k)) { + matched = true; + this.walk(s, b, v[k], pointer(path, k), out); + } + } + if (!matched && node.additionalProperties !== undefined) { + if (node.additionalProperties === false) push("additionalProperties", "unknown field", pointer(path, k)); + else this.walk(node.additionalProperties, b, v[k], pointer(path, k), out); + } + if (node.propertyNames !== undefined && !this.ok(node.propertyNames, b, k)) { + push("propertyNames", "invalid property name", pointer(path, k)); + } + } + const count = Object.keys(v).length; + if (typeof node.minProperties === "number" && count < node.minProperties) { + push("minProperties", `expected at least ${node.minProperties} properties`); + } + if (typeof node.maxProperties === "number" && count > node.maxProperties) { + push("maxProperties", `expected at most ${node.maxProperties} properties`); + } + const dependencies = { + ...(this.c.draft === "7" || !isSchemaObject(node.dependentRequired) ? {} : node.dependentRequired), + ...(isSchemaObject(node.dependencies) ? node.dependencies : {}), + }; + for (const [k, d] of Object.entries(dependencies)) { + if (!Object.hasOwn(v, k)) continue; + if (!Array.isArray(d)) this.walk(d, b, v, path, out); + else + for (const r of d) + if (!Object.hasOwn(v, r)) push("dependentRequired", `required when ${k} is present`, pointer(path, r)); + } + if (this.c.draft !== "7" && isSchemaObject(node.dependentSchemas)) { + for (const [k, s] of Object.entries(node.dependentSchemas)) + if (Object.hasOwn(v, k)) this.walk(s, b, v, path, out); + } + } + + private walkArray( + node: SchemaObject, + b: string, + v: unknown[], + path: string, + out: ValidationError[], + push: Push, + ): void { + let prefix: readonly unknown[] = []; + let rest: unknown; + if (Array.isArray(node.items)) { + prefix = node.items; + rest = node.additionalItems; + } else { + if (this.c.draft !== "7" && Array.isArray(node.prefixItems)) prefix = node.prefixItems; + rest = node.items; + } + for (let i = 0; i < v.length; i++) { + const s = i < prefix.length ? prefix[i] : rest; + if (s !== undefined) this.walk(s, b, v[i], `${path}/${i}`, out); + } + if (typeof node.minItems === "number" && v.length < node.minItems) { + push("minItems", `expected at least ${node.minItems} items`); + } + if (typeof node.maxItems === "number" && v.length > node.maxItems) { + push("maxItems", `expected at most ${node.maxItems} items`); + } + if (node.uniqueItems === true && !uniqueCheck(v)) push("uniqueItems", "expected unique items"); + if (node.contains !== undefined) { + const draft7 = this.c.draft === "7"; + const min = draft7 || typeof node.minContains !== "number" ? 1 : node.minContains; + const max = draft7 || typeof node.maxContains !== "number" ? Number.POSITIVE_INFINITY : node.maxContains; + if (!containsCheck(this.c.check(node.contains, b), min, max)(v)) { + push("contains", `expected between ${min} and ${max} matching items`); + } + } + } + + private walkUnion( + branches: readonly unknown[], + b: string, + v: unknown, + path: string, + out: ValidationError[], + exactlyOne: boolean, + ): void { + const matches = branches.filter((s) => this.ok(s, b, v)).length; + if (matches === 1 || (!exactlyOne && matches > 0)) return; + const keyword = exactlyOne ? "oneOf" : "anyOf"; + if (matches > 1) { + out.push({ path, keyword, message: "matches more than one alternative" }); + return; + } + // A tagged union reports the branch its tag selects; otherwise the branch whose first error is deepest. + const tag = this.tagOf(branches, b); + if (tag !== undefined && isObject(v)) { + const i = tag.values.indexOf(v[tag.key]); + if (i < 0) + out.push({ + path: pointer(path, tag.key), + keyword, + message: `expected one of ${JSON.stringify(tag.values)}`, + }); + else this.walk(branches[i], b, v, path, out); + return; + } + let best: ValidationError[] | undefined; + for (const s of branches) { + const errors: ValidationError[] = []; + this.walk(s, b, v, path, errors); + if (errors.length > 0 && (best === undefined || errors[0].path.length > best[0].path.length)) best = errors; + } + out.push(...(best ?? [{ path, keyword, message: "matches no alternative" }])); + } + + private tagOf(branches: readonly unknown[], b: string): { key: string; values: unknown[] } | undefined { + const resolved = branches.map((s) => + isSchemaObject(s) && typeof s.$ref === "string" ? this.c.r.resolve(s.$ref, b).node : s, + ); + const first = resolved[0]; + if (!isSchemaObject(first) || !isSchemaObject(first.properties)) return undefined; + for (const key of Object.keys(first.properties)) { + const values = resolved.map((s) => { + const tag = isSchemaObject(s) && isSchemaObject(s.properties) ? s.properties[key] : undefined; + return isSchemaObject(tag) ? tag.const : undefined; + }); + const primitive = values.every((x) => x !== undefined && (typeof x !== "object" || x === null)); + if (primitive && new Set(values).size === values.length) return { key, values }; + } + return undefined; + } + + private walkUnevaluated(node: SchemaObject, b: string, v: unknown, path: string, out: ValidationError[]): void { + const { unevaluatedProperties, unevaluatedItems, ...rest } = node; + const a = newAnnotations(); + this.c.annotating(rest, b)(v, a); + if (isObject(v) && unevaluatedProperties !== undefined && !a.allProps) { + for (const k of Object.keys(v)) { + if (a.props?.has(k)) continue; + if (unevaluatedProperties === false) { + out.push({ path: pointer(path, k), keyword: "unevaluatedProperties", message: "unknown field" }); + } else this.walk(unevaluatedProperties, b, v[k], pointer(path, k), out); + } + } + if (Array.isArray(v) && unevaluatedItems !== undefined && !a.allItems) { + for (let i = a.items; i < v.length; i++) { + if (a.indexes?.has(i)) continue; + if (unevaluatedItems === false) { + out.push({ path: `${path}/${i}`, keyword: "unevaluatedItems", message: "unexpected item" }); + } else this.walk(unevaluatedItems, b, v[i], `${path}/${i}`, out); + } + } + } +} + +type Push = (keyword: string, message: string, at?: string) => void; + +function pointer(path: string, key: string | number): string { + return `${path}/${String(key).replace(/~/g, "~0").replace(/\//g, "~1")}`; +} + +function typeName(v: unknown): string { + if (v === null) return "null"; + if (Array.isArray(v)) return "array"; + if (v instanceof Uint8Array) return "bytes"; + if (typeof v === "number" && Number.isInteger(v)) return "integer"; + return typeof v; +} diff --git a/packages/schema/src/formats/index.ts b/packages/schema/src/formats/index.ts new file mode 100644 index 000000000..6919b9256 --- /dev/null +++ b/packages/schema/src/formats/index.ts @@ -0,0 +1,46 @@ +// The format checks of JSON Schema 2020-12, for compile()'s `formats` option. An optional entry point: schemas +// compiled without it treat `format` as an annotation and do not bundle these checks. + +import type { FormatCheck } from "../types.ts"; +import { IsDate } from "./typebox/date.ts"; +import { IsDateTime } from "./typebox/date_time.ts"; +import { IsDuration } from "./typebox/duration.ts"; +import { IsEmail } from "./typebox/email.ts"; +import { IsHostname } from "./typebox/hostname.ts"; +import { IsIdnEmail } from "./typebox/idn_email.ts"; +import { IsIdnHostname } from "./typebox/idn_hostname.ts"; +import { IsIPv4 } from "./typebox/ipv4.ts"; +import { IsIPv6 } from "./typebox/ipv6.ts"; +import { IsIri } from "./typebox/iri.ts"; +import { IsIriReference } from "./typebox/iri_reference.ts"; +import { IsJsonPointer } from "./typebox/json_pointer.ts"; +import { IsRegex } from "./typebox/regex.ts"; +import { IsRelativeJsonPointer } from "./typebox/relative_json_pointer.ts"; +import { IsTime } from "./typebox/time.ts"; +import { IsUri } from "./typebox/uri.ts"; +import { IsUriReference } from "./typebox/uri_reference.ts"; +import { IsUriTemplate } from "./typebox/uri_template.ts"; +import { IsUuid } from "./typebox/uuid.ts"; + +/** Every format JSON Schema 2020-12 defines, by name. The checks are adapted from TypeBox 1.3.27 (MIT). */ +export const formats: Readonly> = Object.freeze({ + date: IsDate, + "date-time": IsDateTime, + duration: IsDuration, + email: IsEmail, + hostname: IsHostname, + "idn-email": IsIdnEmail, + "idn-hostname": IsIdnHostname, + ipv4: IsIPv4, + ipv6: IsIPv6, + iri: IsIri, + "iri-reference": IsIriReference, + "json-pointer": IsJsonPointer, + regex: IsRegex, + "relative-json-pointer": IsRelativeJsonPointer, + time: IsTime, + uri: IsUri, + "uri-reference": IsUriReference, + "uri-template": IsUriTemplate, + uuid: IsUuid, +}); diff --git a/packages/schema/src/formats/typebox/date.ts b/packages/schema/src/formats/typebox/date.ts new file mode 100644 index 000000000..e35cf0770 --- /dev/null +++ b/packages/schema/src/formats/typebox/date.ts @@ -0,0 +1,47 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +const DAYS = [0, 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]; +const DATE = /^(\d\d\d\d)-(\d\d)-(\d\d)$/; + +function IsLeapYear(year: number): boolean { + return year % 4 === 0 && (year % 100 !== 0 || year % 400 === 0); +} +/** + * Returns true if the value is a ISO8601 Date component string + * @description https://datatracker.ietf.org/doc/html/rfc3339 + * @example `2020-12-12` + */ +export function IsDate(value: string): boolean { + const matches: string[] | null = DATE.exec(value); + if (!matches) return false; + const year: number = +matches[1]; + const month: number = +matches[2]; + const day: number = +matches[3]; + return month >= 1 && month <= 12 && day >= 1 && day <= (month === 2 && IsLeapYear(year) ? 29 : DAYS[month]); +} diff --git a/packages/schema/src/formats/typebox/date_time.ts b/packages/schema/src/formats/typebox/date_time.ts new file mode 100644 index 000000000..df4174a40 --- /dev/null +++ b/packages/schema/src/formats/typebox/date_time.ts @@ -0,0 +1,40 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +import { IsDate } from "./date.ts"; +import { IsTime } from "./time.ts"; + +/** + * Returns true if the value is a ISO8601 DateTime string + * @specification https://datatracker.ietf.org/doc/html/rfc3339 + * @example `2020-12-12T20:20:40+00:00` + */ +export function IsDateTime(value: string): boolean { + const dateTime: string[] = value.split(/T/i); + return dateTime.length === 2 && IsDate(dateTime[0]) && IsTime(dateTime[1]); +} diff --git a/packages/schema/src/formats/typebox/duration.ts b/packages/schema/src/formats/typebox/duration.ts new file mode 100644 index 000000000..20d812c4f --- /dev/null +++ b/packages/schema/src/formats/typebox/duration.ts @@ -0,0 +1,38 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +const Duration = + /^P((\d+Y(\d+M(\d+D)?)?|\d+M(\d+D)?|\d+D)(T(\d+H(\d+M(\d+S)?)?|\d+M(\d+S)?|\d+S))?|T(\d+H(\d+M(\d+S)?)?|\d+M(\d+S)?|\d+S)|\d+W)$/; + +/** + * Returns true if the value is a valid ISO-8601 duration. + * @specification https://tools.ietf.org/html/rfc3339 + */ +export function IsDuration(value: string): boolean { + return Duration.test(value); +} diff --git a/packages/schema/src/formats/typebox/email.ts b/packages/schema/src/formats/typebox/email.ts new file mode 100644 index 000000000..a6840daed --- /dev/null +++ b/packages/schema/src/formats/typebox/email.ts @@ -0,0 +1,38 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +const Email = + /^(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[^"\\]|\\[\x20-\x7e])*")@(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?(?:\.[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?)*|\[(?:IPv6:[a-f0-9:]+|(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])(?:\.(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])){3})\])$/i; + +/** + * Returns true if the value is an Email + * @specification https://datatracker.ietf.org/doc/html/rfc5322 + */ +export function IsEmail(value: string): boolean { + return Email.test(value); +} diff --git a/packages/schema/src/formats/typebox/hostname.ts b/packages/schema/src/formats/typebox/hostname.ts new file mode 100644 index 000000000..68c3710fb --- /dev/null +++ b/packages/schema/src/formats/typebox/hostname.ts @@ -0,0 +1,39 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +import * as Idna from "./idna/index.ts"; + +/** + * Returns true if the value is a valid hostname. + * @specification https://tools.ietf.org/html/rfc1123 + * @specification https://tools.ietf.org/html/rfc5891 + * @specification https://tools.ietf.org/html/rfc5892 + */ +export function IsHostname(value: string): boolean { + return Idna.IsHostname(value); +} diff --git a/packages/schema/src/formats/typebox/idn_email.ts b/packages/schema/src/formats/typebox/idn_email.ts new file mode 100644 index 000000000..92e3abdaf --- /dev/null +++ b/packages/schema/src/formats/typebox/idn_email.ts @@ -0,0 +1,38 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +const IdnEmail = + /^(?:[A-Za-z0-9!#$%&'*+/=?^_`{|}~\u{0080}-\u{10FFFF}-]+(?:\.[A-Za-z0-9!#$%&'*+/=?^_`{|}~\u{0080}-\u{10FFFF}-]+)*|"(?:[^"\\]|\\.)*")@[\p{L}\p{N}](?:[\p{L}\p{N}-]{0,62})(?> 1; + delta += Math.floor(delta / numPoints); + let k = 0; + while (delta > ((PUNYCODE_BASE - PUNYCODE_TMIN) * PUNYCODE_TMAX) >> 1) { + delta = Math.floor(delta / (PUNYCODE_BASE - PUNYCODE_TMIN)); + k += PUNYCODE_BASE; + } + return k + Math.floor(((PUNYCODE_BASE - PUNYCODE_TMIN + 1) * delta) / (delta + PUNYCODE_SKEW)); +} +// ------------------------------------------------------------------ +// Decode +// ------------------------------------------------------------------ +export function Decode(value: string): string { + const output: number[] = []; + let n = PUNYCODE_INITIAL_N; + let i = 0; + let bias = PUNYCODE_INITIAL_BIAS; + const delimIdx = value.lastIndexOf("-"); + if (delimIdx > 0) { + for (let j = 0; j < delimIdx; j++) { + const cp = value.charCodeAt(j); + if (cp >= 128) ThrowDontCare(); // throw new Error('Invalid punycode: non-basic before delimiter') + output.push(cp); + } + } + let inIdx = delimIdx < 0 ? 0 : delimIdx + 1; + while (inIdx < value.length) { + const oldi = i; + let w = 1; + let k = PUNYCODE_BASE; + while (true) { + if (inIdx >= value.length) ThrowDontCare(); // throw new Error('Invalid punycode: unexpected end of input') + const ch = value.charCodeAt(inIdx++); + let digit: number; + if (ch >= 0x61 && ch <= 0x7a) + digit = ch - 0x61; // a-z => 0-25 + else if (ch >= 0x30 && ch <= 0x39) + digit = ch - 0x30 + 26; // 0-9 => 26-35 + // else if (ch >= 0x41 && ch <= 0x5a) digit = ch - 0x41 // A-Z => 0-25 // (no-spec-coverage) + else ThrowDontCare(); // throw new Error('Invalid punycode: bad digit character') + i += digit * w; + const t = k <= bias ? PUNYCODE_TMIN : k >= bias + PUNYCODE_TMAX ? PUNYCODE_TMAX : k - bias; + if (digit < t) break; + w *= PUNYCODE_BASE - t; + k += PUNYCODE_BASE; + } + const outLen = output.length + 1; + bias = Adapt(i - oldi, outLen, oldi === 0); + n += Math.floor(i / outLen); + i %= outLen; + output.splice(i, 0, n); + i++; + } + return String.fromCodePoint(...output); +} +// ------------------------------------------------------------------ +// DigitToChar +// ------------------------------------------------------------------ +function DigitToChar(digit: number): string { + return digit < 26 ? String.fromCharCode(digit + 0x61) : String.fromCharCode(digit - 26 + 0x30); +} +// ------------------------------------------------------------------ +// Encode +// +// Encodes a Unicode string into its Punycode payload (the part that +// follows the 'xn--' ACE prefix). Used to compute the true A-label +// length of a U-label, since the 63-octet label length limit (RFC +// 5891 §4.2.4) applies to the encoded form, not the code point count. +// ------------------------------------------------------------------ +const RE_REPLACE_NON_ASCII = new RegExp(Pattern.RE_NON_ASCII.source, "gu"); +export function Encode(input: string): string { + // Extract basic ASCII characters and set initial output array state + const basic = input.replace(RE_REPLACE_NON_ASCII, ""); + const basicLength = basic.length; + const result = basicLength > 0 ? [basic, "-"] : []; + // Extract unicode codepoints + const codePoints = Array.from(input, (char) => char.codePointAt(0)!); + let n = PUNYCODE_INITIAL_N; + let delta = 0; + let bias = PUNYCODE_INITIAL_BIAS; + let handledCPCount = basicLength; + while (handledCPCount < codePoints.length) { + let m = Infinity; + for (const cp of codePoints) { + if (cp >= n && cp < m) m = cp; + } + delta += (m - n) * (handledCPCount + 1); + n = m; + for (const cp of codePoints) { + if (cp < n) delta++; + if (cp === n) { + let q = delta; + for (let k = PUNYCODE_BASE; ; k += PUNYCODE_BASE) { + const t = k <= bias ? PUNYCODE_TMIN : k >= bias + PUNYCODE_TMAX ? PUNYCODE_TMAX : k - bias; + if (q < t) break; + const digit = t + ((q - t) % (PUNYCODE_BASE - t)); + result.push(DigitToChar(digit)); + q = Math.floor((q - t) / (PUNYCODE_BASE - t)); + } + result.push(DigitToChar(q)); + bias = Adapt(delta, handledCPCount + 1, handledCPCount === basicLength); + delta = 0; + handledCPCount++; + } + } + delta++; + n++; + } + return result.join(""); +} diff --git a/packages/schema/src/formats/typebox/idna/hostname.ts b/packages/schema/src/formats/typebox/idna/hostname.ts new file mode 100644 index 000000000..ade2322b4 --- /dev/null +++ b/packages/schema/src/formats/typebox/idna/hostname.ts @@ -0,0 +1,42 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +import * as LabelAscii from "./label/ascii.ts"; +import * as LabelPuny from "./label/puny.ts"; + +function IsValidLabelLength(value: string): boolean { + return value.length > 0 && value.length <= 63; +} +function IsLabel(value: string): boolean { + return IsValidLabelLength(value) && (LabelPuny.IsPunyLabel(value) || LabelAscii.IsAsciiLabel(value)); +} +export function IsHostname(value: string): boolean { + if (value.length === 0 || value.length > 253) return false; + if (value.charCodeAt(value.length - 1) === 46) return false; + return value.split(".").every((label) => IsLabel(label)); +} diff --git a/packages/schema/src/formats/typebox/idna/idn-hostname.ts b/packages/schema/src/formats/typebox/idna/idn-hostname.ts new file mode 100644 index 000000000..d4517faac --- /dev/null +++ b/packages/schema/src/formats/typebox/idna/idn-hostname.ts @@ -0,0 +1,72 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +import * as FormatBidi from "./format/bidi.ts"; +import * as LabelPuny from "./label/puny.ts"; +import * as LabelUnicode from "./label/unicode.ts"; + +// ------------------------------------------------------------------ +// IsLabel +// ------------------------------------------------------------------ +function IsValidLabelLength(value: string): boolean { + return value.length > 0 && value.length <= 63; +} +function IsLabel(value: string): boolean { + return IsValidLabelLength(value) && (LabelPuny.IsPunyLabel(value) || LabelUnicode.IsUnicodeLabel(value)); +} +// ------------------------------------------------------------------ +// NormalizeHostname +// +// Normalizes a hostname for label validation. Maps fullwidth +// codepoints to their ASCII equivalent (e.g. 'ａ' U+FF41 maps to +// 'a'), then normalizes for NFC such that equivalent codepoint +// sequences can be uniformly compared. It also removes codepoints +// that IDNA mapping ignores, and converts full stop variants to +// ASCII '.' so the hostname can be split into labels. +// ------------------------------------------------------------------ +function NormalizeHostname(value: string): string { + return ( + value + .replace(/[\uff01-\uff5e]/g, (char) => String.fromCharCode(char.charCodeAt(0) - 0xfee0)) // RE_FULLWIDTH_MAPPING + .normalize("NFC") + // biome-ignore lint/suspicious/noMisleadingCharacterClass: a set of single code points to remove (u flag), by design. + .replace(/[\u00ad\u034f\u180b-\u180d\u200b\ufe00-\ufe0f\u{e0100}-\u{e01ef}]/gu, "") // RE_IGNORED_MAPPING + .replace(/[\u002E\u3002\uFF0E\uFF61]/g, ".") + ); // RE_FULL_STOP_MAPPING +} +// ------------------------------------------------------------------ +// IsIdnHostname +// ------------------------------------------------------------------ +export function IsIdnHostname(value: string): boolean { + if (value.length === 0 || value.includes(" ")) return false; + const normalized = NormalizeHostname(value); + if (normalized.length > 253) return false; + const labels = normalized.split("."); + const hasBidiChars = labels.some((label) => FormatBidi.HasBidiChars(label)); + return labels.every((label) => IsLabel(label) && (!hasBidiChars || FormatBidi.SatisfiesBidiRule(label))); +} diff --git a/packages/schema/src/formats/typebox/idna/index.ts b/packages/schema/src/formats/typebox/idna/index.ts new file mode 100644 index 000000000..c5d10175e --- /dev/null +++ b/packages/schema/src/formats/typebox/idna/index.ts @@ -0,0 +1,30 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +export * from "./hostname.ts"; +export * from "./idn-hostname.ts"; diff --git a/packages/schema/src/formats/typebox/idna/label/ascii.ts b/packages/schema/src/formats/typebox/idna/label/ascii.ts new file mode 100644 index 000000000..704e8f944 --- /dev/null +++ b/packages/schema/src/formats/typebox/idna/label/ascii.ts @@ -0,0 +1,40 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +import * as Pattern from "../pattern/pattern.ts"; + +// ------------------------------------------------------------------- +// Validates an ASCII LDH label per RFC 5891 §4.2.3.1 +// ------------------------------------------------------------------- +export function IsAsciiLabel(value: string): boolean { + return ( + Pattern.RE_RULE_HYPHEN_PLACEMENT.test(value) && + Pattern.RE_RULE_NOT_RESERVED_ACE.test(value) && + Pattern.RE_ASCII_LDH.test(value) + ); +} diff --git a/packages/schema/src/formats/typebox/idna/label/puny.ts b/packages/schema/src/formats/typebox/idna/label/puny.ts new file mode 100644 index 000000000..23b288ce2 --- /dev/null +++ b/packages/schema/src/formats/typebox/idna/label/puny.ts @@ -0,0 +1,53 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +import * as FormatPuny from "../format/puny.ts"; +import * as Pattern from "../pattern/pattern.ts"; +import * as LabelUnicode from "./unicode.ts"; + +// ------------------------------------------------------------------ +// IsPunyLabel +// ------------------------------------------------------------------ +export function IsPunyLabel(value: string): boolean { + if (!FormatPuny.IsAcePrefixed(value)) return false; + try { + const body = value.slice(4).toLowerCase(); + // A payload consisting of only a hyphen, or starting with the delimiter + // (like "-9uc"), has 0 basic ASCII characters preceding it and is malformed. + if (body.lastIndexOf("-") === 0) return false; + const decoded = FormatPuny.Decode(body); + // RFC 5890 §2.3.2.1 - a U-label must contain at least one non-ASCII + // character. Code-unit level check: any non-ASCII codepoint has at least + // one UTF-16 unit >= 0x80 (surrogate halves included), so no need to + // decode codepoints or allocate a character array to answer this. + if (!Pattern.RE_NON_ASCII.test(decoded)) return false; + return LabelUnicode.IsUnicodeLabel(decoded); + } catch { + return false; + } +} diff --git a/packages/schema/src/formats/typebox/idna/label/unicode.ts b/packages/schema/src/formats/typebox/idna/label/unicode.ts new file mode 100644 index 000000000..615942427 --- /dev/null +++ b/packages/schema/src/formats/typebox/idna/label/unicode.ts @@ -0,0 +1,122 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +import * as FormatBidi from "../format/bidi.ts"; +import * as FormatPuny from "../format/puny.ts"; +import * as Pattern from "../pattern/pattern.ts"; + +// ------------------------------------------------------------------ +// ExceedsMaxALabelLength +// +// RFC 5890 §2.3.2.1: an A-label (the ACE-encoded form of a label, +// "xn--" + punycode) must not exceed 63 octets. Only labels +// containing non-ASCII characters need checking - a pure-ASCII label +// has no "xn--" prefix and is used as-is, so its punycode-encoded +// length is irrelevant. The ASCII check is a cheap short-circuit that +// avoids running the punycode encoder on labels that don't need it. +// ------------------------------------------------------------------ +function ExceedsMaxALabelLength(value: string): boolean { + return Pattern.RE_NON_ASCII.test(value) && FormatPuny.Encode(value).length + 4 > 63; +} +// ------------------------------------------------------------------ +// HasInvalidHyphens +// +// RFC 5891 §4.2.3.2: label must not start or end with a hyphen, and +// must not have hyphens in the 3rd and 4th codepoint positions (the +// ACE prefix reservation, e.g. "xn--"). Operates on `chars` (the +// per-codepoint string array) rather than raw string indices so +// supplementary-plane characters earlier in the label don't throw +// off the codepoint-position count. +// ------------------------------------------------------------------ +function HasInvalidHyphens(chars: string[]): boolean { + if (chars[0] === "-" || chars[chars.length - 1] === "-") return true; + return chars.slice(2).join("").startsWith("--"); +} +// ------------------------------------------------------------------ +// IsUnicodeLabel +// ------------------------------------------------------------------ +export function IsUnicodeLabel(value: string): boolean { + // RFC 5890 §2.3.2.1: Ensure Max Length + if (ExceedsMaxALabelLength(value)) return false; + // RFC 5893 - Bidi Rule (applies whenever this label itself is RTL) + if (FormatBidi.HasRightToLeftCharacters(value) && !FormatBidi.SatisfiesBidiRule(value)) return false; + const chars = [...value]; + const codePoints = chars.map((c) => c.codePointAt(0)!); + const length = codePoints.length; + // RFC 5891 §4.2.3.2: Hyphen rules + if (HasInvalidHyphens(chars)) return false; + // RFC 5891 §4.2.3.2 - Must not begin with a combining mark + if (Pattern.RE_COMBINING_MARK.test(chars[0])) return false; + let hasJapanese = false; + // let hasArabicIndic = false // (no-spec-coverage) + // let hasExtendedArabicIndic = false // (no-spec-coverage) + for (let i = 0; i < length; i++) { + const codePoint = codePoints[i]; + const char = chars[i]; + // 1. DISALLOWED exceptions + if (Pattern.RE_RFC5892_DISALLOWED.test(char)) return false; + // 1b. General category (PVALID) enforcement + if (!Pattern.RE_PERMITTED_CATEGORY.test(char)) return false; + // 2. Collect Flags + if (Pattern.RE_SCRIPT_JAPANESE.test(char)) hasJapanese = true; + // if (Pattern.RE_ARABIC_INDIC_DIGIT.test(char)) hasArabicIndic = true // (no-spec-coverage) + // if (Pattern.RE_EXT_ARABIC_INDIC_DIGIT.test(char)) hasExtendedArabicIndic = true // (no-spec-coverage) + // 3. CONTEXTO / CONTEXTJ Neighbor Rules + const prev = codePoints[i - 1], + next = codePoints[i + 1]; + switch (codePoint) { + case 0x00b7: + if (prev !== 0x006c || next !== 0x006c) return false; + break; // MIDDLE DOT (Catalan) + case 0x0375: + if (!next || !Pattern.RE_SCRIPT_GREEK.test(chars[i + 1])) return false; + break; // Greek KERAIA + case 0x05f3: + case 0x05f4: + if (!prev || !Pattern.RE_SCRIPT_HEBREW.test(chars[i - 1])) return false; + break; // Hebrew GERESH + case 0x200c: // ZWNJ - RFC 5892 Appendix A.1 + // Note: Full validation requires a Unicode joining type table. We reject + // only when preceded by ASCII (U+0000-U+007F), which can never satisfy + // the Virama or cursive-joining rules. + if (!prev || (prev < 0x0080 && !Pattern.RE_VIRAMA.test(chars[i - 1]))) return false; + break; + case 0x200d: // ZWJ: RFC 5892 Appendix A.2 + if (!prev || !Pattern.RE_VIRAMA.test(chars[i - 1])) return false; + break; + case 0x30fb: // Checked at end via hasJapanese + break; // KATAKANA MIDDLE DOT + } + } + // 4. Global Context Validations (Post-loop) + // RFC 5892 Appendix A.7 - Katakana Middle Dot requirement + if (value.includes("\u30fb") && !hasJapanese) return false; + // RFC 5892 Appendix A.8/A.9 - Mixing Arabic Digits + // if (hasArabicIndic && hasExtendedArabicIndic) return false // (no-spec-coverage) + return true; +} diff --git a/packages/schema/src/formats/typebox/idna/pattern/pattern.ts b/packages/schema/src/formats/typebox/idna/pattern/pattern.ts new file mode 100644 index 000000000..dfb55c1d6 --- /dev/null +++ b/packages/schema/src/formats/typebox/idna/pattern/pattern.ts @@ -0,0 +1,116 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +// deno-fmt-ignore-file + +// ------------------------------------------------------------------- +// Validation Rules +// ------------------------------------------------------------------- +export const RE_RULE_HYPHEN_PLACEMENT = /^(?!-).*(? regexp.source).join("|"), + "u", +); + +export const RE_PERMITTED_CATEGORY = new RegExp( + [ + RE_LETTER, + RE_EUROPEAN_SEPARATOR, + RE_COMMON_SEPARATOR, + RE_NUMBER_DECIMAL, + RE_MARK_NONSPACING, + RE_MARK_SPACING_COMBINING, + RE_CONTEXTO_EXCEPTIONS, + RE_PVALID_EXCEPTIONS, + ] + .map((regexp) => regexp.source) + .join("|"), + "u", +); diff --git a/packages/schema/src/formats/typebox/ipv4.ts b/packages/schema/src/formats/typebox/ipv4.ts new file mode 100644 index 000000000..2c6bf3c7b --- /dev/null +++ b/packages/schema/src/formats/typebox/ipv4.ts @@ -0,0 +1,37 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +const IPv4 = /^(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)$/; + +/** + * Returns true if the value is a IPV4 address + * @specification http://tools.ietf.org/html/rfc2673#section-3.2 + */ +export function IsIPv4(value: string): boolean { + return IPv4.test(value); +} diff --git a/packages/schema/src/formats/typebox/ipv6.ts b/packages/schema/src/formats/typebox/ipv6.ts new file mode 100644 index 000000000..4cc46deac --- /dev/null +++ b/packages/schema/src/formats/typebox/ipv6.ts @@ -0,0 +1,38 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +const IPv6 = + /^(?:(?:(?:[0-9a-f]{1,4}:){6}|::(?:[0-9a-f]{1,4}:){5}|(?:[0-9a-f]{1,4})?::(?:[0-9a-f]{1,4}:){4}|(?:(?:[0-9a-f]{1,4}:)?[0-9a-f]{1,4})?::(?:[0-9a-f]{1,4}:){3}|(?:(?:[0-9a-f]{1,4}:){0,2}[0-9a-f]{1,4})?::(?:[0-9a-f]{1,4}:){2}|(?:(?:[0-9a-f]{1,4}:){0,3}[0-9a-f]{1,4})?::[0-9a-f]{1,4}:|(?:(?:[0-9a-f]{1,4}:){0,4}[0-9a-f]{1,4})?::)(?:[0-9a-f]{1,4}:[0-9a-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:(?:[0-9a-f]{1,4}:){0,5}[0-9a-f]{1,4})?::[0-9a-f]{1,4}|(?:(?:[0-9a-f]{1,4}:){0,6}[0-9a-f]{1,4})?::)$/i; + +/** + * Returns true if the value is an IPv6 address + * @specification http://tools.ietf.org/html/rfc2373#section-2.2 + */ +export function IsIPv6(value: string): boolean { + return IPv6.test(value); +} diff --git a/packages/schema/src/formats/typebox/iri.ts b/packages/schema/src/formats/typebox/iri.ts new file mode 100644 index 000000000..8ee1d0540 --- /dev/null +++ b/packages/schema/src/formats/typebox/iri.ts @@ -0,0 +1,63 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +// deno-lint-ignore-file no-control-regex + +const IpvFutureMatchMaxLength = 2048; +const IpvFutureMatch = /\[[vV][0-9a-fA-F]+\.[^\]]+\]/; // Guarded By IpvFutureMatchMaxLength +const InvalidIriChars = /[\x00-\x20<>^`{|}\\]/; +const InvalidPercentEncoding = /%(?![0-9a-fA-F]{2})/; + +// ------------------------------------------------------------------ +// NarrowIpvFuture +// +// Substitutes an IPvFuture address with a standard IPv6 loopback +// address ([::1]). We do this because the native URL.canParse +// (WHATWG standard) rejects IPvFuture literals, which are +// otherwise valid in RFC 3987. Because regex substitution can +// be expensive on large strings, this operation is strictly +// limited to inputs under a defined length threshold. +// +// (review-optimization) +// +// ------------------------------------------------------------------ +function NarrowIpvFuture(value: string): string { + return value.length < IpvFutureMatchMaxLength ? value.replace(IpvFutureMatch, "[::1]") : value; +} +/** + * Returns true if the value is a valid Internationalized Resource Identifier. + * @specification https://datatracker.ietf.org/doc/html/rfc3987 + */ +export function IsIri(value: string): boolean { + // 1. Reject strings containing unencoded whitespace or illegal control characters. + if (InvalidIriChars.test(value)) return false; + // 2. Reject malformed percent-encoding triplets. + if (InvalidPercentEncoding.test(value)) return false; + // 3. Delegate to the native URL parser, patching the IPvFuture edge case beforehand. + return URL.canParse(NarrowIpvFuture(value)); +} diff --git a/packages/schema/src/formats/typebox/iri_reference.ts b/packages/schema/src/formats/typebox/iri_reference.ts new file mode 100644 index 000000000..5e45c75ee --- /dev/null +++ b/packages/schema/src/formats/typebox/iri_reference.ts @@ -0,0 +1,42 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +// deno-lint-ignore-file no-control-regex + +// Rejects unencoded spaces (0x20), backslashes, control characters (0x00-0x1F, 0x7F), or malformed percent-encodings +const InvalidIriChars = /[\x00-\x20\x7F\\]|%(?![0-9a-fA-F]{2})/; +// Detects malformed scheme/authority patterns lacking a colon (e.g. "httpx//example.com") +const MalformedScheme = /^[a-zA-Z][a-zA-Z0-9+\-.]*\/\//; + +/** + * Returns true if the value is an IRI reference + * @specification https://tools.ietf.org/html/rfc3987 + */ +export function IsIriReference(value: string): boolean { + return !InvalidIriChars.test(value) && !MalformedScheme.test(value) && URL.canParse(value, "http://example.com"); +} diff --git a/packages/schema/src/formats/typebox/json_pointer.ts b/packages/schema/src/formats/typebox/json_pointer.ts new file mode 100644 index 000000000..c2d171aef --- /dev/null +++ b/packages/schema/src/formats/typebox/json_pointer.ts @@ -0,0 +1,38 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +const JsonPointer = /^(?:\/(?:[^~/]|~0|~1)*)*$/; + +/** + * Returns true if the value is a json pointer + * @specification https://datatracker.ietf.org/doc/html/rfc6901 + * @source https://github.com/ajv-validator/ajv-formats + */ +export function IsJsonPointer(value: string): boolean { + return JsonPointer.test(value); +} diff --git a/packages/schema/src/formats/typebox/regex.ts b/packages/schema/src/formats/typebox/regex.ts new file mode 100644 index 000000000..52cff8d2a --- /dev/null +++ b/packages/schema/src/formats/typebox/regex.ts @@ -0,0 +1,40 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +/** + * Returns true if the value is a regular expression string pattern + * @specification https://ecma-international.org/ecma-262 + */ +export function IsRegex(value: string): boolean { + try { + new RegExp(value, "u"); + return true; + } catch { + return false; + } +} diff --git a/packages/schema/src/formats/typebox/relative_json_pointer.ts b/packages/schema/src/formats/typebox/relative_json_pointer.ts new file mode 100644 index 000000000..f2dfd6fbd --- /dev/null +++ b/packages/schema/src/formats/typebox/relative_json_pointer.ts @@ -0,0 +1,38 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +const RelativeJsonPointer = /^(?:0|[1-9][0-9]*)(?:#|(?:\/(?:[^~/]|~0|~1)*)*)$/; + +/** + * Returns true if the value is a relative json pointer + * @specification https://datatracker.ietf.org/doc/html/rfc6901 + * @source https://github.com/ajv-validator/ajv-formats + */ +export function IsRelativeJsonPointer(value: string): boolean { + return RelativeJsonPointer.test(value); +} diff --git a/packages/schema/src/formats/typebox/time.ts b/packages/schema/src/formats/typebox/time.ts new file mode 100644 index 000000000..bbc9c6113 --- /dev/null +++ b/packages/schema/src/formats/typebox/time.ts @@ -0,0 +1,56 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +const TIME = /^(\d\d):(\d\d):(\d\d)(?:\.\d+)?(?:([Zz])|([+-])(\d\d):(\d\d))?$/; + +/** + * Returns true if the value is an ISO time string + * @specification https://datatracker.ietf.org/doc/html/rfc3339 + */ +export function IsTime(value: string, strictTimeZone: boolean = true): boolean { + const matches = TIME.exec(value); + if (!matches) return false; + // Require timezone offset or 'Z'/'z' when strictTimeZone is true + if (strictTimeZone && !matches[4] && !matches[5]) return false; + const hr = +matches[1]; + const min = +matches[2]; + const sec = +matches[3]; + if (hr > 23 || min > 59 || sec > 60) return false; + if (matches[5]) { + const tzH = +matches[6]; + const tzM = +matches[7]; + if (tzH > 23 || tzM > 59) return false; + } + if (sec < 60) return true; + // Leap second handling: must normalize to 23:59:60 UTC (1439 total UTC minutes) + const tzSign = matches[5] === "-" ? -1 : 1; + const tzH = +(matches[6] || 0); + const tzM = +(matches[7] || 0); + const totalUtcMin = hr * 60 + min - tzSign * (tzH * 60 + tzM); + return ((totalUtcMin % 1440) + 1440) % 1440 === 1439; +} diff --git a/packages/schema/src/formats/typebox/uri.ts b/packages/schema/src/formats/typebox/uri.ts new file mode 100644 index 000000000..5aadeab17 --- /dev/null +++ b/packages/schema/src/formats/typebox/uri.ts @@ -0,0 +1,38 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +const Uri = + /^[a-z][a-z0-9+\-.]*:(?:\/\/(?:(?:[-a-z0-9._~!$&'()*+,;=:]|%[0-9a-f]{2})*@)?(?:\[(?:(?:(?:[\da-f]{1,4}:){6}(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|::(?:[\da-f]{1,4}:){5}(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:[\da-f]{1,4})?::(?:[\da-f]{1,4}:){4}(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:(?:[\da-f]{1,4}:){0,1}[\da-f]{1,4})?::(?:[\da-f]{1,4}:){3}(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:(?:[\da-f]{1,4}:){0,2}[\da-f]{1,4})?::(?:[\da-f]{1,4}:){2}(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:(?:[\da-f]{1,4}:){0,3}[\da-f]{1,4})?::[\da-f]{1,4}:(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:(?:[\da-f]{1,4}:){0,4}[\da-f]{1,4})?::(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:(?:[\da-f]{1,4}:){0,5}[\da-f]{1,4})?::[\da-f]{1,4}|(?:(?:[\da-f]{1,4}:){0,6}[\da-f]{1,4})?::)|v[0-9a-f]+\.[-a-z0-9._~!$&'()*+,;=:]+)\]|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)|(?:[-a-z0-9._~!$&'()*+,;=]|%[0-9a-f]{2})*)(?::\d*)?(?:\/(?:[-a-z0-9._~!$&'()*+,;=:@]|%[0-9a-f]{2})*)*|\/(?:(?:[-a-z0-9._~!$&'()*+,;=:@]|%[0-9a-f]{2})+(?:\/(?:[-a-z0-9._~!$&'()*+,;=:@]|%[0-9a-f]{2})*)*)?|(?:[-a-z0-9._~!$&'()*+,;=:@]|%[0-9a-f]{2})+(?:\/(?:[-a-z0-9._~!$&'()*+,;=:@]|%[0-9a-f]{2})*)*)?(?:\?(?:[-a-z0-9._~!$&'()*+,;=:@/?]|%[0-9a-f]{2})*)?(?:#(?:[-a-z0-9._~!$&'()*+,;=:@/?]|%[0-9a-f]{2})*)?$/i; + +/** + * Returns true if the value is a valid Uniform Resource Identifier. + * @specification https://tools.ietf.org/html/rfc3986 + */ +export function IsUri(value: string): boolean { + return Uri.test(value); +} diff --git a/packages/schema/src/formats/typebox/uri_reference.ts b/packages/schema/src/formats/typebox/uri_reference.ts new file mode 100644 index 000000000..215058ded --- /dev/null +++ b/packages/schema/src/formats/typebox/uri_reference.ts @@ -0,0 +1,38 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +const UriReference = + /^(?:[a-z][a-z0-9+\-.]*:(?:\/\/(?:(?:[-a-z0-9._~!$&'()*+,;=:]|%[0-9a-f]{2})*@)?(?:\[(?:(?:(?:[\da-f]{1,4}:){6}(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|::(?:[\da-f]{1,4}:){5}(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:[\da-f]{1,4})?::(?:[\da-f]{1,4}:){4}(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:(?:[\da-f]{1,4}:){0,1}[\da-f]{1,4})?::(?:[\da-f]{1,4}:){3}(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:(?:[\da-f]{1,4}:){0,2}[\da-f]{1,4})?::(?:[\da-f]{1,4}:){2}(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:(?:[\da-f]{1,4}:){0,3}[\da-f]{1,4})?::[\da-f]{1,4}:(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:(?:[\da-f]{1,4}:){0,4}[\da-f]{1,4})?::(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:(?:[\da-f]{1,4}:){0,5}[\da-f]{1,4})?::[\da-f]{1,4}|(?:(?:[\da-f]{1,4}:){0,6}[\da-f]{1,4})?::)|v[0-9a-f]+\.[-a-z0-9._~!$&'()*+,;=:]+)\]|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)|(?:[-a-z0-9._~!$&'()*+,;=]|%[0-9a-f]{2})*)(?::\d*)?(?:\/(?:[-a-z0-9._~!$&'()*+,;=:@]|%[0-9a-f]{2})*)*|\/(?:(?:[-a-z0-9._~!$&'()*+,;=:@]|%[0-9a-f]{2})+(?:\/(?:[-a-z0-9._~!$&'()*+,;=:@]|%[0-9a-f]{2})*)*)?|(?:[-a-z0-9._~!$&'()*+,;=:@]|%[0-9a-f]{2})+(?:\/(?:[-a-z0-9._~!$&'()*+,;=:@]|%[0-9a-f]{2})*)*)?|(?:\/\/(?:(?:[-a-z0-9._~!$&'()*+,;=:]|%[0-9a-f]{2})*@)?(?:\[(?:(?:(?:[\da-f]{1,4}:){6}(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|::(?:[\da-f]{1,4}:){5}(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:[\da-f]{1,4})?::(?:[\da-f]{1,4}:){4}(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:(?:[\da-f]{1,4}:){0,1}[\da-f]{1,4})?::(?:[\da-f]{1,4}:){3}(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:(?:[\da-f]{1,4}:){0,2}[\da-f]{1,4})?::(?:[\da-f]{1,4}:){2}(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:(?:[\da-f]{1,4}:){0,3}[\da-f]{1,4})?::[\da-f]{1,4}:(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:(?:[\da-f]{1,4}:){0,4}[\da-f]{1,4})?::(?:[\da-f]{1,4}:[\da-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))|(?:(?:[\da-f]{1,4}:){0,5}[\da-f]{1,4})?::[\da-f]{1,4}|(?:(?:[\da-f]{1,4}:){0,6}[\da-f]{1,4})?::)|v[0-9a-f]+\.[-a-z0-9._~!$&'()*+,;=:]+)\]|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)|(?:[-a-z0-9._~!$&'()*+,;=]|%[0-9a-f]{2})*)(?::\d*)?(?:\/(?:[-a-z0-9._~!$&'()*+,;=:@]|%[0-9a-f]{2})*)*|\/(?:(?:[-a-z0-9._~!$&'()*+,;=:@]|%[0-9a-f]{2})+(?:\/(?:[-a-z0-9._~!$&'()*+,;=:@]|%[0-9a-f]{2})*)*)?|(?:[-a-z0-9._~!$&'()*+,;=@]|%[0-9a-f]{2})+(?:\/(?:[-a-z0-9._~!$&'()*+,;=:@]|%[0-9a-f]{2})*)*)?)(?:\?(?:[-a-z0-9._~!$&'()*+,;=:@/?]|%[0-9a-f]{2})*)?(?:#(?:[-a-z0-9._~!$&'()*+,;=:@/?]|%[0-9a-f]{2})*)?$/i; + +/** + * Returns true if the value is a valid Uri Reference. + * @specification https://tools.ietf.org/html/rfc3986 + */ +export function IsUriReference(value: string): boolean { + return UriReference.test(value); +} diff --git a/packages/schema/src/formats/typebox/uri_template.ts b/packages/schema/src/formats/typebox/uri_template.ts new file mode 100644 index 000000000..ca364fe15 --- /dev/null +++ b/packages/schema/src/formats/typebox/uri_template.ts @@ -0,0 +1,40 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +// deno-lint-ignore-file no-control-regex + +const UriTemplate = + /^(?:(?:[^\x00-\x20"<>%\\^`{|}\x7f]|%[0-9a-f]{2})|\{[+#./;?&=,!@|]?(?:[a-z0-9_]|%[0-9a-f]{2})+(?:\.(?:[a-z0-9_]|%[0-9a-f]{2})+)*(?::[1-9]\d{0,3}|\*)?(?:,(?:[a-z0-9_]|%[0-9a-f]{2})+(?:\.(?:[a-z0-9_]|%[0-9a-f]{2})+)*(?::[1-9]\d{0,3}|\*)?)*\})*$/i; + +/** + * Returns true if the value is a valid Uri Template + * @specification https://datatracker.ietf.org/doc/html/rfc6570 + */ +export function IsUriTemplate(value: string): boolean { + return UriTemplate.test(value); +} diff --git a/packages/schema/src/formats/typebox/uuid.ts b/packages/schema/src/formats/typebox/uuid.ts new file mode 100644 index 000000000..cf26ffa6e --- /dev/null +++ b/packages/schema/src/formats/typebox/uuid.ts @@ -0,0 +1,38 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +const Uuid = /^[0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12}$/i; + +/** + * Returns true if the value is a uuid + * @specification https://www.rfc-editor.org/info/rfc4122/ + * @specification https://www.rfc-editor.org/info/rfc9562/ + */ +export function IsUuid(value: string): boolean { + return Uuid.test(value); +} diff --git a/packages/schema/src/index.ts b/packages/schema/src/index.ts new file mode 100644 index 000000000..c421d6fdb --- /dev/null +++ b/packages/schema/src/index.ts @@ -0,0 +1,31 @@ +// JSON Schema validation (drafts 2020-12 and 7) without eval: compile() builds a schema into plain closures once. +// The builders return plain JSON Schema objects typed with the values they accept. + +export { + type Annotations, + arr, + bool, + bytes, + int, + type LengthBounds, + lit, + map, + type NumberBounds, + nul, + nullable, + num, + Optional, + obj, + oneOf, + opt, + type Properties, + type Schema, + type Static, + type StaticObject, + str, + tuple, + union, + unknown, +} from "./builders.ts"; +export type { Check, CompileOptions, FormatCheck, JsonSchema, ValidationError } from "./types.ts"; +export { compile, Validator } from "./validator.ts"; diff --git a/packages/schema/src/resolver.ts b/packages/schema/src/resolver.ts new file mode 100644 index 000000000..a10d245b6 --- /dev/null +++ b/packages/schema/src/resolver.ts @@ -0,0 +1,152 @@ +// Reference resolution of one compile: schema resources, anchors and JSON Pointers over the root schema and the +// registered documents. A Resolver lives only while a schema compiles (or while its errors are explained). + +import type { CompileOptions, JsonSchema, SchemaObject } from "./types.ts"; + +/** A schema and the base URI its references resolve against. */ +export interface Resolved { + readonly node: unknown; + readonly base: string; +} + +/** The base URI of a root schema without `$id`; relative references resolve against it. */ +export const DEFAULT_BASE = "https://json-schema.invalid/root.json"; + +/** Indexes the root schema and resolves references in it and in the registered documents. */ +export class Resolver { + readonly draft: "2020-12" | "7"; + readonly rootBase: string; + /** Schema object to the base URI it was indexed under. */ + readonly baseOf = new Map(); + /** URI (with a `#name` fragment) of every `$dynamicAnchor`, to its schema. */ + readonly dynamicAnchors = new Map(); + private readonly registry = new Map(); + private readonly resources = new Map(); + private readonly anchors = new Map(); + + constructor(root: JsonSchema, options: CompileOptions) { + this.draft = options.draft === "7" ? "7" : "2020-12"; + for (const [uri, schema] of Object.entries(options.schemas ?? {})) { + this.registry.set(documentOf(resolveUri(uri, DEFAULT_BASE)), schema); + } + this.resources.set(DEFAULT_BASE, root); + this.index(root, DEFAULT_BASE); + this.rootBase = (isSchemaObject(root) && this.baseOf.get(root)) || DEFAULT_BASE; + } + + /** The schema a reference names, with the base URI it compiles under. Throws if it names nothing. */ + resolve(ref: string, base: string): Resolved { + const uri = resolveUri(ref, base); + const doc = documentOf(uri); + const fragment = uri.length > doc.length ? decodeURIComponent(uri.slice(doc.length + 1)) : ""; + const root = this.resources.has(doc) ? this.resources.get(doc) : this.load(doc); + if (root === undefined) throw new Error(`Unresolved $ref ${ref} (no schema registered for ${doc})`); + if (fragment === "") return { node: root, base: doc }; + if (fragment.startsWith("/")) return this.pointer(root, doc, fragment); + const target = this.anchors.get(`${doc}#${fragment}`); + if (target === undefined) throw new Error(`Unresolved $ref ${ref} (no anchor ${fragment} in ${doc})`); + return { node: target, base: this.baseOf.get(target) ?? doc }; + } + + /** Whether `node` is the root of the schema resource at `base`. */ + isResource(node: object, base: string): boolean { + return this.resources.get(base) === node; + } + + // Walks only keywords that hold subschemas: an `$id` inside `enum`, `const` or an unknown keyword names nothing. + private index(node: unknown, base: string): void { + if (!isSchemaObject(node) || this.baseOf.has(node)) return; + let b = base; + // In draft 7 a `$ref` makes every sibling keyword, `$id` included, inert. + const refOnly = this.draft === "7" && typeof node.$ref === "string"; + if (!refOnly && typeof node.$id === "string") { + if (this.draft === "7" && node.$id.startsWith("#")) this.anchors.set(documentOf(b) + node.$id, node); + else { + const uri = resolveUri(node.$id, b); + b = documentOf(uri); + this.resources.set(b, node); + if (uri.length > b.length + 1) this.anchors.set(uri, node); + } + } + this.baseOf.set(node, b); + if (refOnly) return; + if (this.draft !== "7") { + if (typeof node.$anchor === "string") this.anchors.set(`${b}#${node.$anchor}`, node); + if (typeof node.$dynamicAnchor === "string") { + this.anchors.set(`${b}#${node.$dynamicAnchor}`, node); + this.dynamicAnchors.set(`${b}#${node.$dynamicAnchor}`, node); + } + } + for (const key of SCHEMA_KEYWORDS) this.index(node[key], b); + for (const key of MAP_KEYWORDS) { + const map = node[key]; + if (isSchemaObject(map)) for (const k of Object.keys(map)) this.index(map[k], b); + } + for (const key of ARRAY_KEYWORDS) { + const list = node[key]; + if (Array.isArray(list)) for (const s of list) this.index(s, b); + } + } + + private load(doc: string): JsonSchema | undefined { + const schema = this.registry.get(doc); + if (schema === undefined) return undefined; + this.resources.set(doc, schema); + this.index(schema, doc); + return schema; + } + + private pointer(root: unknown, base: string, fragment: string): Resolved { + let node = root; + let b = base; + for (const raw of fragment.split("/").slice(1)) { + const key = raw.replace(/~1/g, "/").replace(/~0/g, "~"); + node = + typeof node === "object" && node !== null && Object.hasOwn(node, key) + ? (node as SchemaObject)[key] + : undefined; + if (node === undefined) throw new Error(`Unresolved $ref #${fragment} in ${base}`); + if (typeof node === "object" && node !== null) b = this.baseOf.get(node) ?? b; + } + return { node, base: b }; + } +} + +const SCHEMA_KEYWORDS = [ + "additionalProperties", + "propertyNames", + "contains", + "not", + "if", + "then", + "else", + "unevaluatedProperties", + "unevaluatedItems", + "additionalItems", + "contentSchema", + "items", +]; + +const MAP_KEYWORDS = ["properties", "patternProperties", "$defs", "definitions", "dependentSchemas", "dependencies"]; + +const ARRAY_KEYWORDS = ["allOf", "anyOf", "oneOf", "prefixItems", "items"]; + +/** Whether `value` is a schema object (not `true`/`false`, an array or a primitive). */ +export function isSchemaObject(value: unknown): value is SchemaObject { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +/** `ref` resolved against `base`. URNs and other bases a URL parser rejects take fragment-only references. */ +export function resolveUri(ref: string, base: string): string { + try { + return new URL(ref, base).href; + } catch { + return ref.startsWith("#") ? documentOf(base) + ref : ref; + } +} + +/** `uri` without its fragment. */ +export function documentOf(uri: string): string { + const hash = uri.indexOf("#"); + return hash < 0 ? uri : uri.slice(0, hash); +} diff --git a/packages/schema/src/shared.ts b/packages/schema/src/shared.ts new file mode 100644 index 000000000..bb3867d3d --- /dev/null +++ b/packages/schema/src/shared.ts @@ -0,0 +1,50 @@ +// Values shared across all compiled schemas while one still uses them: leaf checks and regular expressions. +// Both caches hold their values weakly and drop an entry once its value is collected, so they keep nothing alive. + +import type { Check } from "./types.ts"; + +/** The check shared under `key`, made by `make` when no live schema holds one. */ +export function sharedCheck(key: string, make: () => Check): Check { + const hit = checks.get(key)?.deref(); + if (hit !== undefined) return hit; + const check = make(); + checks.set(key, new WeakRef(check)); + checkCleanup.register(check, key); + return check; +} + +const checks = new Map>(); + +const checkCleanup = new FinalizationRegistry((key) => { + if (checks.get(key)?.deref() === undefined) checks.delete(key); +}); + +/** + * The regular expression of a `pattern` source, shared by source. Patterns are ECMAScript regular expressions with + * the `u` flag; a source that is invalid with it (some older schemas escape characters `u` rejects) compiles + * without it. + */ +export function sharedRegex(source: string): RegExp { + const hit = regexes.get(source)?.deref(); + if (hit !== undefined) return hit; + let re: RegExp; + try { + re = new RegExp(source, "u"); + } catch { + re = new RegExp(source); + } + regexes.set(source, new WeakRef(re)); + regexCleanup.register(re, source); + return re; +} + +const regexes = new Map>(); + +const regexCleanup = new FinalizationRegistry((source) => { + if (regexes.get(source)?.deref() === undefined) regexes.delete(source); +}); + +/** The number of entries in both caches (for tests). */ +export function sharedEntries(): { checks: number; regexes: number } { + return { checks: checks.size, regexes: regexes.size }; +} diff --git a/packages/schema/src/types.ts b/packages/schema/src/types.ts new file mode 100644 index 000000000..a9e2135d1 --- /dev/null +++ b/packages/schema/src/types.ts @@ -0,0 +1,36 @@ +// Public types: the schema accepted by compile(), its options, and the errors a failed value produces. + +/** + * A JSON Schema: `true`, `false` or a schema object. Schemas come from tool definitions, MCP servers and other + * untrusted sources, so compile() reads every keyword defensively instead of trusting a static shape. + */ +export type JsonSchema = boolean | object; + +/** A format check: whether `value` (a string) has the format. */ +export type FormatCheck = (value: string) => boolean; + +/** Options of compile(). */ +export interface CompileOptions { + /** The draft whose semantics apply: "2020-12" (default) or "7". */ + readonly draft?: "2020-12" | "7"; + /** + * Format name to check. Without it `format` is an annotation, as draft 2020-12 specifies. The standard formats are + * in `@earendil-works/pi-schema/formats`. + */ + readonly formats?: Readonly>; + /** Schema documents by URI, for `$ref`s to other documents. */ + readonly schemas?: Readonly>; +} + +/** One failure of a value: where it is (a JSON Pointer, "" for the value itself), the failing keyword and a message. */ +export interface ValidationError { + readonly path: string; + readonly keyword: string; + readonly message: string; +} + +/** A schema object as compile() reads it: any keyword, of any value, checked where it is used. */ +export type SchemaObject = { readonly [keyword: string]: unknown }; + +/** A compiled boolean check. */ +export type Check = (value: unknown) => boolean; diff --git a/packages/schema/src/validator.ts b/packages/schema/src/validator.ts new file mode 100644 index 000000000..3d915b33b --- /dev/null +++ b/packages/schema/src/validator.ts @@ -0,0 +1,58 @@ +// compile(): a schema to a Validator, whose `check` is the fast boolean path and whose `errors` explains failures. + +import type { Static } from "./builders.ts"; +import { Compiler, RESTART } from "./compiler.ts"; +import { Explainer } from "./explain.ts"; +import type { Check, CompileOptions, JsonSchema, ValidationError } from "./types.ts"; + +/** + * Compiles `schema` into a Validator. Throws if the schema is malformed or a `$ref` names nothing (register other + * documents with `options.schemas`). The schema is read, never modified. + */ +export function compile(schema: S, options: CompileOptions = NO_OPTIONS): Validator> { + return new Validator(schema, options, compileCheck(schema, options)); +} + +const NO_OPTIONS: CompileOptions = Object.freeze({}); + +/** A compiled schema whose valid values have type `T` (`unknown` unless the schema was built with the builders). */ +export class Validator { + /** Whether `value` is valid. A plain function: it may be passed around without its validator. */ + readonly check: (value: unknown) => value is T; + private readonly schema: JsonSchema; + private readonly options: CompileOptions; + private explainer: Explainer | undefined = undefined; + + constructor(schema: JsonSchema, options: CompileOptions, check: Check) { + // A compiled check is exactly the test that a value has the schema's static type. + this.check = check as (value: unknown) => value is T; + this.schema = schema; + this.options = options; + } + + /** + * Why `value` is invalid: one error per failing keyword, `[]` when it is valid. Built on the first failure and + * kept, so a validator whose values always pass never pays for it. + */ + errors(value: unknown): ValidationError[] { + if (this.check(value)) return []; + this.explainer ??= new Explainer(compiler(this.schema, this.options), this.schema); + return this.explainer.errors(value); + } +} + +function compileCheck(schema: JsonSchema, options: CompileOptions): Check { + return compiler(schema, options).root(); +} + +// Most schemas compile without tracking the dynamic scope; one with a bookended $dynamicRef compiles again with it. +function compiler(schema: JsonSchema, options: CompileOptions): Compiler { + const plain = new Compiler(schema, options, false); + try { + plain.root(); + return plain; + } catch (error) { + if (error !== RESTART) throw error; + return new Compiler(schema, options, true); + } +} diff --git a/packages/schema/test/bench/all.ts b/packages/schema/test/bench/all.ts new file mode 100644 index 000000000..bb8421fc2 --- /dev/null +++ b/packages/schema/test/bench/all.ts @@ -0,0 +1,36 @@ +// Runs every benchmark, each in its own process, and writes the results as JSON next to the printed tables. +// Speed runs twice: normally and with code generation from strings disabled, as under a strict CSP. +// Run: npm run bench [-- --out ] (default: pi-schema-bench-results.json in the temp directory) + +import { execFileSync } from "node:child_process"; +import { readFileSync, rmSync, writeFileSync } from "node:fs"; +import { cpus, tmpdir } from "node:os"; +import { join } from "node:path"; + +const dir = import.meta.dirname; +const outIndex = process.argv.indexOf("--out"); +const out = outIndex >= 0 ? process.argv[outIndex + 1] : join(tmpdir(), "pi-schema-bench-results.json"); + +const runs: [string, string, string[]][] = [ + ["conformance", "conformance.ts", []], + ["speed", "speed.ts", []], + ["speedNoEval", "speed.ts", ["--disallow-code-generation-from-strings"]], + ["setup", "setup.ts", []], + ["memory", "memory.ts", []], + ["size", "size.ts", []], +]; + +const results: Record = { + node: process.version, + cpu: cpus()[0]?.model ?? "unknown", + date: new Date().toISOString(), +}; +for (const [key, file, flags] of runs) { + const json = join(tmpdir(), `pi-schema-bench-${key}.json`); + console.log(`\n## ${key}${flags.length > 0 ? ` (${flags.join(" ")})` : ""}\n`); + execFileSync(process.execPath, [...flags, join(dir, file), "--json", json], { stdio: "inherit" }); + results[key] = JSON.parse(readFileSync(json, "utf8")); + rmSync(json); +} +writeFileSync(out, JSON.stringify(results, null, "\t")); +console.log(`\nResults: ${out}`); diff --git a/packages/schema/test/bench/candidates.ts b/packages/schema/test/bench/candidates.ts new file mode 100644 index 000000000..8b231c855 --- /dev/null +++ b/packages/schema/test/bench/candidates.ts @@ -0,0 +1,101 @@ +// The validators the benchmarks compare, each behind the same interface: compile a schema, then check values. +// Every library is used through its fastest boolean path and keeps the object it returns, as an application would. + +import { Validator as CfValidator } from "@cfworker/json-schema"; +import { fromJsonSchema } from "@oh-my-pi/omptype"; +import { Compile } from "typebox/compile"; +import { ResetExternal, ResetFunctions } from "typebox/schema"; +import { Settings } from "typebox/system"; +import { compile, type FormatCheck, type JsonSchema } from "../../src/index.ts"; + +/** How a benchmark compiles a schema. */ +export interface PrepareOptions { + readonly draft: "2020-12" | "7"; + readonly schemas?: Readonly>; + readonly formats?: Readonly>; +} + +/** A compiled schema: its boolean check and the object the library returned (kept alive by the caller). */ +export interface Prepared { + readonly check: (value: unknown) => boolean; + readonly validator: unknown; +} + +/** A library under comparison. */ +export interface Candidate { + readonly name: string; + readonly prepare: (schema: JsonSchema, options: PrepareOptions) => Prepared; +} + +/** Whether this process runs with code generation from strings disabled, as under a strict CSP. */ +export const noEval = process.execArgv.includes("--disallow-code-generation-from-strings"); + +const piSchema: Candidate = { + name: "pi-schema", + prepare(schema, options) { + const validator = compile(schema, options); + return { check: validator.check, validator }; + }, +}; + +// TypeBox keeps its last build in module-level accumulators; reset them so no build state is retained between schemas. +function typeboxCompile(schema: JsonSchema, options: PrepareOptions, accelerate: boolean) { + Settings.Set({ useAcceleration: accelerate }); + try { + return Compile((options.schemas ?? {}) as never, schema as never); + } finally { + ResetFunctions(); + ResetExternal(); + Settings.Set({ useAcceleration: true }); + } +} + +const typebox: Candidate = { + name: noEval ? "typebox (no eval: interpreter)" : "typebox", + prepare(schema, options) { + const validator = typeboxCompile(schema, options, true); + return { check: (v) => validator.Check(v), validator }; + }, +}; + +const typeboxInterpreter: Candidate = { + name: "typebox interpreter", + prepare(schema, options) { + const validator = typeboxCompile(schema, options, false); + return { check: (v) => validator.Check(v), validator }; + }, +}; + +const cfworker: Candidate = { + name: "cfworker", + prepare(schema, options) { + const validator = new CfValidator(schema as never, options.draft, true); + for (const [uri, remote] of Object.entries(options.schemas ?? {})) validator.addSchema(remote as never, uri); + return { check: (v) => validator.validate(v).valid, validator }; + }, +}; + +// omptype compiles with new Function on the first allows() call. Without eval that call throws EvalError once per +// schema; omptype then keeps validating through its interpreter, so the benchmark absorbs that one throw. +const omptype: Candidate = { + name: noEval ? "omptype (no eval: interpreter)" : "omptype", + prepare(schema) { + const validator = fromJsonSchema(schema); + if (noEval) { + try { + validator.allows(null); + } catch (error) { + if (!(error instanceof EvalError)) throw error; + } + } + return { check: (v) => validator.allows(v), validator }; + }, +}; + +/** Every candidate, ours first. */ +export const candidates: readonly Candidate[] = [piSchema, typebox, cfworker, omptype]; + +/** Candidates by name, including TypeBox's interpreter (what TypeBox runs when eval is unavailable). */ +export const byName: ReadonlyMap = new Map( + [...candidates, typeboxInterpreter].map((c) => [c.name.split(" (")[0], c]), +); diff --git a/packages/schema/test/bench/cases.ts b/packages/schema/test/bench/cases.ts new file mode 100644 index 000000000..d6821df1e --- /dev/null +++ b/packages/schema/test/bench/cases.ts @@ -0,0 +1,261 @@ +// Speed cases: realistic schemas with a valid and an invalid value each. Argument checks shaped like Chord's +// contracts (closed objects, tagged unions), pi's built-in tool schemas, and MCP tool schemas with $ref and anyOf. + +import type { JsonSchema } from "../../src/index.ts"; + +/** A schema, the draft it is written in, and one valid and one invalid value. */ +export interface Case { + readonly name: string; + readonly schema: JsonSchema; + readonly valid: unknown; + readonly invalid: unknown; +} + +type Props = Record; +const closed = (properties: Props, optional: string[] = []) => ({ + type: "object", + properties, + required: Object.keys(properties).filter((k) => !optional.includes(k)), + additionalProperties: false, +}); +const str = (o: object = {}) => ({ type: "string", ...o }); +const int = (o: object = {}) => ({ type: "integer", ...o }); +const num = (o: object = {}) => ({ type: "number", ...o }); +const lit = (v: string) => ({ const: v }); +const arr = (items: object, o: object = {}) => ({ type: "array", items, ...o }); +const uuid = "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$"; + +const block = { + anyOf: [ + closed({ kind: lit("text"), text: str() }), + closed({ kind: lit("image"), src: str(), width: int({ minimum: 0 }), height: int({ minimum: 0 }), alt: str() }, [ + "alt", + ]), + closed({ kind: lit("code"), lang: { anyOf: [lit("ts"), lit("js"), lit("py")] }, source: str() }), + ], +}; + +const document = closed( + { + id: str({ pattern: uuid }), + title: str({ minLength: 1, maxLength: 200 }), + tags: arr(str(), { maxItems: 16 }), + author: closed({ name: str(), email: str() }, ["email"]), + blocks: arr(block), + score: num({ minimum: 0, maximum: 1 }), + draft: { type: "boolean" }, + }, + ["score", "draft"], +); + +const documentValue = { + id: "01890a5d-ac96-774b-bcce-b302099a8057", + title: "Quarterly notes", + tags: ["a", "b", "c"], + author: { name: "Mario", email: "m@example.com" }, + blocks: [ + { kind: "text", text: "Hello" }, + { kind: "image", src: "a.png", width: 640, height: 480 }, + { kind: "code", lang: "ts", source: "const x = 1;" }, + { kind: "text", text: "World" }, + { kind: "image", src: "b.png", width: 10, height: 20, alt: "b" }, + ], + score: 0.5, +}; + +const id = int({ minimum: 0 }); +const message = { + anyOf: [ + closed({ + kind: lit("hello"), + protocol: int({ minimum: 0 }), + incarnation: str({ pattern: uuid }), + calls: int({ minimum: 1 }), + }), + closed( + { + kind: lit("hold"), + hold: id, + type: str(), + version: int({ minimum: 1 }), + fingerprint: str({ pattern: "^[0-9a-f]{16}$" }), + params: {}, + address: closed({ kind: str(), params: {} }), + chain: arr(str({ pattern: uuid }), { maxItems: 64 }), + }, + ["chain"], + ), + closed({ kind: lit("release"), hold: id }), + closed({ kind: lit("call"), call: id, hold: id, method: id, args: {} }, ["args"]), + closed({ kind: lit("abort"), call: id }), + ], +}; + +const tool = (properties: Props, required: string[]) => ({ type: "object", properties, required }); + +const mcpIssue = { + $schema: "http://json-schema.org/draft-07/schema#", + type: "object", + properties: { + owner: str({ minLength: 1, pattern: "^[A-Za-z0-9-]+$" }), + repo: str({ minLength: 1, pattern: "^[A-Za-z0-9._-]+$" }), + title: str({ minLength: 1 }), + body: str(), + labels: arr(str()), + assignees: arr(str()), + milestone: { anyOf: [{ type: "number" }, { type: "null" }] }, + state: { type: "string", enum: ["open", "closed"] }, + }, + required: ["owner", "repo", "title"], + additionalProperties: false, +}; + +const mcpEdit = { + $schema: "https://json-schema.org/draft/2020-12/schema", + type: "object", + $defs: { + Range: closed({ start: int({ minimum: 1 }), end: int({ minimum: 1 }) }), + Replace: closed({ oldText: str(), newText: str() }), + LineEdit: closed({ range: { $ref: "#/$defs/Range" }, text: str() }), + }, + properties: { + path: str({ pattern: "^(/|\\./|[A-Za-z]:\\\\)?[^\\u0000]+$" }), + edits: arr({ anyOf: [{ $ref: "#/$defs/Replace" }, { $ref: "#/$defs/LineEdit" }] }, { minItems: 1 }), + dryRun: { type: "boolean" }, + }, + required: ["path", "edits"], + additionalProperties: false, +}; + +const mcpPage = { + type: "object", + $defs: { + RichText: closed( + { + type: { enum: ["text", "mention"] }, + content: str({ maxLength: 2000 }), + annotations: closed({ bold: { type: "boolean" }, italic: { type: "boolean" }, color: str() }, [ + "bold", + "italic", + "color", + ]), + }, + ["annotations"], + ), + Block: { + type: "object", + properties: { + type: { enum: ["paragraph", "heading_1", "heading_2", "bulleted_list_item", "to_do"] }, + rich_text: arr({ $ref: "#/$defs/RichText" }), + checked: { type: "boolean" }, + children: arr({ $ref: "#/$defs/Block" }), + }, + required: ["type", "rich_text"], + additionalProperties: false, + }, + }, + properties: { + parent: { oneOf: [closed({ page_id: str({ pattern: uuid }) }), closed({ database_id: str({ pattern: uuid }) })] }, + title: str({ minLength: 1 }), + due: str({ pattern: "^\\d{4}-\\d{2}-\\d{2}$" }), + children: arr({ $ref: "#/$defs/Block" }, { maxItems: 100 }), + }, + required: ["parent", "title"], + additionalProperties: false, +}; + +const rt = (content: string) => ({ + type: "text", + content, + annotations: { bold: false, italic: false, color: "default" }, +}); +const pageValue = { + parent: { page_id: "01890a5d-ac96-774b-bcce-b302099a8057" }, + title: "Plan", + due: "2026-11-01", + children: [ + { type: "heading_1", rich_text: [rt("Plan")] }, + { + type: "bulleted_list_item", + rich_text: [rt("one")], + children: [{ type: "to_do", rich_text: [rt("sub")], checked: true }], + }, + { type: "paragraph", rich_text: [rt("a"), rt("b"), rt("c")] }, + ], +}; +const pageInvalid = structuredClone(pageValue); +pageInvalid.children[1].children = [{ type: "to_do", rich_text: [rt("sub")], checked: "yes" as unknown as boolean }]; + +/** The speed cases. */ +export const cases: readonly Case[] = [ + { name: "small closed object", schema: closed({ text: str() }), valid: { text: "hello" }, invalid: { text: 1 } }, + { + name: "nested object with tagged unions", + schema: document, + valid: documentValue, + invalid: { + ...documentValue, + blocks: [...documentValue.blocks.slice(0, 4), { kind: "image", src: "b.png", width: -1, height: 20 }], + }, + }, + { + name: "message union of 5", + schema: message, + valid: { kind: "call", call: 7, hold: 3, method: 1, args: { a: 1 } }, + invalid: { kind: "call", call: 7, hold: 3, method: 1, extra: true }, + }, + { + name: "tool: read", + schema: tool({ path: str({ description: "Path to the file" }), offset: num(), limit: num() }, ["path"]), + valid: { path: "src/a.ts", offset: 10, limit: 50 }, + invalid: { offset: 1 }, + }, + { + name: "tool: edit", + schema: tool({ path: str(), edits: arr(tool({ oldText: str(), newText: str() }, ["oldText", "newText"])) }, [ + "path", + "edits", + ]), + valid: { + path: "a.ts", + edits: [ + { oldText: "a", newText: "b" }, + { oldText: "c", newText: "d" }, + ], + }, + invalid: { path: "a.ts", edits: [{ oldText: "a" }] }, + }, + { + name: "tool: bash", + schema: tool({ command: str(), timeout: num() }, ["command"]), + valid: { command: "ls -la", timeout: 30 }, + invalid: { command: 5 }, + }, + { + name: "MCP create_issue (pattern, anyOf, enum)", + schema: mcpIssue, + valid: { + owner: "earendil-works", + repo: "pi", + title: "Bug", + body: "...", + labels: ["bug"], + milestone: null, + state: "open", + }, + invalid: { owner: "earendil works", repo: "pi", title: "Bug" }, + }, + { + name: "MCP edit_file ($ref, anyOf)", + schema: mcpEdit, + valid: { + path: "./src/a.ts", + edits: [ + { oldText: "a", newText: "b" }, + { range: { start: 1, end: 3 }, text: "x" }, + ], + }, + invalid: { path: "./src/a.ts", edits: [{ range: { start: 0, end: 3 }, text: "x" }] }, + }, + { name: "MCP create_page (recursive $ref, oneOf)", schema: mcpPage, valid: pageValue, invalid: pageInvalid }, +]; diff --git a/packages/schema/test/bench/conformance.ts b/packages/schema/test/bench/conformance.ts new file mode 100644 index 000000000..d0cc72a7e --- /dev/null +++ b/packages/schema/test/bench/conformance.ts @@ -0,0 +1,54 @@ +// Conformance of every candidate on the vendored JSON-Schema-Test-Suite (drafts 2020-12 and 7), by section: +// core, optional and optional/format. Every candidate gets the remote documents and meta-schemas; format tests run +// with pi-schema's format checks (the other libraries check formats on their own or not at all). +// Run: node test/bench/conformance.ts [--json ] + +import { writeFileSync } from "node:fs"; +import { formats } from "../../src/formats/index.ts"; +import { suiteFiles, suiteSchemas } from "../suite.ts"; +import { candidates } from "./candidates.ts"; + +/** Passed and total tests by section ("draft2020-12 core", ...) and candidate name. */ +export type ConformanceResults = Record }>; + +const results: ConformanceResults = {}; +for (const file of suiteFiles()) { + const kind = file.name.includes("/optional/format/") + ? "format" + : file.name.includes("/optional/") + ? "optional" + : "core"; + const section = `${file.name.split("/")[0]} ${kind}`; + results[section] ??= { total: 0, passed: Object.fromEntries(candidates.map((c) => [c.name, 0])) }; + for (const group of file.groups) { + results[section].total += group.tests.length; + for (const candidate of candidates) { + let check: ((v: unknown) => boolean) | undefined; + try { + check = candidate.prepare(structuredClone(group.schema), { + draft: file.draft, + schemas: suiteSchemas(), + formats: kind === "format" ? formats : undefined, + }).check; + } catch { + continue; // A schema the library cannot compile fails all its tests. + } + for (const test of group.tests) { + try { + if (check(test.data) === test.valid) results[section].passed[candidate.name]++; + } catch { + // A throw while checking fails the test. + } + } + } + } +} + +const names = candidates.map((c) => c.name); +console.log(`| section | tests | ${names.join(" | ")} |`); +console.log(`|---|---:|${names.map(() => "---:").join("|")}|`); +for (const [section, r] of Object.entries(results)) { + console.log(`| ${section} | ${r.total} | ${names.map((n) => r.passed[n]).join(" | ")} |`); +} +const out = process.argv.indexOf("--json"); +if (out >= 0) writeFileSync(process.argv[out + 1], JSON.stringify(results, null, "\t")); diff --git a/packages/schema/test/bench/corpus.ts b/packages/schema/test/bench/corpus.ts new file mode 100644 index 000000000..2a985db8e --- /dev/null +++ b/packages/schema/test/bench/corpus.ts @@ -0,0 +1,122 @@ +// A deterministic corpus of distinct tool-like JSON Schemas with one valid value each: random property names, +// descriptions, leaf constraints, enums, nested objects, arrays and some $defs/$ref. No two schemas are copies. + +/** A schema and a value it accepts. */ +export interface CorpusItem { + readonly schema: Record; + readonly value: unknown; +} + +const SYLLABLES = [ + "ka", + "lo", + "mi", + "re", + "su", + "ta", + "ne", + "vo", + "pi", + "da", + "ru", + "xe", + "bo", + "fi", + "gu", + "ha", + "jo", + "ze", +]; +const PATTERNS: readonly (readonly [string, string])[] = [ + ["^[a-z0-9-]+$", "a-1"], + ["^[A-Za-z0-9._-]+$", "A.b_c"], + ["^\\d{4}-\\d{2}-\\d{2}$", "2026-10-31"], + ["^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$", "01890a5d-ac96-774b-bcce-b302099a8057"], + ["^#[0-9a-f]{6}$", "#00ff00"], +]; + +/** `count` distinct schemas, the same for the same seed. */ +export function corpus(count: number, seed = 42): CorpusItem[] { + const random = mulberry32(seed); + const int = (lo: number, hi: number) => lo + Math.floor(random() * (hi - lo + 1)); + const pick = (list: readonly T[]): T => list[Math.floor(random() * list.length)]; + const word = () => Array.from({ length: int(2, 4) }, () => pick(SYLLABLES)).join(""); + const sentence = () => `${Array.from({ length: int(4, 12) }, word).join(" ")}.`; + + function leaf(depth: number): [Record, unknown] { + const r = random(); + if (r < 0.3) return [{ type: "string", description: sentence() }, word()]; + if (r < 0.4) { + const min = int(0, 3); + return [ + { type: "string", minLength: min, maxLength: min + int(5, 200), description: sentence() }, + "x".repeat(min + 1), + ]; + } + if (r < 0.47) { + const [pattern, value] = pick(PATTERNS); + return [{ type: "string", pattern, description: sentence() }, value]; + } + if (r < 0.55) { + const values = Array.from({ length: int(2, 5) }, word); + return [{ type: "string", enum: values, description: sentence() }, values[0]]; + } + if (r < 0.65) { + const min = int(0, 10); + return [{ type: "integer", minimum: min, maximum: min + int(10, 10000), description: sentence() }, min + 1]; + } + if (r < 0.72) return [{ type: "number", description: sentence() }, random() * 100]; + if (r < 0.8) return [{ type: "boolean", description: sentence() }, random() < 0.5]; + if (r < 0.88) { + return [{ type: "array", items: { type: "string" }, maxItems: int(1, 50), description: sentence() }, [word()]]; + } + if (depth < 2) return object(depth + 1, int(2, 5)); + return [{ type: "string", description: sentence() }, word()]; + } + + function object(depth: number, n: number): [Record, Record] { + const properties: Record> = {}; + const value: Record = {}; + const required: string[] = []; + for (let i = 0; i < n; i++) { + const name = word() + (i > 0 ? i : ""); + const [schema, v] = leaf(depth); + properties[name] = schema; + if (random() < 0.6) { + required.push(name); + value[name] = v; + } else if (random() < 0.5) value[name] = v; + } + const schema: Record = { type: "object", properties, required }; + if (random() < 0.5) schema.additionalProperties = false; + return [schema, value]; + } + + const items: CorpusItem[] = []; + for (let i = 0; i < count; i++) { + const [schema, value] = object(0, int(2, 10)); + const properties = schema.properties as Record>; + if (random() < 0.2) { + // Move one nested object into $defs and reference it, as MCP servers built with zod often do. + const key = Object.keys(properties).find((k) => properties[k].type === "object"); + if (key !== undefined) { + const name = `${word()}Def`; + schema.$defs = { [name]: properties[key] }; + properties[key] = { $ref: `#/$defs/${name}` }; + } + } + schema.description = sentence(); + items.push({ schema, value }); + } + return items; +} + +function mulberry32(seed: number): () => number { + let state = seed; + return () => { + state = (state + 0x6d2b79f5) | 0; + let t = Math.imul(state ^ (state >>> 15), 1 | state); + t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + }; +} diff --git a/packages/schema/test/bench/memory.ts b/packages/schema/test/bench/memory.ts new file mode 100644 index 000000000..9f2c1546b --- /dev/null +++ b/packages/schema/test/bench/memory.ts @@ -0,0 +1,73 @@ +// Retained heap of 3,200 distinct compiled schemas per candidate. Each candidate runs in its own process with +// --expose-gc: the schemas and values are built before the baseline, every library's returned validator is kept, +// and the heap is read after forced collections, once after compiling and once after checking each value 3 times +// (when libraries that compile lazily, like omptype, have compiled). +// Run: node test/bench/memory.ts [--json ] + +import { execFileSync } from "node:child_process"; +import { writeFileSync } from "node:fs"; +import { byName, candidates } from "./candidates.ts"; +import { corpus } from "./corpus.ts"; + +const COUNT = Number(process.env.COUNT ?? 3200); + +/** MB retained by the schemas themselves, and by each candidate after compiling and after checking. */ +export interface MemoryResults { + readonly count: number; + readonly schemasMB: number; + readonly candidates: Record; +} + +const child = process.argv.indexOf("--child"); +if (child >= 0) measure(process.argv[child + 1]); +else run(); + +function run(): void { + const results: MemoryResults = { count: COUNT, schemasMB: 0, candidates: {} }; + const names = [...candidates.map((c) => c.name), "typebox interpreter"]; + for (const name of names) { + const output = execFileSync(process.execPath, ["--expose-gc", import.meta.filename, "--child", name], { + encoding: "utf8", + env: { ...process.env, COUNT: String(COUNT) }, + }); + const result = JSON.parse(output); + Object.assign(results, { schemasMB: result.schemasMB }); + results.candidates[name] = { compiledMB: result.compiledMB, checkedMB: result.checkedMB }; + } + console.log(`${COUNT} distinct schemas take ${results.schemasMB.toFixed(1)} MB themselves (not counted below).\n`); + console.log("| candidate | after compiling | after checking |"); + console.log("|---|---:|---:|"); + for (const [name, r] of Object.entries(results.candidates)) { + console.log(`| ${name} | ${r.compiledMB.toFixed(1)} MB | ${r.checkedMB.toFixed(1)} MB |`); + } + const out = process.argv.indexOf("--json"); + if (out >= 0) writeFileSync(process.argv[out + 1], JSON.stringify(results, null, "\t")); +} + +function measure(name: string): void { + const candidate = byName.get(name); + if (candidate === undefined) throw new Error(`Unknown candidate ${name}`); + const gc = (globalThis as { gc?: () => void }).gc; + if (gc === undefined) throw new Error("Run with --expose-gc"); + const heap = () => { + for (let i = 0; i < 4; i++) gc(); + return process.memoryUsage().heapUsed; + }; + // Warm the library's own code on another corpus, so its lazily compiled functions are not counted. + for (const { schema, value } of corpus(100, 7)) { + const { check } = candidate.prepare(schema, { draft: "2020-12" }); + for (let i = 0; i < 4; i++) check(value); + } + const empty = heap(); + const items = corpus(COUNT, 42); + const base = heap(); + const prepared = items.map(({ schema }) => candidate.prepare(schema, { draft: "2020-12" })); + const compiled = heap(); + for (let i = 0; i < items.length; i++) for (let k = 0; k < 3; k++) prepared[i].check(items[i].value); + const checked = heap(); + (globalThis as { keep?: unknown }).keep = [items, prepared]; + const mb = (bytes: number) => bytes / 1048576; + console.log( + JSON.stringify({ schemasMB: mb(base - empty), compiledMB: mb(compiled - base), checkedMB: mb(checked - base) }), + ); +} diff --git a/packages/schema/test/bench/setup.ts b/packages/schema/test/bench/setup.ts new file mode 100644 index 000000000..ccf188c2b --- /dev/null +++ b/packages/schema/test/bench/setup.ts @@ -0,0 +1,51 @@ +// Setup cost: compiling a schema the library has not seen, and the time of its first checks. Uses distinct +// schemas from the corpus after warming each library's own code, so it measures a new schema, not a new process. +// Run: node test/bench/setup.ts [--json ] + +import { writeFileSync } from "node:fs"; +import { candidates } from "./candidates.ts"; +import { corpus } from "./corpus.ts"; + +const COUNT = Number(process.env.COUNT ?? 1000); + +/** Mean µs per schema: compile, then each of the first three checks and the mean of checks 4-10. */ +export type SetupResults = Record; + +const results: SetupResults = {}; +for (const candidate of candidates) { + for (const { schema, value } of corpus(200, 7)) { + const { check } = candidate.prepare(schema, { draft: "2020-12" }); + for (let i = 0; i < 10; i++) check(value); + } + const items = corpus(COUNT, 99); + let compile = 0; + const checks = [0, 0, 0, 0]; + for (const { schema, value } of items) { + let start = process.hrtime.bigint(); + const { check } = candidate.prepare(schema, { draft: "2020-12" }); + compile += Number(process.hrtime.bigint() - start); + for (let k = 0; k < 3; k++) { + start = process.hrtime.bigint(); + check(value); + checks[k] += Number(process.hrtime.bigint() - start); + } + start = process.hrtime.bigint(); + for (let k = 3; k < 10; k++) check(value); + checks[3] += Number(process.hrtime.bigint() - start) / 7; + } + const us = (ns: number) => ns / COUNT / 1000; + results[candidate.name] = { + compileUs: us(compile), + checkUs: [us(checks[0]), us(checks[1]), us(checks[2]), us(checks[3])], + }; +} + +console.log("| candidate | compile | check 1 | check 2 | check 3 | checks 4-10 |"); +console.log("|---|---:|---:|---:|---:|---:|"); +for (const [name, r] of Object.entries(results)) { + console.log( + `| ${name} | ${r.compileUs.toFixed(1)} µs | ${r.checkUs.map((x) => `${x.toFixed(2)} µs`).join(" | ")} |`, + ); +} +const out = process.argv.indexOf("--json"); +if (out >= 0) writeFileSync(process.argv[out + 1], JSON.stringify(results, null, "\t")); diff --git a/packages/schema/test/bench/size.ts b/packages/schema/test/bench/size.ts new file mode 100644 index 000000000..b6996af60 --- /dev/null +++ b/packages/schema/test/bench/size.ts @@ -0,0 +1,43 @@ +// Bundle size of what a validator needs from each library: esbuild bundles and minifies the import, then gzip -9. +// Run: node test/bench/size.ts [--json ] + +import { writeFileSync } from "node:fs"; +import { gzipSync } from "node:zlib"; +import { build } from "esbuild"; + +const src = new URL("../../src/", import.meta.url).pathname; + +const entries: Record = { + "pi-schema": `export { compile } from ${JSON.stringify(`${src}index.ts`)};`, + "pi-schema + formats": `export { compile } from ${JSON.stringify(`${src}index.ts`)}; export { formats } from ${JSON.stringify(`${src}formats/index.ts`)};`, + typebox: `export { Compile } from "typebox/compile";`, + cfworker: `export { Validator } from "@cfworker/json-schema";`, + omptype: `export { fromJsonSchema } from "@oh-my-pi/omptype";`, +}; + +/** Minified and gzipped bytes by entry. */ +export type SizeResults = Record; + +const results: SizeResults = {}; +for (const [name, contents] of Object.entries(entries)) { + const output = await build({ + stdin: { contents, resolveDir: src, loader: "ts" }, + bundle: true, + minify: true, + format: "esm", + platform: "neutral", + mainFields: ["module", "main"], + write: false, + logLevel: "silent", + }); + const code = output.outputFiles[0].contents; + results[name] = { minified: code.length, gzip: gzipSync(code, { level: 9 }).length }; +} + +console.log("| library | minified | gzip |"); +console.log("|---|---:|---:|"); +for (const [name, r] of Object.entries(results)) { + console.log(`| ${name} | ${(r.minified / 1024).toFixed(1)} KB | ${(r.gzip / 1024).toFixed(1)} KB |`); +} +const out = process.argv.indexOf("--json"); +if (out >= 0) writeFileSync(process.argv[out + 1], JSON.stringify(results, null, "\t")); diff --git a/packages/schema/test/bench/speed.ts b/packages/schema/test/bench/speed.ts new file mode 100644 index 000000000..6f470951b --- /dev/null +++ b/packages/schema/test/bench/speed.ts @@ -0,0 +1,62 @@ +// Hot validation speed: ns per check for each case and candidate, after warm-up. Each round times every +// candidate once, in a rotated order; the result is the median of the rounds. +// Run: node test/bench/speed.ts [--json ] (add --disallow-code-generation-from-strings for no eval) + +import { writeFileSync } from "node:fs"; +import { candidates } from "./candidates.ts"; +import { cases } from "./cases.ts"; + +const ROUNDS = Number(process.env.ROUNDS ?? 15); +const ROUND_MS = Number(process.env.ROUND_MS ?? 30); +let sink = 0; + +function time(check: (v: unknown) => boolean, value: unknown, n: number): number { + const start = process.hrtime.bigint(); + for (let i = 0; i < n; i++) if (check(value)) sink++; + return Number(process.hrtime.bigint() - start) / n; +} + +function median(values: number[]): number { + return [...values].sort((a, b) => a - b)[values.length >> 1]; +} + +/** Median ns per check, by case name, input ("valid"/"invalid") and candidate name. */ +export type SpeedResults = Record>>; + +const results: SpeedResults = {}; +for (const c of cases) { + results[c.name] = { valid: {}, invalid: {} }; + for (const input of ["valid", "invalid"] as const) { + const value = c[input]; + const runs = candidates.map((candidate) => { + const { check } = candidate.prepare(c.schema, { draft: "2020-12" }); + if (check(value) !== (input === "valid")) throw new Error(`${candidate.name} gets ${c.name} (${input}) wrong`); + let n = 1000; + while (time(check, value, n) * n < ROUND_MS * 1e6 && n < 1e8) n *= 2; + return { candidate, check, n, samples: [] as number[] }; + }); + for (let round = 0; round < ROUNDS; round++) { + for (let i = 0; i < runs.length; i++) { + const run = runs[(i + round) % runs.length]; + run.samples.push(time(run.check, value, run.n)); + } + } + for (const run of runs) results[c.name][input][run.candidate.name] = median(run.samples); + } +} + +const names = candidates.map((c) => c.name); +console.log(`| case | input | ${names.join(" | ")} |`); +console.log(`|---|---|${names.map(() => "---:").join("|")}|`); +for (const [name, inputs] of Object.entries(results)) { + for (const [input, ns] of Object.entries(inputs)) { + console.log(`| ${name} | ${input} | ${names.map((n) => format(ns[n])).join(" | ")} |`); + } +} +const out = process.argv.indexOf("--json"); +if (out >= 0) writeFileSync(process.argv[out + 1], JSON.stringify(results, null, "\t")); +if (sink < 0) console.log(sink); + +function format(ns: number): string { + return ns >= 1000 ? `${(ns / 1000).toFixed(ns >= 10000 ? 0 : 1)} µs` : `${ns.toFixed(0)} ns`; +} diff --git a/packages/schema/test/builders.test.ts b/packages/schema/test/builders.test.ts new file mode 100644 index 000000000..2d13886c7 --- /dev/null +++ b/packages/schema/test/builders.test.ts @@ -0,0 +1,104 @@ +// The builders: the JSON Schema they produce, the static types they carry, and how they compile. + +import { describe, expect, expectTypeOf, it } from "vitest"; +import { + arr, + bool, + bytes, + compile, + int, + lit, + map, + nul, + nullable, + num, + obj, + oneOf, + opt, + type Static, + str, + tuple, + union, + unknown, +} from "../src/index.ts"; + +describe("builders", () => { + it("produce plain JSON Schema", () => { + const schema = obj({ + path: str({ description: "File path", minLength: 1 }), + offset: opt(int({ minimum: 1 })), + mode: oneOf(["read", "write"]), + }); + expect(JSON.parse(JSON.stringify(schema))).toEqual({ + type: "object", + properties: { + path: { type: "string", description: "File path", minLength: 1 }, + offset: { type: "integer", minimum: 1 }, + mode: { enum: ["read", "write"] }, + }, + required: ["path", "mode"], + additionalProperties: false, + }); + }); + + it("produce every leaf", () => { + expect([str(), num(), int(), bool(), nul(), unknown(), lit(3), bytes({ maxLength: 8 })]).toEqual([ + { type: "string" }, + { type: "number" }, + { type: "integer" }, + { type: "boolean" }, + { type: "null" }, + {}, + { const: 3 }, + { type: "bytes", maxLength: 8 }, + ]); + expect(arr(str(), { maxItems: 2 })).toEqual({ type: "array", items: { type: "string" }, maxItems: 2 }); + expect(map(num())).toEqual({ type: "object", additionalProperties: { type: "number" } }); + expect(tuple([str(), num()])).toEqual({ + type: "array", + prefixItems: [{ type: "string" }, { type: "number" }], + items: false, + minItems: 2, + }); + expect(nullable(str())).toEqual({ anyOf: [{ type: "string" }, { type: "null" }] }); + expect(obj({ a: str() }, { open: true })).toEqual({ + type: "object", + properties: { a: { type: "string" } }, + required: ["a"], + }); + }); + + it("carry the static type of valid values", () => { + const message = union([ + obj({ kind: lit("text"), text: str() }), + obj({ kind: lit("image"), url: str(), alt: opt(str()) }), + ]); + expectTypeOf>().toEqualTypeOf< + { kind: "text"; text: string } | { kind: "image"; url: string; alt?: string } + >(); + const pair = tuple([str(), arr(int())]); + expectTypeOf>().toEqualTypeOf<[string, number[]]>(); + expectTypeOf>>>().toEqualTypeOf(); + expectTypeOf>().toEqualTypeOf(); + }); + + it("narrow values through a compiled check", () => { + const point = obj({ x: num(), y: num(), label: opt(str()) }); + const value: unknown = { x: 1, y: 2 }; + const { check } = compile(point); + if (!check(value)) throw new Error("expected a point"); + expectTypeOf(value).toEqualTypeOf<{ x: number; y: number; label?: string }>(); + expect(value.x + value.y).toBe(3); + }); + + it("compile into checks with the declared semantics", () => { + const validator = compile(obj({ name: str({ minLength: 1 }), tags: opt(arr(str(), { uniqueItems: true })) })); + expect(validator.check({ name: "a" })).toBe(true); + expect(validator.check({ name: "a", tags: ["x", "y"] })).toBe(true); + expect(validator.errors({ name: "", tags: ["x", "x"], other: 1 })).toEqual([ + { path: "/name", keyword: "minLength", message: "expected at least 1 characters" }, + { path: "/tags", keyword: "uniqueItems", message: "expected unique items" }, + { path: "/other", keyword: "additionalProperties", message: "unknown field" }, + ]); + }); +}); diff --git a/packages/schema/test/conformance.test.ts b/packages/schema/test/conformance.test.ts new file mode 100644 index 000000000..5702220c5 --- /dev/null +++ b/packages/schema/test/conformance.test.ts @@ -0,0 +1,51 @@ +// The official JSON-Schema-Test-Suite, drafts 2020-12 and 7: every test, including optional and format tests. +// A test listed in KNOWN_FAILURES must fail; when one starts passing, remove it from the list. + +import { describe, expect, it } from "vitest"; +import { formats } from "../src/formats/index.ts"; +import { compile } from "../src/index.ts"; +import { suiteFiles, suiteSchemas } from "./suite.ts"; + +const KNOWN_FAILURES = new Set([ + // Vocabularies switched off by a custom meta-schema are not supported. + "draft2020-12/vocabulary > schema that uses custom metaschema with with no validation vocabulary > no validation: invalid number, but it still validates", + "draft2020-12/optional/format-assertion > schema that uses custom metaschema with format-assertion: false > format-assertion: false: invalid string", + "draft2020-12/optional/format-assertion > schema that uses custom metaschema with format-assertion: true > format-assertion: true: invalid string", + // A $ref to a document of another draft is evaluated with the compile's draft. + "draft2020-12/optional/cross-draft > refs to historic drafts are processed as historic drafts > first item not a string is valid", + "draft7/optional/cross-draft > refs to future drafts are processed as future drafts > missing bar is invalid", + // contentMediaType and contentEncoding are annotations. + "draft7/optional/content > validation of string-encoded content based on media type > an invalid JSON document", + "draft7/optional/content > validation of binary string-encoding > an invalid base64 string (% is not a valid character)", + "draft7/optional/content > validation of binary-encoded media type documents > a validly-encoded invalid JSON document", + "draft7/optional/content > validation of binary-encoded media type documents > an invalid base64 string that is valid JSON", +]); + +const schemas = suiteSchemas(); + +for (const file of suiteFiles()) { + // Format tests assert formats; everywhere else `format` is an annotation, as draft 2020-12 specifies. + const options = { + draft: file.draft, + schemas, + formats: file.name.includes("/optional/format/") ? formats : undefined, + }; + describe(file.name, () => { + for (const group of file.groups) { + describe(group.description, () => { + for (const test of group.tests) { + const known = KNOWN_FAILURES.has(`${file.name} > ${group.description} > ${test.description}`); + (known ? it.fails : it)(test.description, () => { + const validator = compile(group.schema, options); + expect(validator.check(test.data)).toBe(test.valid); + }); + it(`${test.description} (errors agree with check)`, () => { + const validator = compile(group.schema, options); + const errors = validator.errors(test.data); + expect(errors.length === 0).toBe(validator.check(test.data)); + }); + } + }); + } + }); +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/LICENSE b/packages/schema/test/fixtures/json-schema-test-suite/LICENSE new file mode 100644 index 000000000..c28adbadd --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/LICENSE @@ -0,0 +1,19 @@ +Copyright (c) 2012 Julian Berman + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. diff --git a/packages/schema/test/fixtures/json-schema-test-suite/README.md b/packages/schema/test/fixtures/json-schema-test-suite/README.md new file mode 100644 index 000000000..324ff0a0c --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/README.md @@ -0,0 +1,9 @@ +# JSON-Schema-Test-Suite (subset) + +From https://github.com/json-schema-org/JSON-Schema-Test-Suite at commit +`3f05891cac5d51ffa068d7076d1372130e00c829`, unmodified, under its MIT license (`LICENSE`). + +Only what the drafts this package supports use: + +- `tests/draft2020-12/` and `tests/draft7/`, complete (core, `optional/`, `optional/format/`); +- `remotes/`: the documents those tests reference (served as `http://localhost:1234/...` in the tests). diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/baseUriChange/folderInteger.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/baseUriChange/folderInteger.json new file mode 100644 index 000000000..8b50ea308 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/baseUriChange/folderInteger.json @@ -0,0 +1,3 @@ +{ + "type": "integer" +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/baseUriChangeFolder/folderInteger.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/baseUriChangeFolder/folderInteger.json new file mode 100644 index 000000000..8b50ea308 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/baseUriChangeFolder/folderInteger.json @@ -0,0 +1,3 @@ +{ + "type": "integer" +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/baseUriChangeFolderInSubschema/folderInteger.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/baseUriChangeFolderInSubschema/folderInteger.json new file mode 100644 index 000000000..8b50ea308 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/baseUriChangeFolderInSubschema/folderInteger.json @@ -0,0 +1,3 @@ +{ + "type": "integer" +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2019-09/dependentRequired.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2019-09/dependentRequired.json new file mode 100644 index 000000000..0d691d964 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2019-09/dependentRequired.json @@ -0,0 +1,7 @@ +{ + "$id": "http://localhost:1234/draft2019-09/dependentRequired.json", + "$schema": "https://json-schema.org/draft/2019-09/schema", + "dependentRequired": { + "foo": ["bar"] + } +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2019-09/ignore-prefixItems.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2019-09/ignore-prefixItems.json new file mode 100644 index 000000000..b5ef3928c --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2019-09/ignore-prefixItems.json @@ -0,0 +1,7 @@ +{ + "$id": "http://localhost:1234/draft2019-09/ignore-prefixItems.json", + "$schema": "https://json-schema.org/draft/2019-09/schema", + "prefixItems": [ + {"type": "string"} + ] +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/baseUriChange/folderInteger.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/baseUriChange/folderInteger.json new file mode 100644 index 000000000..1f44a6313 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/baseUriChange/folderInteger.json @@ -0,0 +1,4 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "integer" +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/baseUriChangeFolder/folderInteger.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/baseUriChangeFolder/folderInteger.json new file mode 100644 index 000000000..1f44a6313 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/baseUriChangeFolder/folderInteger.json @@ -0,0 +1,4 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "integer" +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/baseUriChangeFolderInSubschema/folderInteger.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/baseUriChangeFolderInSubschema/folderInteger.json new file mode 100644 index 000000000..1f44a6313 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/baseUriChangeFolderInSubschema/folderInteger.json @@ -0,0 +1,4 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "integer" +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/detached-dynamicref.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/detached-dynamicref.json new file mode 100644 index 000000000..07cce1dac --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/detached-dynamicref.json @@ -0,0 +1,13 @@ +{ + "$id": "http://localhost:1234/draft2020-12/detached-dynamicref.json", + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "foo": { + "$dynamicRef": "#detached" + }, + "detached": { + "$dynamicAnchor": "detached", + "type": "integer" + } + } +} \ No newline at end of file diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/detached-ref.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/detached-ref.json new file mode 100644 index 000000000..9c2dca93c --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/detached-ref.json @@ -0,0 +1,13 @@ +{ + "$id": "http://localhost:1234/draft2020-12/detached-ref.json", + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "foo": { + "$ref": "#detached" + }, + "detached": { + "$anchor": "detached", + "type": "integer" + } + } +} \ No newline at end of file diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/different-id-ref-string.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/different-id-ref-string.json new file mode 100644 index 000000000..5d2644d3f --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/different-id-ref-string.json @@ -0,0 +1,5 @@ +{ + "$id": "http://localhost:1234/draft2020-12/real-id-ref-string.json", + "$defs": {"bar": {"type": "string"}}, + "$ref": "#/$defs/bar" +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/extendible-dynamic-ref.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/extendible-dynamic-ref.json new file mode 100644 index 000000000..65bc0c217 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/extendible-dynamic-ref.json @@ -0,0 +1,21 @@ +{ + "description": "extendible array", + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://localhost:1234/draft2020-12/extendible-dynamic-ref.json", + "type": "object", + "properties": { + "elements": { + "type": "array", + "items": { + "$dynamicRef": "#elements" + } + } + }, + "required": ["elements"], + "additionalProperties": false, + "$defs": { + "elements": { + "$dynamicAnchor": "elements" + } + } +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/format-assertion-false.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/format-assertion-false.json new file mode 100644 index 000000000..43a711c9d --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/format-assertion-false.json @@ -0,0 +1,13 @@ +{ + "$id": "http://localhost:1234/draft2020-12/format-assertion-false.json", + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$vocabulary": { + "https://json-schema.org/draft/2020-12/vocab/core": true, + "https://json-schema.org/draft/2020-12/vocab/format-assertion": false + }, + "$dynamicAnchor": "meta", + "allOf": [ + { "$ref": "https://json-schema.org/draft/2020-12/meta/core" }, + { "$ref": "https://json-schema.org/draft/2020-12/meta/format-assertion" } + ] +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/format-assertion-true.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/format-assertion-true.json new file mode 100644 index 000000000..39c6b0abf --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/format-assertion-true.json @@ -0,0 +1,13 @@ +{ + "$id": "http://localhost:1234/draft2020-12/format-assertion-true.json", + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$vocabulary": { + "https://json-schema.org/draft/2020-12/vocab/core": true, + "https://json-schema.org/draft/2020-12/vocab/format-assertion": true + }, + "$dynamicAnchor": "meta", + "allOf": [ + { "$ref": "https://json-schema.org/draft/2020-12/meta/core" }, + { "$ref": "https://json-schema.org/draft/2020-12/meta/format-assertion" } + ] +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/integer.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/integer.json new file mode 100644 index 000000000..1f44a6313 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/integer.json @@ -0,0 +1,4 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "integer" +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/locationIndependentIdentifier.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/locationIndependentIdentifier.json new file mode 100644 index 000000000..6565a1ee0 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/locationIndependentIdentifier.json @@ -0,0 +1,12 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "refToInteger": { + "$ref": "#foo" + }, + "A": { + "$anchor": "foo", + "type": "integer" + } + } +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/metaschema-no-validation.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/metaschema-no-validation.json new file mode 100644 index 000000000..71be8b5da --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/metaschema-no-validation.json @@ -0,0 +1,13 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://localhost:1234/draft2020-12/metaschema-no-validation.json", + "$vocabulary": { + "https://json-schema.org/draft/2020-12/vocab/applicator": true, + "https://json-schema.org/draft/2020-12/vocab/core": true + }, + "$dynamicAnchor": "meta", + "allOf": [ + { "$ref": "https://json-schema.org/draft/2020-12/meta/applicator" }, + { "$ref": "https://json-schema.org/draft/2020-12/meta/core" } + ] +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/metaschema-optional-vocabulary.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/metaschema-optional-vocabulary.json new file mode 100644 index 000000000..a6963e548 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/metaschema-optional-vocabulary.json @@ -0,0 +1,14 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://localhost:1234/draft2020-12/metaschema-optional-vocabulary.json", + "$vocabulary": { + "https://json-schema.org/draft/2020-12/vocab/validation": true, + "https://json-schema.org/draft/2020-12/vocab/core": true, + "http://localhost:1234/draft/2020-12/vocab/custom": false + }, + "$dynamicAnchor": "meta", + "allOf": [ + { "$ref": "https://json-schema.org/draft/2020-12/meta/validation" }, + { "$ref": "https://json-schema.org/draft/2020-12/meta/core" } + ] +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/name-defs.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/name-defs.json new file mode 100644 index 000000000..67bc33c51 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/name-defs.json @@ -0,0 +1,16 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "orNull": { + "anyOf": [ + { + "type": "null" + }, + { + "$ref": "#" + } + ] + } + }, + "type": "string" +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/nested-absolute-ref-to-string.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/nested-absolute-ref-to-string.json new file mode 100644 index 000000000..3d51ded5c --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/nested-absolute-ref-to-string.json @@ -0,0 +1,9 @@ +{ + "$defs": { + "bar": { + "$id": "http://localhost:1234/draft2020-12/the-nested-id.json", + "type": "string" + } + }, + "$ref": "http://localhost:1234/draft2020-12/the-nested-id.json" +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/nested/foo-ref-string.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/nested/foo-ref-string.json new file mode 100644 index 000000000..29661ff9f --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/nested/foo-ref-string.json @@ -0,0 +1,7 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "foo": {"$ref": "string.json"} + } +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/nested/string.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/nested/string.json new file mode 100644 index 000000000..6607ac534 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/nested/string.json @@ -0,0 +1,4 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "string" +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/prefixItems.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/prefixItems.json new file mode 100644 index 000000000..acd8293c6 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/prefixItems.json @@ -0,0 +1,7 @@ +{ + "$id": "http://localhost:1234/draft2020-12/prefixItems.json", + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [ + {"type": "string"} + ] +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/ref-and-defs.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/ref-and-defs.json new file mode 100644 index 000000000..16d30fa3a --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/ref-and-defs.json @@ -0,0 +1,12 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://localhost:1234/draft2020-12/ref-and-defs.json", + "$defs": { + "inner": { + "properties": { + "bar": { "type": "string" } + } + } + }, + "$ref": "#/$defs/inner" +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/subSchemas.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/subSchemas.json new file mode 100644 index 000000000..1bb4846d7 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/subSchemas.json @@ -0,0 +1,11 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "integer": { + "type": "integer" + }, + "refToInteger": { + "$ref": "#/$defs/integer" + } + } +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/tree.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/tree.json new file mode 100644 index 000000000..b07555fb3 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/tree.json @@ -0,0 +1,17 @@ +{ + "description": "tree schema, extensible", + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://localhost:1234/draft2020-12/tree.json", + "$dynamicAnchor": "node", + + "type": "object", + "properties": { + "data": true, + "children": { + "type": "array", + "items": { + "$dynamicRef": "#node" + } + } + } +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/urn-ref-string.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/urn-ref-string.json new file mode 100644 index 000000000..796f37021 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft2020-12/urn-ref-string.json @@ -0,0 +1,5 @@ +{ + "$id": "urn:uuid:feebdaed-ffff-0000-2020-1200deadbeef", + "$defs": {"bar": {"type": "string"}}, + "$ref": "#/$defs/bar" +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/detached-ref.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/detached-ref.json new file mode 100644 index 000000000..27f2ec80a --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/detached-ref.json @@ -0,0 +1,13 @@ +{ + "$id": "http://localhost:1234/draft7/detached-ref.json", + "$schema": "http://json-schema.org/draft-07/schema#", + "definitions": { + "foo": { + "$ref": "#detached" + }, + "detached": { + "$id": "#detached", + "type": "integer" + } + } +} \ No newline at end of file diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/ignore-dependentRequired.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/ignore-dependentRequired.json new file mode 100644 index 000000000..13fd0df68 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/ignore-dependentRequired.json @@ -0,0 +1,7 @@ +{ + "$id": "http://localhost:1234/draft7/ignore-dependentRequired.json", + "$schema": "http://json-schema.org/draft-07/schema#", + "dependentRequired": { + "foo": ["bar"] + } +} \ No newline at end of file diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/locationIndependentIdentifier.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/locationIndependentIdentifier.json new file mode 100644 index 000000000..e72815cd5 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/locationIndependentIdentifier.json @@ -0,0 +1,11 @@ +{ + "definitions": { + "refToInteger": { + "$ref": "#foo" + }, + "A": { + "$id": "#foo", + "type": "integer" + } + } +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/name.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/name.json new file mode 100644 index 000000000..fceacb809 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/name.json @@ -0,0 +1,15 @@ +{ + "definitions": { + "orNull": { + "anyOf": [ + { + "type": "null" + }, + { + "$ref": "#" + } + ] + } + }, + "type": "string" +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/ref-and-definitions.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/ref-and-definitions.json new file mode 100644 index 000000000..d5929380c --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/ref-and-definitions.json @@ -0,0 +1,11 @@ +{ + "$id": "http://localhost:1234/draft7/ref-and-definitions.json", + "definitions": { + "inner": { + "properties": { + "bar": { "type": "string" } + } + } + }, + "allOf": [ { "$ref": "#/definitions/inner" } ] +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/subSchemas.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/subSchemas.json new file mode 100644 index 000000000..6e9b3de35 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/draft7/subSchemas.json @@ -0,0 +1,10 @@ +{ + "definitions": { + "integer": { + "type": "integer" + }, + "refToInteger": { + "$ref": "#/definitions/integer" + } + } +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/integer.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/integer.json new file mode 100644 index 000000000..8b50ea308 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/integer.json @@ -0,0 +1,3 @@ +{ + "type": "integer" +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/nested/foo-ref-string.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/nested/foo-ref-string.json new file mode 100644 index 000000000..9cd2527ed --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/nested/foo-ref-string.json @@ -0,0 +1,6 @@ +{ + "type": "object", + "properties": { + "foo": {"$ref": "string.json"} + } +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/remotes/nested/string.json b/packages/schema/test/fixtures/json-schema-test-suite/remotes/nested/string.json new file mode 100644 index 000000000..c2d48c068 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/remotes/nested/string.json @@ -0,0 +1,3 @@ +{ + "type": "string" +} diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/additionalProperties.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/additionalProperties.json new file mode 100644 index 000000000..9618575e2 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/additionalProperties.json @@ -0,0 +1,219 @@ +[ + { + "description": + "additionalProperties being false does not allow other properties", + "specification": [ { "core":"10.3.2.3", "quote": "The value of \"additionalProperties\" MUST be a valid JSON Schema. Boolean \"false\" forbids everything." } ], + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": {"foo": {}, "bar": {}}, + "patternProperties": { "^v": {} }, + "additionalProperties": false + }, + "tests": [ + { + "description": "no additional properties is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "an additional property is invalid", + "data": {"foo" : 1, "bar" : 2, "quux" : "boom"}, + "valid": false + }, + { + "description": "ignores arrays", + "data": [1, 2, 3], + "valid": true + }, + { + "description": "ignores strings", + "data": "foobarbaz", + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + }, + { + "description": "patternProperties are not additional properties", + "data": {"foo":1, "vroom": 2}, + "valid": true + } + ] + }, + { + "description": "non-ASCII pattern with additionalProperties", + "specification": [ { "core":"10.3.2.3"} ], + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "patternProperties": {"^á": {}}, + "additionalProperties": false + }, + "tests": [ + { + "description": "matching the pattern is valid", + "data": {"ármányos": 2}, + "valid": true + }, + { + "description": "not matching the pattern is invalid", + "data": {"élmény": 2}, + "valid": false + } + ] + }, + { + "description": "additionalProperties with schema", + "specification": [ { "core":"10.3.2.3", "quote": "The value of \"additionalProperties\" MUST be a valid JSON Schema." } ], + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": {"foo": {}, "bar": {}}, + "additionalProperties": {"type": "boolean"} + }, + "tests": [ + { + "description": "no additional properties is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "an additional valid property is valid", + "data": {"foo" : 1, "bar" : 2, "quux" : true}, + "valid": true + }, + { + "description": "an additional invalid property is invalid", + "data": {"foo" : 1, "bar" : 2, "quux" : 12}, + "valid": false + } + ] + }, + { + "description": "additionalProperties can exist by itself", + "specification": [ { "core":"10.3.2.3", "quote": "With no other applicator applying to object instances. This validates all the instance values irrespective of their property names" } ], + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "additionalProperties": {"type": "boolean"} + }, + "tests": [ + { + "description": "an additional valid property is valid", + "data": {"foo" : true}, + "valid": true + }, + { + "description": "an additional invalid property is invalid", + "data": {"foo" : 1}, + "valid": false + } + ] + }, + { + "description": "additionalProperties are allowed by default", + "specification": [ { "core":"10.3.2.3", "quote": "Omitting this keyword has the same assertion behavior as an empty schema." } ], + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": {"foo": {}, "bar": {}} + }, + "tests": [ + { + "description": "additional properties are allowed", + "data": {"foo": 1, "bar": 2, "quux": true}, + "valid": true + } + ] + }, + { + "description": "additionalProperties does not look in applicators", + "specification":[ { "core": "10.2", "quote": "Subschemas of applicator keywords evaluate the instance completely independently such that the results of one such subschema MUST NOT impact the results of sibling subschemas." } ], + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + {"properties": {"foo": {}}} + ], + "additionalProperties": {"type": "boolean"} + }, + "tests": [ + { + "description": "properties defined in allOf are not examined", + "data": {"foo": 1, "bar": true}, + "valid": false + } + ] + }, + { + "description": "additionalProperties with null valued instance properties", + "specification": [ { "core":"10.3.2.3" } ], + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "additionalProperties": { + "type": "null" + } + }, + "tests": [ + { + "description": "allows null values", + "data": {"foo": null}, + "valid": true + } + ] + }, + { + "description": "additionalProperties with propertyNames", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "propertyNames": { + "maxLength": 5 + }, + "additionalProperties": { + "type": "number" + } + }, + "tests": [ + { + "description": "Valid against both keywords", + "data": { "apple": 4 }, + "valid": true + }, + { + "description": "Valid against propertyNames, but not additionalProperties", + "data": { "fig": 2, "pear": "available" }, + "valid": false + } + ] + }, + { + "description": "dependentSchemas with additionalProperties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": {"foo2": {}}, + "dependentSchemas": { + "foo" : {}, + "foo2": { + "properties": { + "bar": {} + } + } + }, + "additionalProperties": false + }, + "tests": [ + { + "description": "additionalProperties doesn't consider dependentSchemas", + "data": {"foo": ""}, + "valid": false + }, + { + "description": "additionalProperties can't see bar", + "data": {"bar": ""}, + "valid": false + }, + { + "description": "additionalProperties can't see bar even when foo2 is present", + "data": {"foo2": "", "bar": ""}, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/allOf.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/allOf.json new file mode 100644 index 000000000..9e87903fe --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/allOf.json @@ -0,0 +1,312 @@ +[ + { + "description": "allOf", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + { + "properties": { + "bar": {"type": "integer"} + }, + "required": ["bar"] + }, + { + "properties": { + "foo": {"type": "string"} + }, + "required": ["foo"] + } + ] + }, + "tests": [ + { + "description": "allOf", + "data": {"foo": "baz", "bar": 2}, + "valid": true + }, + { + "description": "mismatch second", + "data": {"foo": "baz"}, + "valid": false + }, + { + "description": "mismatch first", + "data": {"bar": 2}, + "valid": false + }, + { + "description": "wrong type", + "data": {"foo": "baz", "bar": "quux"}, + "valid": false + } + ] + }, + { + "description": "allOf with base schema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": {"bar": {"type": "integer"}}, + "required": ["bar"], + "allOf" : [ + { + "properties": { + "foo": {"type": "string"} + }, + "required": ["foo"] + }, + { + "properties": { + "baz": {"type": "null"} + }, + "required": ["baz"] + } + ] + }, + "tests": [ + { + "description": "valid", + "data": {"foo": "quux", "bar": 2, "baz": null}, + "valid": true + }, + { + "description": "mismatch base schema", + "data": {"foo": "quux", "baz": null}, + "valid": false + }, + { + "description": "mismatch first allOf", + "data": {"bar": 2, "baz": null}, + "valid": false + }, + { + "description": "mismatch second allOf", + "data": {"foo": "quux", "bar": 2}, + "valid": false + }, + { + "description": "mismatch both", + "data": {"bar": 2}, + "valid": false + } + ] + }, + { + "description": "allOf simple types", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + {"maximum": 30}, + {"minimum": 20} + ] + }, + "tests": [ + { + "description": "valid", + "data": 25, + "valid": true + }, + { + "description": "mismatch one", + "data": 35, + "valid": false + } + ] + }, + { + "description": "allOf with boolean schemas, all true", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [true, true] + }, + "tests": [ + { + "description": "any value is valid", + "data": "foo", + "valid": true + } + ] + }, + { + "description": "allOf with boolean schemas, some false", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [true, false] + }, + "tests": [ + { + "description": "any value is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "allOf with boolean schemas, all false", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [false, false] + }, + "tests": [ + { + "description": "any value is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "allOf with one empty schema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + {} + ] + }, + "tests": [ + { + "description": "any data is valid", + "data": 1, + "valid": true + } + ] + }, + { + "description": "allOf with two empty schemas", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + {}, + {} + ] + }, + "tests": [ + { + "description": "any data is valid", + "data": 1, + "valid": true + } + ] + }, + { + "description": "allOf with the first empty schema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + {}, + { "type": "number" } + ] + }, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "string is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "allOf with the last empty schema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + { "type": "number" }, + {} + ] + }, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "string is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "nested allOf, to check validation semantics", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + { + "allOf": [ + { + "type": "null" + } + ] + } + ] + }, + "tests": [ + { + "description": "null is valid", + "data": null, + "valid": true + }, + { + "description": "anything non-null is invalid", + "data": 123, + "valid": false + } + ] + }, + { + "description": "allOf combined with anyOf, oneOf", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ { "multipleOf": 2 } ], + "anyOf": [ { "multipleOf": 3 } ], + "oneOf": [ { "multipleOf": 5 } ] + }, + "tests": [ + { + "description": "allOf: false, anyOf: false, oneOf: false", + "data": 1, + "valid": false + }, + { + "description": "allOf: false, anyOf: false, oneOf: true", + "data": 5, + "valid": false + }, + { + "description": "allOf: false, anyOf: true, oneOf: false", + "data": 3, + "valid": false + }, + { + "description": "allOf: false, anyOf: true, oneOf: true", + "data": 15, + "valid": false + }, + { + "description": "allOf: true, anyOf: false, oneOf: false", + "data": 2, + "valid": false + }, + { + "description": "allOf: true, anyOf: false, oneOf: true", + "data": 10, + "valid": false + }, + { + "description": "allOf: true, anyOf: true, oneOf: false", + "data": 6, + "valid": false + }, + { + "description": "allOf: true, anyOf: true, oneOf: true", + "data": 30, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/anchor.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/anchor.json new file mode 100644 index 000000000..99143fa11 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/anchor.json @@ -0,0 +1,120 @@ +[ + { + "description": "Location-independent identifier", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "#foo", + "$defs": { + "A": { + "$anchor": "foo", + "type": "integer" + } + } + }, + "tests": [ + { + "data": 1, + "description": "match", + "valid": true + }, + { + "data": "a", + "description": "mismatch", + "valid": false + } + ] + }, + { + "description": "Location-independent identifier with absolute URI", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "http://localhost:1234/draft2020-12/bar#foo", + "$defs": { + "A": { + "$id": "http://localhost:1234/draft2020-12/bar", + "$anchor": "foo", + "type": "integer" + } + } + }, + "tests": [ + { + "data": 1, + "description": "match", + "valid": true + }, + { + "data": "a", + "description": "mismatch", + "valid": false + } + ] + }, + { + "description": "Location-independent identifier with base URI change in subschema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://localhost:1234/draft2020-12/root", + "$ref": "http://localhost:1234/draft2020-12/nested.json#foo", + "$defs": { + "A": { + "$id": "nested.json", + "$defs": { + "B": { + "$anchor": "foo", + "type": "integer" + } + } + } + } + }, + "tests": [ + { + "data": 1, + "description": "match", + "valid": true + }, + { + "data": "a", + "description": "mismatch", + "valid": false + } + ] + }, + { + "description": "same $anchor with different base uri", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://localhost:1234/draft2020-12/foobar", + "$defs": { + "A": { + "$id": "child1", + "allOf": [ + { + "$id": "child2", + "$anchor": "my_anchor", + "type": "number" + }, + { + "$anchor": "my_anchor", + "type": "string" + } + ] + } + }, + "$ref": "child1#my_anchor" + }, + "tests": [ + { + "description": "$ref resolves to /$defs/A/allOf/1", + "data": "a", + "valid": true + }, + { + "description": "$ref does not resolve to /$defs/A/allOf/0", + "data": 1, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/anyOf.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/anyOf.json new file mode 100644 index 000000000..89b192dbd --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/anyOf.json @@ -0,0 +1,203 @@ +[ + { + "description": "anyOf", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "anyOf": [ + { + "type": "integer" + }, + { + "minimum": 2 + } + ] + }, + "tests": [ + { + "description": "first anyOf valid", + "data": 1, + "valid": true + }, + { + "description": "second anyOf valid", + "data": 2.5, + "valid": true + }, + { + "description": "both anyOf valid", + "data": 3, + "valid": true + }, + { + "description": "neither anyOf valid", + "data": 1.5, + "valid": false + } + ] + }, + { + "description": "anyOf with base schema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "string", + "anyOf" : [ + { + "maxLength": 2 + }, + { + "minLength": 4 + } + ] + }, + "tests": [ + { + "description": "mismatch base schema", + "data": 3, + "valid": false + }, + { + "description": "one anyOf valid", + "data": "foobar", + "valid": true + }, + { + "description": "both anyOf invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "anyOf with boolean schemas, all true", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "anyOf": [true, true] + }, + "tests": [ + { + "description": "any value is valid", + "data": "foo", + "valid": true + } + ] + }, + { + "description": "anyOf with boolean schemas, some true", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "anyOf": [true, false] + }, + "tests": [ + { + "description": "any value is valid", + "data": "foo", + "valid": true + } + ] + }, + { + "description": "anyOf with boolean schemas, all false", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "anyOf": [false, false] + }, + "tests": [ + { + "description": "any value is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "anyOf complex types", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "anyOf": [ + { + "properties": { + "bar": {"type": "integer"} + }, + "required": ["bar"] + }, + { + "properties": { + "foo": {"type": "string"} + }, + "required": ["foo"] + } + ] + }, + "tests": [ + { + "description": "first anyOf valid (complex)", + "data": {"bar": 2}, + "valid": true + }, + { + "description": "second anyOf valid (complex)", + "data": {"foo": "baz"}, + "valid": true + }, + { + "description": "both anyOf valid (complex)", + "data": {"foo": "baz", "bar": 2}, + "valid": true + }, + { + "description": "neither anyOf valid (complex)", + "data": {"foo": 2, "bar": "quux"}, + "valid": false + } + ] + }, + { + "description": "anyOf with one empty schema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "anyOf": [ + { "type": "number" }, + {} + ] + }, + "tests": [ + { + "description": "string is valid", + "data": "foo", + "valid": true + }, + { + "description": "number is valid", + "data": 123, + "valid": true + } + ] + }, + { + "description": "nested anyOf, to check validation semantics", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "anyOf": [ + { + "anyOf": [ + { + "type": "null" + } + ] + } + ] + }, + "tests": [ + { + "description": "null is valid", + "data": null, + "valid": true + }, + { + "description": "anything non-null is invalid", + "data": 123, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/boolean_schema.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/boolean_schema.json new file mode 100644 index 000000000..6d40f23f2 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/boolean_schema.json @@ -0,0 +1,104 @@ +[ + { + "description": "boolean schema 'true'", + "schema": true, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "string is valid", + "data": "foo", + "valid": true + }, + { + "description": "boolean true is valid", + "data": true, + "valid": true + }, + { + "description": "boolean false is valid", + "data": false, + "valid": true + }, + { + "description": "null is valid", + "data": null, + "valid": true + }, + { + "description": "object is valid", + "data": {"foo": "bar"}, + "valid": true + }, + { + "description": "empty object is valid", + "data": {}, + "valid": true + }, + { + "description": "array is valid", + "data": ["foo"], + "valid": true + }, + { + "description": "empty array is valid", + "data": [], + "valid": true + } + ] + }, + { + "description": "boolean schema 'false'", + "schema": false, + "tests": [ + { + "description": "number is invalid", + "data": 1, + "valid": false + }, + { + "description": "string is invalid", + "data": "foo", + "valid": false + }, + { + "description": "boolean true is invalid", + "data": true, + "valid": false + }, + { + "description": "boolean false is invalid", + "data": false, + "valid": false + }, + { + "description": "null is invalid", + "data": null, + "valid": false + }, + { + "description": "object is invalid", + "data": {"foo": "bar"}, + "valid": false + }, + { + "description": "empty object is invalid", + "data": {}, + "valid": false + }, + { + "description": "array is invalid", + "data": ["foo"], + "valid": false + }, + { + "description": "empty array is invalid", + "data": [], + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/const.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/const.json new file mode 100644 index 000000000..72ea3f042 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/const.json @@ -0,0 +1,431 @@ +[ + { + "description": "const validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "const": 2 + }, + "tests": [ + { + "description": "same value is valid", + "data": 2, + "valid": true + }, + { + "description": "another value is invalid", + "data": 5, + "valid": false + }, + { + "description": "another type is invalid", + "data": "a", + "valid": false + } + ] + }, + { + "description": "const with object", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "const": {"foo": "bar", "baz": "bax"} + }, + "tests": [ + { + "description": "same object is valid", + "data": {"foo": "bar", "baz": "bax"}, + "valid": true + }, + { + "description": "same object with different property order is valid", + "data": {"baz": "bax", "foo": "bar"}, + "valid": true + }, + { + "description": "another object is invalid", + "data": {"foo": "bar"}, + "valid": false + }, + { + "description": "another type is invalid", + "data": [1, 2], + "valid": false + } + ] + }, + { + "description": "const with array", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "const": [{ "foo": "bar" }] + }, + "tests": [ + { + "description": "same array is valid", + "data": [{"foo": "bar"}], + "valid": true + }, + { + "description": "another array item is invalid", + "data": [2], + "valid": false + }, + { + "description": "array with additional items is invalid", + "data": [1, 2, 3], + "valid": false + } + ] + }, + { + "description": "const with null", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "const": null + }, + "tests": [ + { + "description": "null is valid", + "data": null, + "valid": true + }, + { + "description": "not null is invalid", + "data": 0, + "valid": false + } + ] + }, + { + "description": "const with false does not match 0", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "const": false + }, + "tests": [ + { + "description": "false is valid", + "data": false, + "valid": true + }, + { + "description": "integer zero is invalid", + "data": 0, + "valid": false + }, + { + "description": "float zero is invalid", + "data": 0.0, + "valid": false + } + ] + }, + { + "description": "const with true does not match 1", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "const": true + }, + "tests": [ + { + "description": "true is valid", + "data": true, + "valid": true + }, + { + "description": "integer one is invalid", + "data": 1, + "valid": false + }, + { + "description": "float one is invalid", + "data": 1.0, + "valid": false + } + ] + }, + { + "description": "const with [false] does not match [0]", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "const": [false] + }, + "tests": [ + { + "description": "[false] is valid", + "data": [false], + "valid": true + }, + { + "description": "[0] is invalid", + "data": [0], + "valid": false + }, + { + "description": "[0.0] is invalid", + "data": [0.0], + "valid": false + } + ] + }, + { + "description": "const with [true] does not match [1]", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "const": [true] + }, + "tests": [ + { + "description": "[true] is valid", + "data": [true], + "valid": true + }, + { + "description": "[1] is invalid", + "data": [1], + "valid": false + }, + { + "description": "[1.0] is invalid", + "data": [1.0], + "valid": false + } + ] + }, + { + "description": "const with {\"a\": false} does not match {\"a\": 0}", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "const": {"a": false} + }, + "tests": [ + { + "description": "{\"a\": false} is valid", + "data": {"a": false}, + "valid": true + }, + { + "description": "{\"a\": 0} is invalid", + "data": {"a": 0}, + "valid": false + }, + { + "description": "{\"a\": 0.0} is invalid", + "data": {"a": 0.0}, + "valid": false + } + ] + }, + { + "description": "const with {\"a\": true} does not match {\"a\": 1}", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "const": {"a": true} + }, + "tests": [ + { + "description": "{\"a\": true} is valid", + "data": {"a": true}, + "valid": true + }, + { + "description": "{\"a\": 1} is invalid", + "data": {"a": 1}, + "valid": false + }, + { + "description": "{\"a\": 1.0} is invalid", + "data": {"a": 1.0}, + "valid": false + } + ] + }, + { + "description": "const with 0 does not match other zero-like types", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "const": 0 + }, + "tests": [ + { + "description": "false is invalid", + "data": false, + "valid": false + }, + { + "description": "integer zero is valid", + "data": 0, + "valid": true + }, + { + "description": "float zero is valid", + "data": 0.0, + "valid": true + }, + { + "description": "empty object is invalid", + "data": {}, + "valid": false + }, + { + "description": "empty array is invalid", + "data": [], + "valid": false + }, + { + "description": "empty string is invalid", + "data": "", + "valid": false + } + ] + }, + { + "description": "const with 1 does not match true", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "const": 1 + }, + "tests": [ + { + "description": "true is invalid", + "data": true, + "valid": false + }, + { + "description": "integer one is valid", + "data": 1, + "valid": true + }, + { + "description": "float one is valid", + "data": 1.0, + "valid": true + } + ] + }, + { + "description": "const with -2.0 matches integer and float types", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "const": -2.0 + }, + "tests": [ + { + "description": "integer -2 is valid", + "data": -2, + "valid": true + }, + { + "description": "integer 2 is invalid", + "data": 2, + "valid": false + }, + { + "description": "float -2.0 is valid", + "data": -2.0, + "valid": true + }, + { + "description": "float 2.0 is invalid", + "data": 2.0, + "valid": false + }, + { + "description": "float -2.00001 is invalid", + "data": -2.00001, + "valid": false + } + ] + }, + { + "description": "float and integers are equal up to 64-bit representation limits", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "const": 9007199254740992 + }, + "tests": [ + { + "description": "integer is valid", + "data": 9007199254740992, + "valid": true + }, + { + "description": "integer minus one is invalid", + "data": 9007199254740991, + "valid": false + }, + { + "description": "float is valid", + "data": 9007199254740992.0, + "valid": true + }, + { + "description": "float minus one is invalid", + "data": 9007199254740991.0, + "valid": false + } + ] + }, + { + "description": "nul characters in strings", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "const": "hello\u0000there" + }, + "tests": [ + { + "description": "match string with nul", + "data": "hello\u0000there", + "valid": true + }, + { + "description": "do not match string lacking nul", + "data": "hellothere", + "valid": false + } + ] + }, + { + "description": "characters with the same visual representation but different codepoint", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "const": "μ", + "$comment": "U+03BC" + }, + "tests": [ + { + "description": "character uses the same codepoint", + "data": "μ", + "comment": "U+03BC", + "valid": true + }, + { + "description": "character looks the same but uses a different codepoint", + "data": "µ", + "comment": "U+00B5", + "valid": false + } + ] + }, + { + "description": "characters with the same visual representation, but different number of codepoints", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "const": "ä", + "$comment": "U+00E4" + }, + "tests": [ + { + "description": "character uses the same codepoint", + "data": "ä", + "comment": "U+00E4", + "valid": true + }, + { + "description": "character looks the same but uses combining marks", + "data": "ä", + "comment": "a, U+0308", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/contains.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/contains.json new file mode 100644 index 000000000..08a00a753 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/contains.json @@ -0,0 +1,176 @@ +[ + { + "description": "contains keyword validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contains": {"minimum": 5} + }, + "tests": [ + { + "description": "array with item matching schema (5) is valid", + "data": [3, 4, 5], + "valid": true + }, + { + "description": "array with item matching schema (6) is valid", + "data": [3, 4, 6], + "valid": true + }, + { + "description": "array with two items matching schema (5, 6) is valid", + "data": [3, 4, 5, 6], + "valid": true + }, + { + "description": "array without items matching schema is invalid", + "data": [2, 3, 4], + "valid": false + }, + { + "description": "empty array is invalid", + "data": [], + "valid": false + }, + { + "description": "not array is valid", + "data": {}, + "valid": true + } + ] + }, + { + "description": "contains keyword with const keyword", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contains": { "const": 5 } + }, + "tests": [ + { + "description": "array with item 5 is valid", + "data": [3, 4, 5], + "valid": true + }, + { + "description": "array with two items 5 is valid", + "data": [3, 4, 5, 5], + "valid": true + }, + { + "description": "array without item 5 is invalid", + "data": [1, 2, 3, 4], + "valid": false + } + ] + }, + { + "description": "contains keyword with boolean schema true", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contains": true + }, + "tests": [ + { + "description": "any non-empty array is valid", + "data": ["foo"], + "valid": true + }, + { + "description": "empty array is invalid", + "data": [], + "valid": false + } + ] + }, + { + "description": "contains keyword with boolean schema false", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contains": false + }, + "tests": [ + { + "description": "any non-empty array is invalid", + "data": ["foo"], + "valid": false + }, + { + "description": "empty array is invalid", + "data": [], + "valid": false + }, + { + "description": "non-arrays are valid", + "data": "contains does not apply to strings", + "valid": true + } + ] + }, + { + "description": "items + contains", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "items": { "multipleOf": 2 }, + "contains": { "multipleOf": 3 } + }, + "tests": [ + { + "description": "matches items, does not match contains", + "data": [ 2, 4, 8 ], + "valid": false + }, + { + "description": "does not match items, matches contains", + "data": [ 3, 6, 9 ], + "valid": false + }, + { + "description": "matches both items and contains", + "data": [ 6, 12 ], + "valid": true + }, + { + "description": "matches neither items nor contains", + "data": [ 1, 5 ], + "valid": false + } + ] + }, + { + "description": "contains with false if subschema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contains": { + "if": false, + "else": true + } + }, + "tests": [ + { + "description": "any non-empty array is valid", + "data": ["foo"], + "valid": true + }, + { + "description": "empty array is invalid", + "data": [], + "valid": false + } + ] + }, + { + "description": "contains with null instance elements", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contains": { + "type": "null" + } + }, + "tests": [ + { + "description": "allows null items", + "data": [ null ], + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/content.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/content.json new file mode 100644 index 000000000..698f78057 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/content.json @@ -0,0 +1,131 @@ +[ + { + "description": "validation of string-encoded content based on media type", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contentMediaType": "application/json" + }, + "tests": [ + { + "description": "a valid JSON document", + "data": "{\"foo\": \"bar\"}", + "valid": true + }, + { + "description": "an invalid JSON document; validates true", + "data": "{:}", + "valid": true + }, + { + "description": "ignores non-strings", + "data": 100, + "valid": true + } + ] + }, + { + "description": "validation of binary string-encoding", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contentEncoding": "base64" + }, + "tests": [ + { + "description": "a valid base64 string", + "data": "eyJmb28iOiAiYmFyIn0K", + "valid": true + }, + { + "description": "an invalid base64 string (% is not a valid character); validates true", + "data": "eyJmb28iOi%iYmFyIn0K", + "valid": true + }, + { + "description": "ignores non-strings", + "data": 100, + "valid": true + } + ] + }, + { + "description": "validation of binary-encoded media type documents", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contentMediaType": "application/json", + "contentEncoding": "base64" + }, + "tests": [ + { + "description": "a valid base64-encoded JSON document", + "data": "eyJmb28iOiAiYmFyIn0K", + "valid": true + }, + { + "description": "a validly-encoded invalid JSON document; validates true", + "data": "ezp9Cg==", + "valid": true + }, + { + "description": "an invalid base64 string that is valid JSON; validates true", + "data": "{}", + "valid": true + }, + { + "description": "ignores non-strings", + "data": 100, + "valid": true + } + ] + }, + { + "description": "validation of binary-encoded media type documents with schema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contentMediaType": "application/json", + "contentEncoding": "base64", + "contentSchema": { "type": "object", "required": ["foo"], "properties": { "foo": { "type": "string" } } } + }, + "tests": [ + { + "description": "a valid base64-encoded JSON document", + "data": "eyJmb28iOiAiYmFyIn0K", + "valid": true + }, + { + "description": "another valid base64-encoded JSON document", + "data": "eyJib28iOiAyMCwgImZvbyI6ICJiYXoifQ==", + "valid": true + }, + { + "description": "an invalid base64-encoded JSON document; validates true", + "data": "eyJib28iOiAyMH0=", + "valid": true + }, + { + "description": "an empty object as a base64-encoded JSON document; validates true", + "data": "e30=", + "valid": true + }, + { + "description": "an empty array as a base64-encoded JSON document", + "data": "W10=", + "valid": true + }, + { + "description": "a validly-encoded invalid JSON document; validates true", + "data": "ezp9Cg==", + "valid": true + }, + { + "description": "an invalid base64 string that is valid JSON; validates true", + "data": "{}", + "valid": true + }, + { + "description": "ignores non-strings", + "data": 100, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/default.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/default.json new file mode 100644 index 000000000..ceb3ae271 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/default.json @@ -0,0 +1,82 @@ +[ + { + "description": "invalid type for default", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { + "type": "integer", + "default": [] + } + } + }, + "tests": [ + { + "description": "valid when property is specified", + "data": {"foo": 13}, + "valid": true + }, + { + "description": "still valid when the invalid default is used", + "data": {}, + "valid": true + } + ] + }, + { + "description": "invalid string value for default", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "bar": { + "type": "string", + "minLength": 4, + "default": "bad" + } + } + }, + "tests": [ + { + "description": "valid when property is specified", + "data": {"bar": "good"}, + "valid": true + }, + { + "description": "still valid when the invalid default is used", + "data": {}, + "valid": true + } + ] + }, + { + "description": "the default keyword does not do anything if the property is missing", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "alpha": { + "type": "number", + "maximum": 3, + "default": 5 + } + } + }, + "tests": [ + { + "description": "an explicit property value is checked against maximum (passing)", + "data": { "alpha": 1 }, + "valid": true + }, + { + "description": "an explicit property value is checked against maximum (failing)", + "data": { "alpha": 5 }, + "valid": false + }, + { + "description": "missing properties are not filled in with the default", + "data": {}, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/defs.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/defs.json new file mode 100644 index 000000000..da2a503bf --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/defs.json @@ -0,0 +1,21 @@ +[ + { + "description": "validate definition against metaschema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "https://json-schema.org/draft/2020-12/schema" + }, + "tests": [ + { + "description": "valid definition schema", + "data": {"$defs": {"foo": {"type": "integer"}}}, + "valid": true + }, + { + "description": "invalid definition schema", + "data": {"$defs": {"foo": {"type": 1}}}, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/dependentRequired.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/dependentRequired.json new file mode 100644 index 000000000..2baa38e9f --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/dependentRequired.json @@ -0,0 +1,152 @@ +[ + { + "description": "single dependency", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "dependentRequired": {"bar": ["foo"]} + }, + "tests": [ + { + "description": "neither", + "data": {}, + "valid": true + }, + { + "description": "nondependant", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "with dependency", + "data": {"foo": 1, "bar": 2}, + "valid": true + }, + { + "description": "missing dependency", + "data": {"bar": 2}, + "valid": false + }, + { + "description": "ignores arrays", + "data": ["bar"], + "valid": true + }, + { + "description": "ignores strings", + "data": "foobar", + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + } + ] + }, + { + "description": "empty dependents", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "dependentRequired": {"bar": []} + }, + "tests": [ + { + "description": "empty object", + "data": {}, + "valid": true + }, + { + "description": "object with one property", + "data": {"bar": 2}, + "valid": true + }, + { + "description": "non-object is valid", + "data": 1, + "valid": true + } + ] + }, + { + "description": "multiple dependents required", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "dependentRequired": {"quux": ["foo", "bar"]} + }, + "tests": [ + { + "description": "neither", + "data": {}, + "valid": true + }, + { + "description": "nondependants", + "data": {"foo": 1, "bar": 2}, + "valid": true + }, + { + "description": "with dependencies", + "data": {"foo": 1, "bar": 2, "quux": 3}, + "valid": true + }, + { + "description": "missing dependency", + "data": {"foo": 1, "quux": 2}, + "valid": false + }, + { + "description": "missing other dependency", + "data": {"bar": 1, "quux": 2}, + "valid": false + }, + { + "description": "missing both dependencies", + "data": {"quux": 1}, + "valid": false + } + ] + }, + { + "description": "dependencies with escaped characters", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "dependentRequired": { + "foo\nbar": ["foo\rbar"], + "foo\"bar": ["foo'bar"] + } + }, + "tests": [ + { + "description": "CRLF", + "data": { + "foo\nbar": 1, + "foo\rbar": 2 + }, + "valid": true + }, + { + "description": "quoted quotes", + "data": { + "foo'bar": 1, + "foo\"bar": 2 + }, + "valid": true + }, + { + "description": "CRLF missing dependent", + "data": { + "foo\nbar": 1, + "foo": 2 + }, + "valid": false + }, + { + "description": "quoted quotes missing dependent", + "data": { + "foo\"bar": 2 + }, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/dependentSchemas.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/dependentSchemas.json new file mode 100644 index 000000000..1c5f0574a --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/dependentSchemas.json @@ -0,0 +1,171 @@ +[ + { + "description": "single dependency", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "dependentSchemas": { + "bar": { + "properties": { + "foo": {"type": "integer"}, + "bar": {"type": "integer"} + } + } + } + }, + "tests": [ + { + "description": "valid", + "data": {"foo": 1, "bar": 2}, + "valid": true + }, + { + "description": "no dependency", + "data": {"foo": "quux"}, + "valid": true + }, + { + "description": "wrong type", + "data": {"foo": "quux", "bar": 2}, + "valid": false + }, + { + "description": "wrong type other", + "data": {"foo": 2, "bar": "quux"}, + "valid": false + }, + { + "description": "wrong type both", + "data": {"foo": "quux", "bar": "quux"}, + "valid": false + }, + { + "description": "ignores arrays", + "data": ["bar"], + "valid": true + }, + { + "description": "ignores strings", + "data": "foobar", + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + } + ] + }, + { + "description": "boolean subschemas", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "dependentSchemas": { + "foo": true, + "bar": false + } + }, + "tests": [ + { + "description": "object with property having schema true is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "object with property having schema false is invalid", + "data": {"bar": 2}, + "valid": false + }, + { + "description": "object with both properties is invalid", + "data": {"foo": 1, "bar": 2}, + "valid": false + }, + { + "description": "empty object is valid", + "data": {}, + "valid": true + } + ] + }, + { + "description": "dependencies with escaped characters", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "dependentSchemas": { + "foo\tbar": {"minProperties": 4}, + "foo'bar": {"required": ["foo\"bar"]} + } + }, + "tests": [ + { + "description": "quoted tab", + "data": { + "foo\tbar": 1, + "a": 2, + "b": 3, + "c": 4 + }, + "valid": true + }, + { + "description": "quoted quote", + "data": { + "foo'bar": {"foo\"bar": 1} + }, + "valid": false + }, + { + "description": "quoted tab invalid under dependent schema", + "data": { + "foo\tbar": 1, + "a": 2 + }, + "valid": false + }, + { + "description": "quoted quote invalid under dependent schema", + "data": {"foo'bar": 1}, + "valid": false + } + ] + }, + { + "description": "dependent subschema incompatible with root", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": {} + }, + "dependentSchemas": { + "foo": { + "properties": { + "bar": {} + }, + "additionalProperties": false + } + } + }, + "tests": [ + { + "description": "matches root", + "data": {"foo": 1}, + "valid": false + }, + { + "description": "matches dependency", + "data": {"bar": 1}, + "valid": true + }, + { + "description": "matches both", + "data": {"foo": 1, "bar": 2}, + "valid": false + }, + { + "description": "no dependency", + "data": {"baz": 1}, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/dynamicRef.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/dynamicRef.json new file mode 100644 index 000000000..eba4b441b --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/dynamicRef.json @@ -0,0 +1,874 @@ +[ + { + "description": "A $dynamicRef to a $dynamicAnchor in the same schema resource behaves like a normal $ref to an $anchor", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://test.json-schema.org/dynamicRef-dynamicAnchor-same-schema/root", + "type": "array", + "items": { "$dynamicRef": "#items" }, + "$defs": { + "foo": { + "$dynamicAnchor": "items", + "type": "string" + } + } + }, + "tests": [ + { + "description": "An array of strings is valid", + "data": ["foo", "bar"], + "valid": true + }, + { + "description": "An array containing non-strings is invalid", + "data": ["foo", 42], + "valid": false + } + ] + }, + { + "description": "A $dynamicRef to an $anchor in the same schema resource behaves like a normal $ref to an $anchor", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://test.json-schema.org/dynamicRef-anchor-same-schema/root", + "type": "array", + "items": { "$dynamicRef": "#items" }, + "$defs": { + "foo": { + "$anchor": "items", + "type": "string" + } + } + }, + "tests": [ + { + "description": "An array of strings is valid", + "data": ["foo", "bar"], + "valid": true + }, + { + "description": "An array containing non-strings is invalid", + "data": ["foo", 42], + "valid": false + } + ] + }, + { + "description": "A $ref to a $dynamicAnchor in the same schema resource behaves like a normal $ref to an $anchor", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://test.json-schema.org/ref-dynamicAnchor-same-schema/root", + "type": "array", + "items": { "$ref": "#items" }, + "$defs": { + "foo": { + "$dynamicAnchor": "items", + "type": "string" + } + } + }, + "tests": [ + { + "description": "An array of strings is valid", + "data": ["foo", "bar"], + "valid": true + }, + { + "description": "An array containing non-strings is invalid", + "data": ["foo", 42], + "valid": false + } + ] + }, + { + "description": "A $dynamicRef resolves to the first $dynamicAnchor still in scope that is encountered when the schema is evaluated", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://test.json-schema.org/typical-dynamic-resolution/root", + "$ref": "list", + "$defs": { + "foo": { + "$dynamicAnchor": "items", + "type": "string" + }, + "list": { + "$id": "list", + "type": "array", + "items": { "$dynamicRef": "#items" }, + "$defs": { + "items": { + "$comment": "This is only needed to satisfy the bookending requirement", + "$dynamicAnchor": "items" + } + } + } + } + }, + "tests": [ + { + "description": "An array of strings is valid", + "data": ["foo", "bar"], + "valid": true + }, + { + "description": "An array containing non-strings is invalid", + "data": ["foo", 42], + "valid": false + } + ] + }, + { + "description": "A $dynamicRef without anchor in fragment behaves identical to $ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://test.json-schema.org/dynamicRef-without-anchor/root", + "$ref": "list", + "$defs": { + "foo": { + "$dynamicAnchor": "items", + "type": "string" + }, + "list": { + "$id": "list", + "type": "array", + "items": { "$dynamicRef": "#/$defs/items" }, + "$defs": { + "items": { + "$comment": "This is only needed to satisfy the bookending requirement", + "$dynamicAnchor": "items", + "type": "number" + } + } + } + } + }, + "tests": [ + { + "description": "An array of strings is invalid", + "data": ["foo", "bar"], + "valid": false + }, + { + "description": "An array of numbers is valid", + "data": [24, 42], + "valid": true + } + ] + }, + { + "description": "A $dynamicRef with intermediate scopes that don't include a matching $dynamicAnchor does not affect dynamic scope resolution", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://test.json-schema.org/dynamic-resolution-with-intermediate-scopes/root", + "$ref": "intermediate-scope", + "$defs": { + "foo": { + "$dynamicAnchor": "items", + "type": "string" + }, + "intermediate-scope": { + "$id": "intermediate-scope", + "$ref": "list" + }, + "list": { + "$id": "list", + "type": "array", + "items": { "$dynamicRef": "#items" }, + "$defs": { + "items": { + "$comment": "This is only needed to satisfy the bookending requirement", + "$dynamicAnchor": "items" + } + } + } + } + }, + "tests": [ + { + "description": "An array of strings is valid", + "data": ["foo", "bar"], + "valid": true + }, + { + "description": "An array containing non-strings is invalid", + "data": ["foo", 42], + "valid": false + } + ] + }, + { + "description": "An $anchor with the same name as a $dynamicAnchor is not used for dynamic scope resolution", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://test.json-schema.org/dynamic-resolution-ignores-anchors/root", + "$ref": "list", + "$defs": { + "foo": { + "$anchor": "items", + "type": "string" + }, + "list": { + "$id": "list", + "type": "array", + "items": { "$dynamicRef": "#items" }, + "$defs": { + "items": { + "$comment": "This is only needed to satisfy the bookending requirement", + "$dynamicAnchor": "items" + } + } + } + } + }, + "tests": [ + { + "description": "Any array is valid", + "data": ["foo", 42], + "valid": true + } + ] + }, + { + "description": "A $dynamicRef without a matching $dynamicAnchor in the same schema resource behaves like a normal $ref to $anchor", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://test.json-schema.org/dynamic-resolution-without-bookend/root", + "$ref": "list", + "$defs": { + "foo": { + "$dynamicAnchor": "items", + "type": "string" + }, + "list": { + "$id": "list", + "type": "array", + "items": { "$dynamicRef": "#items" }, + "$defs": { + "items": { + "$comment": "This is only needed to give the reference somewhere to resolve to when it behaves like $ref", + "$anchor": "items" + } + } + } + } + }, + "tests": [ + { + "description": "Any array is valid", + "data": ["foo", 42], + "valid": true + } + ] + }, + { + "description": "A $dynamicRef with a non-matching $dynamicAnchor in the same schema resource behaves like a normal $ref to $anchor", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://test.json-schema.org/unmatched-dynamic-anchor/root", + "$ref": "list", + "$defs": { + "foo": { + "$dynamicAnchor": "items", + "type": "string" + }, + "list": { + "$id": "list", + "type": "array", + "items": { "$dynamicRef": "#items" }, + "$defs": { + "items": { + "$comment": "This is only needed to give the reference somewhere to resolve to when it behaves like $ref", + "$anchor": "items", + "$dynamicAnchor": "foo" + } + } + } + } + }, + "tests": [ + { + "description": "Any array is valid", + "data": ["foo", 42], + "valid": true + } + ] + }, + { + "description": "A $dynamicRef that initially resolves to a schema with a matching $dynamicAnchor resolves to the first $dynamicAnchor in the dynamic scope", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://test.json-schema.org/relative-dynamic-reference/root", + "$dynamicAnchor": "meta", + "type": "object", + "properties": { + "foo": { "const": "pass" } + }, + "$ref": "extended", + "$defs": { + "extended": { + "$id": "extended", + "$dynamicAnchor": "meta", + "type": "object", + "properties": { + "bar": { "$ref": "bar" } + } + }, + "bar": { + "$id": "bar", + "type": "object", + "properties": { + "baz": { "$dynamicRef": "extended#meta" } + } + } + } + }, + "tests": [ + { + "description": "The recursive part is valid against the root", + "data": { + "foo": "pass", + "bar": { + "baz": { "foo": "pass" } + } + }, + "valid": true + }, + { + "description": "The recursive part is not valid against the root", + "data": { + "foo": "pass", + "bar": { + "baz": { "foo": "fail" } + } + }, + "valid": false + } + ] + }, + { + "description": "A $dynamicRef that initially resolves to a schema without a matching $dynamicAnchor behaves like a normal $ref to $anchor", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://test.json-schema.org/relative-dynamic-reference-without-bookend/root", + "$dynamicAnchor": "meta", + "type": "object", + "properties": { + "foo": { "const": "pass" } + }, + "$ref": "extended", + "$defs": { + "extended": { + "$id": "extended", + "$anchor": "meta", + "type": "object", + "properties": { + "bar": { "$ref": "bar" } + } + }, + "bar": { + "$id": "bar", + "type": "object", + "properties": { + "baz": { "$dynamicRef": "extended#meta" } + } + } + } + }, + "tests": [ + { + "description": "The recursive part doesn't need to validate against the root", + "data": { + "foo": "pass", + "bar": { + "baz": { "foo": "fail" } + } + }, + "valid": true + } + ] + }, + { + "description": "multiple dynamic paths to the $dynamicRef keyword", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://test.json-schema.org/dynamic-ref-with-multiple-paths/main", + "if": { + "properties": { + "kindOfList": { "const": "numbers" } + }, + "required": ["kindOfList"] + }, + "then": { "$ref": "numberList" }, + "else": { "$ref": "stringList" }, + + "$defs": { + "genericList": { + "$id": "genericList", + "properties": { + "list": { + "items": { "$dynamicRef": "#itemType" } + } + }, + "$defs": { + "defaultItemType": { + "$comment": "Only needed to satisfy bookending requirement", + "$dynamicAnchor": "itemType" + } + } + }, + "numberList": { + "$id": "numberList", + "$defs": { + "itemType": { + "$dynamicAnchor": "itemType", + "type": "number" + } + }, + "$ref": "genericList" + }, + "stringList": { + "$id": "stringList", + "$defs": { + "itemType": { + "$dynamicAnchor": "itemType", + "type": "string" + } + }, + "$ref": "genericList" + } + } + }, + "tests": [ + { + "description": "number list with number values", + "data": { + "kindOfList": "numbers", + "list": [1.1] + }, + "valid": true + }, + { + "description": "number list with string values", + "data": { + "kindOfList": "numbers", + "list": ["foo"] + }, + "valid": false + }, + { + "description": "string list with number values", + "data": { + "kindOfList": "strings", + "list": [1.1] + }, + "valid": false + }, + { + "description": "string list with string values", + "data": { + "kindOfList": "strings", + "list": ["foo"] + }, + "valid": true + } + ] + }, + { + "description": "after leaving a dynamic scope, it is not used by a $dynamicRef", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://test.json-schema.org/dynamic-ref-leaving-dynamic-scope/main", + "if": { + "$id": "first_scope", + "$defs": { + "thingy": { + "$comment": "this is first_scope#thingy", + "$dynamicAnchor": "thingy", + "type": "number" + } + } + }, + "then": { + "$id": "second_scope", + "$ref": "start", + "$defs": { + "thingy": { + "$comment": "this is second_scope#thingy, the final destination of the $dynamicRef", + "$dynamicAnchor": "thingy", + "type": "null" + } + } + }, + "$defs": { + "start": { + "$comment": "this is the landing spot from $ref", + "$id": "start", + "$dynamicRef": "inner_scope#thingy" + }, + "thingy": { + "$comment": "this is the first stop for the $dynamicRef", + "$id": "inner_scope", + "$dynamicAnchor": "thingy", + "type": "string" + } + } + }, + "tests": [ + { + "description": "string matches /$defs/thingy, but the $dynamicRef does not stop here", + "data": "a string", + "valid": false + }, + { + "description": "first_scope is not in dynamic scope for the $dynamicRef", + "data": 42, + "valid": false + }, + { + "description": "/then/$defs/thingy is the final stop for the $dynamicRef", + "data": null, + "valid": true + } + ] + }, + { + "description": "strict-tree schema, guards against misspelled properties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://localhost:1234/draft2020-12/strict-tree.json", + "$dynamicAnchor": "node", + + "$ref": "tree.json", + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "instance with misspelled field", + "data": { + "children": [{ + "daat": 1 + }] + }, + "valid": false + }, + { + "description": "instance with correct field", + "data": { + "children": [{ + "data": 1 + }] + }, + "valid": true + } + ] + }, + { + "description": "tests for implementation dynamic anchor and reference link", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://localhost:1234/draft2020-12/strict-extendible.json", + "$ref": "extendible-dynamic-ref.json", + "$defs": { + "elements": { + "$dynamicAnchor": "elements", + "properties": { + "a": true + }, + "required": ["a"], + "additionalProperties": false + } + } + }, + "tests": [ + { + "description": "incorrect parent schema", + "data": { + "a": true + }, + "valid": false + }, + { + "description": "incorrect extended schema", + "data": { + "elements": [ + { "b": 1 } + ] + }, + "valid": false + }, + { + "description": "correct extended schema", + "data": { + "elements": [ + { "a": 1 } + ] + }, + "valid": true + } + ] + }, + { + "description": "$ref and $dynamicAnchor are independent of order - $defs first", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://localhost:1234/draft2020-12/strict-extendible-allof-defs-first.json", + "allOf": [ + { + "$ref": "extendible-dynamic-ref.json" + }, + { + "$defs": { + "elements": { + "$dynamicAnchor": "elements", + "properties": { + "a": true + }, + "required": ["a"], + "additionalProperties": false + } + } + } + ] + }, + "tests": [ + { + "description": "incorrect parent schema", + "data": { + "a": true + }, + "valid": false + }, + { + "description": "incorrect extended schema", + "data": { + "elements": [ + { "b": 1 } + ] + }, + "valid": false + }, + { + "description": "correct extended schema", + "data": { + "elements": [ + { "a": 1 } + ] + }, + "valid": true + } + ] + }, + { + "description": "$ref and $dynamicAnchor are independent of order - $ref first", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://localhost:1234/draft2020-12/strict-extendible-allof-ref-first.json", + "allOf": [ + { + "$defs": { + "elements": { + "$dynamicAnchor": "elements", + "properties": { + "a": true + }, + "required": ["a"], + "additionalProperties": false + } + } + }, + { + "$ref": "extendible-dynamic-ref.json" + } + ] + }, + "tests": [ + { + "description": "incorrect parent schema", + "data": { + "a": true + }, + "valid": false + }, + { + "description": "incorrect extended schema", + "data": { + "elements": [ + { "b": 1 } + ] + }, + "valid": false + }, + { + "description": "correct extended schema", + "data": { + "elements": [ + { "a": 1 } + ] + }, + "valid": true + } + ] + }, + { + "description": "$ref to $dynamicRef finds detached $dynamicAnchor", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "http://localhost:1234/draft2020-12/detached-dynamicref.json#/$defs/foo" + }, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "non-number is invalid", + "data": "a", + "valid": false + } + ] + }, + { + "description": "$dynamicRef points to a boolean schema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "true": true, + "false": false + }, + "properties": { + "true": { + "$dynamicRef": "#/$defs/true" + }, + "false": { + "$dynamicRef": "#/$defs/false" + } + } + }, + "tests": [ + { + "description": "follow $dynamicRef to a true schema", + "data": { "true": 1 }, + "valid": true + }, + { + "description": "follow $dynamicRef to a false schema", + "data": { "false": 1 }, + "valid": false + } + ] + }, + { + "description": "$dynamicRef skips over intermediate resources - direct reference", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://test.json-schema.org/dynamic-ref-skips-intermediate-resource/main", + "type": "object", + "properties": { + "bar-item": { + "$ref": "item" + } + }, + "$defs": { + "bar": { + "$id": "bar", + "type": "array", + "items": { + "$ref": "item" + }, + "$defs": { + "item": { + "$id": "item", + "type": "object", + "properties": { + "content": { + "$dynamicRef": "#content" + } + }, + "$defs": { + "defaultContent": { + "$dynamicAnchor": "content", + "type": "integer" + } + } + }, + "content": { + "$dynamicAnchor": "content", + "type": "string" + } + } + } + } + }, + "tests": [ + { + "description": "integer property passes", + "data": { "bar-item": { "content": 42 } }, + "valid": true + }, + { + "description": "string property fails", + "data": { "bar-item": { "content": "value" } }, + "valid": false + } + ] + }, + { + "description": "$dynamicRef avoids the root of each schema, but scopes are still registered", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://test.json-schema.org/dynamic-ref-avoids-root-of-each-schema/base", + "$ref": "first#/$defs/stuff", + "$defs": { + "first": { + "$id": "first", + "$defs": { + "stuff": { + "$ref": "second#/$defs/stuff" + }, + "length": { + "$comment": "unused, because there is no $dynamicAnchor here", + "maxLength": 1 + } + } + }, + "second": { + "$id": "second", + "$defs": { + "stuff": { + "$ref": "third#/$defs/stuff" + }, + "length": { + "$dynamicAnchor": "length", + "maxLength": 2 + } + } + }, + "third": { + "$id": "third", + "$defs": { + "stuff": { + "$dynamicRef": "#length" + }, + "length": { + "$dynamicAnchor": "length", + "maxLength": 3 + } + } + } + } + }, + "tests": [ + { + "description": "data is sufficient for schema at second#/$defs/length", + "data": "hi", + "valid": true + }, + { + "description": "data is not sufficient for schema at second#/$defs/length", + "data": "hey", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/enum.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/enum.json new file mode 100644 index 000000000..dbc5e0d31 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/enum.json @@ -0,0 +1,419 @@ +[ + { + "description": "simple enum validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "enum": [1, 2, 3] + }, + "tests": [ + { + "description": "one of the enum is valid", + "data": 1, + "valid": true + }, + { + "description": "something else is invalid", + "data": 4, + "valid": false + } + ] + }, + { + "description": "heterogeneous enum validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "enum": [6, "foo", [], true, {"foo": 12}] + }, + "tests": [ + { + "description": "one of the enum is valid", + "data": [], + "valid": true + }, + { + "description": "something else is invalid", + "data": null, + "valid": false + }, + { + "description": "objects are deep compared", + "data": {"foo": false}, + "valid": false + }, + { + "description": "valid object matches", + "data": {"foo": 12}, + "valid": true + }, + { + "description": "extra properties in object is invalid", + "data": {"foo": 12, "boo": 42}, + "valid": false + } + ] + }, + { + "description": "heterogeneous enum-with-null validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "enum": [6, null] + }, + "tests": [ + { + "description": "null is valid", + "data": null, + "valid": true + }, + { + "description": "number is valid", + "data": 6, + "valid": true + }, + { + "description": "something else is invalid", + "data": "test", + "valid": false + } + ] + }, + { + "description": "enums in properties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type":"object", + "properties": { + "foo": {"enum":["foo"]}, + "bar": {"enum":["bar"]} + }, + "required": ["bar"] + }, + "tests": [ + { + "description": "both properties are valid", + "data": {"foo":"foo", "bar":"bar"}, + "valid": true + }, + { + "description": "wrong foo value", + "data": {"foo":"foot", "bar":"bar"}, + "valid": false + }, + { + "description": "wrong bar value", + "data": {"foo":"foo", "bar":"bart"}, + "valid": false + }, + { + "description": "missing optional property is valid", + "data": {"bar":"bar"}, + "valid": true + }, + { + "description": "missing required property is invalid", + "data": {"foo":"foo"}, + "valid": false + }, + { + "description": "missing all properties is invalid", + "data": {}, + "valid": false + } + ] + }, + { + "description": "enum with escaped characters", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "enum": ["foo\nbar", "foo\rbar"] + }, + "tests": [ + { + "description": "member 1 is valid", + "data": "foo\nbar", + "valid": true + }, + { + "description": "member 2 is valid", + "data": "foo\rbar", + "valid": true + }, + { + "description": "another string is invalid", + "data": "abc", + "valid": false + } + ] + }, + { + "description": "enum with false does not match 0", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "enum": [false] + }, + "tests": [ + { + "description": "false is valid", + "data": false, + "valid": true + }, + { + "description": "integer zero is invalid", + "data": 0, + "valid": false + }, + { + "description": "float zero is invalid", + "data": 0.0, + "valid": false + } + ] + }, + { + "description": "enum with [false] does not match [0]", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "enum": [[false]] + }, + "tests": [ + { + "description": "[false] is valid", + "data": [false], + "valid": true + }, + { + "description": "[0] is invalid", + "data": [0], + "valid": false + }, + { + "description": "[0.0] is invalid", + "data": [0.0], + "valid": false + } + ] + }, + { + "description": "enum with true does not match 1", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "enum": [true] + }, + "tests": [ + { + "description": "true is valid", + "data": true, + "valid": true + }, + { + "description": "integer one is invalid", + "data": 1, + "valid": false + }, + { + "description": "float one is invalid", + "data": 1.0, + "valid": false + } + ] + }, + { + "description": "enum with [true] does not match [1]", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "enum": [[true]] + }, + "tests": [ + { + "description": "[true] is valid", + "data": [true], + "valid": true + }, + { + "description": "[1] is invalid", + "data": [1], + "valid": false + }, + { + "description": "[1.0] is invalid", + "data": [1.0], + "valid": false + } + ] + }, + { + "description": "enum with 0 does not match false", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "enum": [0] + }, + "tests": [ + { + "description": "false is invalid", + "data": false, + "valid": false + }, + { + "description": "integer zero is valid", + "data": 0, + "valid": true + }, + { + "description": "float zero is valid", + "data": 0.0, + "valid": true + } + ] + }, + { + "description": "enum with [0] does not match [false]", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "enum": [[0]] + }, + "tests": [ + { + "description": "[false] is invalid", + "data": [false], + "valid": false + }, + { + "description": "[0] is valid", + "data": [0], + "valid": true + }, + { + "description": "[0.0] is valid", + "data": [0.0], + "valid": true + } + ] + }, + { + "description": "enum with 1 does not match true", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "enum": [1] + }, + "tests": [ + { + "description": "true is invalid", + "data": true, + "valid": false + }, + { + "description": "integer one is valid", + "data": 1, + "valid": true + }, + { + "description": "float one is valid", + "data": 1.0, + "valid": true + } + ] + }, + { + "description": "enum with [1] does not match [true]", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "enum": [[1]] + }, + "tests": [ + { + "description": "[true] is invalid", + "data": [true], + "valid": false + }, + { + "description": "[1] is valid", + "data": [1], + "valid": true + }, + { + "description": "[1.0] is valid", + "data": [1.0], + "valid": true + } + ] + }, + { + "description": "nul characters in strings", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "enum": [ "hello\u0000there" ] + }, + "tests": [ + { + "description": "match string with nul", + "data": "hello\u0000there", + "valid": true + }, + { + "description": "do not match string lacking nul", + "data": "hellothere", + "valid": false + } + ] + }, + { + "description": "empty enum", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "enum": [] + }, + "tests": [ + { + "description": "string is invalid", + "data": "foo", + "valid": false + }, + { + "description": "number is invalid", + "data": 42, + "valid": false + }, + { + "description": "null is invalid", + "data": null, + "valid": false + }, + { + "description": "object is invalid", + "data": {}, + "valid": false + }, + { + "description": "array is invalid", + "data": [], + "valid": false + }, + { + "description": "boolean is invalid", + "data": false, + "valid": false + } + ] + }, + { + "description": "characters with the same visual representation, but different number of codepoints", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "enum": ["ä"], + "$comment": "U+00E4" + }, + "tests": [ + { + "description": "character uses the same codepoint", + "data": "ä", + "comment": "U+00E4", + "valid": true + }, + { + "description": "character looks the same but uses combining marks", + "data": "ä", + "comment": "a, U+0308", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/exclusiveMaximum.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/exclusiveMaximum.json new file mode 100644 index 000000000..05db23351 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/exclusiveMaximum.json @@ -0,0 +1,31 @@ +[ + { + "description": "exclusiveMaximum validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "exclusiveMaximum": 3.0 + }, + "tests": [ + { + "description": "below the exclusiveMaximum is valid", + "data": 2.2, + "valid": true + }, + { + "description": "boundary point is invalid", + "data": 3.0, + "valid": false + }, + { + "description": "above the exclusiveMaximum is invalid", + "data": 3.5, + "valid": false + }, + { + "description": "ignores non-numbers", + "data": "x", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/exclusiveMinimum.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/exclusiveMinimum.json new file mode 100644 index 000000000..00af9d7ff --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/exclusiveMinimum.json @@ -0,0 +1,31 @@ +[ + { + "description": "exclusiveMinimum validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "exclusiveMinimum": 1.1 + }, + "tests": [ + { + "description": "above the exclusiveMinimum is valid", + "data": 1.2, + "valid": true + }, + { + "description": "boundary point is invalid", + "data": 1.1, + "valid": false + }, + { + "description": "below the exclusiveMinimum is invalid", + "data": 0.6, + "valid": false + }, + { + "description": "ignores non-numbers", + "data": "x", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/format.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/format.json new file mode 100644 index 000000000..01adcbda3 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/format.json @@ -0,0 +1,838 @@ +[ + { + "description": "email format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "email" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid email string is only an annotation by default", + "data": "2962", + "valid": true + } + ] + }, + { + "description": "idn-email format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "idn-email" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid idn-email string is only an annotation by default", + "data": "2962", + "valid": true + } + ] + }, + { + "description": "regex format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "regex" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid regex string is only an annotation by default", + "data": "^(abc]", + "valid": true + } + ] + }, + { + "description": "ipv4 format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "ipv4" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid ipv4 string is only an annotation by default", + "data": "127.0.0.0.1", + "valid": true + } + ] + }, + { + "description": "ipv6 format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "ipv6" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid ipv6 string is only an annotation by default", + "data": "12345::", + "valid": true + } + ] + }, + { + "description": "idn-hostname format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "idn-hostname" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid idn-hostname string is only an annotation by default", + "data": "〮실례.테스트", + "valid": true + } + ] + }, + { + "description": "hostname format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "hostname" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid hostname string is only an annotation by default", + "data": "-a-host-name-that-starts-with--", + "valid": true + } + ] + }, + { + "description": "date format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "date" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid date string is only an annotation by default", + "data": "06/19/1963", + "valid": true + } + ] + }, + { + "description": "date-time format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "date-time" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid date-time string is only an annotation by default", + "data": "1990-02-31T15:59:60.123-08:00", + "valid": true + } + ] + }, + { + "description": "time format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "time" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid time string is only an annotation by default", + "data": "08:30:06 PST", + "valid": true + } + ] + }, + { + "description": "json-pointer format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "json-pointer" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid json-pointer string is only an annotation by default", + "data": "/foo/bar~", + "valid": true + } + ] + }, + { + "description": "relative-json-pointer format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "relative-json-pointer" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid relative-json-pointer string is only an annotation by default", + "data": "/foo/bar", + "valid": true + } + ] + }, + { + "description": "iri format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "iri" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid iri string is only an annotation by default", + "data": "http://2001:0db8:85a3:0000:0000:8a2e:0370:7334", + "valid": true + } + ] + }, + { + "description": "iri-reference format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "iri-reference" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid iri-reference string is only an annotation by default", + "data": "\\\\WINDOWS\\filëßåré", + "valid": true + } + ] + }, + { + "description": "uri format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "uri" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid uri string is only an annotation by default", + "data": "//foo.bar/?baz=qux#quux", + "valid": true + } + ] + }, + { + "description": "uri-reference format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "uri-reference" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid uri-reference string is only an annotation by default", + "data": "\\\\WINDOWS\\fileshare", + "valid": true + } + ] + }, + { + "description": "uri-template format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "uri-template" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid uri-template string is only an annotation by default", + "data": "http://example.com/dictionary/{term:1}/{term", + "valid": true + } + ] + }, + { + "description": "uuid format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "uuid" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid uuid string is only an annotation by default", + "data": "2eb8aa08-aa98-11ea-b4aa-73b441d1638", + "valid": true + } + ] + }, + { + "description": "duration format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "duration" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "invalid duration string is only an annotation by default", + "data": "PT1D", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/if-then-else.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/if-then-else.json new file mode 100644 index 000000000..9abf26f71 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/if-then-else.json @@ -0,0 +1,308 @@ +[ + { + "description": "ignore if without then or else", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "if": { + "const": 0 + } + }, + "tests": [ + { + "description": "valid when valid against lone if", + "data": 0, + "valid": true + }, + { + "description": "valid when invalid against lone if", + "data": "hello", + "valid": true + } + ] + }, + { + "description": "ignore then without if", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "then": { + "const": 0 + } + }, + "tests": [ + { + "description": "valid when valid against lone then", + "data": 0, + "valid": true + }, + { + "description": "valid when invalid against lone then", + "data": "hello", + "valid": true + } + ] + }, + { + "description": "ignore else without if", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "else": { + "const": 0 + } + }, + "tests": [ + { + "description": "valid when valid against lone else", + "data": 0, + "valid": true + }, + { + "description": "valid when invalid against lone else", + "data": "hello", + "valid": true + } + ] + }, + { + "description": "if and then without else", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "if": { + "exclusiveMaximum": 0 + }, + "then": { + "minimum": -10 + } + }, + "tests": [ + { + "description": "valid through then", + "data": -1, + "valid": true + }, + { + "description": "invalid through then", + "data": -100, + "valid": false + }, + { + "description": "valid when if test fails", + "data": 3, + "valid": true + } + ] + }, + { + "description": "if and else without then", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "if": { + "exclusiveMaximum": 0 + }, + "else": { + "multipleOf": 2 + } + }, + "tests": [ + { + "description": "valid when if test passes", + "data": -1, + "valid": true + }, + { + "description": "valid through else", + "data": 4, + "valid": true + }, + { + "description": "invalid through else", + "data": 3, + "valid": false + } + ] + }, + { + "description": "validate against correct branch, then vs else", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "if": { + "exclusiveMaximum": 0 + }, + "then": { + "minimum": -10 + }, + "else": { + "multipleOf": 2 + } + }, + "tests": [ + { + "description": "valid through then", + "data": -1, + "valid": true + }, + { + "description": "invalid through then", + "data": -100, + "valid": false + }, + { + "description": "valid through else", + "data": 4, + "valid": true + }, + { + "description": "invalid through else", + "data": 3, + "valid": false + } + ] + }, + { + "description": "non-interference across combined schemas", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + { + "if": { + "exclusiveMaximum": 0 + } + }, + { + "then": { + "minimum": -10 + } + }, + { + "else": { + "multipleOf": 2 + } + } + ] + }, + "tests": [ + { + "description": "valid, but would have been invalid through then", + "data": -100, + "valid": true + }, + { + "description": "valid, but would have been invalid through else", + "data": 3, + "valid": true + } + ] + }, + { + "description": "if with boolean schema true", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "if": true, + "then": { "const": "then" }, + "else": { "const": "else" } + }, + "tests": [ + { + "description": "boolean schema true in if always chooses the then path (valid)", + "data": "then", + "valid": true + }, + { + "description": "boolean schema true in if always chooses the then path (invalid)", + "data": "else", + "valid": false + } + ] + }, + { + "description": "if with boolean schema false", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "if": false, + "then": { "const": "then" }, + "else": { "const": "else" } + }, + "tests": [ + { + "description": "boolean schema false in if always chooses the else path (invalid)", + "data": "then", + "valid": false + }, + { + "description": "boolean schema false in if always chooses the else path (valid)", + "data": "else", + "valid": true + } + ] + }, + { + "description": "if appears at the end when serialized (keyword processing sequence)", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "then": { "const": "yes" }, + "else": { "const": "other" }, + "if": { "maxLength": 4 } + }, + "tests": [ + { + "description": "yes redirects to then and passes", + "data": "yes", + "valid": true + }, + { + "description": "other redirects to else and passes", + "data": "other", + "valid": true + }, + { + "description": "no redirects to then and fails", + "data": "no", + "valid": false + }, + { + "description": "invalid redirects to else and fails", + "data": "invalid", + "valid": false + } + ] + }, + { + "description": "then: false fails when condition matches", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "if": { "const": 1 }, + "then": false + }, + "tests": [ + { + "description": "matches if → then=false → invalid", + "data": 1, + "valid": false + }, + { + "description": "does not match if → then ignored → valid", + "data": 2, + "valid": true + } + ] + }, + { + "description": "else: false fails when condition does not match", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "if": { "const": 1 }, + "else": false + }, + "tests": [ + { + "description": "matches if → else ignored → valid", + "data": 1, + "valid": true + }, + { + "description": "does not match if → else executes → invalid", + "data": 2, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/infinite-loop-detection.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/infinite-loop-detection.json new file mode 100644 index 000000000..46f157a35 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/infinite-loop-detection.json @@ -0,0 +1,37 @@ +[ + { + "description": "evaluating the same schema location against the same data location twice is not a sign of an infinite loop", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "int": { "type": "integer" } + }, + "allOf": [ + { + "properties": { + "foo": { + "$ref": "#/$defs/int" + } + } + }, + { + "additionalProperties": { + "$ref": "#/$defs/int" + } + } + ] + }, + "tests": [ + { + "description": "passing case", + "data": { "foo": 1 }, + "valid": true + }, + { + "description": "failing case", + "data": { "foo": "a string" }, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/items.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/items.json new file mode 100644 index 000000000..6a3e1cf26 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/items.json @@ -0,0 +1,304 @@ +[ + { + "description": "a schema given for items", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "items": {"type": "integer"} + }, + "tests": [ + { + "description": "valid items", + "data": [ 1, 2, 3 ], + "valid": true + }, + { + "description": "wrong type of items", + "data": [1, "x"], + "valid": false + }, + { + "description": "ignores non-arrays", + "data": {"foo" : "bar"}, + "valid": true + }, + { + "description": "JavaScript pseudo-array is valid", + "data": { + "0": "invalid", + "length": 1 + }, + "valid": true + } + ] + }, + { + "description": "items with boolean schema (true)", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "items": true + }, + "tests": [ + { + "description": "any array is valid", + "data": [ 1, "foo", true ], + "valid": true + }, + { + "description": "empty array is valid", + "data": [], + "valid": true + } + ] + }, + { + "description": "items with boolean schema (false)", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "items": false + }, + "tests": [ + { + "description": "any non-empty array is invalid", + "data": [ 1, "foo", true ], + "valid": false + }, + { + "description": "empty array is valid", + "data": [], + "valid": true + } + ] + }, + { + "description": "items and subitems", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "item": { + "type": "array", + "items": false, + "prefixItems": [ + { "$ref": "#/$defs/sub-item" }, + { "$ref": "#/$defs/sub-item" } + ] + }, + "sub-item": { + "type": "object", + "required": ["foo"] + } + }, + "type": "array", + "items": false, + "prefixItems": [ + { "$ref": "#/$defs/item" }, + { "$ref": "#/$defs/item" }, + { "$ref": "#/$defs/item" } + ] + }, + "tests": [ + { + "description": "valid items", + "data": [ + [ {"foo": null}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ] + ], + "valid": true + }, + { + "description": "too many items", + "data": [ + [ {"foo": null}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ] + ], + "valid": false + }, + { + "description": "too many sub-items", + "data": [ + [ {"foo": null}, {"foo": null}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ] + ], + "valid": false + }, + { + "description": "wrong item", + "data": [ + {"foo": null}, + [ {"foo": null}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ] + ], + "valid": false + }, + { + "description": "wrong sub-item", + "data": [ + [ {}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ] + ], + "valid": false + }, + { + "description": "fewer items is valid", + "data": [ + [ {"foo": null} ], + [ {"foo": null} ] + ], + "valid": true + } + ] + }, + { + "description": "nested items", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "array", + "items": { + "type": "array", + "items": { + "type": "array", + "items": { + "type": "array", + "items": { + "type": "number" + } + } + } + } + }, + "tests": [ + { + "description": "valid nested array", + "data": [[[[1]], [[2],[3]]], [[[4], [5], [6]]]], + "valid": true + }, + { + "description": "nested array with invalid type", + "data": [[[["1"]], [[2],[3]]], [[[4], [5], [6]]]], + "valid": false + }, + { + "description": "not deep enough", + "data": [[[1], [2],[3]], [[4], [5], [6]]], + "valid": false + } + ] + }, + { + "description": "prefixItems with no additional items allowed", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [{}, {}, {}], + "items": false + }, + "tests": [ + { + "description": "empty array", + "data": [ ], + "valid": true + }, + { + "description": "fewer number of items present (1)", + "data": [ 1 ], + "valid": true + }, + { + "description": "fewer number of items present (2)", + "data": [ 1, 2 ], + "valid": true + }, + { + "description": "equal number of items present", + "data": [ 1, 2, 3 ], + "valid": true + }, + { + "description": "additional items are not permitted", + "data": [ 1, 2, 3, 4 ], + "valid": false + } + ] + }, + { + "description": "items does not look in applicators, valid case", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + { "prefixItems": [ { "minimum": 3 } ] } + ], + "items": { "minimum": 5 } + }, + "tests": [ + { + "description": "prefixItems in allOf does not constrain items, invalid case", + "data": [ 3, 5 ], + "valid": false + }, + { + "description": "prefixItems in allOf does not constrain items, valid case", + "data": [ 5, 5 ], + "valid": true + } + ] + }, + { + "description": "prefixItems validation adjusts the starting index for items", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [ { "type": "string" } ], + "items": { "type": "integer" } + }, + "tests": [ + { + "description": "valid items", + "data": [ "x", 2, 3 ], + "valid": true + }, + { + "description": "wrong type of second item", + "data": [ "x", "y" ], + "valid": false + } + ] + }, + { + "description": "items with heterogeneous array", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [{}], + "items": false + }, + "tests": [ + { + "description": "heterogeneous invalid instance", + "data": [ "foo", "bar", 37 ], + "valid": false + }, + { + "description": "valid instance", + "data": [ null ], + "valid": true + } + ] + }, + { + "description": "items with null instance elements", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "items": { + "type": "null" + } + }, + "tests": [ + { + "description": "allows null elements", + "data": [ null ], + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maxContains.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maxContains.json new file mode 100644 index 000000000..9a42b45d6 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maxContains.json @@ -0,0 +1,123 @@ +[ + { + "description": "maxContains without contains is ignored", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "maxContains": 1 + }, + "tests": [ + { + "description": "one item valid against lone maxContains", + "data": [ 1 ], + "valid": true + }, + { + "description": "two items still valid against lone maxContains", + "data": [ 1, 2 ], + "valid": true + } + ] + }, + { + "description": "maxContains with contains", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contains": {"const": 1}, + "maxContains": 1 + }, + "tests": [ + { + "description": "empty data", + "data": [ ], + "valid": false + }, + { + "description": "all elements match, valid maxContains", + "data": [ 1 ], + "valid": true + }, + { + "description": "all elements match, invalid maxContains", + "data": [ 1, 1 ], + "valid": false + }, + { + "description": "some elements match, valid maxContains", + "data": [ 1, 2 ], + "valid": true + }, + { + "description": "some elements match, invalid maxContains", + "data": [ 1, 2, 1 ], + "valid": false + } + ] + }, + { + "description": "maxContains with contains, value with a decimal", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contains": {"const": 1}, + "maxContains": 1.0 + }, + "tests": [ + { + "description": "one element matches, valid maxContains", + "data": [ 1 ], + "valid": true + }, + { + "description": "too many elements match, invalid maxContains", + "data": [ 1, 1 ], + "valid": false + } + ] + }, + { + "description": "minContains < maxContains", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contains": {"const": 1}, + "minContains": 1, + "maxContains": 3 + }, + "tests": [ + { + "description": "actual < minContains < maxContains", + "data": [ ], + "valid": false + }, + { + "description": "minContains < actual < maxContains", + "data": [ 1, 1 ], + "valid": true + }, + { + "description": "minContains < maxContains < actual", + "data": [ 1, 1, 1, 1 ], + "valid": false + } + ] + }, + { + "description": "maxContains = 0 with minContains = 0", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contains": {"const": 1}, + "minContains": 0, + "maxContains": 0 + }, + "tests": [ + { + "description": "empty array", + "data": [ ], + "valid": true + }, + { + "description": "one matching item", + "data": [ 1 ], + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maxItems.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maxItems.json new file mode 100644 index 000000000..f6a6b7c9a --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maxItems.json @@ -0,0 +1,50 @@ +[ + { + "description": "maxItems validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "maxItems": 2 + }, + "tests": [ + { + "description": "shorter is valid", + "data": [1], + "valid": true + }, + { + "description": "exact length is valid", + "data": [1, 2], + "valid": true + }, + { + "description": "too long is invalid", + "data": [1, 2, 3], + "valid": false + }, + { + "description": "ignores non-arrays", + "data": "foobar", + "valid": true + } + ] + }, + { + "description": "maxItems validation with a decimal", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "maxItems": 2.0 + }, + "tests": [ + { + "description": "shorter is valid", + "data": [1], + "valid": true + }, + { + "description": "too long is invalid", + "data": [1, 2, 3], + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maxLength.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maxLength.json new file mode 100644 index 000000000..7462726d7 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maxLength.json @@ -0,0 +1,55 @@ +[ + { + "description": "maxLength validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "maxLength": 2 + }, + "tests": [ + { + "description": "shorter is valid", + "data": "f", + "valid": true + }, + { + "description": "exact length is valid", + "data": "fo", + "valid": true + }, + { + "description": "too long is invalid", + "data": "foo", + "valid": false + }, + { + "description": "ignores non-strings", + "data": 100, + "valid": true + }, + { + "description": "two graphemes is long enough", + "data": "\uD83D\uDCA9\uD83D\uDCA9", + "valid": true + } + ] + }, + { + "description": "maxLength validation with a decimal", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "maxLength": 2.0 + }, + "tests": [ + { + "description": "shorter is valid", + "data": "f", + "valid": true + }, + { + "description": "too long is invalid", + "data": "foo", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maxProperties.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maxProperties.json new file mode 100644 index 000000000..9cb745cad --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maxProperties.json @@ -0,0 +1,94 @@ +[ + { + "description": "maxProperties validation", + "specification": [ + { + "validation": "6.5.1" + } + ], + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "maxProperties": 2 + }, + "tests": [ + { + "description": "shorter is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "exact length is valid", + "data": {"foo": 1, "bar": 2}, + "valid": true + }, + { + "description": "too long is invalid", + "data": {"foo": 1, "bar": 2, "baz": 3}, + "valid": false + }, + { + "description": "ignores arrays", + "data": [1, 2, 3], + "valid": true + }, + { + "description": "ignores strings", + "data": "foobar", + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + } + ] + }, + { + "description": "maxProperties validation with a decimal", + "specification": [ + { + "validation": "6.5.1" + } + ], + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "maxProperties": 2.0 + }, + "tests": [ + { + "description": "shorter is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "too long is invalid", + "data": {"foo": 1, "bar": 2, "baz": 3}, + "valid": false + } + ] + }, + { + "description": "maxProperties = 0 means the object is empty", + "specification": [ + { + "validation": "6.5.1" + } + ], + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "maxProperties": 0 + }, + "tests": [ + { + "description": "no properties is valid", + "data": {}, + "valid": true + }, + { + "description": "one property is invalid", + "data": { "foo": 1 }, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maximum.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maximum.json new file mode 100644 index 000000000..7eeab2877 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/maximum.json @@ -0,0 +1,60 @@ +[ + { + "description": "maximum validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "maximum": 3.0 + }, + "tests": [ + { + "description": "below the maximum is valid", + "data": 2.6, + "valid": true + }, + { + "description": "boundary point is valid", + "data": 3.0, + "valid": true + }, + { + "description": "above the maximum is invalid", + "data": 3.5, + "valid": false + }, + { + "description": "ignores non-numbers", + "data": "x", + "valid": true + } + ] + }, + { + "description": "maximum validation with unsigned integer", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "maximum": 300 + }, + "tests": [ + { + "description": "below the maximum is valid", + "data": 299.97, + "valid": true + }, + { + "description": "boundary point integer is valid", + "data": 300, + "valid": true + }, + { + "description": "boundary point float is valid", + "data": 300.00, + "valid": true + }, + { + "description": "above the maximum is invalid", + "data": 300.5, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minContains.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minContains.json new file mode 100644 index 000000000..ee72d7d62 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minContains.json @@ -0,0 +1,224 @@ +[ + { + "description": "minContains without contains is ignored", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "minContains": 1 + }, + "tests": [ + { + "description": "one item valid against lone minContains", + "data": [ 1 ], + "valid": true + }, + { + "description": "zero items still valid against lone minContains", + "data": [], + "valid": true + } + ] + }, + { + "description": "minContains=1 with contains", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contains": {"const": 1}, + "minContains": 1 + }, + "tests": [ + { + "description": "empty data", + "data": [ ], + "valid": false + }, + { + "description": "no elements match", + "data": [ 2 ], + "valid": false + }, + { + "description": "single element matches, valid minContains", + "data": [ 1 ], + "valid": true + }, + { + "description": "some elements match, valid minContains", + "data": [ 1, 2 ], + "valid": true + }, + { + "description": "all elements match, valid minContains", + "data": [ 1, 1 ], + "valid": true + } + ] + }, + { + "description": "minContains=2 with contains", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contains": {"const": 1}, + "minContains": 2 + }, + "tests": [ + { + "description": "empty data", + "data": [ ], + "valid": false + }, + { + "description": "all elements match, invalid minContains", + "data": [ 1 ], + "valid": false + }, + { + "description": "some elements match, invalid minContains", + "data": [ 1, 2 ], + "valid": false + }, + { + "description": "all elements match, valid minContains (exactly as needed)", + "data": [ 1, 1 ], + "valid": true + }, + { + "description": "all elements match, valid minContains (more than needed)", + "data": [ 1, 1, 1 ], + "valid": true + }, + { + "description": "some elements match, valid minContains", + "data": [ 1, 2, 1 ], + "valid": true + } + ] + }, + { + "description": "minContains=2 with contains with a decimal value", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contains": {"const": 1}, + "minContains": 2.0 + }, + "tests": [ + { + "description": "one element matches, invalid minContains", + "data": [ 1 ], + "valid": false + }, + { + "description": "both elements match, valid minContains", + "data": [ 1, 1 ], + "valid": true + } + ] + }, + { + "description": "maxContains = minContains", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contains": {"const": 1}, + "maxContains": 2, + "minContains": 2 + }, + "tests": [ + { + "description": "empty data", + "data": [ ], + "valid": false + }, + { + "description": "all elements match, invalid minContains", + "data": [ 1 ], + "valid": false + }, + { + "description": "all elements match, invalid maxContains", + "data": [ 1, 1, 1 ], + "valid": false + }, + { + "description": "all elements match, valid maxContains and minContains", + "data": [ 1, 1 ], + "valid": true + } + ] + }, + { + "description": "maxContains < minContains", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contains": {"const": 1}, + "maxContains": 1, + "minContains": 3 + }, + "tests": [ + { + "description": "empty data", + "data": [ ], + "valid": false + }, + { + "description": "invalid minContains", + "data": [ 1 ], + "valid": false + }, + { + "description": "invalid maxContains", + "data": [ 1, 1, 1 ], + "valid": false + }, + { + "description": "invalid maxContains and minContains", + "data": [ 1, 1 ], + "valid": false + } + ] + }, + { + "description": "minContains = 0", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contains": {"const": 1}, + "minContains": 0 + }, + "tests": [ + { + "description": "empty data", + "data": [ ], + "valid": true + }, + { + "description": "minContains = 0 makes contains always pass", + "data": [ 2 ], + "valid": true + } + ] + }, + { + "description": "minContains = 0 with maxContains", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contains": {"const": 1}, + "minContains": 0, + "maxContains": 1 + }, + "tests": [ + { + "description": "empty data", + "data": [ ], + "valid": true + }, + { + "description": "not more than maxContains", + "data": [ 1 ], + "valid": true + }, + { + "description": "too many", + "data": [ 1, 1 ], + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minItems.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minItems.json new file mode 100644 index 000000000..9d6a8b6d2 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minItems.json @@ -0,0 +1,50 @@ +[ + { + "description": "minItems validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "minItems": 1 + }, + "tests": [ + { + "description": "longer is valid", + "data": [1, 2], + "valid": true + }, + { + "description": "exact length is valid", + "data": [1], + "valid": true + }, + { + "description": "too short is invalid", + "data": [], + "valid": false + }, + { + "description": "ignores non-arrays", + "data": "", + "valid": true + } + ] + }, + { + "description": "minItems validation with a decimal", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "minItems": 1.0 + }, + "tests": [ + { + "description": "longer is valid", + "data": [1, 2], + "valid": true + }, + { + "description": "too short is invalid", + "data": [], + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minLength.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minLength.json new file mode 100644 index 000000000..5076c5a92 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minLength.json @@ -0,0 +1,55 @@ +[ + { + "description": "minLength validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "minLength": 2 + }, + "tests": [ + { + "description": "longer is valid", + "data": "foo", + "valid": true + }, + { + "description": "exact length is valid", + "data": "fo", + "valid": true + }, + { + "description": "too short is invalid", + "data": "f", + "valid": false + }, + { + "description": "ignores non-strings", + "data": 1, + "valid": true + }, + { + "description": "one grapheme is not long enough", + "data": "\uD83D\uDCA9", + "valid": false + } + ] + }, + { + "description": "minLength validation with a decimal", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "minLength": 2.0 + }, + "tests": [ + { + "description": "longer is valid", + "data": "foo", + "valid": true + }, + { + "description": "too short is invalid", + "data": "f", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minProperties.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minProperties.json new file mode 100644 index 000000000..12587a5f8 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minProperties.json @@ -0,0 +1,70 @@ +[ + { + "description": "minProperties validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "minProperties": 1 + }, + "tests": [ + { + "description": "longer is valid", + "data": {"foo": 1, "bar": 2}, + "valid": true + }, + { + "description": "exact length is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "too short is invalid", + "data": {}, + "valid": false + }, + { + "description": "ignores arrays", + "data": [], + "valid": true + }, + { + "description": "ignores strings", + "data": "", + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + }, + { + "description": "ignores null", + "data": null, + "valid": true + }, + { + "description": "ignores booleans", + "data": true, + "valid": true + } + ] + }, + { + "description": "minProperties validation with a decimal", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "minProperties": 1.0 + }, + "tests": [ + { + "description": "longer is valid", + "data": {"foo": 1, "bar": 2}, + "valid": true + }, + { + "description": "too short is invalid", + "data": {}, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minimum.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minimum.json new file mode 100644 index 000000000..dc4405278 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/minimum.json @@ -0,0 +1,75 @@ +[ + { + "description": "minimum validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "minimum": 1.1 + }, + "tests": [ + { + "description": "above the minimum is valid", + "data": 2.6, + "valid": true + }, + { + "description": "boundary point is valid", + "data": 1.1, + "valid": true + }, + { + "description": "below the minimum is invalid", + "data": 0.6, + "valid": false + }, + { + "description": "ignores non-numbers", + "data": "x", + "valid": true + } + ] + }, + { + "description": "minimum validation with signed integer", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "minimum": -2 + }, + "tests": [ + { + "description": "negative above the minimum is valid", + "data": -1, + "valid": true + }, + { + "description": "positive above the minimum is valid", + "data": 0, + "valid": true + }, + { + "description": "boundary point is valid", + "data": -2, + "valid": true + }, + { + "description": "boundary point with float is valid", + "data": -2.0, + "valid": true + }, + { + "description": "float below the minimum is invalid", + "data": -2.0001, + "valid": false + }, + { + "description": "int below the minimum is invalid", + "data": -3, + "valid": false + }, + { + "description": "ignores non-numbers", + "data": "x", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/multipleOf.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/multipleOf.json new file mode 100644 index 000000000..76fa4765b --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/multipleOf.json @@ -0,0 +1,102 @@ +[ + { + "description": "by int", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "multipleOf": 2 + }, + "tests": [ + { + "description": "int by int", + "data": 10, + "valid": true + }, + { + "description": "int by int fail", + "data": 7, + "valid": false + }, + { + "description": "ignores non-numbers", + "data": "foo", + "valid": true + } + ] + }, + { + "description": "by number", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "multipleOf": 1.5 + }, + "tests": [ + { + "description": "zero is multiple of anything", + "data": 0, + "valid": true + }, + { + "description": "4.5 is multiple of 1.5", + "data": 4.5, + "valid": true + }, + { + "description": "-4.5 is multiple of 1.5", + "data": -4.5, + "valid": true + }, + { + "description": "35 is not multiple of 1.5", + "data": 35, + "valid": false + } + ] + }, + { + "description": "by small number", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "multipleOf": 0.0001 + }, + "tests": [ + { + "description": "0.0075 is multiple of 0.0001", + "data": 0.0075, + "valid": true + }, + { + "description": "0.00751 is not multiple of 0.0001", + "data": 0.00751, + "valid": false + } + ] + }, + { + "description": "float division = inf", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "integer", "multipleOf": 0.123456789 + }, + "tests": [ + { + "description": "always invalid, but naive implementations may raise an overflow error", + "data": 1e308, + "valid": false + } + ] + }, + { + "description": "small multiple of large integer", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "integer", "multipleOf": 1e-8 + }, + "tests": [ + { + "description": "any integer is a multiple of 1e-8", + "data": 12391239123, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/not.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/not.json new file mode 100644 index 000000000..d0f2b6e84 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/not.json @@ -0,0 +1,301 @@ +[ + { + "description": "not", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "not": {"type": "integer"} + }, + "tests": [ + { + "description": "allowed", + "data": "foo", + "valid": true + }, + { + "description": "disallowed", + "data": 1, + "valid": false + } + ] + }, + { + "description": "not multiple types", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "not": {"type": ["integer", "boolean"]} + }, + "tests": [ + { + "description": "valid", + "data": "foo", + "valid": true + }, + { + "description": "mismatch", + "data": 1, + "valid": false + }, + { + "description": "other mismatch", + "data": true, + "valid": false + } + ] + }, + { + "description": "not more complex schema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "not": { + "type": "object", + "properties": { + "foo": { + "type": "string" + } + } + } + }, + "tests": [ + { + "description": "match", + "data": 1, + "valid": true + }, + { + "description": "other match", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "mismatch", + "data": {"foo": "bar"}, + "valid": false + } + ] + }, + { + "description": "forbidden property", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { + "not": {} + } + } + }, + "tests": [ + { + "description": "property present", + "data": {"foo": 1, "bar": 2}, + "valid": false + }, + { + "description": "property absent", + "data": {"bar": 1, "baz": 2}, + "valid": true + } + ] + }, + { + "description": "forbid everything with empty schema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "not": {} + }, + "tests": [ + { + "description": "number is invalid", + "data": 1, + "valid": false + }, + { + "description": "string is invalid", + "data": "foo", + "valid": false + }, + { + "description": "boolean true is invalid", + "data": true, + "valid": false + }, + { + "description": "boolean false is invalid", + "data": false, + "valid": false + }, + { + "description": "null is invalid", + "data": null, + "valid": false + }, + { + "description": "object is invalid", + "data": {"foo": "bar"}, + "valid": false + }, + { + "description": "empty object is invalid", + "data": {}, + "valid": false + }, + { + "description": "array is invalid", + "data": ["foo"], + "valid": false + }, + { + "description": "empty array is invalid", + "data": [], + "valid": false + } + ] + }, + { + "description": "forbid everything with boolean schema true", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "not": true + }, + "tests": [ + { + "description": "number is invalid", + "data": 1, + "valid": false + }, + { + "description": "string is invalid", + "data": "foo", + "valid": false + }, + { + "description": "boolean true is invalid", + "data": true, + "valid": false + }, + { + "description": "boolean false is invalid", + "data": false, + "valid": false + }, + { + "description": "null is invalid", + "data": null, + "valid": false + }, + { + "description": "object is invalid", + "data": {"foo": "bar"}, + "valid": false + }, + { + "description": "empty object is invalid", + "data": {}, + "valid": false + }, + { + "description": "array is invalid", + "data": ["foo"], + "valid": false + }, + { + "description": "empty array is invalid", + "data": [], + "valid": false + } + ] + }, + { + "description": "allow everything with boolean schema false", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "not": false + }, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "string is valid", + "data": "foo", + "valid": true + }, + { + "description": "boolean true is valid", + "data": true, + "valid": true + }, + { + "description": "boolean false is valid", + "data": false, + "valid": true + }, + { + "description": "null is valid", + "data": null, + "valid": true + }, + { + "description": "object is valid", + "data": {"foo": "bar"}, + "valid": true + }, + { + "description": "empty object is valid", + "data": {}, + "valid": true + }, + { + "description": "array is valid", + "data": ["foo"], + "valid": true + }, + { + "description": "empty array is valid", + "data": [], + "valid": true + } + ] + }, + { + "description": "double negation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "not": { "not": {} } + }, + "tests": [ + { + "description": "any value is valid", + "data": "foo", + "valid": true + } + ] + }, + { + "description": "collect annotations inside a 'not', even if collection is disabled", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "not": { + "$comment": "this subschema must still produce annotations internally, even though the 'not' will ultimately discard them", + "anyOf": [ + true, + { "properties": { "foo": true } } + ], + "unevaluatedProperties": false + } + }, + "tests": [ + { + "description": "unevaluated property", + "data": { "bar": 1 }, + "valid": true + }, + { + "description": "annotations are still collected inside a 'not'", + "data": { "foo": 1 }, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/oneOf.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/oneOf.json new file mode 100644 index 000000000..7a7c7ffe3 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/oneOf.json @@ -0,0 +1,293 @@ +[ + { + "description": "oneOf", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "oneOf": [ + { + "type": "integer" + }, + { + "minimum": 2 + } + ] + }, + "tests": [ + { + "description": "first oneOf valid", + "data": 1, + "valid": true + }, + { + "description": "second oneOf valid", + "data": 2.5, + "valid": true + }, + { + "description": "both oneOf valid", + "data": 3, + "valid": false + }, + { + "description": "neither oneOf valid", + "data": 1.5, + "valid": false + } + ] + }, + { + "description": "oneOf with base schema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "string", + "oneOf" : [ + { + "minLength": 2 + }, + { + "maxLength": 4 + } + ] + }, + "tests": [ + { + "description": "mismatch base schema", + "data": 3, + "valid": false + }, + { + "description": "one oneOf valid", + "data": "foobar", + "valid": true + }, + { + "description": "both oneOf valid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "oneOf with boolean schemas, all true", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "oneOf": [true, true, true] + }, + "tests": [ + { + "description": "any value is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "oneOf with boolean schemas, one true", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "oneOf": [true, false, false] + }, + "tests": [ + { + "description": "any value is valid", + "data": "foo", + "valid": true + } + ] + }, + { + "description": "oneOf with boolean schemas, more than one true", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "oneOf": [true, true, false] + }, + "tests": [ + { + "description": "any value is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "oneOf with boolean schemas, all false", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "oneOf": [false, false, false] + }, + "tests": [ + { + "description": "any value is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "oneOf complex types", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "oneOf": [ + { + "properties": { + "bar": {"type": "integer"} + }, + "required": ["bar"] + }, + { + "properties": { + "foo": {"type": "string"} + }, + "required": ["foo"] + } + ] + }, + "tests": [ + { + "description": "first oneOf valid (complex)", + "data": {"bar": 2}, + "valid": true + }, + { + "description": "second oneOf valid (complex)", + "data": {"foo": "baz"}, + "valid": true + }, + { + "description": "both oneOf valid (complex)", + "data": {"foo": "baz", "bar": 2}, + "valid": false + }, + { + "description": "neither oneOf valid (complex)", + "data": {"foo": 2, "bar": "quux"}, + "valid": false + } + ] + }, + { + "description": "oneOf with empty schema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "oneOf": [ + { "type": "number" }, + {} + ] + }, + "tests": [ + { + "description": "one valid - valid", + "data": "foo", + "valid": true + }, + { + "description": "both valid - invalid", + "data": 123, + "valid": false + } + ] + }, + { + "description": "oneOf with required", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "oneOf": [ + { "required": ["foo", "bar"] }, + { "required": ["foo", "baz"] } + ] + }, + "tests": [ + { + "description": "both invalid - invalid", + "data": {"bar": 2}, + "valid": false + }, + { + "description": "first valid - valid", + "data": {"foo": 1, "bar": 2}, + "valid": true + }, + { + "description": "second valid - valid", + "data": {"foo": 1, "baz": 3}, + "valid": true + }, + { + "description": "both valid - invalid", + "data": {"foo": 1, "bar": 2, "baz" : 3}, + "valid": false + } + ] + }, + { + "description": "oneOf with missing optional property", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "oneOf": [ + { + "properties": { + "bar": true, + "baz": true + }, + "required": ["bar"] + }, + { + "properties": { + "foo": true + }, + "required": ["foo"] + } + ] + }, + "tests": [ + { + "description": "first oneOf valid", + "data": {"bar": 8}, + "valid": true + }, + { + "description": "second oneOf valid", + "data": {"foo": "foo"}, + "valid": true + }, + { + "description": "both oneOf valid", + "data": {"foo": "foo", "bar": 8}, + "valid": false + }, + { + "description": "neither oneOf valid", + "data": {"baz": "quux"}, + "valid": false + } + ] + }, + { + "description": "nested oneOf, to check validation semantics", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "oneOf": [ + { + "oneOf": [ + { + "type": "null" + } + ] + } + ] + }, + "tests": [ + { + "description": "null is valid", + "data": null, + "valid": true + }, + { + "description": "anything non-null is invalid", + "data": 123, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/anchor.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/anchor.json new file mode 100644 index 000000000..6d6713be5 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/anchor.json @@ -0,0 +1,60 @@ +[ + { + "description": "$anchor inside an enum is not a real identifier", + "comment": "the implementation must not be confused by an $anchor buried in the enum", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "anchor_in_enum": { + "enum": [ + { + "$anchor": "my_anchor", + "type": "null" + } + ] + }, + "real_identifier_in_schema": { + "$anchor": "my_anchor", + "type": "string" + }, + "zzz_anchor_in_const": { + "const": { + "$anchor": "my_anchor", + "type": "null" + } + } + }, + "anyOf": [ + { "$ref": "#/$defs/anchor_in_enum" }, + { "$ref": "#my_anchor" } + ] + }, + "tests": [ + { + "description": "exact match to enum, and type matches", + "data": { + "$anchor": "my_anchor", + "type": "null" + }, + "valid": true + }, + { + "description": "in implementations that strip $anchor, this may match either $def", + "data": { + "type": "null" + }, + "valid": false + }, + { + "description": "match $ref to $anchor", + "data": "a string to match #/$defs/anchor_in_enum", + "valid": true + }, + { + "description": "no match on enum or $ref to $anchor", + "data": 1, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/bignum.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/bignum.json new file mode 100644 index 000000000..d69b29e85 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/bignum.json @@ -0,0 +1,110 @@ +[ + { + "description": "integer", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "integer" + }, + "tests": [ + { + "description": "a bignum is an integer", + "data": 12345678910111213141516171819202122232425262728293031, + "valid": true + }, + { + "description": "a negative bignum is an integer", + "data": -12345678910111213141516171819202122232425262728293031, + "valid": true + } + ] + }, + { + "description": "number", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "number" + }, + "tests": [ + { + "description": "a bignum is a number", + "data": 98249283749234923498293171823948729348710298301928331, + "valid": true + }, + { + "description": "a negative bignum is a number", + "data": -98249283749234923498293171823948729348710298301928331, + "valid": true + } + ] + }, + { + "description": "string", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "string" + }, + "tests": [ + { + "description": "a bignum is not a string", + "data": 98249283749234923498293171823948729348710298301928331, + "valid": false + } + ] + }, + { + "description": "maximum integer comparison", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "maximum": 18446744073709551615 + }, + "tests": [ + { + "description": "comparison works for high numbers", + "data": 18446744073709551600, + "valid": true + } + ] + }, + { + "description": "float comparison with high precision", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "exclusiveMaximum": 972783798187987123879878123.18878137 + }, + "tests": [ + { + "description": "comparison works for high numbers", + "data": 972783798187987123879878123.188781371, + "valid": false + } + ] + }, + { + "description": "minimum integer comparison", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "minimum": -18446744073709551615 + }, + "tests": [ + { + "description": "comparison works for very negative numbers", + "data": -18446744073709551600, + "valid": true + } + ] + }, + { + "description": "float comparison with high precision on negative numbers", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "exclusiveMinimum": -972783798187987123879878123.18878137 + }, + "tests": [ + { + "description": "comparison works for very negative numbers", + "data": -972783798187987123879878123.188781371, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/cross-draft.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/cross-draft.json new file mode 100644 index 000000000..5113bd649 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/cross-draft.json @@ -0,0 +1,18 @@ +[ + { + "description": "refs to historic drafts are processed as historic drafts", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "array", + "$ref": "http://localhost:1234/draft2019-09/ignore-prefixItems.json" + }, + "tests": [ + { + "description": "first item not a string is valid", + "comment": "if the implementation is not processing the $ref as a 2019-09 schema, this test will fail", + "data": [1, 2, 3], + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/dependencies-compatibility.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/dependencies-compatibility.json new file mode 100644 index 000000000..47d5bd79d --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/dependencies-compatibility.json @@ -0,0 +1,282 @@ +[ + { + "description": "single dependency", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "dependencies": {"bar": ["foo"]} + }, + "tests": [ + { + "description": "neither", + "data": {}, + "valid": true + }, + { + "description": "nondependant", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "with dependency", + "data": {"foo": 1, "bar": 2}, + "valid": true + }, + { + "description": "missing dependency", + "data": {"bar": 2}, + "valid": false + }, + { + "description": "ignores arrays", + "data": ["bar"], + "valid": true + }, + { + "description": "ignores strings", + "data": "foobar", + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + } + ] + }, + { + "description": "empty dependents", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "dependencies": {"bar": []} + }, + "tests": [ + { + "description": "empty object", + "data": {}, + "valid": true + }, + { + "description": "object with one property", + "data": {"bar": 2}, + "valid": true + }, + { + "description": "non-object is valid", + "data": 1, + "valid": true + } + ] + }, + { + "description": "multiple dependents required", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "dependencies": {"quux": ["foo", "bar"]} + }, + "tests": [ + { + "description": "neither", + "data": {}, + "valid": true + }, + { + "description": "nondependants", + "data": {"foo": 1, "bar": 2}, + "valid": true + }, + { + "description": "with dependencies", + "data": {"foo": 1, "bar": 2, "quux": 3}, + "valid": true + }, + { + "description": "missing dependency", + "data": {"foo": 1, "quux": 2}, + "valid": false + }, + { + "description": "missing other dependency", + "data": {"bar": 1, "quux": 2}, + "valid": false + }, + { + "description": "missing both dependencies", + "data": {"quux": 1}, + "valid": false + } + ] + }, + { + "description": "dependencies with escaped characters", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "dependencies": { + "foo\nbar": ["foo\rbar"], + "foo\"bar": ["foo'bar"] + } + }, + "tests": [ + { + "description": "CRLF", + "data": { + "foo\nbar": 1, + "foo\rbar": 2 + }, + "valid": true + }, + { + "description": "quoted quotes", + "data": { + "foo'bar": 1, + "foo\"bar": 2 + }, + "valid": true + }, + { + "description": "CRLF missing dependent", + "data": { + "foo\nbar": 1, + "foo": 2 + }, + "valid": false + }, + { + "description": "quoted quotes missing dependent", + "data": { + "foo\"bar": 2 + }, + "valid": false + } + ] + }, + { + "description": "single schema dependency", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "dependencies": { + "bar": { + "properties": { + "foo": {"type": "integer"}, + "bar": {"type": "integer"} + } + } + } + }, + "tests": [ + { + "description": "valid", + "data": {"foo": 1, "bar": 2}, + "valid": true + }, + { + "description": "no dependency", + "data": {"foo": "quux"}, + "valid": true + }, + { + "description": "wrong type", + "data": {"foo": "quux", "bar": 2}, + "valid": false + }, + { + "description": "wrong type other", + "data": {"foo": 2, "bar": "quux"}, + "valid": false + }, + { + "description": "wrong type both", + "data": {"foo": "quux", "bar": "quux"}, + "valid": false + }, + { + "description": "ignores arrays", + "data": ["bar"], + "valid": true + }, + { + "description": "ignores strings", + "data": "foobar", + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + } + ] + }, + { + "description": "boolean subschemas", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "dependencies": { + "foo": true, + "bar": false + } + }, + "tests": [ + { + "description": "object with property having schema true is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "object with property having schema false is invalid", + "data": {"bar": 2}, + "valid": false + }, + { + "description": "object with both properties is invalid", + "data": {"foo": 1, "bar": 2}, + "valid": false + }, + { + "description": "empty object is valid", + "data": {}, + "valid": true + } + ] + }, + { + "description": "schema dependencies with escaped characters", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "dependencies": { + "foo\tbar": {"minProperties": 4}, + "foo'bar": {"required": ["foo\"bar"]} + } + }, + "tests": [ + { + "description": "quoted tab", + "data": { + "foo\tbar": 1, + "a": 2, + "b": 3, + "c": 4 + }, + "valid": true + }, + { + "description": "quoted quote", + "data": { + "foo'bar": {"foo\"bar": 1} + }, + "valid": false + }, + { + "description": "quoted tab invalid under dependent schema", + "data": { + "foo\tbar": 1, + "a": 2 + }, + "valid": false + }, + { + "description": "quoted quote invalid under dependent schema", + "data": {"foo'bar": 1}, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/dynamicRef.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/dynamicRef.json new file mode 100644 index 000000000..7e63f209a --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/dynamicRef.json @@ -0,0 +1,56 @@ +[ + { + "description": "$dynamicRef skips over intermediate resources - pointer reference across resource boundary", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://test.json-schema.org/dynamic-ref-skips-intermediate-resource/optional/main", + "type": "object", + "properties": { + "bar-item": { + "$ref": "bar#/$defs/item" + } + }, + "$defs": { + "bar": { + "$id": "bar", + "type": "array", + "items": { + "$ref": "item" + }, + "$defs": { + "item": { + "$id": "item", + "type": "object", + "properties": { + "content": { + "$dynamicRef": "#content" + } + }, + "$defs": { + "defaultContent": { + "$dynamicAnchor": "content", + "type": "integer" + } + } + }, + "content": { + "$dynamicAnchor": "content", + "type": "string" + } + } + } + } + }, + "tests": [ + { + "description": "integer property passes", + "data": { "bar-item": { "content": 42 } }, + "valid": true + }, + { + "description": "string property fails", + "data": { "bar-item": { "content": "value" } }, + "valid": false + } + ] + }] \ No newline at end of file diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/ecmascript-regex.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/ecmascript-regex.json new file mode 100644 index 000000000..a4d62e0cf --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/ecmascript-regex.json @@ -0,0 +1,582 @@ +[ + { + "description": "ECMA 262 regex $ does not match trailing newline", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "string", + "pattern": "^abc$" + }, + "tests": [ + { + "description": "matches in Python, but not in ECMA 262", + "data": "abc\\n", + "valid": false + }, + { + "description": "matches", + "data": "abc", + "valid": true + } + ] + }, + { + "description": "ECMA 262 regex converts \\t to horizontal tab", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "string", + "pattern": "^\\t$" + }, + "tests": [ + { + "description": "does not match", + "data": "\\t", + "valid": false + }, + { + "description": "matches", + "data": "\u0009", + "valid": true + } + ] + }, + { + "description": "ECMA 262 regex escapes control codes with \\c and upper letter", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "string", + "pattern": "^\\cC$" + }, + "tests": [ + { + "description": "does not match", + "data": "\\cC", + "valid": false + }, + { + "description": "matches", + "data": "\u0003", + "valid": true + } + ] + }, + { + "description": "ECMA 262 regex escapes control codes with \\c and lower letter", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "string", + "pattern": "^\\cc$" + }, + "tests": [ + { + "description": "does not match", + "data": "\\cc", + "valid": false + }, + { + "description": "matches", + "data": "\u0003", + "valid": true + } + ] + }, + { + "description": "ECMA 262 \\d matches ascii digits only", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "string", + "pattern": "^\\d$" + }, + "tests": [ + { + "description": "ASCII zero matches", + "data": "0", + "valid": true + }, + { + "description": "NKO DIGIT ZERO does not match (unlike e.g. Python)", + "data": "߀", + "valid": false + }, + { + "description": "NKO DIGIT ZERO (as \\u escape) does not match", + "data": "\u07c0", + "valid": false + } + ] + }, + { + "description": "ECMA 262 \\D matches everything but ascii digits", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "string", + "pattern": "^\\D$" + }, + "tests": [ + { + "description": "ASCII zero does not match", + "data": "0", + "valid": false + }, + { + "description": "NKO DIGIT ZERO matches (unlike e.g. Python)", + "data": "߀", + "valid": true + }, + { + "description": "NKO DIGIT ZERO (as \\u escape) matches", + "data": "\u07c0", + "valid": true + } + ] + }, + { + "description": "ECMA 262 \\w matches ascii letters only", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "string", + "pattern": "^\\w$" + }, + "tests": [ + { + "description": "ASCII 'a' matches", + "data": "a", + "valid": true + }, + { + "description": "latin-1 e-acute does not match (unlike e.g. Python)", + "data": "é", + "valid": false + } + ] + }, + { + "description": "ECMA 262 \\W matches everything but ascii letters", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "string", + "pattern": "^\\W$" + }, + "tests": [ + { + "description": "ASCII 'a' does not match", + "data": "a", + "valid": false + }, + { + "description": "latin-1 e-acute matches (unlike e.g. Python)", + "data": "é", + "valid": true + } + ] + }, + { + "description": "ECMA 262 \\s matches whitespace", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "string", + "pattern": "^\\s$" + }, + "tests": [ + { + "description": "ASCII space matches", + "data": " ", + "valid": true + }, + { + "description": "Character tabulation matches", + "data": "\t", + "valid": true + }, + { + "description": "Line tabulation matches", + "data": "\u000b", + "valid": true + }, + { + "description": "Form feed matches", + "data": "\u000c", + "valid": true + }, + { + "description": "latin-1 non-breaking-space matches", + "data": "\u00a0", + "valid": true + }, + { + "description": "zero-width whitespace matches", + "data": "\ufeff", + "valid": true + }, + { + "description": "line feed matches (line terminator)", + "data": "\u000a", + "valid": true + }, + { + "description": "paragraph separator matches (line terminator)", + "data": "\u2029", + "valid": true + }, + { + "description": "EM SPACE matches (Space_Separator)", + "data": "\u2003", + "valid": true + }, + { + "description": "Non-whitespace control does not match", + "data": "\u0001", + "valid": false + }, + { + "description": "Non-whitespace does not match", + "data": "\u2013", + "valid": false + } + ] + }, + { + "description": "ECMA 262 \\S matches everything but whitespace", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "string", + "pattern": "^\\S$" + }, + "tests": [ + { + "description": "ASCII space does not match", + "data": " ", + "valid": false + }, + { + "description": "Character tabulation does not match", + "data": "\t", + "valid": false + }, + { + "description": "Line tabulation does not match", + "data": "\u000b", + "valid": false + }, + { + "description": "Form feed does not match", + "data": "\u000c", + "valid": false + }, + { + "description": "latin-1 non-breaking-space does not match", + "data": "\u00a0", + "valid": false + }, + { + "description": "zero-width whitespace does not match", + "data": "\ufeff", + "valid": false + }, + { + "description": "line feed does not match (line terminator)", + "data": "\u000a", + "valid": false + }, + { + "description": "paragraph separator does not match (line terminator)", + "data": "\u2029", + "valid": false + }, + { + "description": "EM SPACE does not match (Space_Separator)", + "data": "\u2003", + "valid": false + }, + { + "description": "Non-whitespace control matches", + "data": "\u0001", + "valid": true + }, + { + "description": "Non-whitespace matches", + "data": "\u2013", + "valid": true + } + ] + }, + { + "description": "patterns always use unicode semantics with pattern", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "pattern": "\\p{Letter}cole" + }, + "tests": [ + { + "description": "ascii character in json string", + "data": "Les hivers de mon enfance etaient des saisons longues, longues. Nous vivions en trois lieux: l'ecole, l'eglise et la patinoire; mais la vraie vie etait sur la patinoire.", + "valid": true + }, + { + "description": "literal unicode character in json string", + "data": "Les hivers de mon enfance étaient des saisons longues, longues. Nous vivions en trois lieux: l'école, l'église et la patinoire; mais la vraie vie était sur la patinoire.", + "valid": true + }, + { + "description": "unicode character in hex format in string", + "data": "Les hivers de mon enfance étaient des saisons longues, longues. Nous vivions en trois lieux: l'\u00e9cole, l'église et la patinoire; mais la vraie vie était sur la patinoire.", + "valid": true + }, + { + "description": "unicode matching is case-sensitive", + "data": "LES HIVERS DE MON ENFANCE ÉTAIENT DES SAISONS LONGUES, LONGUES. NOUS VIVIONS EN TROIS LIEUX: L'ÉCOLE, L'ÉGLISE ET LA PATINOIRE; MAIS LA VRAIE VIE ÉTAIT SUR LA PATINOIRE.", + "valid": false + } + ] + }, + { + "description": "\\w in patterns matches [A-Za-z0-9_], not unicode letters", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "pattern": "\\wcole" + }, + "tests": [ + { + "description": "ascii character in json string", + "data": "Les hivers de mon enfance etaient des saisons longues, longues. Nous vivions en trois lieux: l'ecole, l'eglise et la patinoire; mais la vraie vie etait sur la patinoire.", + "valid": true + }, + { + "description": "literal unicode character in json string", + "data": "Les hivers de mon enfance étaient des saisons longues, longues. Nous vivions en trois lieux: l'école, l'église et la patinoire; mais la vraie vie était sur la patinoire.", + "valid": false + }, + { + "description": "unicode character in hex format in string", + "data": "Les hivers de mon enfance étaient des saisons longues, longues. Nous vivions en trois lieux: l'\u00e9cole, l'église et la patinoire; mais la vraie vie était sur la patinoire.", + "valid": false + }, + { + "description": "unicode matching is case-sensitive", + "data": "LES HIVERS DE MON ENFANCE ÉTAIENT DES SAISONS LONGUES, LONGUES. NOUS VIVIONS EN TROIS LIEUX: L'ÉCOLE, L'ÉGLISE ET LA PATINOIRE; MAIS LA VRAIE VIE ÉTAIT SUR LA PATINOIRE.", + "valid": false + } + ] + }, + { + "description": "pattern with ASCII ranges", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "pattern": "[a-z]cole" + }, + "tests": [ + { + "description": "literal unicode character in json string", + "data": "Les hivers de mon enfance étaient des saisons longues, longues. Nous vivions en trois lieux: l'école, l'église et la patinoire; mais la vraie vie était sur la patinoire.", + "valid": false + }, + { + "description": "unicode character in hex format in string", + "data": "Les hivers de mon enfance étaient des saisons longues, longues. Nous vivions en trois lieux: l'\u00e9cole, l'église et la patinoire; mais la vraie vie était sur la patinoire.", + "valid": false + }, + { + "description": "ascii characters match", + "data": "Les hivers de mon enfance etaient des saisons longues, longues. Nous vivions en trois lieux: l'ecole, l'eglise et la patinoire; mais la vraie vie etait sur la patinoire.", + "valid": true + } + ] + }, + { + "description": "\\d in pattern matches [0-9], not unicode digits", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "pattern": "^\\d+$" + }, + "tests": [ + { + "description": "ascii digits", + "data": "42", + "valid": true + }, + { + "description": "ascii non-digits", + "data": "-%#", + "valid": false + }, + { + "description": "non-ascii digits (BENGALI DIGIT FOUR, BENGALI DIGIT TWO)", + "data": "৪২", + "valid": false + } + ] + }, + { + "description": "pattern with non-ASCII digits", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "pattern": "^\\p{digit}+$" + }, + "tests": [ + { + "description": "ascii digits", + "data": "42", + "valid": true + }, + { + "description": "ascii non-digits", + "data": "-%#", + "valid": false + }, + { + "description": "non-ascii digits (BENGALI DIGIT FOUR, BENGALI DIGIT TWO)", + "data": "৪২", + "valid": true + } + ] + }, + { + "description": "patterns always use unicode semantics with patternProperties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "patternProperties": { + "\\p{Letter}cole": true + }, + "additionalProperties": false + }, + "tests": [ + { + "description": "ascii character in json string", + "data": { "l'ecole": "pas de vraie vie" }, + "valid": true + }, + { + "description": "literal unicode character in json string", + "data": { "l'école": "pas de vraie vie" }, + "valid": true + }, + { + "description": "unicode character in hex format in string", + "data": { "l'\u00e9cole": "pas de vraie vie" }, + "valid": true + }, + { + "description": "unicode matching is case-sensitive", + "data": { "L'ÉCOLE": "PAS DE VRAIE VIE" }, + "valid": false + } + ] + }, + { + "description": "\\w in patternProperties matches [A-Za-z0-9_], not unicode letters", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "patternProperties": { + "\\wcole": true + }, + "additionalProperties": false + }, + "tests": [ + { + "description": "ascii character in json string", + "data": { "l'ecole": "pas de vraie vie" }, + "valid": true + }, + { + "description": "literal unicode character in json string", + "data": { "l'école": "pas de vraie vie" }, + "valid": false + }, + { + "description": "unicode character in hex format in string", + "data": { "l'\u00e9cole": "pas de vraie vie" }, + "valid": false + }, + { + "description": "unicode matching is case-sensitive", + "data": { "L'ÉCOLE": "PAS DE VRAIE VIE" }, + "valid": false + } + ] + }, + { + "description": "patternProperties with ASCII ranges", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "patternProperties": { + "[a-z]cole": true + }, + "additionalProperties": false + }, + "tests": [ + { + "description": "literal unicode character in json string", + "data": { "l'école": "pas de vraie vie" }, + "valid": false + }, + { + "description": "unicode character in hex format in string", + "data": { "l'\u00e9cole": "pas de vraie vie" }, + "valid": false + }, + { + "description": "ascii characters match", + "data": { "l'ecole": "pas de vraie vie" }, + "valid": true + } + ] + }, + { + "description": "\\d in patternProperties matches [0-9], not unicode digits", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "patternProperties": { + "^\\d+$": true + }, + "additionalProperties": false + }, + "tests": [ + { + "description": "ascii digits", + "data": { "42": "life, the universe, and everything" }, + "valid": true + }, + { + "description": "ascii non-digits", + "data": { "-%#": "spending the year dead for tax reasons" }, + "valid": false + }, + { + "description": "non-ascii digits (BENGALI DIGIT FOUR, BENGALI DIGIT TWO)", + "data": { "৪২": "khajit has wares if you have coin" }, + "valid": false + } + ] + }, + { + "description": "patternProperties with non-ASCII digits", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "patternProperties": { + "^\\p{digit}+$": true + }, + "additionalProperties": false + }, + "tests": [ + { + "description": "ascii digits", + "data": { "42": "life, the universe, and everything" }, + "valid": true + }, + { + "description": "ascii non-digits", + "data": { "-%#": "spending the year dead for tax reasons" }, + "valid": false + }, + { + "description": "non-ascii digits (BENGALI DIGIT FOUR, BENGALI DIGIT TWO)", + "data": { "৪২": "khajit has wares if you have coin" }, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/float-overflow.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/float-overflow.json new file mode 100644 index 000000000..f5ae8b18f --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/float-overflow.json @@ -0,0 +1,17 @@ +[ + { + "description": "all integers are multiples of 0.5, if overflow is handled", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "integer", + "multipleOf": 0.5 + }, + "tests": [ + { + "description": "valid if optional overflow handling is implemented", + "data": 1e308, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format-assertion.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format-assertion.json new file mode 100644 index 000000000..96f26fefb --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format-assertion.json @@ -0,0 +1,44 @@ +[ + { + "description": "schema that uses custom metaschema with format-assertion: false", + "comment": "The true/false boolean in $vocabulary only controls behavior for implementations that do not recognize the vocabulary. For implementations that do understand format-assertion, the boolean has no impact on validation behavior — hence both test groups produce identical results.", + "schema": { + "$id": "https://schema/using/format-assertion/false", + "$schema": "http://localhost:1234/draft2020-12/format-assertion-false.json", + "format": "ipv4" + }, + "tests": [ + { + "description": "format-assertion: false: valid string", + "data": "127.0.0.1", + "valid": true + }, + { + "description": "format-assertion: false: invalid string", + "comment": "valid: false is intentional — the false value in $vocabulary only affects unknown vocabulary handling, not validation behavior itself", + "data": "not-an-ipv4", + "valid": false + } + ] + }, + { + "description": "schema that uses custom metaschema with format-assertion: true", + "schema": { + "$id": "https://schema/using/format-assertion/true", + "$schema": "http://localhost:1234/draft2020-12/format-assertion-true.json", + "format": "ipv4" + }, + "tests": [ + { + "description": "format-assertion: true: valid string", + "data": "127.0.0.1", + "valid": true + }, + { + "description": "format-assertion: true: invalid string", + "data": "not-an-ipv4", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/date-time.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/date-time.json new file mode 100644 index 000000000..590f2f678 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/date-time.json @@ -0,0 +1,250 @@ +[ + { + "description": "validation of date-time strings", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "date-time" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid date-time string", + "data": "1963-06-19T08:30:06.283185Z", + "valid": true + }, + { + "description": "a valid date-time string without second fraction", + "data": "1963-06-19T08:30:06Z", + "valid": true + }, + { + "description": "a valid date-time string with plus offset", + "data": "1937-01-01T12:00:27.87+00:20", + "valid": true + }, + { + "description": "a valid date-time string with minus offset", + "data": "1990-12-31T15:59:50.123-08:00", + "valid": true + }, + { + "description": "a valid date-time with a leap second, UTC", + "data": "1998-12-31T23:59:60Z", + "valid": true + }, + { + "description": "a valid date-time with a leap second, with minus offset", + "data": "1998-12-31T15:59:60.123-08:00", + "valid": true + }, + { + "description": "an invalid date-time past leap second, UTC", + "data": "1998-12-31T23:59:61Z", + "valid": false + }, + { + "description": "an invalid date-time with leap second on a wrong minute, UTC", + "data": "1998-12-31T23:58:60Z", + "valid": false + }, + { + "description": "an invalid date-time with leap second on a wrong hour, UTC", + "data": "1998-12-31T22:59:60Z", + "valid": false + }, + { + "description": "an invalid day in date-time string", + "data": "1990-02-31T15:59:59.123-08:00", + "valid": false + }, + { + "description": "an invalid offset in date-time string", + "data": "1990-12-31T15:59:59-24:00", + "valid": false + }, + { + "description": "an invalid closing Z after time-zone offset", + "data": "1963-06-19T08:30:06.28123+01:00Z", + "valid": false + }, + { + "description": "an invalid hour in date-time string", + "data": "1990-12-31T24:00:00Z", + "valid": false + }, + { + "description": "an invalid minute in date-time string", + "data": "1990-12-31T15:60:00Z", + "valid": false + }, + { + "description": "an invalid offset minute in date-time string", + "data": "1990-12-31T10:00:00+10:60", + "valid": false + }, + { + "description": "an invalid date-time string", + "data": "06/19/1963 08:30:06 PST", + "valid": false + }, + { + "description": "case-insensitive T and Z", + "data": "1963-06-19t08:30:06.283185z", + "valid": true + }, + { + "description": "only RFC3339 not all of ISO 8601 are valid", + "data": "2013-350T01:01:01", + "valid": false + }, + { + "description": "invalid non-padded month dates", + "data": "1963-6-19T08:30:06.283185Z", + "valid": false + }, + { + "description": "invalid non-padded day dates", + "data": "1963-06-1T08:30:06.283185Z", + "valid": false + }, + { + "description": "invalid non-ASCII '৪' (a Bengali 4) in date portion", + "data": "1963-06-1৪T00:00:00Z", + "valid": false + }, + { + "description": "invalid non-ASCII '৪' (a Bengali 4) in time portion", + "data": "1963-06-11T0৪:00:00Z", + "valid": false + }, + { + "description": "invalid extended year", + "data": "+11963-06-19T08:30:06.283185Z", + "valid": false + }, + { + "description": "a numeric offset without minutes is invalid", + "data": "1985-04-12T23:20:50+01", + "valid": false + }, + { + "description": "hour 24 is invalid even with a leap second", + "data": "2016-12-31T24:59:60+01:00", + "valid": false + }, + { + "description": "a second fraction of fifteen nines is valid", + "data": "1985-04-12T00:59:59.999999999999999Z", + "valid": true + }, + { + "description": "a trailing newline is invalid", + "data": "1985-04-12T23:20:50Z\n", + "valid": false + }, + { + "description": "an invalid date-time string without seconds", + "data": "1985-04-12T23:20Z", + "valid": false + }, + { + "description": "an invalid date-time string with trailing content after the offset", + "data": "1985-04-12T23:20:50Ztail", + "valid": false + }, + { + "description": "an invalid minute in date-time string, with a numeric offset", + "data": "1985-04-12T23:60:00+00:01", + "valid": false + }, + { + "description": "a valid date-time string with 28 days in February (normal)", + "data": "2021-02-28T00:00:00Z", + "valid": true + }, + { + "description": "an invalid date-time string with 30 days in February (leap)", + "data": "2020-02-30T00:00:00Z", + "valid": false + }, + { + "description": "2021 is not a leap year", + "data": "2021-02-29T00:00:00Z", + "valid": false + }, + { + "description": "2020 is a leap year", + "data": "2020-02-29T00:00:00Z", + "valid": true + }, + { + "description": "century year 0100 is not a leap year", + "data": "0100-02-29T00:00:00Z", + "valid": false + }, + { + "description": "century year 0400 is a leap year", + "data": "0400-02-29T00:00:00Z", + "valid": true + }, + { + "description": "century year 2100 is not a leap year", + "data": "2100-02-29T00:00:00Z", + "valid": false + }, + { + "description": "a second fraction with no digits", + "comment": "RFC 3339 section 5.6: time-secfrac = \".\" 1*DIGIT requires at least one digit", + "data": "1985-04-12T23:20:50.Z", + "valid": false + }, + { + "description": "a comma decimal separator with a time-offset", + "comment": "RFC 3339 section 5.6: the decimal separator in time-secfrac is \".\" - the ISO 8601 comma is not part of this profile", + "data": "1985-04-12T23:20:50,5Z", + "valid": false + }, + { + "description": "leading whitespace before a valid date-time", + "comment": "RFC 3339 section 5.6: no production of date-time emits whitespace", + "data": " 1985-04-12T23:20:50Z", + "valid": false + }, + { + "description": "trailing whitespace after a valid date-time", + "comment": "RFC 3339 section 5.6: no production of date-time emits whitespace", + "data": "1985-04-12T23:20:50Z ", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/date.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/date.json new file mode 100644 index 000000000..a8a19261a --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/date.json @@ -0,0 +1,423 @@ +[ + { + "description": "validation of date strings", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "date" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid date string", + "data": "1963-06-19", + "valid": true + }, + { + "description": "a valid date string with 31 days in January", + "data": "2020-01-31", + "valid": true + }, + { + "description": "a invalid date string with 32 days in January", + "data": "2020-01-32", + "valid": false + }, + { + "description": "a valid date string with 28 days in February (normal)", + "data": "2021-02-28", + "valid": true + }, + { + "description": "a invalid date string with 30 days in February (leap)", + "data": "2020-02-30", + "valid": false + }, + { + "description": "a valid date string with 31 days in March", + "data": "2020-03-31", + "valid": true + }, + { + "description": "a invalid date string with 32 days in March", + "data": "2020-03-32", + "valid": false + }, + { + "description": "a valid date string with 30 days in April", + "data": "2020-04-30", + "valid": true + }, + { + "description": "a invalid date string with 31 days in April", + "data": "2020-04-31", + "valid": false + }, + { + "description": "a valid date string with 31 days in May", + "data": "2020-05-31", + "valid": true + }, + { + "description": "a invalid date string with 32 days in May", + "data": "2020-05-32", + "valid": false + }, + { + "description": "a valid date string with 30 days in June", + "data": "2020-06-30", + "valid": true + }, + { + "description": "a invalid date string with 31 days in June", + "data": "2020-06-31", + "valid": false + }, + { + "description": "a valid date string with 31 days in July", + "data": "2020-07-31", + "valid": true + }, + { + "description": "a invalid date string with 32 days in July", + "data": "2020-07-32", + "valid": false + }, + { + "description": "a valid date string with 31 days in August", + "data": "2020-08-31", + "valid": true + }, + { + "description": "a invalid date string with 32 days in August", + "data": "2020-08-32", + "valid": false + }, + { + "description": "a valid date string with 30 days in September", + "data": "2020-09-30", + "valid": true + }, + { + "description": "a invalid date string with 31 days in September", + "data": "2020-09-31", + "valid": false + }, + { + "description": "a valid date string with 31 days in October", + "data": "2020-10-31", + "valid": true + }, + { + "description": "a invalid date string with 32 days in October", + "data": "2020-10-32", + "valid": false + }, + { + "description": "a valid date string with 30 days in November", + "data": "2020-11-30", + "valid": true + }, + { + "description": "a invalid date string with 31 days in November", + "data": "2020-11-31", + "valid": false + }, + { + "description": "a valid date string with 31 days in December", + "data": "2020-12-31", + "valid": true + }, + { + "description": "a invalid date string with 32 days in December", + "data": "2020-12-32", + "valid": false + }, + { + "description": "an invalid date string", + "data": "06/19/1963", + "valid": false + }, + { + "description": "only RFC3339 not all of ISO 8601 are valid", + "data": "2013-350", + "valid": false + }, + { + "description": "non-padded month dates are not valid", + "data": "1998-1-20", + "valid": false + }, + { + "description": "non-padded day dates are not valid", + "data": "1998-01-1", + "valid": false + }, + { + "description": "invalid month", + "data": "1998-13-01", + "valid": false + }, + { + "description": "2021 is not a leap year", + "data": "2021-02-29", + "valid": false + }, + { + "description": "2020 is a leap year", + "data": "2020-02-29", + "valid": true + }, + { + "description": "invalid non-ASCII '৪' (a Bengali 4)", + "data": "1963-06-1৪", + "valid": false + }, + { + "description": "invalid: non-ASCII Bengali digit in month field", + "data": "2020-0৪-01", + "valid": false + }, + { + "description": "ISO8601 / non-RFC3339: YYYYMMDD without dashes (2023-03-28)", + "data": "20230328", + "valid": false + }, + { + "description": "ISO8601 / non-RFC3339: week number implicit day of week (2023-01-02)", + "data": "2023-W01", + "valid": false + }, + { + "description": "ISO8601 / non-RFC3339: week number with day of week (2023-03-28)", + "data": "2023-W13-2", + "valid": false + }, + { + "description": "ISO8601 / non-RFC3339: week number rollover to next year (2023-01-01)", + "data": "2022W527", + "valid": false + }, + { + "description": "an invalid time string in date-time format", + "data": "2020-11-28T23:55:45Z", + "valid": false + }, + { + "description": "century year 0100 is not a leap year", + "comment": "https://www.rfc-editor.org/rfc/rfc3339#appendix-C — 100 % 100 == 0 but 100 % 400 != 0", + "data": "0100-02-29", + "valid": false + }, + { + "description": "century year 0400 is a leap year", + "comment": "https://www.rfc-editor.org/rfc/rfc3339#appendix-C — 400-year cycle boundary", + "data": "0400-02-29", + "valid": true + }, + { + "description": "century year 2100 is not a leap year", + "comment": "https://www.rfc-editor.org/rfc/rfc3339#appendix-C — future century year", + "data": "2100-02-29", + "valid": false + }, + { + "description": "invalid: leading whitespace is not permitted", + "comment": "https://www.rfc-editor.org/rfc/rfc3339#section-5.6 — full-date grammar does not include whitespace", + "data": " 2024-01-15", + "valid": false + }, + { + "description": "invalid: trailing whitespace is not permitted", + "comment": "https://www.rfc-editor.org/rfc/rfc3339#section-5.6 — full-date grammar does not include whitespace", + "data": "2024-01-15 ", + "valid": false + }, + { + "description": "invalid: month 00 is not valid per date-month range 01-12", + "comment": "https://www.rfc-editor.org/rfc/rfc3339#section-5.6 — date-month = 2DIGIT ; 01-12", + "data": "2024-00-15", + "valid": false + }, + { + "description": "invalid: day 00 is not valid per date-mday minimum of 01", + "comment": "https://www.rfc-editor.org/rfc/rfc3339#section-5.6 — date-mday = 2DIGIT ; 01-28/29/30/31", + "data": "2024-01-00", + "valid": false + }, + { + "description": "invalid: empty string", + "data": "", + "valid": false + }, + { + "description": "invalid: embedded whitespace between year and month", + "data": "2020 -01-01", + "valid": false + }, + { + "description": "invalid: trailing character after valid full-date", + "data": "2020-01-01X", + "valid": false + }, + { + "description": "invalid: trailing Z after full-date", + "data": "2020-01-01Z", + "valid": false + }, + { + "description": "invalid: full-date followed by space and time component", + "data": "2020-01-01 00:00:00Z", + "valid": false + }, + { + "description": "valid: four-digit year 0001", + "data": "0001-01-01", + "valid": true + }, + { + "description": "invalid: two-digit year (N-2 digits)", + "data": "20-01-01", + "valid": false + }, + { + "description": "invalid: three-digit year (N-1 digits)", + "data": "998-01-01", + "valid": false + }, + { + "description": "invalid: five-digit year (N+1 digits)", + "data": "12020-01-01", + "valid": false + }, + { + "description": "invalid: positive sign prefix on year", + "data": "+2020-01-01", + "valid": false + }, + { + "description": "invalid: negative sign prefix on year", + "data": "-2020-01-01", + "valid": false + }, + { + "description": "invalid: non-ASCII Bengali digit in year field", + "data": "২020-01-01", + "valid": false + }, + { + "description": "invalid: alphabetic characters in year field", + "data": "YYYY-01-01", + "valid": false + }, + { + "description": "invalid: three-digit month (N+1 digits)", + "data": "2020-001-01", + "valid": false + }, + { + "description": "invalid: alphabetic characters in month field", + "data": "2020-MM-01", + "valid": false + }, + { + "description": "invalid: three-digit day (N+1 digits)", + "data": "2020-01-001", + "valid": false + }, + { + "description": "invalid: alphabetic characters in day field", + "data": "2020-01-DD", + "valid": false + }, + { + "description": "invalid: colon separators", + "data": "2020:01:01", + "valid": false + }, + { + "description": "invalid: dot separators", + "data": "2020.01.01", + "valid": false + }, + { + "description": "invalid: space separators", + "data": "2020 01 01", + "valid": false + }, + { + "description": "invalid: mixed slash and hyphen separators", + "data": "2020-01/01", + "valid": false + }, + { + "description": "invalid: duplicated first hyphen", + "data": "2020--01-01", + "valid": false + }, + { + "description": "invalid: duplicated second hyphen", + "data": "2020-01--01", + "valid": false + }, + { + "description": "valid: date inside the Julian to Gregorian reform gap", + "data": "1582-10-10", + "valid": true + }, + { + "description": "invalid: year field numerically larger than a 32-bit signed integer", + "data": "2147483648-01-01", + "valid": false + }, + { + "description": "invalid: character just above '9' as the second digit of the day", + "data": "2020-01-0:", + "valid": false + }, + { + "description": "invalid: non-ASCII digit leaving ten UTF-8 bytes but eight characters", + "data": "\u09e80-01-01", + "valid": false + }, + { + "description": "invalid: en dash separators instead of hyphen-minus", + "data": "2020\u201301\u201301", + "valid": false + }, + { + "description": "invalid: NUL character after an otherwise valid full-date", + "data": "2020-01-01\u0000", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/duration.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/duration.json new file mode 100644 index 000000000..e3e184161 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/duration.json @@ -0,0 +1,268 @@ +[ + { + "description": "validation of duration strings", + "comment": "RFC 3339 Appendix A defines the ABNF grammar for ISO-8601 durations used by JSON Schema format 'duration'. These tests enforce only the syntax defined by that grammar.", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "duration" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid duration string", + "data": "P4DT12H30M5S", + "valid": true + }, + { + "description": "an invalid duration string", + "data": "PT1D", + "valid": false + }, + { + "description": "must start with P", + "data": "4DT12H30M5S", + "valid": false + }, + { + "description": "no elements present", + "data": "P", + "valid": false + }, + { + "description": "no time elements present", + "data": "P1YT", + "valid": false + }, + { + "description": "no date or time elements present", + "data": "PT", + "valid": false + }, + { + "description": "elements out of order", + "data": "P2D1Y", + "valid": false + }, + { + "description": "missing time separator", + "data": "P1D2H", + "valid": false + }, + { + "description": "time element in the date position", + "data": "P2S", + "valid": false + }, + { + "description": "four years duration", + "data": "P4Y", + "valid": true + }, + { + "description": "zero time, in seconds", + "data": "PT0S", + "valid": true + }, + { + "description": "zero time, in days", + "data": "P0D", + "valid": true + }, + { + "description": "one month duration", + "data": "P1M", + "valid": true + }, + { + "description": "one minute duration", + "data": "PT1M", + "valid": true + }, + { + "description": "one and a half days, in hours", + "data": "PT36H", + "valid": true + }, + { + "description": "one and a half days, in days and hours", + "data": "P1DT12H", + "valid": true + }, + { + "description": "two weeks", + "data": "P2W", + "valid": true + }, + { + "description": "weeks cannot be combined with other units", + "data": "P1Y2W", + "valid": false + }, + { + "description": "invalid non-ASCII '২' (a Bengali 2)", + "data": "P২Y", + "valid": false + }, + { + "description": "element without unit", + "data": "P1", + "valid": false + }, + { + "description": "all date and time components", + "data": "P1Y2M3DT4H5M6S", + "valid": true + }, + { + "description": "date components only", + "data": "P1Y2M3D", + "valid": true + }, + { + "description": "time components only", + "data": "PT1H2M3S", + "valid": true + }, + { + "description": "month and day", + "data": "P1M2D", + "valid": true + }, + { + "description": "hour and minute", + "data": "PT1H30M", + "valid": true + }, + { + "description": "multi-digit values in all components", + "data": "P10Y10M10DT10H10M10S", + "valid": true + }, + { + "description": "fractional duration is not allowed by RFC 3339 ABNF", + "comment": "numeric components use 1*DIGIT where DIGIT = %x30-39; '.' is not allowed", + "data": "PT0.5S", + "valid": false + }, + { + "description": "leading whitespace is invalid", + "data": " P1D", + "valid": false + }, + { + "description": "trailing whitespace is invalid", + "data": "P1D ", + "valid": false + }, + { + "description": "empty string is invalid", + "data": "", + "valid": false + }, + { + "description": "years and months can appear without days", + "data": "P1Y2M", + "valid": true + }, + { + "description": "years and days cannot appear without months", + "data": "P1Y2D", + "valid": false + }, + { + "description": "hours and minutes can appear without seconds", + "data": "PT1H2M", + "valid": true + }, + { + "description": "hours and seconds cannot appear without minutes", + "data": "PT1H2S", + "valid": false + }, + { + "description": "minutes and seconds can appear without hour", + "data": "PT1M2S", + "valid": true + }, + { + "description": "a leading sign is not allowed", + "data": "-P1D", + "valid": false + }, + { + "description": "a number before the time separator has no unit", + "data": "P1D2T3H", + "valid": false + }, + { + "description": "exponent notation is not allowed in a component", + "data": "P1e2D", + "valid": false + }, + { + "description": "a trailing newline is invalid", + "data": "P1D\n", + "valid": false + }, + { + "description": "weeks cannot be combined with a time component", + "data": "P1WT1H", + "valid": false + }, + { + "description": "weeks cannot be combined with a zero-valued component", + "data": "P0Y1W", + "valid": false + }, + { + "description": "a leading zero in a component is valid", + "data": "P01D", + "valid": true + }, + { + "description": "a component with many digits is valid", + "data": "P999999999999999999999999999999999999999999999999999999999999999999999999999999D", + "valid": true + }, + { + "description": "a comma as the decimal separator is invalid", + "data": "PT0,5S", + "valid": false + }, + { + "description": "a sign inside a component is invalid", + "data": "P-1D", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/ecmascript-regex.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/ecmascript-regex.json new file mode 100644 index 000000000..b9ae3c049 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/ecmascript-regex.json @@ -0,0 +1,116 @@ +[ + { + "description": "\\a is not an ECMA 262 control escape", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "regex" + }, + "tests": [ + { + "description": "when used as a pattern", + "data": "\\a", + "valid": false + } + ] + }, + { + "description": "Python-specific regular expression syntax is not valid ECMA 262", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "regex" + }, + "tests": [ + { + "description": "Python named group (?P...) is not ECMA 262", + "data": "(?Px)", + "valid": false + }, + { + "description": "Python named backreference (?P=name) is not ECMA 262", + "data": "(?Pa)(?P=n)", + "valid": false + }, + { + "description": "inline comment group (?#...) is not ECMA 262", + "data": "(?#comment)a", + "valid": false + } + ] + }, + { + "description": "global inline flag groups are not valid ECMA 262", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "regex" + }, + "tests": [ + { + "description": "a single global inline flag (?i)", + "data": "(?i)abc", + "valid": false + }, + { + "description": "multiple global inline flags (?ims)", + "data": "(?ims)abc", + "valid": false + } + ] + }, + { + "description": "ECMA 262 named groups and backreferences are valid", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "regex" + }, + "tests": [ + { + "description": "an ECMA 262 named group (?...)", + "data": "(?x)", + "valid": true + }, + { + "description": "an ECMA 262 named backreference \\k", + "data": "(?a)\\k", + "valid": true + } + ] + }, + { + "description": "ECMA 262 lookbehind is valid, including variable width", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "regex" + }, + "tests": [ + { + "description": "a variable-width lookbehind (ES2018)", + "data": "(?<=a+)b", + "valid": true + } + ] + }, + { + "description": "ECMA 262 character classes and escapes", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "regex" + }, + "tests": [ + { + "description": "an empty character class is valid ECMA 262", + "data": "[]", + "valid": true + }, + { + "description": "a negated empty character class is valid ECMA 262", + "data": "[^]", + "valid": true + }, + { + "description": "a control escape \\cA is valid ECMA 262", + "data": "\\cA", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/email.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/email.json new file mode 100644 index 000000000..50e214622 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/email.json @@ -0,0 +1,368 @@ +[ + { + "description": "validation of e-mail addresses", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "email" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid e-mail address", + "data": "joe.bloggs@example.com", + "valid": true + }, + { + "description": "an invalid e-mail address", + "data": "2962", + "valid": false + }, + { + "description": "tilde in local part is valid", + "data": "te~st@example.com", + "valid": true + }, + { + "description": "tilde before local part is valid", + "data": "~test@example.com", + "valid": true + }, + { + "description": "tilde after local part is valid", + "data": "test~@example.com", + "valid": true + }, + { + "description": "a quoted string with a space in the local part is valid", + "data": "\"joe bloggs\"@example.com", + "valid": true + }, + { + "description": "a quoted string with a double dot in the local part is valid", + "data": "\"joe..bloggs\"@example.com", + "valid": true + }, + { + "description": "a quoted string with a @ in the local part is valid", + "data": "\"joe@bloggs\"@example.com", + "valid": true + }, + { + "description": "an IPv4-address-literal after the @ is valid", + "data": "joe.bloggs@[127.0.0.1]", + "valid": true + }, + { + "description": "an IPv6-address-literal after the @ is valid", + "data": "joe.bloggs@[IPv6:::1]", + "valid": true + }, + { + "description": "dot before local part is not valid", + "data": ".test@example.com", + "valid": false + }, + { + "description": "dot after local part is not valid", + "data": "test.@example.com", + "valid": false + }, + { + "description": "two separated dots inside local part are valid", + "data": "te.s.t@example.com", + "valid": true + }, + { + "description": "two subsequent dots inside local part are not valid", + "data": "te..st@example.com", + "valid": false + }, + { + "description": "an invalid domain", + "data": "joe.bloggs@invalid=domain.com", + "valid": false + }, + { + "description": "an invalid IPv4-address-literal", + "data": "joe.bloggs@[127.0.0.300]", + "valid": false + }, + { + "description": "two email addresses is not valid", + "data": "user1@oceania.org, user2@oceania.org", + "valid": false + }, + { + "description": "full \"From\" header is invalid", + "data": "\"Winston Smith\" (Records Department)", + "valid": false + }, + { + "description": "local part is required", + "data": "@example.com", + "valid": false + }, + { + "description": "domain is required", + "data": "joe.bloggs@", + "valid": false + }, + { + "description": "unquoted space in local part is invalid", + "data": "joe bloggs@example.com", + "valid": false + }, + { + "description": "a domain that looks like an IPv4 address without brackets is valid", + "data": "test@255.255.255.255", + "valid": true + }, + { + "description": "an empty quoted string in the local part is valid", + "data": "\"\"@iana.org", + "valid": true + }, + { + "description": "a quoted string containing only a space in the local part is valid", + "data": "\" \"@iana.org", + "valid": true + }, + { + "description": "consecutive hyphens inside a domain label are valid", + "data": "test@c--n.com", + "valid": true + }, + { + "description": "a domain label starting with a digit is valid", + "data": "test@123.com", + "valid": true + }, + { + "description": "a non-ASCII character in the local part is not valid", + "data": "a\u00e9@iana.org", + "valid": false + }, + { + "description": "a non-ASCII character in the domain is not valid", + "data": "a@\u00e9.org", + "valid": false + }, + { + "description": "a fullwidth commercial at is not a valid separator", + "data": "a\uff20iana.org", + "valid": false + }, + { + "description": "a non-ASCII character in a quoted pair is not valid", + "data": "\"test\\\u00a9\"@iana.org", + "valid": false + }, + { + "description": "a trailing line feed is not valid", + "data": "test@iana.org\n", + "valid": false + }, + { + "description": "a trailing carriage return is not valid", + "data": "test@iana.org\r", + "valid": false + }, + { + "description": "a trailing space is not valid", + "data": "a@iana.org ", + "valid": false + }, + { + "description": "a leading space is not valid", + "data": " a@iana.org", + "valid": false + }, + { + "description": "a tab in the local part is not valid", + "data": "a\tb@iana.org", + "valid": false + }, + { + "description": "a delete character in the local part is not valid", + "data": "\u007f@iana.org", + "valid": false + }, + { + "description": "a quoted string with no special characters in the local part is valid", + "data": "\"test\"@iana.org", + "valid": true + }, + { + "description": "a quoted pair in the local part is valid", + "data": "\"\\a\"@iana.org", + "valid": true + }, + { + "description": "an escaped double quote in the local part is valid", + "data": "\"\\\"\"@iana.org", + "valid": true + }, + { + "description": "an escaped backslash in the local part is valid", + "data": "\"\\\\\"@iana.org", + "valid": true + }, + { + "description": "an unescaped double quote in an unquoted local part is not valid", + "data": "test\"@iana.org", + "valid": false + }, + { + "description": "text after the closing double quote in the local part is not valid", + "data": "\"test\"test@iana.org", + "valid": false + }, + { + "description": "an unclosed double quote in the local part is not valid", + "data": "\"test@iana.org", + "valid": false + }, + { + "description": "two unquoted @ signs are not valid", + "data": "a@b@c.org", + "valid": false + }, + { + "description": "a backslash-escaped @ in an unquoted local part is not valid", + "data": "test\\@test@iana.org", + "valid": false + }, + { + "description": "an address literal in the local part position is not valid", + "data": "[1.2.3.4]@iana.org", + "valid": false + }, + { + "description": "an empty string is not valid", + "data": "", + "valid": false + }, + { + "description": "a domain label starting with a hyphen is not valid", + "data": "test@-iana.org", + "valid": false + }, + { + "description": "a domain label ending with a hyphen is not valid", + "data": "test@iana-.com", + "valid": false + }, + { + "description": "a domain starting with a dot is not valid", + "data": "test@.iana.org", + "valid": false + }, + { + "description": "a domain ending with a dot is not valid", + "data": "test@iana.org.", + "valid": false + }, + { + "description": "two subsequent dots inside the domain are not valid", + "data": "test@iana..com", + "valid": false + }, + { + "description": "a single-label domain is valid", + "data": "test@io", + "valid": true + }, + { + "description": "an unclosed address literal is not valid", + "data": "test@[1.2.3.4", + "valid": false + }, + { + "description": "empty brackets after the @ are not valid", + "data": "a@[]", + "valid": false + }, + { + "description": "an IPv4-address-literal with three octets is not valid", + "data": "a@[1.2.3]", + "valid": false + }, + { + "description": "an IPv4-address-literal with five octets is not valid", + "data": "a@[1.2.3.4.5]", + "valid": false + }, + { + "description": "bracketed content that is not an address is not valid", + "data": "test@[RFC-5322-domain-literal]", + "valid": false + }, + { + "description": "a domain label before the opening bracket is not valid", + "data": "test@a[255.255.255.255]", + "valid": false + }, + { + "description": "a lowercase IPv6 tag in an address literal is valid", + "data": "a@[ipv6:::1]", + "valid": true + }, + { + "description": "a comma in an unquoted local part is not valid", + "data": "a,b@iana.org", + "valid": false + }, + { + "description": "a semicolon in an unquoted local part is not valid", + "data": "a;b@iana.org", + "valid": false + }, + { + "description": "a colon in an unquoted local part is not valid", + "data": "a:b@iana.org", + "valid": false + }, + { + "description": "a parenthesis in an unquoted local part is not valid", + "comment": "a comment is RFC 5322 syntax, which the RFC 5321 Mailbox grammar does not have", + "data": "test(comment)@iana.org", + "valid": false + }, + { + "description": "a parenthesis in the domain is not valid", + "comment": "a comment is RFC 5322 syntax, which the RFC 5321 Mailbox grammar does not have", + "data": "a@b(c).org", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/hostname.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/hostname.json new file mode 100644 index 000000000..7cacfcfd2 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/hostname.json @@ -0,0 +1,386 @@ +[ + { + "description": "validation of host names", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "hostname" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid host name", + "data": "www.example.com", + "valid": true + }, + { + "description": "single label", + "data": "hostname", + "valid": true + }, + { + "description": "single label with digits", + "data": "h0stn4me", + "valid": true + }, + { + "description": "single label starting with digit", + "data": "1host", + "valid": true + }, + { + "description": "single label ending with digit", + "data": "hostnam3", + "valid": true + }, + { + "description": "empty string", + "data": "", + "valid": false + }, + { + "description": "single dot", + "data": ".", + "valid": false + }, + { + "description": "leading dot", + "data": ".example", + "valid": false + }, + { + "description": "trailing dot", + "data": "example.", + "valid": false + }, + { + "description": "IDN label separator", + "data": "example\uff0ecom", + "valid": false + }, + { + "description": "single label with hyphen", + "data": "host-name", + "valid": true + }, + { + "description": "starts with hyphen", + "data": "-hostname", + "valid": false + }, + { + "description": "ends with hyphen", + "data": "hostname-", + "valid": false + }, + { + "description": "contains underscore", + "data": "host_name", + "valid": false + }, + { + "description": "exceeds maximum overall length (256)", + "data": "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk.abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk.abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk.abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk.com", + "valid": false + }, + { + "description": "maximum label length (63)", + "data": "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk.com", + "valid": true + }, + { + "description": "exceeds maximum label length (63)", + "data": "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijkl.com", + "valid": false + }, + { + "description": "trailing newline is invalid", + "data": "example.com\n", + "valid": false + }, + { + "description": "leading space is invalid", + "data": " example.com", + "valid": false + }, + { + "description": "space inside a label is invalid", + "data": "ex ample.com", + "valid": false + }, + { + "description": "invalid non-ASCII KELVIN SIGN (U+212A)", + "data": "\u212Aelvin.example.com", + "valid": false + }, + { + "description": "consecutive hyphens inside a label", + "data": "a--b.com", + "valid": true + } + ] + }, + { + "description": "validation of A-label (punycode) host names", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "hostname" + }, + "tests": [ + { + "description": "invalid Punycode", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.4 https://tools.ietf.org/html/rfc5890#section-2.3.2.1", + "data": "xn--X", + "valid": false + }, + { + "description": "a valid host name (example.test in Hangul)", + "data": "xn--9n2bp8q.xn--9t4b11yi5a", + "valid": true + }, + { + "description": "contains illegal char U+302E Hangul single dot tone mark", + "data": "xn--07jt112bpxg.xn--9t4b11yi5a", + "valid": false + }, + { + "description": "Begins with a Spacing Combining Mark", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.2", + "data": "xn--hello-txk", + "valid": false + }, + { + "description": "Begins with a Nonspacing Mark", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.2", + "data": "xn--hello-zed", + "valid": false + }, + { + "description": "Begins with an Enclosing Mark", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.2", + "data": "xn--hello-6bf", + "valid": false + }, + { + "description": "Exceptions that are PVALID, left-to-right chars", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.2 https://tools.ietf.org/html/rfc5892#section-2.6", + "data": "xn--zca29lwxobi7a", + "valid": true + }, + { + "description": "Exceptions that are PVALID, right-to-left chars", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.2 https://tools.ietf.org/html/rfc5892#section-2.6", + "data": "xn--qmbc", + "valid": true + }, + { + "description": "Exceptions that are DISALLOWED, right-to-left chars", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.2 https://tools.ietf.org/html/rfc5892#section-2.6", + "data": "xn--chb89f", + "valid": false + }, + { + "description": "Exceptions that are DISALLOWED, left-to-right chars", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.2 https://tools.ietf.org/html/rfc5892#section-2.6 Note: The two combining marks (U+302E and U+302F) are in the middle and not at the start", + "data": "xn--07jceefgh4c", + "valid": false + }, + { + "description": "MIDDLE DOT with no preceding 'l'", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "xn--al-0ea", + "valid": false + }, + { + "description": "MIDDLE DOT with nothing preceding", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "xn--l-fda", + "valid": false + }, + { + "description": "MIDDLE DOT with no following 'l'", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "xn--la-0ea", + "valid": false + }, + { + "description": "MIDDLE DOT with nothing following", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "xn--l-gda", + "valid": false + }, + { + "description": "MIDDLE DOT with surrounding 'l's", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "xn--ll-0ea", + "valid": true + }, + { + "description": "Greek KERAIA not followed by Greek", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.4", + "data": "xn--S-jib3p", + "valid": false + }, + { + "description": "Greek KERAIA not followed by anything", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.4", + "data": "xn--wva3j", + "valid": false + }, + { + "description": "Greek KERAIA followed by Greek", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.4", + "data": "xn--wva3je", + "valid": true + }, + { + "description": "Hebrew GERESH not preceded by Hebrew", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.5", + "data": "xn--A-2hc5h", + "valid": false + }, + { + "description": "Hebrew GERESH not preceded by anything", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.5", + "data": "xn--5db1e", + "valid": false + }, + { + "description": "Hebrew GERESH preceded by Hebrew", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.5", + "data": "xn--4dbc5h", + "valid": true + }, + { + "description": "Hebrew GERSHAYIM not preceded by Hebrew", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.6", + "data": "xn--A-2hc8h", + "valid": false + }, + { + "description": "Hebrew GERSHAYIM not preceded by anything", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.6", + "data": "xn--5db3e", + "valid": false + }, + { + "description": "Hebrew GERSHAYIM preceded by Hebrew", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.6", + "data": "xn--4dbc8h", + "valid": true + }, + { + "description": "KATAKANA MIDDLE DOT with no Hiragana, Katakana, or Han", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "xn--defabc-k64e", + "valid": false + }, + { + "description": "KATAKANA MIDDLE DOT with no other characters", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "xn--vek", + "valid": false + }, + { + "description": "KATAKANA MIDDLE DOT with Hiragana", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "xn--k8j5u", + "valid": true + }, + { + "description": "KATAKANA MIDDLE DOT with Katakana", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "xn--bck0j", + "valid": true + }, + { + "description": "KATAKANA MIDDLE DOT with Han", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "xn--vek778f", + "valid": true + }, + { + "description": "Arabic-Indic digits mixed with Extended Arabic-Indic digits", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.8", + "data": "xn--ngb6iyr", + "valid": false + }, + { + "description": "Arabic-Indic digits not mixed with Extended Arabic-Indic digits", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.8", + "data": "xn--ngba1o", + "valid": true + }, + { + "description": "Extended Arabic-Indic digits not mixed with Arabic-Indic digits", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.9", + "data": "xn--0-gyc", + "valid": true + }, + { + "description": "ZERO WIDTH JOINER not preceded by Virama", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.2 https://www.unicode.org/review/pr-37.pdf", + "data": "xn--11b2er09f", + "valid": false + }, + { + "description": "ZERO WIDTH JOINER not preceded by anything", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.2 https://www.unicode.org/review/pr-37.pdf", + "data": "xn--02b508i", + "valid": false + }, + { + "description": "ZERO WIDTH JOINER preceded by Virama", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.2 https://www.unicode.org/review/pr-37.pdf", + "data": "xn--11b2ezcw70k", + "valid": true + }, + { + "description": "ZERO WIDTH NON-JOINER preceded by Virama", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.1", + "data": "xn--11b2ezcs70k", + "valid": true + }, + { + "description": "ZERO WIDTH NON-JOINER not preceded by Virama but matches regexp", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.1 https://www.w3.org/TR/alreq/#h_disjoining_enforcement", + "data": "xn--ngba5hb2804a", + "valid": true + }, + { + "description": "contains \"--\" in the 3rd and 4th position", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.1 https://tools.ietf.org/html/rfc5890#section-2.3.2.1", + "data": "XN--aa---o47jg78q", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/idn-email.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/idn-email.json new file mode 100644 index 000000000..f46238b6f --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/idn-email.json @@ -0,0 +1,140 @@ +[ + { + "description": "validation of an internationalized e-mail addresses", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "idn-email" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid idn e-mail (example@example.test in Hangul)", + "data": "실례@실례.테스트", + "valid": true + }, + { + "description": "an invalid e-mail/idn e-mail address", + "data": "2962", + "valid": false + }, + { + "description": "a valid e-mail address", + "data": "joe.bloggs@example.com", + "valid": true + }, + { + "description": "a non-ASCII local part with an ASCII domain is valid", + "data": "δοκιμή@example.com", + "valid": true + }, + { + "description": "a non-ASCII quoted local part is valid", + "data": "\"δοκιμή\"@example.com", + "valid": true + }, + { + "description": "a domain label that is not in Unicode NFC is valid", + "data": "user@cafe\u0301.com", + "valid": true + }, + { + "description": "a local part that is not in Unicode NFC is valid", + "data": "cafe\u0301@example.com", + "valid": true + }, + { + "description": "a C1 control in the local part is valid", + "comment": "https://www.rfc-editor.org/rfc/rfc6531#section-3.3 atext =/ UTF8-non-ascii; https://www.rfc-editor.org/rfc/rfc3629#section-4 UTF8-2 admits U+0085", + "data": "\u0085@example.com", + "valid": true + }, + { + "description": "a noncharacter in the local part is valid", + "comment": "https://www.rfc-editor.org/rfc/rfc6531#section-3.3 atext =/ UTF8-non-ascii; https://www.rfc-editor.org/rfc/rfc3629#section-4 UTF8-3 admits U+FFFF", + "data": "\uffff@example.com", + "valid": true + }, + { + "description": "a local part at the 64-octet limit is valid", + "comment": "https://www.rfc-editor.org/rfc/rfc5321#section-4.5.3.1 every implementation must be able to receive a local part of 64 octets", + "data": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa@example.com", + "valid": true + }, + { + "description": "a fullwidth commercial at is not a local-part separator", + "comment": "https://www.rfc-editor.org/rfc/rfc5321#section-4.1.2 Mailbox = Local-part \"@\" ( Domain / address-literal ) splits on U+0040, and any IDN mapping applies only to the domain that split produces, so U+FF20 cannot become the delimiter", + "data": "user\uff20example.com", + "valid": false + }, + { + "description": "a local part with a supplementary-plane character is valid", + "comment": "https://www.rfc-editor.org/rfc/rfc6531#section-3.3 atext =/ UTF8-non-ascii, and https://www.rfc-editor.org/rfc/rfc3629#section-4 UTF8-4 covers U+10000 and above", + "data": "\ud835\udd4f@example.com", + "valid": true + }, + { + "description": "a domain label starting with a hyphen is not valid", + "comment": "https://www.rfc-editor.org/rfc/rfc5321#section-4.1.2 sub-domain = Let-dig [Ldh-str], so a label starts with a letter or digit; RFC 6531 section 3.3 only adds sub-domain =/ U-label", + "data": "δοκιμή@-example.com", + "valid": false + }, + { + "description": "a domain label ending with a hyphen is not valid", + "comment": "https://www.rfc-editor.org/rfc/rfc5321#section-4.1.2 Ldh-str = *( ALPHA / DIGIT / \"-\" ) Let-dig, so a label ends with a letter or digit", + "data": "δοκιμή@example-.com", + "valid": false + }, + { + "description": "a domain starting with a dot is not valid", + "comment": "https://www.rfc-editor.org/rfc/rfc5321#section-4.1.2 Domain = sub-domain *(\".\" sub-domain) admits no empty sub-domain", + "data": "δοκιμή@.example.com", + "valid": false + }, + { + "description": "a domain ending with a dot is not valid", + "data": "δοκιμή@example.com.", + "valid": false + }, + { + "description": "two subsequent dots inside the domain are not valid", + "data": "δοκιμή@example..com", + "valid": false + }, + { + "description": "domain is required", + "comment": "https://www.rfc-editor.org/rfc/rfc5321#section-4.1.2 Mailbox = Local-part \"@\" ( Domain / address-literal )", + "data": "δοκιμή@", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/idn-hostname.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/idn-hostname.json new file mode 100644 index 000000000..62f914d02 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/idn-hostname.json @@ -0,0 +1,588 @@ +[ + { + "description": "validation of internationalized host names", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "idn-hostname" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid host name (example.test in Hangul)", + "data": "실례.테스트", + "valid": true + }, + { + "description": "illegal first char U+302E Hangul single dot tone mark", + "data": "〮실례.테스트", + "valid": false + }, + { + "description": "contains illegal char U+302E Hangul single dot tone mark", + "data": "실〮례.테스트", + "valid": false + }, + { + "description": "a single label of 63 characters is valid", + "data": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "valid": true + }, + { + "description": "a single label of 64 characters is too long", + "data": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "valid": false + }, + { + "description": "invalid label, correct Punycode", + "comment": "https://tools.ietf.org/html/rfc5890#section-2.3.2.1 https://tools.ietf.org/html/rfc5891#section-4.4 https://tools.ietf.org/html/rfc3492#section-7.1", + "data": "-> $1.00 <--", + "valid": false + }, + { + "description": "valid Chinese Punycode", + "comment": "https://tools.ietf.org/html/rfc5890#section-2.3.2.1 https://tools.ietf.org/html/rfc5891#section-4.4", + "data": "xn--ihqwcrb4cv8a8dqg056pqjye", + "valid": true + }, + { + "description": "invalid Punycode", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.4 https://tools.ietf.org/html/rfc5890#section-2.3.2.1", + "data": "xn--X", + "valid": false + }, + { + "description": "U-label contains \"--\" in the 3rd and 4th position", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.1 https://tools.ietf.org/html/rfc5890#section-2.3.2.1", + "data": "XN--aa---o47jg78q", + "valid": false + }, + { + "description": "U-label starts with a dash", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.1", + "data": "-hello", + "valid": false + }, + { + "description": "U-label ends with a dash", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.1", + "data": "hello-", + "valid": false + }, + { + "description": "U-label starts and ends with a dash", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.1", + "data": "-hello-", + "valid": false + }, + { + "description": "Begins with a Spacing Combining Mark", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.2", + "data": "\u0903hello", + "valid": false + }, + { + "description": "Begins with a Nonspacing Mark", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.2", + "data": "\u0300hello", + "valid": false + }, + { + "description": "Begins with an Enclosing Mark", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.2", + "data": "\u0488hello", + "valid": false + }, + { + "description": "Exceptions that are PVALID, left-to-right chars", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.2 https://tools.ietf.org/html/rfc5892#section-2.6", + "data": "\u00df\u03c2\u0f0b\u3007", + "valid": true + }, + { + "description": "Exceptions that are PVALID, right-to-left chars", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.2 https://tools.ietf.org/html/rfc5892#section-2.6", + "data": "\u06fd\u06fe", + "valid": true + }, + { + "description": "Exceptions that are DISALLOWED, right-to-left chars", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.2 https://tools.ietf.org/html/rfc5892#section-2.6", + "data": "\u0640\u07fa", + "valid": false + }, + { + "description": "Exceptions that are DISALLOWED, left-to-right chars", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.2 https://tools.ietf.org/html/rfc5892#section-2.6 Note: The two combining marks (U+302E and U+302F) are in the middle and not at the start", + "data": "\u3031\u3032\u3033\u3034\u3035\u302e\u302f\u303b", + "valid": false + }, + { + "description": "MIDDLE DOT with no preceding 'l'", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "a\u00b7l", + "valid": false + }, + { + "description": "MIDDLE DOT with nothing preceding", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "\u00b7l", + "valid": false + }, + { + "description": "MIDDLE DOT with no following 'l'", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "l\u00b7a", + "valid": false + }, + { + "description": "MIDDLE DOT with nothing following", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "l\u00b7", + "valid": false + }, + { + "description": "MIDDLE DOT with surrounding 'l's", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "l\u00b7l", + "valid": true + }, + { + "description": "Greek KERAIA not followed by Greek", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.4", + "data": "\u03b1\u0375S", + "valid": false + }, + { + "description": "Greek KERAIA not followed by anything", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.4", + "data": "\u03b1\u0375", + "valid": false + }, + { + "description": "Greek KERAIA followed by Greek", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.4", + "data": "\u03b1\u0375\u03b2", + "valid": true + }, + { + "description": "Hebrew GERESH not preceded by Hebrew", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.5", + "data": "A\u05f3\u05d1", + "valid": false + }, + { + "description": "Hebrew GERESH not preceded by anything", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.5", + "data": "\u05f3\u05d1", + "valid": false + }, + { + "description": "Hebrew GERESH preceded by Hebrew", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.5", + "data": "\u05d0\u05f3\u05d1", + "valid": true + }, + { + "description": "Hebrew GERSHAYIM not preceded by Hebrew", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.6", + "data": "A\u05f4\u05d1", + "valid": false + }, + { + "description": "Hebrew GERSHAYIM not preceded by anything", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.6", + "data": "\u05f4\u05d1", + "valid": false + }, + { + "description": "Hebrew GERSHAYIM preceded by Hebrew", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.6", + "data": "\u05d0\u05f4\u05d1", + "valid": true + }, + { + "description": "KATAKANA MIDDLE DOT with no Hiragana, Katakana, or Han", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "def\u30fbabc", + "valid": false + }, + { + "description": "KATAKANA MIDDLE DOT with no other characters", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "\u30fb", + "valid": false + }, + { + "description": "KATAKANA MIDDLE DOT with Hiragana", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "\u30fb\u3041", + "valid": true + }, + { + "description": "KATAKANA MIDDLE DOT with Katakana", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "\u30fb\u30a1", + "valid": true + }, + { + "description": "KATAKANA MIDDLE DOT with Han", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "\u30fb\u4e08", + "valid": true + }, + { + "description": "Arabic-Indic digits mixed with Extended Arabic-Indic digits", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.8", + "data": "\u0628\u0660\u06f0", + "valid": false + }, + { + "description": "Arabic-Indic digits not mixed with Extended Arabic-Indic digits", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.8", + "data": "\u0628\u0660\u0628", + "valid": true + }, + { + "description": "Extended Arabic-Indic digits not mixed with Arabic-Indic digits", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.9", + "data": "\u06f00", + "valid": true + }, + { + "description": "ZERO WIDTH JOINER not preceded by Virama", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.2 https://www.unicode.org/review/pr-37.pdf", + "data": "\u0915\u200d\u0937", + "valid": false + }, + { + "description": "ZERO WIDTH JOINER not preceded by anything", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.2 https://www.unicode.org/review/pr-37.pdf", + "data": "\u200d\u0937", + "valid": false + }, + { + "description": "ZERO WIDTH JOINER preceded by Virama", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.2 https://www.unicode.org/review/pr-37.pdf", + "data": "\u0915\u094d\u200d\u0937", + "valid": true + }, + { + "description": "ZERO WIDTH NON-JOINER preceded by Virama", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.1", + "data": "\u0915\u094d\u200c\u0937", + "valid": true + }, + { + "description": "ZERO WIDTH NON-JOINER not preceded by Virama but matches regexp", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.1 https://www.w3.org/TR/alreq/#h_disjoining_enforcement", + "data": "\u0628\u064a\u200c\u0628\u064a", + "valid": true + }, + { + "description": "single label", + "data": "hostname", + "valid": true + }, + { + "description": "single label with hyphen", + "data": "host-name", + "valid": true + }, + { + "description": "single label with digits", + "data": "h0stn4me", + "valid": true + }, + { + "description": "single label starting with digit", + "data": "1host", + "valid": true + }, + { + "description": "single label ending with digit", + "data": "hostnam3", + "valid": true + }, + { + "description": "empty string", + "data": "", + "valid": false + }, + { + "description": "trailing newline is invalid", + "data": "example.com\n", + "valid": false + }, + { + "description": "leading space is invalid", + "data": " example.com", + "valid": false + }, + { + "description": "space inside a label is invalid", + "data": "ex ample.com", + "valid": false + }, + { + "description": "fullwidth digits are mapped to ASCII digits", + "comment": "IdnaMappingTable.txt maps U+FF11 to 0031, so this label maps to 123", + "data": "\uff11\uff12\uff13", + "valid": true + }, + { + "description": "a zero width space is ignored by the mapping", + "comment": "IdnaMappingTable.txt lists U+200B as ignored, so this label maps to ab", + "data": "a\u200bb", + "valid": true + }, + { + "description": "an A-label with an uppercase Punycode body is valid", + "comment": "https://www.rfc-editor.org/rfc/rfc5891#section-5.3 lowercases the A-label before the round-trip test, and RFC 3492 section 5 gives A-Z and a-z the same digit-values", + "data": "xn--NXASMQ6B", + "valid": true + }, + { + "description": "the two Arabic-Indic digit blocks may appear in different labels", + "comment": "https://www.rfc-editor.org/rfc/rfc5892#appendix-A.8 scopes the rule to \"this label\", so one block per label is valid", + "data": "\u0628\u0660.\u0628\u06f0", + "valid": true + }, + { + "description": "a label of only Arabic-Indic digits is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5893#section-2 condition 1 allows only L, R or AL as the first character, and an Arabic-Indic digit is AN", + "data": "\u0660\u0661", + "valid": false + }, + { + "description": "a non-NFC U-label is valid after UTS 46 normalisation", + "comment": "UTS 46 section 4 normalises to NFC before validating, so the decomposed form is accepted", + "data": "cafe\u0301.com", + "valid": true + }, + { + "description": "a label of 63 fullwidth characters is valid after mapping", + "comment": "UTS 46 maps each U+FF41 to an ASCII letter, so this label is 63 octets in its mapped form", + "data": "\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41", + "valid": true + }, + { + "description": "an ACE prefix produced by the mapping step is decoded as an A-label", + "comment": "UTS 46 maps U+FF58 U+FF4E to \"xn\", so the mapped label is the A-label xn--nxasmq6b", + "data": "\uff58\uff4e--nxasmq6b", + "valid": true + }, + { + "description": "zero width non-joiner must pass at every occurrence", + "comment": "https://www.rfc-editor.org/rfc/rfc5892#appendix-A.1", + "data": "\u0915\u094d\u200c\u0937x\u200cy", + "valid": false + }, + { + "description": "Bidi domain name with a digit-first label is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5893#section-2 a label in a Bidi domain name must start with an L, R or AL character", + "data": "0a.\u05d0", + "valid": false + }, + { + "description": "label starting with a digit before a right-to-left letter is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5893#section-2", + "data": "0\u0627", + "valid": false + }, + { + "description": "left-to-right label containing a right-to-left letter is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5893#section-2", + "data": "a\u05d0", + "valid": false + }, + { + "description": "right-to-left label mixing both digit types is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5893#section-2", + "data": "\u05d00\u0660", + "valid": false + }, + { + "description": "A-label that decodes to a disallowed code point is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5890#section-2.3.2.1 https://www.rfc-editor.org/rfc/rfc5892#section-2.6", + "data": "xn--7a", + "valid": false + }, + { + "description": "A-label that decodes to a Bidi rule violation is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5890#section-2.3.2.1 https://www.rfc-editor.org/rfc/rfc5893", + "data": "xn--0ca24w", + "valid": false + }, + { + "description": "a U-label whose A-label form is longer than 63 octets is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5891#section-4.2.4 the 63-octet limit is on the A-label form, not the code point count", + "data": "\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc", + "valid": false + }, + { + "description": "empty label between two dots is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc1034#section-3.1", + "data": "a..b", + "valid": false + }, + { + "description": "a name longer than 253 characters is invalid", + "data": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "valid": false + }, + { + "description": "A-label that decodes to only ASCII is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5890#section-2.3.2.1 a U-label must have at least one non-ASCII character", + "data": "xn--example-", + "valid": false + }, + { + "description": "non-canonical Punycode that does not re-encode to itself is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5891#section-5.4 a label decoded from Punycode must be identical to the original A-label", + "data": "xn---9uc", + "valid": false + } + ] + }, + { + "description": "validation of separators in internationalized host names", + "specification": [ + {"rfc3490": "3.1", "quote": "Whenever dots are used as label separators, the following characters MUST be recognized as dots: U+002E (full stop), U+3002 (ideographic full stop), U+FF0E (fullwidth full stop), U+FF61(halfwidth ideographic full stop)"} + ], + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "idn-hostname" + }, + "tests": [ + { + "description": "single dot", + "data": ".", + "valid": false + }, + { + "description": "single ideographic full stop", + "data": "\u3002", + "valid": false + }, + { + "description": "single fullwidth full stop", + "data": "\uff0e", + "valid": false + }, + { + "description": "single halfwidth ideographic full stop", + "data": "\uff61", + "valid": false + }, + { + "description": "dot as label separator", + "data": "a.b", + "valid": true + }, + { + "description": "ideographic full stop as label separator", + "data": "a\u3002b", + "valid": true + }, + { + "description": "fullwidth full stop as label separator", + "data": "a\uff0eb", + "valid": true + }, + { + "description": "halfwidth ideographic full stop as label separator", + "data": "a\uff61b", + "valid": true + }, + { + "description": "leading dot", + "data": ".example", + "valid": false + }, + { + "description": "leading ideographic full stop", + "data": "\u3002example", + "valid": false + }, + { + "description": "leading fullwidth full stop", + "data": "\uff0eexample", + "valid": false + }, + { + "description": "leading halfwidth ideographic full stop", + "data": "\uff61example", + "valid": false + }, + { + "description": "trailing dot", + "data": "example.", + "valid": false + }, + { + "description": "trailing ideographic full stop", + "data": "example\u3002", + "valid": false + }, + { + "description": "trailing fullwidth full stop", + "data": "example\uff0e", + "valid": false + }, + { + "description": "trailing halfwidth ideographic full stop", + "data": "example\uff61", + "valid": false + }, + { + "description": "label too long if separator ignored (full stop)", + "data": "παράδειγμαπαράδειγμαπαράδειγμαπαράδειγμαπαράδειγμαπα.com", + "valid": true + }, + { + "description": "label too long if separator ignored (ideographic full stop)", + "data": "παράδειγμαπαράδειγμαπαράδειγμαπαράδειγμαπαράδειγμαπα\u3002com", + "valid": true + }, + { + "description": "label too long if separator ignored (fullwidth full stop)", + "data": "παράδειγμαπαράδειγμαπαράδειγμαπαράδειγμαπαράδειγμαπα\uff0ecom", + "valid": true + }, + { + "description": "label too long if separator ignored (halfwidth ideographic full stop)", + "data": "παράδειγμαπαράδειγμαπαράδειγμαπαράδειγμαπαράδειγμαπα\uff61com", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/ipv4.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/ipv4.json new file mode 100644 index 000000000..696713e90 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/ipv4.json @@ -0,0 +1,217 @@ +[ + { + "description": "validation of IP addresses", + "comment": "RFC 2673, Section 3.2: dotted-quad = decbyte \".\" decbyte \".\" decbyte \".\" decbyte. A 'decbyte' (1*3DIGIT) restricts semantic values to 0-255, allows leading zeros, and strictly forbids symbols, alpha/hex, whitespace, and non-ASCII characters.", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "ipv4" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid IP address", + "data": "192.168.0.1", + "valid": true + }, + { + "description": "an IP address with too many components", + "data": "127.0.0.0.1", + "valid": false + }, + { + "description": "an IP address with out-of-range values", + "data": "256.256.256.256", + "valid": false + }, + { + "description": "an IP address without 4 components", + "data": "127.0", + "valid": false + }, + { + "description": "a 2-part address resolving to a routable IP (inet_aton shorthand)", + "data": "127.1", + "valid": false + }, + { + "description": "a 3-part address resolving to a routable IP (inet_aton shorthand)", + "data": "127.0.1", + "valid": false + }, + { + "description": "an IP address as an integer", + "data": "0x7f000001", + "valid": false + }, + { + "description": "an IP address as an integer (decimal)", + "data": "2130706433", + "valid": false + }, + { + "description": "invalid non-ASCII '২' (a Bengali 2)", + "data": "1২7.0.0.1", + "valid": false + }, + { + "description": "invalid fullwidth digits (non-ASCII)", + "data": "１９２.１６８.１.１", + "valid": false + }, + { + "description": "invalid mathematical bold digits (non-ASCII)", + "data": "𝟏𝟗𝟐.𝟏𝟔𝟖.𝟏.𝟏", + "valid": false + }, + { + "description": "netmask is not a part of ipv4 address", + "data": "192.168.1.0/24", + "valid": false + }, + { + "description": "leading whitespace is invalid", + "data": " 192.168.0.1", + "valid": false + }, + { + "description": "trailing whitespace is invalid", + "data": "192.168.0.1 ", + "valid": false + }, + { + "description": "trailing newline is invalid", + "data": "192.168.0.1\n", + "valid": false + }, + { + "description": "hexadecimal notation is invalid", + "data": "0x7f.0.0.1", + "valid": false + }, + { + "description": "octal notation explicit is invalid", + "data": "0o10.0.0.1", + "valid": false + }, + { + "description": "empty part (double dot) is invalid", + "data": "192.168..1", + "valid": false + }, + { + "description": "leading dot is invalid", + "data": ".192.168.0.1", + "valid": false + }, + { + "description": "trailing dot is invalid", + "data": "192.168.0.1.", + "valid": false + }, + { + "description": "minimum valid IPv4 address", + "data": "0.0.0.0", + "valid": true + }, + { + "description": "maximum valid IPv4 address", + "data": "255.255.255.255", + "valid": true + }, + { + "description": "valid IPv4 with an octet in the 200 to 249 range", + "data": "200.0.0.0", + "valid": true + }, + { + "description": "valid IPv4 with two-digit octets", + "data": "10.20.30.40", + "valid": true + }, + { + "description": "empty string is invalid", + "data": "", + "valid": false + }, + { + "description": "plus sign is invalid", + "data": "+1.2.3.4", + "valid": false + }, + { + "description": "negative sign is invalid", + "data": "-1.2.3.4", + "valid": false + }, + { + "description": "exponential notation is invalid", + "data": "1e2.0.0.1", + "valid": false + }, + { + "description": "alpha characters are invalid", + "data": "192.168.a.1", + "valid": false + }, + { + "description": "internal whitespace is invalid", + "data": "192. 168.0.1", + "valid": false + }, + { + "description": "tab character is invalid", + "data": "192.168.0.1\t", + "valid": false + }, + { + "description": "additional content after an embedded NUL byte", + "data": "192.168.0.1\u0000.evil.com", + "valid": false + }, + { + "description": "with port number is invalid", + "data": "192.168.0.1:80", + "valid": false + }, + { + "description": "an IPv4-mapped IPv6 address is invalid", + "data": "::ffff:192.168.0.1", + "valid": false + }, + { + "description": "single octet out of range in last position", + "data": "192.168.0.256", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/ipv6.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/ipv6.json new file mode 100644 index 000000000..e5b46a00f --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/ipv6.json @@ -0,0 +1,266 @@ +[ + { + "description": "validation of IPv6 addresses", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "ipv6" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid IPv6 address", + "data": "::1", + "valid": true + }, + { + "description": "a group with 5 hex digits is invalid", + "data": "12345::", + "valid": false + }, + { + "description": "trailing 4 hex symbols is valid", + "data": "::abef", + "valid": true + }, + { + "description": "trailing 5 hex symbols is invalid", + "data": "::abcef", + "valid": false + }, + { + "description": "an IPv6 address with too many components", + "data": "1:1:1:1:1:1:1:1:1:1:1:1:1:1:1:1", + "valid": false + }, + { + "description": "an IPv6 address containing illegal characters", + "data": "::laptop", + "valid": false + }, + { + "description": "no digits is valid", + "data": "::", + "valid": true + }, + { + "description": "leading colons is valid", + "data": "::42:ff:1", + "valid": true + }, + { + "description": "trailing colons is valid", + "data": "d6::", + "valid": true + }, + { + "description": "missing leading octet is invalid", + "data": ":2:3:4:5:6:7:8", + "valid": false + }, + { + "description": "missing trailing octet is invalid", + "data": "1:2:3:4:5:6:7:", + "valid": false + }, + { + "description": "missing leading octet with omitted octets later", + "data": ":2:3:4::8", + "valid": false + }, + { + "description": "single set of double colons in the middle is valid", + "data": "1:d6::42", + "valid": true + }, + { + "description": "two sets of double colons is invalid", + "data": "1::d6::42", + "valid": false + }, + { + "description": "mixed format with the ipv4 section as decimal octets", + "data": "1::d6:192.168.0.1", + "valid": true + }, + { + "description": "mixed format with double colons between the sections", + "data": "1:2::192.168.0.1", + "valid": true + }, + { + "description": "mixed format with ipv4 section with octet out of range", + "data": "1::2:192.168.256.1", + "valid": false + }, + { + "description": "mixed format with ipv4 section with a hex octet", + "data": "1::2:192.168.ff.1", + "valid": false + }, + { + "description": "mixed format with leading double colons (ipv4-mapped ipv6 address)", + "data": "::ffff:192.168.0.1", + "valid": true + }, + { + "description": "triple colons is invalid", + "data": "1:2:3:4:5:::8", + "valid": false + }, + { + "description": "8 octets", + "data": "1:2:3:4:5:6:7:8", + "valid": true + }, + { + "description": "insufficient octets without double colons", + "data": "1:2:3:4:5:6:7", + "valid": false + }, + { + "description": "no colons is invalid", + "data": "1", + "valid": false + }, + { + "description": "ipv4 is not ipv6", + "data": "127.0.0.1", + "valid": false + }, + { + "description": "ipv4 segment must have 4 octets", + "data": "1:2:3:4:1.2.3", + "valid": false + }, + { + "description": "leading whitespace is invalid", + "data": " ::1", + "valid": false + }, + { + "description": "trailing whitespace is invalid", + "data": "::1 ", + "valid": false + }, + { + "description": "netmask is not a part of ipv6 address", + "data": "fe80::/64", + "valid": false + }, + { + "description": "zone id is not a part of ipv6 address", + "data": "fe80::a%eth1", + "valid": false + }, + { + "description": "a long valid ipv6", + "data": "1000:1000:1000:1000:1000:1000:255.255.255.255", + "valid": true + }, + { + "description": "a long invalid ipv6, below length limit, first", + "data": "100:100:100:100:100:100:255.255.255.255.255", + "valid": false + }, + { + "description": "a long invalid ipv6, below length limit, second", + "data": "100:100:100:100:100:100:100:255.255.255.255", + "valid": false + }, + { + "description": "invalid non-ASCII '৪' (a Bengali 4)", + "data": "1:2:3:4:5:6:7:৪", + "valid": false + }, + { + "description": "invalid non-ASCII '৪' (a Bengali 4) in the IPv4 portion", + "data": "1:2::192.16৪.0.1", + "valid": false + }, + { + "description": "a bracketed address is invalid", + "data": "[::1]", + "valid": false + }, + { + "description": "a leading zero in the last IPv4 octet is invalid", + "data": "::ffff:192.168.0.01", + "valid": false + }, + { + "description": "uppercase hex digits are valid", + "data": "2001:DB8::1", + "valid": true + }, + { + "description": "leading zeros in every group are valid", + "data": "2001:0db8:0000:0000:0000:0000:0000:0001", + "valid": true + }, + { + "description": "two groups before '::' and four groups after is valid", + "data": "1:2::3:4:5:6", + "valid": true + }, + { + "description": "eight groups alongside '::' is invalid", + "data": "1:2:3:4:5:6:7:8::", + "valid": false + }, + { + "description": "trailing newline is invalid", + "data": "::1\n", + "valid": false + }, + { + "description": "an out-of-range first octet in the embedded IPv4 is invalid", + "data": "::ffff:256.1.1.1", + "valid": false + }, + { + "description": "one group before '::' and five groups after is valid", + "data": "1::2:3:4:5:6", + "valid": true + }, + { + "description": "'::' followed by six groups is valid", + "data": "::1:2:3:4:5:6", + "valid": true + }, + { + "description": "leading '::' before five groups and an embedded IPv4 is valid", + "data": "::1:2:3:4:5:1.2.3.4", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/iri-reference.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/iri-reference.json new file mode 100644 index 000000000..d31cdbbea --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/iri-reference.json @@ -0,0 +1,116 @@ +[ + { + "description": "validation of IRI References", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "iri-reference" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid IRI", + "data": "http://ƒøø.ßår/?∂éœ=πîx#πîüx", + "valid": true + }, + { + "description": "a valid protocol-relative IRI Reference", + "data": "//ƒøø.ßår/?∂éœ=πîx#πîüx", + "valid": true + }, + { + "description": "a valid relative IRI Reference", + "data": "/âππ", + "valid": true + }, + { + "description": "an invalid IRI Reference", + "data": "\\\\WINDOWS\\filëßåré", + "valid": false + }, + { + "description": "a valid IRI Reference", + "data": "âππ", + "valid": true + }, + { + "description": "a valid IRI fragment", + "data": "#ƒrägmênt", + "valid": true + }, + { + "description": "an invalid IRI fragment", + "data": "#ƒräg\\mênt", + "valid": false + }, + { + "description": "an embedded IPv4 with a leading zero is invalid", + "data": "//[::ffff:192.168.0.01]/p", + "valid": false + }, + { + "description": "a valid protocol-relative IRI Reference with a compressed IPv6 host", + "data": "//[2001:db8::1]/p", + "valid": true + }, + { + "description": "a valid relative IRI Reference with a supplementary-plane character", + "data": "/âππ/𐌀", + "valid": true + }, + { + "description": "a query-only IRI Reference with a supplementary private-use char", + "data": "?q=󰀀", + "valid": true + }, + { + "description": "a trailing newline after a valid IRI Reference is invalid", + "data": "/âππ\n", + "valid": false + }, + { + "description": "invalid percent-encoding with non-hex digits", + "data": "/%6G", + "valid": false + }, + { + "description": "incomplete percent-encoding triplet", + "data": "/%A", + "valid": false + }, + { + "description": "lone percent sign is invalid", + "data": "/%", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/iri.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/iri.json new file mode 100644 index 000000000..db27c829a --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/iri.json @@ -0,0 +1,141 @@ +[ + { + "description": "validation of IRIs", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "iri" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid IRI with anchor tag", + "data": "http://ƒøø.ßår/?∂éœ=πîx#πîüx", + "valid": true + }, + { + "description": "a valid IRI with anchor tag and parentheses", + "data": "http://ƒøø.com/blah_(wîkïpédiå)_blah#ßité-1", + "valid": true + }, + { + "description": "a valid IRI with URL-encoded stuff", + "data": "http://ƒøø.ßår/?q=Test%20URL-encoded%20stuff", + "valid": true + }, + { + "description": "invalid percent-encoding with non-hex digits", + "data": "http://ƒøø.ßår/%6G", + "valid": false + }, + { + "description": "incomplete percent-encoding triplet", + "data": "http://ƒøø.ßår/%A", + "valid": false + }, + { + "description": "lone percent sign is invalid", + "data": "http://ƒøø.ßår/%", + "valid": false + }, + { + "description": "a valid IRI with many special characters", + "data": "http://-.~_!$&'()*+,;=:%40:80%2f::::::@example.com", + "valid": true + }, + { + "description": "a valid IRI based on IPv6", + "data": "http://[2001:0db8:85a3:0000:0000:8a2e:0370:7334]", + "valid": true + }, + { + "description": "an IPv6 address without enclosing brackets is invalid", + "data": "http://2001:0db8:85a3:0000:0000:8a2e:0370:7334", + "valid": false + }, + { + "description": "an invalid relative IRI Reference", + "data": "/abc", + "valid": false + }, + { + "description": "an invalid IRI", + "data": "\\\\WINDOWS\\filëßåré", + "valid": false + }, + { + "description": "an invalid IRI though valid IRI reference", + "data": "âππ", + "valid": false + }, + { + "description": "an IPv6 host whose embedded IPv4 has a leading zero is invalid", + "data": "http://[::ffff:192.168.0.01]", + "valid": false + }, + { + "description": "a valid IRI with a compressed IPv6 host", + "data": "http://[2001:db8::1]", + "valid": true + }, + { + "description": "a trailing newline after a valid IRI is invalid", + "data": "http://ƒøø.ßår/\n", + "valid": false + }, + { + "description": "a valid IRI with an IPv4 host", + "data": "http://192.168.0.1/p", + "valid": true + }, + { + "description": "a valid IRI with no authority and a rootless path", + "data": "urn:example:resource", + "valid": true + }, + { + "description": "a valid IRI with no authority and an absolute path", + "data": "file:/etc/hosts", + "valid": true + }, + { + "description": "a valid IRI with a supplementary-plane character in the path", + "data": "http://ƒøø.ßår/𐌀", + "valid": true + }, + { + "description": "a valid IRI with a supplementary-plane private-use char in query", + "data": "http://ƒøø.ßår/?q=󰀀", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/json-pointer.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/json-pointer.json new file mode 100644 index 000000000..d5b9a8bb1 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/json-pointer.json @@ -0,0 +1,211 @@ +[ + { + "description": "validation of JSON-pointers (JSON String Representation)", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "json-pointer" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid JSON-pointer", + "data": "/foo/bar~0/baz~1/%a", + "valid": true + }, + { + "description": "not a valid JSON-pointer (~ not escaped)", + "data": "/foo/bar~", + "valid": false + }, + { + "description": "valid JSON-pointer with empty segment", + "data": "/foo//bar", + "valid": true + }, + { + "description": "valid JSON-pointer with the last empty segment", + "data": "/foo/bar/", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #1", + "data": "", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #2", + "data": "/foo", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #3", + "data": "/foo/0", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #4", + "data": "/", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #5", + "data": "/a~1b", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #6", + "data": "/c%d", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #7", + "data": "/e^f", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #8", + "data": "/g|h", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #9", + "data": "/i\\j", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #10", + "data": "/k\"l", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #11", + "data": "/ ", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #12", + "data": "/m~0n", + "valid": true + }, + { + "description": "valid JSON-pointer used adding to the last array position", + "data": "/foo/-", + "valid": true + }, + { + "description": "valid JSON-pointer (- used as object member name)", + "data": "/foo/-/bar", + "valid": true + }, + { + "description": "valid JSON-pointer (multiple escaped characters)", + "data": "/~1~0~0~1~1", + "valid": true + }, + { + "description": "valid JSON-pointer (escaped with fraction part) #1", + "data": "/~1.1", + "valid": true + }, + { + "description": "valid JSON-pointer (escaped with fraction part) #2", + "data": "/~0.1", + "valid": true + }, + { + "description": "not a valid JSON-pointer (URI Fragment Identifier) #1", + "data": "#", + "valid": false + }, + { + "description": "not a valid JSON-pointer (URI Fragment Identifier) #2", + "data": "#/", + "valid": false + }, + { + "description": "not a valid JSON-pointer (URI Fragment Identifier) #3", + "data": "#a", + "valid": false + }, + { + "description": "not a valid JSON-pointer (some escaped, but not all) #1", + "data": "/~0~", + "valid": false + }, + { + "description": "not a valid JSON-pointer (some escaped, but not all) #2", + "data": "/~0/~", + "valid": false + }, + { + "description": "not a valid JSON-pointer (wrong escape character) #1", + "data": "/~2", + "valid": false + }, + { + "description": "not a valid JSON-pointer (wrong escape character) #2", + "data": "/~-1", + "valid": false + }, + { + "description": "not a valid JSON-pointer (multiple characters not escaped)", + "data": "/~~", + "valid": false + }, + { + "description": "not a valid JSON-pointer (isn't empty nor starts with /) #1", + "data": "a", + "valid": false + }, + { + "description": "not a valid JSON-pointer (isn't empty nor starts with /) #2", + "data": "0", + "valid": false + }, + { + "description": "not a valid JSON-pointer (isn't empty nor starts with /) #3", + "data": "a/a", + "valid": false + }, + { + "description": "valid JSON-pointer (Unicode characters allowed by RFC 6901)", + "data": "/foo/bar/😎", + "valid": true + }, + { + "description": "valid JSON-pointer (control characters allowed after JSON unescaping)", + "data": "/foo\u0000bar\n\tbaz", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/regex.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/regex.json new file mode 100644 index 000000000..dc3e72937 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/regex.json @@ -0,0 +1,103 @@ +[ + { + "description": "validation of regular expressions", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "regex" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid regular expression", + "data": "([abc])+\\s+$", + "valid": true + }, + { + "description": "a regular expression with unclosed parens is invalid", + "data": "^(abc]", + "valid": false + }, + { + "description": "a quantifier with nothing to repeat is invalid", + "comment": "https://tc39.es/ecma262/#sec-patterns a Quantifier follows something (Term :: Atom Quantifier), so a pattern cannot begin with one", + "data": "*a", + "valid": false + }, + { + "description": "a character class range in reverse order is invalid", + "comment": "https://tc39.es/ecma262/#sec-patterns-static-semantics-early-errors it is a Syntax Error if the first ClassAtom of a range has a greater CharacterValue than the second", + "data": "[z-a]", + "valid": false + }, + { + "description": "a trailing backslash is invalid", + "data": "\\", + "valid": false + }, + { + "description": "an unmatched closing parenthesis is invalid", + "data": "a)", + "valid": false + }, + { + "description": "an unterminated character class is invalid", + "data": "[abc", + "valid": false + }, + { + "description": "an empty regular expression is valid", + "data": "", + "valid": true + }, + { + "description": "alternation is valid", + "data": "a|b", + "valid": true + }, + { + "description": "a bounded quantifier is valid", + "data": "a{2,3}", + "valid": true + }, + { + "description": "a negated range character class is valid", + "data": "[^a-z]", + "valid": true + }, + { + "description": "a lazy quantifier is valid", + "data": "a+?", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/relative-json-pointer.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/relative-json-pointer.json new file mode 100644 index 000000000..413c03fd8 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/relative-json-pointer.json @@ -0,0 +1,141 @@ +[ + { + "description": "validation of Relative JSON Pointers (RJP)", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "relative-json-pointer" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid upwards RJP", + "data": "1", + "valid": true + }, + { + "description": "a valid downwards RJP", + "data": "0/foo/bar", + "valid": true + }, + { + "description": "a valid up and then down RJP, with array index", + "data": "2/0/baz/1/zip", + "valid": true + }, + { + "description": "a valid RJP taking the member or index name", + "data": "0#", + "valid": true + }, + { + "description": "an invalid RJP that is a valid JSON Pointer", + "data": "/foo/bar", + "valid": false + }, + { + "description": "a json-pointer part that does not start with a slash is invalid", + "data": "1foo", + "valid": false + }, + { + "description": "negative prefix", + "data": "-1/foo/bar", + "valid": false + }, + { + "description": "explicit positive prefix", + "data": "+1/foo/bar", + "valid": false + }, + { + "description": "non-ASCII digit in the prefix is not allowed", + "data": "١/foo", + "valid": false + }, + { + "description": "## is not a valid json-pointer", + "data": "0##", + "valid": false + }, + { + "description": "zero cannot be followed by other digits, plus json-pointer", + "data": "01/a", + "valid": false + }, + { + "description": "zero cannot be followed by other digits, plus octothorpe", + "data": "01#", + "valid": false + }, + { + "description": "empty string", + "data": "", + "valid": false + }, + { + "description": "multi-digit integer prefix", + "data": "120/foo/bar", + "valid": true + }, + { + "description": "multi-digit prefix with a zero followed by another digit", + "data": "100", + "valid": true + }, + { + "description": "tilde not followed by 0 or 1 in the json-pointer part", + "data": "0/~2", + "valid": false + }, + { + "description": "unescaped tilde at the end of the json-pointer part", + "data": "0/foo/bar~", + "valid": false + }, + { + "description": "octothorpe followed by a json-pointer", + "data": "1#/foo/bar", + "valid": false + }, + { + "description": "empty reference tokens in the json-pointer part", + "data": "0//", + "valid": true + }, + { + "description": "trailing newline after the non-negative integer", + "data": "1\n", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/time.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/time.json new file mode 100644 index 000000000..2f4b6353f --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/time.json @@ -0,0 +1,303 @@ +[ + { + "description": "validation of time strings", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "time" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid time string", + "data": "08:30:06Z", + "valid": true + }, + { + "description": "invalid time string with extra leading zeros", + "data": "008:030:006Z", + "valid": false + }, + { + "description": "invalid time string with no leading zero for single digit", + "data": "8:3:6Z", + "valid": false + }, + { + "description": "hour, minute, second must be two digits", + "data": "8:0030:6Z", + "valid": false + }, + { + "description": "a valid time string with leap second, Zulu", + "data": "23:59:60Z", + "valid": true + }, + { + "description": "invalid leap second, Zulu (wrong hour)", + "data": "22:59:60Z", + "valid": false + }, + { + "description": "invalid leap second, Zulu (wrong minute)", + "data": "23:58:60Z", + "valid": false + }, + { + "description": "valid leap second, zero time-offset", + "data": "23:59:60+00:00", + "valid": true + }, + { + "description": "invalid leap second, zero time-offset (wrong hour)", + "data": "22:59:60+00:00", + "valid": false + }, + { + "description": "invalid leap second, zero time-offset (wrong minute)", + "data": "23:58:60+00:00", + "valid": false + }, + { + "description": "valid leap second, positive time-offset", + "data": "01:29:60+01:30", + "valid": true + }, + { + "description": "valid leap second, large positive time-offset", + "data": "23:29:60+23:30", + "valid": true + }, + { + "description": "invalid leap second, positive time-offset (wrong hour)", + "data": "23:59:60+01:00", + "valid": false + }, + { + "description": "invalid leap second, positive time-offset (wrong minute)", + "data": "23:59:60+00:30", + "valid": false + }, + { + "description": "valid leap second, negative time-offset", + "data": "15:59:60-08:00", + "valid": true + }, + { + "description": "valid leap second, large negative time-offset", + "data": "00:29:60-23:30", + "valid": true + }, + { + "description": "invalid leap second, negative time-offset (wrong hour)", + "data": "23:59:60-01:00", + "valid": false + }, + { + "description": "invalid leap second, negative time-offset (wrong minute)", + "data": "23:59:60-00:30", + "valid": false + }, + { + "description": "a valid time string with second fraction", + "data": "23:20:50.52Z", + "valid": true + }, + { + "description": "a valid time string with precise second fraction", + "data": "08:30:06.283185Z", + "valid": true + }, + { + "description": "a valid time string with plus offset", + "data": "08:30:06+00:20", + "valid": true + }, + { + "description": "a valid time string with minus offset", + "data": "08:30:06-08:00", + "valid": true + }, + { + "description": "time with unknown local offset is valid", + "comment": "RFC 3339 section 4.3 (unknown local offset)", + "data": "12:34:56-00:00", + "valid": true + }, + { + "description": "hour, minute in time-offset must be two digits", + "data": "08:30:06-8:000", + "valid": false + }, + { + "description": "a valid time string with case-insensitive Z", + "data": "08:30:06z", + "valid": true + }, + { + "description": "an invalid time string with invalid hour", + "data": "24:00:00Z", + "valid": false + }, + { + "description": "an invalid time string with invalid minute", + "data": "00:60:00Z", + "valid": false + }, + { + "description": "an invalid time string with invalid second", + "data": "00:00:61Z", + "valid": false + }, + { + "description": "an invalid time string with invalid time numoffset hour", + "data": "01:02:03+24:00", + "valid": false + }, + { + "description": "an invalid time string with invalid time numoffset minute", + "data": "01:02:03+00:60", + "valid": false + }, + { + "description": "an invalid time string with invalid time with both Z and numoffset", + "data": "01:02:03Z+00:30", + "valid": false + }, + { + "description": "an invalid offset indicator", + "data": "08:30:06 PST", + "valid": false + }, + { + "description": "only RFC3339 not all of ISO 8601 are valid", + "data": "01:01:01,1111", + "valid": false + }, + { + "description": "no time offset", + "data": "12:00:00", + "valid": false + }, + { + "description": "no time offset with second fraction", + "data": "12:00:00.52", + "valid": false + }, + { + "description": "invalid non-ASCII '২' (a Bengali 2)", + "data": "1২:00:00Z", + "valid": false + }, + { + "description": "offset not starting with plus or minus", + "data": "08:30:06#00:20", + "valid": false + }, + { + "description": "contains letters", + "data": "ab:cd:ef", + "valid": false + }, + { + "description": "an invalid time string in date-time format", + "data": "2020-11-28T23:55:45Z", + "valid": false + }, + { + "description": "a numeric time-offset requires the colon separator", + "comment": "RFC 3339 section 5.6: time-numoffset = (\"+\" / \"-\") time-hour \":\" time-minute", + "data": "08:30:06+0130", + "valid": false + }, + { + "description": "a numeric time-offset requires the minutes component", + "comment": "RFC 3339 section 5.6: time-numoffset = (\"+\" / \"-\") time-hour \":\" time-minute", + "data": "08:30:06+01", + "valid": false + }, + { + "description": "a trailing newline after a valid time", + "comment": "RFC 3339 section 5.6: no production of full-time emits a line feed", + "data": "08:30:06Z\n", + "valid": false + }, + { + "description": "hour 24 is invalid with a numeric time-offset", + "comment": "RFC 3339 section 5.6: time-hour = 2DIGIT ; 00-23, and section 5.7 only allows 00 to 23", + "data": "24:59:00+01:00", + "valid": false + }, + { + "description": "minute 60 is invalid with a numeric time-offset", + "comment": "RFC 3339 section 5.6: time-minute = 2DIGIT ; 00-59", + "data": "23:60:00+00:01", + "valid": false + }, + { + "description": "a valid time string with a 15 digit second fraction", + "comment": "RFC 3339 section 5.6: time-secfrac = \".\" 1*DIGIT, with no upper bound on the number of digits", + "data": "00:59:59.999999999999999Z", + "valid": true + }, + { + "description": "a second fraction with no digits", + "comment": "RFC 3339 section 5.6: time-secfrac = \".\" 1*DIGIT requires at least one digit", + "data": "08:30:06.Z", + "valid": false + }, + { + "description": "partial-time requires the seconds component", + "comment": "RFC 3339 section 5.6: partial-time = time-hour \":\" time-minute \":\" time-second [time-secfrac]", + "data": "12:00Z", + "valid": false + }, + { + "description": "a comma decimal separator with a time-offset", + "comment": "RFC 3339 section 5.6: the decimal separator in time-secfrac is \".\" - the ISO 8601 comma is not part of this profile", + "data": "08:30:06,5Z", + "valid": false + }, + { + "description": "leading whitespace before a valid time", + "comment": "RFC 3339 section 5.6: no production of full-time emits whitespace", + "data": " 08:30:06Z", + "valid": false + }, + { + "description": "trailing whitespace after a valid time", + "comment": "RFC 3339 section 5.6: no production of full-time emits whitespace", + "data": "08:30:06Z ", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/unknown.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/unknown.json new file mode 100644 index 000000000..7fc35f536 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/unknown.json @@ -0,0 +1,46 @@ +[ + { + "description": "unknown format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "unknown" + }, + "tests": [ + { + "description": "unknown formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "unknown formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "unknown formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "unknown formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "unknown formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "unknown formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "unknown formats ignore strings", + "data": "string", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/uri-reference.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/uri-reference.json new file mode 100644 index 000000000..08c655e00 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/uri-reference.json @@ -0,0 +1,182 @@ +[ + { + "description": "validation of URI References", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "uri-reference" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid URI", + "data": "http://foo.bar/?baz=qux#quux", + "valid": true + }, + { + "description": "a valid protocol-relative URI Reference", + "data": "//foo.bar/?baz=qux#quux", + "valid": true + }, + { + "description": "a valid relative URI Reference", + "data": "/abc", + "valid": true + }, + { + "description": "an invalid URI Reference", + "data": "\\\\WINDOWS\\fileshare", + "valid": false + }, + { + "description": "a valid URI Reference", + "data": "abc", + "valid": true + }, + { + "description": "a valid URI fragment", + "data": "#fragment", + "valid": true + }, + { + "description": "an invalid URI fragment", + "data": "#frag\\ment", + "valid": false + }, + { + "description": "unescaped non US-ASCII characters", + "data": "/foobar®.txt", + "valid": false + }, + { + "description": "invalid backslash character", + "data": "https://example.org/foobar\\.txt", + "valid": false + }, + { + "description": "URI with leading-zero IPv4 is structurally valid as a reg-name", + "comment": "RFC 3986, Section 3.2.2: If an IP literal fails strict IPv4address parsing, it falls back to reg-name. A string of digits and dots is valid under unreserved characters. JSON Schema format asserts syntax, not scheme-specific DNS semantics.", + "data": "http://087.10.0.1/", + "valid": true + }, + { + "description": "URI with out-of-bounds IPv4 is structurally valid as a reg-name", + "comment": "RFC 3986, Section 3.2.2: Fallback to reg-name allows digits and dots.", + "data": "http://999.999.999.999/", + "valid": true + }, + { + "description": "an empty URI reference", + "comment": "RFC 3986, Section 4.2: relative-part includes path-empty, which is zero characters.", + "data": "", + "valid": true + }, + { + "description": "a query-only reference", + "comment": "RFC 3986, Section 4.2: relative-ref = relative-part [ \"?\" query ] [ \"#\" fragment ], so a query with an empty path is a reference.", + "data": "?query=1", + "valid": true + }, + { + "description": "a network-path reference with an empty authority", + "comment": "RFC 3986, Section 3.2.2: host = IP-literal / IPv4address / reg-name, and reg-name may be empty.", + "data": "//", + "valid": true + }, + { + "description": "an incomplete percent-encoding", + "comment": "RFC 3986, Section 2.1: pct-encoded = \"%\" HEXDIG HEXDIG.", + "data": "/%zz", + "valid": false + }, + { + "description": "incomplete percent-encoding triplet", + "comment": "RFC 3986, Section 2.1: pct-encoded = \"%\" HEXDIG HEXDIG.", + "data": "/%A", + "valid": false + }, + { + "description": "lone percent sign is invalid", + "comment": "RFC 3986, Section 2.1: pct-encoded = \"%\" HEXDIG HEXDIG.", + "data": "/%", + "valid": false + }, + { + "description": "a double quote in a path", + "comment": "RFC 3986, Appendix A: pchar = unreserved / pct-encoded / sub-delims / \":\" / \"@\", and the double quote is in none of them.", + "data": "/a\"b", + "valid": false + }, + { + "description": "square brackets outside an authority", + "comment": "RFC 3986, Section 3.2.2: square brackets are reserved for an IP-literal inside the authority. path-absolute is built from pchar, which excludes them.", + "data": "/[::1]", + "valid": false + }, + { + "description": "a non-numeric port in a network-path reference", + "comment": "RFC 3986, Section 3.2.3: port = *DIGIT. reg-name admits no colon, so the text after the colon can only be a port.", + "data": "//example.com:abc/p", + "valid": false + }, + { + "description": "more than one at-sign in the authority", + "comment": "RFC 3986, Section 3.2: authority = [ userinfo \"@\" ] host [ \":\" port ]. Neither userinfo nor host admits a literal at-sign.", + "data": "//a@b@example.com/", + "valid": false + }, + { + "description": "a leading zero in the IPv4 part of an IPv6 literal", + "comment": "RFC 3986, Section 3.2.2: inside an IP-literal the dotted quad must be an IPv4address, and dec-octet has no leading-zero alternative. There is no reg-name fallback inside brackets.", + "data": "//[::ffff:192.168.0.01]/p", + "valid": false + }, + { + "description": "a colon in the first segment of a relative-path reference", + "comment": "RFC 3986, Section 4.2: relative-part uses path-noscheme, whose first segment is segment-nz-nc - a non-zero-length segment without any colon. A scheme must start with ALPHA, so 1:b cannot be read as a URI either.", + "data": "1:b", + "valid": false + }, + { + "description": "a colon in the first segment preceded by a dot-segment", + "comment": "RFC 3986, Section 4.2: \"Such a segment must be preceded by a dot-segment (e.g., \\\"./this:that\\\") to make a relative-path reference.\"", + "data": "./this:that", + "valid": true + }, + { + "description": "a trailing line feed", + "comment": "RFC 3986, Appendix A: no production admits %x0A.", + "data": "/p\n", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/uri-template.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/uri-template.json new file mode 100644 index 000000000..58d28f9de --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/uri-template.json @@ -0,0 +1,251 @@ +[ + { + "description": "format: uri-template", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "uri-template" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid uri-template", + "data": "http://example.com/dictionary/{term:1}/{term}", + "valid": true + }, + { + "description": "an invalid uri-template", + "data": "http://example.com/dictionary/{term:1}/{term", + "valid": false + }, + { + "description": "a valid uri-template without variables", + "data": "http://example.com/dictionary", + "valid": true + }, + { + "description": "a valid relative uri-template", + "data": "dictionary/{term:1}/{term}", + "valid": true + }, + { + "description": "an empty expression is invalid", + "data": "{}", + "valid": false + }, + { + "description": "an empty varspec inside the variable list is invalid", + "data": "{a,,b}", + "valid": false + }, + { + "description": "a zero prefix max-length is invalid", + "data": "{v:0}", + "valid": false + }, + { + "description": "a five-digit prefix max-length is invalid", + "data": "{v:10000}", + "valid": false + }, + { + "description": "an empty string is a valid uri-template", + "data": "", + "valid": true + }, + { + "description": "an unmatched closing brace is invalid", + "data": "foo}bar", + "valid": false + }, + { + "description": "a percent-encoded triplet in a literal is valid", + "data": "a%41b", + "valid": true + }, + { + "description": "a literal with a lone percent sign is invalid", + "data": "a%", + "valid": false + }, + { + "description": "a literal with an incomplete percent-encoded triplet is invalid", + "data": "a%4", + "valid": false + }, + { + "description": "a literal with non-hex percent encoding is invalid", + "data": "a%GG", + "valid": false + }, + { + "description": "a supplementary plane character in a literal is valid", + "data": "a\ud83d\ude00b", + "valid": true + }, + { + "description": "a private-use character in a literal is valid", + "data": "a\ue000b", + "valid": true + }, + { + "description": "a supplementary private-use character in a literal is valid", + "data": "a\udb80\udc00b", + "valid": true + }, + { + "description": "reserved expansion operator", + "data": "{+var}", + "valid": true + }, + { + "description": "fragment expansion operator", + "data": "{#var}", + "valid": true + }, + { + "description": "label expansion operator", + "data": "{.var}", + "valid": true + }, + { + "description": "path segment expansion operator", + "data": "{/var}", + "valid": true + }, + { + "description": "path style parameter expansion operator", + "data": "{;var}", + "valid": true + }, + { + "description": "form style query expansion operator", + "data": "{?var}", + "valid": true + }, + { + "description": "form style query continuation operator", + "data": "{&var}", + "valid": true + }, + { + "description": "the explode modifier is valid", + "data": "{var*}", + "valid": true + }, + { + "description": "combined prefix and explode modifiers are invalid", + "data": "{var:1*}", + "valid": false + }, + { + "description": "a variable list with more than one varspec is valid", + "data": "{x,y}", + "valid": true + }, + { + "description": "a dotted variable name is valid", + "data": "{a.b}", + "valid": true + }, + { + "description": "a trailing dot in a variable name is invalid", + "data": "{a.}", + "valid": false + }, + { + "description": "a double dot in a variable name is invalid", + "data": "{a..b}", + "valid": false + }, + { + "description": "a variable name may start with a percent-encoded triplet", + "data": "{%41}", + "valid": true + }, + { + "description": "a variable name with a lone percent sign is invalid", + "data": "{%}", + "valid": false + }, + { + "description": "a variable name with an incomplete percent-encoded triplet is invalid", + "data": "{%4}", + "valid": false + }, + { + "description": "a variable name with non-hex percent encoding is invalid", + "data": "{%GG}", + "valid": false + }, + { + "description": "a four-digit prefix max-length is valid", + "data": "{v:1000}", + "valid": true + }, + { + "description": "a leading zero in the prefix max-length is invalid", + "data": "{v:01}", + "valid": false + }, + { + "description": "a space in a literal is invalid", + "data": "a b", + "valid": false + }, + { + "description": "a delete character in a literal is invalid", + "data": "a\u007fb", + "valid": false + }, + { + "description": "a trailing comma in the variable list is invalid", + "data": "{a,}", + "valid": false + }, + { + "description": "a default value in a varspec is invalid", + "data": "{var=def}", + "valid": false + }, + { + "description": "an operator in place of the first varspec is invalid", + "data": "{,+var}", + "valid": false + }, + { + "description": "an apostrophe in a literal is valid", + "data": "a'b", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/uri.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/uri.json new file mode 100644 index 000000000..e3736ccaf --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/uri.json @@ -0,0 +1,253 @@ +[ + { + "description": "validation of URIs", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "uri" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid URL with anchor tag", + "data": "http://foo.bar/?baz=qux#quux", + "valid": true + }, + { + "description": "a valid URL with anchor tag and parentheses", + "data": "http://foo.com/blah_(wikipedia)_blah#cite-1", + "valid": true + }, + { + "description": "a valid URL with URL-encoded stuff", + "data": "http://foo.bar/?q=Test%20URL-encoded%20stuff", + "valid": true + }, + { + "description": "a valid puny-coded URL ", + "data": "http://xn--nw2a.xn--j6w193g/", + "valid": true + }, + { + "description": "a valid URL with many special characters", + "data": "http://-.~_!$&'()*+,;=:%40:80%2f::::::@example.com", + "valid": true + }, + { + "description": "a valid URL based on IPv4", + "data": "http://223.255.255.254", + "valid": true + }, + { + "description": "a valid URL with ftp scheme", + "data": "ftp://ftp.is.co.za/rfc/rfc1808.txt", + "valid": true + }, + { + "description": "a valid URL for a simple text file", + "data": "http://www.ietf.org/rfc/rfc2396.txt", + "valid": true + }, + { + "description": "a valid URL", + "data": "ldap://[2001:db8::7]/c=GB?objectClass?one", + "valid": true + }, + { + "description": "a valid mailto URI", + "data": "mailto:John.Doe@example.com", + "valid": true + }, + { + "description": "a valid newsgroup URI", + "data": "news:comp.infosystems.www.servers.unix", + "valid": true + }, + { + "description": "a valid tel URI", + "data": "tel:+1-816-555-1212", + "valid": true + }, + { + "description": "a valid URN", + "data": "urn:oasis:names:specification:docbook:dtd:xml:4.1.2", + "valid": true + }, + { + "description": "an invalid protocol-relative URI Reference", + "data": "//foo.bar/?baz=qux#quux", + "valid": false + }, + { + "description": "an invalid relative URI Reference", + "data": "/abc", + "valid": false + }, + { + "description": "an invalid URI", + "data": "\\\\WINDOWS\\fileshare", + "valid": false + }, + { + "description": "an invalid URI though valid URI reference", + "data": "abc", + "valid": false + }, + { + "description": "an invalid URI with spaces", + "data": "http:// shouldfail.com", + "valid": false + }, + { + "description": "an invalid URI with spaces and missing scheme", + "data": ":// should fail", + "valid": false + }, + { + "description": "an invalid URI with comma in scheme", + "data": "bar,baz:foo", + "valid": false + }, + { + "description": "invalid userinfo", + "data": "https://[@example.org/test.txt", + "valid": false + }, + { + "description": "unescaped non US-ASCII characters", + "data": "https://example.org/foobar®.txt", + "valid": false + }, + { + "description": "invalid backslash character", + "data": "https://example.org/foobar\\.txt", + "valid": false + }, + { + "description": "invalid \" character", + "data": "https://example.org/foobar\".txt", + "valid": false + }, + { + "description": "invalid <> characters", + "data": "https://example.org/foobar<>.txt", + "valid": false + }, + { + "description": "invalid {} characters", + "data": "https://example.org/foobar{}.txt", + "valid": false + }, + { + "description": "invalid ^ character", + "data": "https://example.org/foobar^.txt", + "valid": false + }, + { + "description": "invalid ` character", + "data": "https://example.org/foobar`.txt", + "valid": false + }, + { + "description": "invalid SPACE character", + "data": "https://example.org/foo bar.txt", + "valid": false + }, + { + "description": "invalid | character", + "data": "https://example.org/foobar|.txt", + "valid": false + }, + { + "description": "URI with leading-zero IPv4 is structurally valid as a reg-name", + "comment": "RFC 3986, Section 3.2.2: If an IP literal fails strict IPv4address parsing, it falls back to reg-name. A string of digits and dots is valid under unreserved characters. JSON Schema format asserts syntax, not scheme-specific DNS semantics.", + "data": "http://087.10.0.1/", + "valid": true + }, + { + "description": "URI with out-of-bounds IPv4 is structurally valid as a reg-name", + "comment": "RFC 3986, Section 3.2.2: Fallback to reg-name allows digits and dots.", + "data": "http://999.999.999.999/", + "valid": true + }, + { + "description": "invalid percent-encoding with non-hex digits", + "data": "http://example.com/%6G", + "valid": false + }, + { + "description": "incomplete percent-encoding triplet", + "data": "http://example.com/%A", + "valid": false + }, + { + "description": "lone percent sign is invalid", + "data": "http://example.com/%", + "valid": false + }, + { + "description": "scheme must start with a letter", + "data": "1http://example.com", + "valid": false + }, + { + "description": "invalid character in scheme", + "data": "ht_tp://example.com", + "valid": false + }, + { + "description": "non-numeric port is invalid", + "data": "http://example.com:abc/path", + "valid": false + }, + { + "description": "leading zero in an embedded IPv4 address is invalid", + "data": "http://[::ffff:01.2.3.4]", + "valid": false + }, + { + "description": "square brackets are not allowed in a path segment", + "data": "http:/[::1]", + "valid": false + }, + { + "description": "a trailing newline after a valid URI is invalid", + "data": "http://foo.bar/\n", + "valid": false + }, + { + "description": "an IPv6 address without enclosing brackets is invalid", + "data": "http://2001:0db8:85a3:0000:0000:8a2e:0370:7334", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/uuid.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/uuid.json new file mode 100644 index 000000000..4679e996a --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/format/uuid.json @@ -0,0 +1,156 @@ +[ + { + "description": "uuid format", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "format": "uuid" + }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "all upper-case", + "data": "2EB8AA08-AA98-11EA-B4AA-73B441D16380", + "valid": true + }, + { + "description": "all lower-case", + "data": "2eb8aa08-aa98-11ea-b4aa-73b441d16380", + "valid": true + }, + { + "description": "mixed case", + "data": "2eb8aa08-AA98-11ea-B4Aa-73B441D16380", + "valid": true + }, + { + "description": "all zeroes is valid", + "data": "00000000-0000-0000-0000-000000000000", + "valid": true + }, + { + "description": "wrong length", + "data": "2eb8aa08-aa98-11ea-b4aa-73b441d1638", + "valid": false + }, + { + "description": "missing section", + "data": "2eb8aa08-aa98-11ea-73b441d16380", + "valid": false + }, + { + "description": "bad characters (not hex)", + "data": "2eb8aa08-aa98-11ea-b4ga-73b441d16380", + "valid": false + }, + { + "description": "no dashes", + "data": "2eb8aa08aa9811eab4aa73b441d16380", + "valid": false + }, + { + "description": "too few dashes", + "data": "2eb8aa08aa98-11ea-b4aa73b441d16380", + "valid": false + }, + { + "description": "too many dashes", + "data": "2eb8-aa08-aa98-11ea-b4aa73b44-1d16380", + "valid": false + }, + { + "description": "dashes in the wrong spot", + "data": "2eb8aa08aa9811eab4aa73b441d16380----", + "valid": false + }, + { + "description": "shifted dashes", + "data": "2eb8aa0-8aa98-11e-ab4aa7-3b441d16380", + "valid": false + }, + { + "description": "valid version 4", + "data": "98d80576-482e-427f-8434-7f86890ab222", + "valid": true + }, + { + "description": "valid version 5", + "data": "99c17cbb-656f-564a-940f-1a4568f03487", + "valid": true + }, + { + "description": "hypothetical version 6", + "data": "99c17cbb-656f-664a-940f-1a4568f03487", + "valid": true + }, + { + "description": "hypothetical version 15", + "data": "99c17cbb-656f-f64a-940f-1a4568f03487", + "valid": true + }, + { + "description": "URN prefixed UUID is invalid", + "data": "urn:uuid:2eb8aa08-aa98-11ea-b4aa-73b441d16380", + "valid": false + }, + { + "description": "a UUID wrapped in curly braces is invalid", + "data": "{2eb8aa08-aa98-11ea-b4aa-73b441d16380}", + "valid": false + }, + { + "description": "trailing hyphen after a complete UUID is invalid", + "data": "2eb8aa08-aa98-11ea-b4aa-73b441d16380-", + "valid": false + }, + { + "description": "non-ASCII digit '২' (a Bengali 2) is invalid", + "data": "২eb8aa08-aa98-11ea-b4aa-73b441d16380", + "valid": false + }, + { + "description": "an underscore in place of a hex digit is invalid", + "data": "2eb8aa08-aa98-11ea-b4aa-73b441d1_380", + "valid": false + }, + { + "description": "a trailing newline after a complete UUID is invalid", + "data": "2eb8aa08-aa98-11ea-b4aa-73b441d16380\n", + "valid": false + }, + { + "description": "a variant nibble not defined by RFC 4122 is valid", + "data": "2eb8aa08-aa98-11ea-f4aa-73b441d16380", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/id.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/id.json new file mode 100644 index 000000000..0b7df4e80 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/id.json @@ -0,0 +1,53 @@ +[ + { + "description": "$id inside an enum is not a real identifier", + "comment": "the implementation must not be confused by an $id buried in the enum", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "id_in_enum": { + "enum": [ + { + "$id": "https://localhost:1234/draft2020-12/id/my_identifier.json", + "type": "null" + } + ] + }, + "real_id_in_schema": { + "$id": "https://localhost:1234/draft2020-12/id/my_identifier.json", + "type": "string" + }, + "zzz_id_in_const": { + "const": { + "$id": "https://localhost:1234/draft2020-12/id/my_identifier.json", + "type": "null" + } + } + }, + "anyOf": [ + { "$ref": "#/$defs/id_in_enum" }, + { "$ref": "https://localhost:1234/draft2020-12/id/my_identifier.json" } + ] + }, + "tests": [ + { + "description": "exact match to enum, and type matches", + "data": { + "$id": "https://localhost:1234/draft2020-12/id/my_identifier.json", + "type": "null" + }, + "valid": true + }, + { + "description": "match $ref to $id", + "data": "a string to match #/$defs/id_in_enum", + "valid": true + }, + { + "description": "no match on enum or $ref to $id", + "data": 1, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/no-schema.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/no-schema.json new file mode 100644 index 000000000..676e6b53a --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/no-schema.json @@ -0,0 +1,26 @@ +[ + { + "description": "validation without $schema", + "comment": "minLength is the same across all drafts", + "schema": { + "minLength": 2 + }, + "tests": [ + { + "description": "a 3-character string is valid", + "data": "foo", + "valid": true + }, + { + "description": "a 1-character string is not valid", + "data": "a", + "valid": false + }, + { + "description": "a non-string is valid", + "data": 5, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/non-bmp-regex.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/non-bmp-regex.json new file mode 100644 index 000000000..d2efb3ef6 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/non-bmp-regex.json @@ -0,0 +1,86 @@ +[ + { + "description": "Proper UTF-16 surrogate pair handling: pattern", + "comment": "Optional because .Net doesn't correctly handle 32-bit Unicode characters", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "pattern": "^🐲*$" + }, + "tests": [ + { + "description": "matches empty", + "data": "", + "valid": true + }, + { + "description": "matches single", + "data": "🐲", + "valid": true + }, + { + "description": "matches two", + "data": "🐲🐲", + "valid": true + }, + { + "description": "doesn't match one", + "data": "🐉", + "valid": false + }, + { + "description": "doesn't match two", + "data": "🐉🐉", + "valid": false + }, + { + "description": "doesn't match one ASCII", + "data": "D", + "valid": false + }, + { + "description": "doesn't match two ASCII", + "data": "DD", + "valid": false + } + ] + }, + { + "description": "Proper UTF-16 surrogate pair handling: patternProperties", + "comment": "Optional because .Net doesn't correctly handle 32-bit Unicode characters", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "patternProperties": { + "^🐲*$": { + "type": "integer" + } + } + }, + "tests": [ + { + "description": "matches empty", + "data": { "": 1 }, + "valid": true + }, + { + "description": "matches single", + "data": { "🐲": 1 }, + "valid": true + }, + { + "description": "matches two", + "data": { "🐲🐲": 1 }, + "valid": true + }, + { + "description": "doesn't match one", + "data": { "🐲": "hello" }, + "valid": false + }, + { + "description": "doesn't match two", + "data": { "🐲🐲": "hello" }, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/refOfUnknownKeyword.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/refOfUnknownKeyword.json new file mode 100644 index 000000000..ab1510dac --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/refOfUnknownKeyword.json @@ -0,0 +1,112 @@ +[ + { + "description": "reference of a root arbitrary keyword ", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "unknown-keyword": {"type": "integer"}, + "properties": { + "bar": {"$ref": "#/unknown-keyword"} + } + }, + "tests": [ + { + "description": "match", + "data": {"bar": 3}, + "valid": true + }, + { + "description": "mismatch", + "data": {"bar": true}, + "valid": false + } + ] + }, + { + "description": "reference of a root arbitrary keyword with encoded ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "unknown/keyword": {"type": "integer"}, + "properties": { + "bar": {"$ref": "#/unknown~1keyword"} + } + }, + "tests": [ + { + "description": "match", + "data": {"bar": 3}, + "valid": true + }, + { + "description": "mismatch", + "data": {"bar": true}, + "valid": false + } + ] + }, + { + "description": "reference of an arbitrary keyword of a sub-schema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": {"unknown-keyword": {"type": "integer"}}, + "bar": {"$ref": "#/properties/foo/unknown-keyword"} + } + }, + "tests": [ + { + "description": "match", + "data": {"bar": 3}, + "valid": true + }, + { + "description": "mismatch", + "data": {"bar": true}, + "valid": false + } + ] + }, + { + "description": "reference internals of known non-applicator", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "examples": [ + { "type": "string" } + ], + "$ref": "#/examples/0" + }, + "tests": [ + { + "description": "match", + "data": "a string", + "valid": true + }, + { + "description": "mismatch", + "data": 42, + "valid": false + } + ] + }, + { + "description": "reference of an arbitrary keyword of a sub-schema with encoded ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": {"unknown/keyword": {"type": "integer"}}, + "bar": {"$ref": "#/properties/foo/unknown~1keyword"} + } + }, + "tests": [ + { + "description": "match", + "data": {"bar": 3}, + "valid": true + }, + { + "description": "mismatch", + "data": {"bar": true}, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/unknownKeyword.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/unknownKeyword.json new file mode 100644 index 000000000..28b0c4ce9 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/optional/unknownKeyword.json @@ -0,0 +1,57 @@ +[ + { + "description": "$id inside an unknown keyword is not a real identifier", + "comment": "the implementation must not be confused by an $id in locations we do not know how to parse", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "id_in_unknown0": { + "not": { + "array_of_schemas": [ + { + "$id": "https://localhost:1234/draft2020-12/unknownKeyword/my_identifier.json", + "type": "null" + } + ] + } + }, + "real_id_in_schema": { + "$id": "https://localhost:1234/draft2020-12/unknownKeyword/my_identifier.json", + "type": "string" + }, + "id_in_unknown1": { + "not": { + "object_of_schemas": { + "foo": { + "$id": "https://localhost:1234/draft2020-12/unknownKeyword/my_identifier.json", + "type": "integer" + } + } + } + } + }, + "anyOf": [ + { "$ref": "#/$defs/id_in_unknown0" }, + { "$ref": "#/$defs/id_in_unknown1" }, + { "$ref": "https://localhost:1234/draft2020-12/unknownKeyword/my_identifier.json" } + ] + }, + "tests": [ + { + "description": "type matches second anyOf, which has a real schema in it", + "data": "a string", + "valid": true + }, + { + "description": "type matches non-schema in first anyOf", + "data": null, + "valid": false + }, + { + "description": "type matches non-schema in third anyOf", + "data": 1, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/pattern.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/pattern.json new file mode 100644 index 000000000..ed41f79a7 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/pattern.json @@ -0,0 +1,90 @@ +[ + { + "description": "pattern validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "pattern": "^a*$" + }, + "tests": [ + { + "description": "a matching pattern is valid", + "data": "aaa", + "valid": true + }, + { + "description": "a non-matching pattern is invalid", + "data": "abc", + "valid": false + }, + { + "description": "ignores booleans", + "data": true, + "valid": true + }, + { + "description": "ignores integers", + "data": 123, + "valid": true + }, + { + "description": "ignores floats", + "data": 1.0, + "valid": true + }, + { + "description": "ignores objects", + "data": {}, + "valid": true + }, + { + "description": "ignores arrays", + "data": [], + "valid": true + }, + { + "description": "ignores null", + "data": null, + "valid": true + } + ] + }, + { + "description": "pattern is not anchored", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "pattern": "a+" + }, + "tests": [ + { + "description": "matches a substring", + "data": "xxaayy", + "valid": true + } + ] + }, + { + "description": "pattern with Unicode property escape requires unicode mode", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "string", + "pattern": "^\\p{Letter}+$" + }, + "tests": [ + { + "description": "ASCII letters match", + "data": "Hello", + "valid": true + }, + { + "description": "Non-ASCII letters match", + "data": "π", + "valid": true + }, + { + "description": "Digits do not match", + "data": "123", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/patternProperties.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/patternProperties.json new file mode 100644 index 000000000..5a0ce941b --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/patternProperties.json @@ -0,0 +1,211 @@ +[ + { + "description": + "patternProperties validates properties matching a regex", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "patternProperties": { + "f.*o": {"type": "integer"} + } + }, + "tests": [ + { + "description": "a single valid match is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "multiple valid matches is valid", + "data": {"foo": 1, "foooooo" : 2}, + "valid": true + }, + { + "description": "a single invalid match is invalid", + "data": {"foo": "bar", "fooooo": 2}, + "valid": false + }, + { + "description": "multiple invalid matches is invalid", + "data": {"foo": "bar", "foooooo" : "baz"}, + "valid": false + }, + { + "description": "ignores arrays", + "data": ["foo"], + "valid": true + }, + { + "description": "ignores strings", + "data": "foo", + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + } + ] + }, + { + "description": "multiple simultaneous patternProperties are validated", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "patternProperties": { + "a*": {"type": "integer"}, + "aaa*": {"maximum": 20} + } + }, + "tests": [ + { + "description": "a single valid match is valid", + "data": {"a": 21}, + "valid": true + }, + { + "description": "a simultaneous match is valid", + "data": {"aaaa": 18}, + "valid": true + }, + { + "description": "multiple matches is valid", + "data": {"a": 21, "aaaa": 18}, + "valid": true + }, + { + "description": "an invalid due to one is invalid", + "data": {"a": "bar"}, + "valid": false + }, + { + "description": "an invalid due to the other is invalid", + "data": {"aaaa": 31}, + "valid": false + }, + { + "description": "an invalid due to both is invalid", + "data": {"aaa": "foo", "aaaa": 31}, + "valid": false + } + ] + }, + { + "description": "regexes are not anchored by default and are case sensitive", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "patternProperties": { + "[0-9]{2,}": { "type": "boolean" }, + "X_": { "type": "string" } + } + }, + "tests": [ + { + "description": "non recognized members are ignored", + "data": { "answer 1": "42" }, + "valid": true + }, + { + "description": "recognized members are accounted for", + "data": { "a31b": null }, + "valid": false + }, + { + "description": "regexes are case sensitive", + "data": { "a_x_3": 3 }, + "valid": true + }, + { + "description": "regexes are case sensitive, 2", + "data": { "a_X_3": 3 }, + "valid": false + } + ] + }, + { + "description": "patternProperties with boolean schemas", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "patternProperties": { + "f.*": true, + "b.*": false + } + }, + "tests": [ + { + "description": "object with property matching schema true is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "object with property matching schema false is invalid", + "data": {"bar": 2}, + "valid": false + }, + { + "description": "object with both properties is invalid", + "data": {"foo": 1, "bar": 2}, + "valid": false + }, + { + "description": "object with a property matching both true and false is invalid", + "data": {"foobar":1}, + "valid": false + }, + { + "description": "empty object is valid", + "data": {}, + "valid": true + } + ] + }, + { + "description": "patternProperties with null valued instance properties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "patternProperties": { + "^.*bar$": {"type": "null"} + } + }, + "tests": [ + { + "description": "allows null values", + "data": {"foobar": null}, + "valid": true + } + ] + }, + { + "description": "patternProperties with Unicode property escape", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "patternProperties": { + "^\\p{Letter}+$": { + "type": "number" + } + } + }, + "tests": [ + { + "description": "Unicode letter property name matches", + "data": { + "π": 1 + }, + "valid": true + }, + { + "description": "Unicode letter property name matches but value type is incorrect", + "data": { + "π": "1" + }, + "valid": false + }, + { + "description": "Non-letter property name does not match pattern", + "data": { + "123": 1 + }, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/prefixItems.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/prefixItems.json new file mode 100644 index 000000000..0adfc069e --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/prefixItems.json @@ -0,0 +1,104 @@ +[ + { + "description": "a schema given for prefixItems", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [ + {"type": "integer"}, + {"type": "string"} + ] + }, + "tests": [ + { + "description": "correct types", + "data": [ 1, "foo" ], + "valid": true + }, + { + "description": "wrong types", + "data": [ "foo", 1 ], + "valid": false + }, + { + "description": "incomplete array of items", + "data": [ 1 ], + "valid": true + }, + { + "description": "array with additional items", + "data": [ 1, "foo", true ], + "valid": true + }, + { + "description": "empty array", + "data": [ ], + "valid": true + }, + { + "description": "JavaScript pseudo-array is valid", + "data": { + "0": "invalid", + "1": "valid", + "length": 2 + }, + "valid": true + } + ] + }, + { + "description": "prefixItems with boolean schemas", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [true, false] + }, + "tests": [ + { + "description": "array with one item is valid", + "data": [ 1 ], + "valid": true + }, + { + "description": "array with two items is invalid", + "data": [ 1, "foo" ], + "valid": false + }, + { + "description": "empty array is valid", + "data": [], + "valid": true + } + ] + }, + { + "description": "additional items are allowed by default", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [{"type": "integer"}] + }, + "tests": [ + { + "description": "only the first item is validated", + "data": [1, "foo", false], + "valid": true + } + ] + }, + { + "description": "prefixItems with null instance elements", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [ + { + "type": "null" + } + ] + }, + "tests": [ + { + "description": "allows null elements", + "data": [ null ], + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/properties.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/properties.json new file mode 100644 index 000000000..eb66fa8bd --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/properties.json @@ -0,0 +1,242 @@ +[ + { + "description": "object properties validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": {"type": "integer"}, + "bar": {"type": "string"} + } + }, + "tests": [ + { + "description": "both properties present and valid is valid", + "data": {"foo": 1, "bar": "baz"}, + "valid": true + }, + { + "description": "one property invalid is invalid", + "data": {"foo": 1, "bar": {}}, + "valid": false + }, + { + "description": "both properties invalid is invalid", + "data": {"foo": [], "bar": {}}, + "valid": false + }, + { + "description": "doesn't invalidate other properties", + "data": {"quux": []}, + "valid": true + }, + { + "description": "ignores arrays", + "data": [], + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + } + ] + }, + { + "description": + "properties, patternProperties, additionalProperties interaction", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": {"type": "array", "maxItems": 3}, + "bar": {"type": "array"} + }, + "patternProperties": {"f.o": {"minItems": 2}}, + "additionalProperties": {"type": "integer"} + }, + "tests": [ + { + "description": "property validates property", + "data": {"foo": [1, 2]}, + "valid": true + }, + { + "description": "property invalidates property", + "data": {"foo": [1, 2, 3, 4]}, + "valid": false + }, + { + "description": "patternProperty invalidates property", + "data": {"foo": []}, + "valid": false + }, + { + "description": "patternProperty validates nonproperty", + "data": {"fxo": [1, 2]}, + "valid": true + }, + { + "description": "patternProperty invalidates nonproperty", + "data": {"fxo": []}, + "valid": false + }, + { + "description": "additionalProperty ignores property", + "data": {"bar": []}, + "valid": true + }, + { + "description": "additionalProperty validates others", + "data": {"quux": 3}, + "valid": true + }, + { + "description": "additionalProperty invalidates others", + "data": {"quux": "foo"}, + "valid": false + } + ] + }, + { + "description": "properties with boolean schema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": true, + "bar": false + } + }, + "tests": [ + { + "description": "no property present is valid", + "data": {}, + "valid": true + }, + { + "description": "only 'true' property present is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "only 'false' property present is invalid", + "data": {"bar": 2}, + "valid": false + }, + { + "description": "both properties present is invalid", + "data": {"foo": 1, "bar": 2}, + "valid": false + } + ] + }, + { + "description": "properties with escaped characters", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo\nbar": {"type": "number"}, + "foo\"bar": {"type": "number"}, + "foo\\bar": {"type": "number"}, + "foo\rbar": {"type": "number"}, + "foo\tbar": {"type": "number"}, + "foo\fbar": {"type": "number"} + } + }, + "tests": [ + { + "description": "object with all numbers is valid", + "data": { + "foo\nbar": 1, + "foo\"bar": 1, + "foo\\bar": 1, + "foo\rbar": 1, + "foo\tbar": 1, + "foo\fbar": 1 + }, + "valid": true + }, + { + "description": "object with strings is invalid", + "data": { + "foo\nbar": "1", + "foo\"bar": "1", + "foo\\bar": "1", + "foo\rbar": "1", + "foo\tbar": "1", + "foo\fbar": "1" + }, + "valid": false + } + ] + }, + { + "description": "properties with null valued instance properties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": {"type": "null"} + } + }, + "tests": [ + { + "description": "allows null values", + "data": {"foo": null}, + "valid": true + } + ] + }, + { + "description": "properties whose names are Javascript object property names", + "comment": "Ensure JS implementations don't universally consider e.g. __proto__ to always be present in an object.", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "__proto__": {"type": "number"}, + "toString": { + "properties": { "length": { "type": "string" } } + }, + "constructor": {"type": "number"} + } + }, + "tests": [ + { + "description": "ignores arrays", + "data": [], + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + }, + { + "description": "none of the properties mentioned", + "data": {}, + "valid": true + }, + { + "description": "__proto__ not valid", + "data": { "__proto__": "foo" }, + "valid": false + }, + { + "description": "toString not valid", + "data": { "toString": { "length": 37 } }, + "valid": false + }, + { + "description": "constructor not valid", + "data": { "constructor": { "length": 37 } }, + "valid": false + }, + { + "description": "all present and valid", + "data": { + "__proto__": 12, + "toString": { "length": "foo" }, + "constructor": 37 + }, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/propertyNames.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/propertyNames.json new file mode 100644 index 000000000..84ca2b84e --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/propertyNames.json @@ -0,0 +1,178 @@ +[ + { + "description": "propertyNames validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "propertyNames": {"maxLength": 3} + }, + "tests": [ + { + "description": "all property names valid", + "data": { + "f": {}, + "foo": {} + }, + "valid": true + }, + { + "description": "some property names invalid", + "data": { + "foo": {}, + "foobar": {} + }, + "valid": false + }, + { + "description": "object without properties is valid", + "data": {}, + "valid": true + }, + { + "description": "ignores arrays", + "data": [1, 2, 3, 4], + "valid": true + }, + { + "description": "ignores strings", + "data": "foobar", + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + }, + { + "description": "ignores null", + "data": null, + "valid": true + }, + { + "description": "ignores booleans", + "data": true, + "valid": true + } + ] + }, + { + "description": "propertyNames validation with pattern", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "propertyNames": { "pattern": "^a+$" } + }, + "tests": [ + { + "description": "matching property names valid", + "data": { + "a": {}, + "aa": {}, + "aaa": {} + }, + "valid": true + }, + { + "description": "non-matching property name is invalid", + "data": { + "aaA": {} + }, + "valid": false + }, + { + "description": "object without properties is valid", + "data": {}, + "valid": true + } + ] + }, + { + "description": "propertyNames with boolean schema true", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "propertyNames": true + }, + "tests": [ + { + "description": "object with any properties is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "empty object is valid", + "data": {}, + "valid": true + } + ] + }, + { + "description": "propertyNames with boolean schema false", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "propertyNames": false + }, + "tests": [ + { + "description": "object with any properties is invalid", + "data": {"foo": 1}, + "valid": false + }, + { + "description": "empty object is valid", + "data": {}, + "valid": true + } + ] + }, + { + "description": "propertyNames with const", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "propertyNames": {"const": "foo"} + }, + "tests": [ + { + "description": "object with property foo is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "object with any other property is invalid", + "data": {"bar": 1}, + "valid": false + }, + { + "description": "empty object is valid", + "data": {}, + "valid": true + } + ] + }, + { + "description": "propertyNames with enum", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "propertyNames": {"enum": ["foo", "bar"]} + }, + "tests": [ + { + "description": "object with property foo is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "object with property foo and bar is valid", + "data": {"foo": 1, "bar": 1}, + "valid": true + }, + { + "description": "object with any other property is invalid", + "data": {"baz": 1}, + "valid": false + }, + { + "description": "empty object is valid", + "data": {}, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/ref.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/ref.json new file mode 100644 index 000000000..9ca82eeb0 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/ref.json @@ -0,0 +1,1085 @@ +[ + { + "description": "root pointer ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": {"$ref": "#"} + }, + "additionalProperties": false + }, + "tests": [ + { + "description": "match", + "data": {"foo": false}, + "valid": true + }, + { + "description": "recursive match", + "data": {"foo": {"foo": false}}, + "valid": true + }, + { + "description": "mismatch", + "data": {"bar": false}, + "valid": false + }, + { + "description": "recursive mismatch", + "data": {"foo": {"bar": false}}, + "valid": false + } + ] + }, + { + "description": "relative pointer ref to object", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": {"type": "integer"}, + "bar": {"$ref": "#/properties/foo"} + } + }, + "tests": [ + { + "description": "match", + "data": {"bar": 3}, + "valid": true + }, + { + "description": "mismatch", + "data": {"bar": true}, + "valid": false + } + ] + }, + { + "description": "relative pointer ref to array", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [ + {"type": "integer"}, + {"$ref": "#/prefixItems/0"} + ] + }, + "tests": [ + { + "description": "match array", + "data": [1, 2], + "valid": true + }, + { + "description": "mismatch array", + "data": [1, "foo"], + "valid": false + } + ] + }, + { + "description": "escaped pointer ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "tilde~field": {"type": "integer"}, + "slash/field": {"type": "integer"}, + "percent%field": {"type": "integer"} + }, + "properties": { + "tilde": {"$ref": "#/$defs/tilde~0field"}, + "slash": {"$ref": "#/$defs/slash~1field"}, + "percent": {"$ref": "#/$defs/percent%25field"} + } + }, + "tests": [ + { + "description": "slash invalid", + "data": {"slash": "aoeu"}, + "valid": false + }, + { + "description": "tilde invalid", + "data": {"tilde": "aoeu"}, + "valid": false + }, + { + "description": "percent invalid", + "data": {"percent": "aoeu"}, + "valid": false + }, + { + "description": "slash valid", + "data": {"slash": 123}, + "valid": true + }, + { + "description": "tilde valid", + "data": {"tilde": 123}, + "valid": true + }, + { + "description": "percent valid", + "data": {"percent": 123}, + "valid": true + } + ] + }, + { + "description": "nested refs", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "a": {"type": "integer"}, + "b": {"$ref": "#/$defs/a"}, + "c": {"$ref": "#/$defs/b"} + }, + "$ref": "#/$defs/c" + }, + "tests": [ + { + "description": "nested ref valid", + "data": 5, + "valid": true + }, + { + "description": "nested ref invalid", + "data": "a", + "valid": false + } + ] + }, + { + "description": "ref applies alongside sibling keywords", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "reffed": { + "type": "array" + } + }, + "properties": { + "foo": { + "$ref": "#/$defs/reffed", + "maxItems": 2 + } + } + }, + "tests": [ + { + "description": "ref valid, maxItems valid", + "data": { "foo": [] }, + "valid": true + }, + { + "description": "ref valid, maxItems invalid", + "data": { "foo": [1, 2, 3] }, + "valid": false + }, + { + "description": "ref invalid", + "data": { "foo": "string" }, + "valid": false + } + ] + }, + { + "description": "remote ref, containing refs itself", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "https://json-schema.org/draft/2020-12/schema" + }, + "tests": [ + { + "description": "remote ref valid", + "data": {"minLength": 1}, + "valid": true + }, + { + "description": "remote ref invalid", + "data": {"minLength": -1}, + "valid": false + } + ] + }, + { + "description": "property named $ref that is not a reference", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "$ref": {"type": "string"} + } + }, + "tests": [ + { + "description": "property named $ref valid", + "data": {"$ref": "a"}, + "valid": true + }, + { + "description": "property named $ref invalid", + "data": {"$ref": 2}, + "valid": false + } + ] + }, + { + "description": "property named $ref, containing an actual $ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "$ref": {"$ref": "#/$defs/is-string"} + }, + "$defs": { + "is-string": { + "type": "string" + } + } + }, + "tests": [ + { + "description": "property named $ref valid", + "data": {"$ref": "a"}, + "valid": true + }, + { + "description": "property named $ref invalid", + "data": {"$ref": 2}, + "valid": false + } + ] + }, + { + "description": "$ref to boolean schema true", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "#/$defs/bool", + "$defs": { + "bool": true + } + }, + "tests": [ + { + "description": "any value is valid", + "data": "foo", + "valid": true + } + ] + }, + { + "description": "$ref to boolean schema false", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "#/$defs/bool", + "$defs": { + "bool": false + } + }, + "tests": [ + { + "description": "any value is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "Recursive references between schemas", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://localhost:1234/draft2020-12/tree", + "description": "tree of nodes", + "type": "object", + "properties": { + "meta": {"type": "string"}, + "nodes": { + "type": "array", + "items": {"$ref": "node"} + } + }, + "required": ["meta", "nodes"], + "$defs": { + "node": { + "$id": "http://localhost:1234/draft2020-12/node", + "description": "node", + "type": "object", + "properties": { + "value": {"type": "number"}, + "subtree": {"$ref": "tree"} + }, + "required": ["value"] + } + } + }, + "tests": [ + { + "description": "valid tree", + "data": { + "meta": "root", + "nodes": [ + { + "value": 1, + "subtree": { + "meta": "child", + "nodes": [ + {"value": 1.1}, + {"value": 1.2} + ] + } + }, + { + "value": 2, + "subtree": { + "meta": "child", + "nodes": [ + {"value": 2.1}, + {"value": 2.2} + ] + } + } + ] + }, + "valid": true + }, + { + "description": "invalid tree", + "data": { + "meta": "root", + "nodes": [ + { + "value": 1, + "subtree": { + "meta": "child", + "nodes": [ + {"value": "string is invalid"}, + {"value": 1.2} + ] + } + }, + { + "value": 2, + "subtree": { + "meta": "child", + "nodes": [ + {"value": 2.1}, + {"value": 2.2} + ] + } + } + ] + }, + "valid": false + } + ] + }, + { + "description": "refs with quote", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo\"bar": {"$ref": "#/$defs/foo%22bar"} + }, + "$defs": { + "foo\"bar": {"type": "number"} + } + }, + "tests": [ + { + "description": "object with numbers is valid", + "data": { + "foo\"bar": 1 + }, + "valid": true + }, + { + "description": "object with strings is invalid", + "data": { + "foo\"bar": "1" + }, + "valid": false + } + ] + }, + { + "description": "ref creates new scope when adjacent to keywords", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "A": { + "unevaluatedProperties": false + } + }, + "properties": { + "prop1": { + "type": "string" + } + }, + "$ref": "#/$defs/A" + }, + "tests": [ + { + "description": "referenced subschema doesn't see annotations from properties", + "data": { + "prop1": "match" + }, + "valid": false + } + ] + }, + { + "description": "naive replacement of $ref with its destination is not correct", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "a_string": { "type": "string" } + }, + "enum": [ + { "$ref": "#/$defs/a_string" } + ] + }, + "tests": [ + { + "description": "do not evaluate the $ref inside the enum, matching any string", + "data": "this is a string", + "valid": false + }, + { + "description": "do not evaluate the $ref inside the enum, definition exact match", + "data": { "type": "string" }, + "valid": false + }, + { + "description": "match the enum exactly", + "data": { "$ref": "#/$defs/a_string" }, + "valid": true + } + ] + }, + { + "description": "refs with relative uris and defs", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://example.com/schema-relative-uri-defs1.json", + "properties": { + "foo": { + "$id": "schema-relative-uri-defs2.json", + "$defs": { + "inner": { + "properties": { + "bar": { "type": "string" } + } + } + }, + "$ref": "#/$defs/inner" + } + }, + "$ref": "schema-relative-uri-defs2.json" + }, + "tests": [ + { + "description": "invalid on inner field", + "data": { + "foo": { + "bar": 1 + }, + "bar": "a" + }, + "valid": false + }, + { + "description": "invalid on outer field", + "data": { + "foo": { + "bar": "a" + }, + "bar": 1 + }, + "valid": false + }, + { + "description": "valid on both fields", + "data": { + "foo": { + "bar": "a" + }, + "bar": "a" + }, + "valid": true + } + ] + }, + { + "description": "relative refs with absolute uris and defs", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://example.com/schema-refs-absolute-uris-defs1.json", + "properties": { + "foo": { + "$id": "http://example.com/schema-refs-absolute-uris-defs2.json", + "$defs": { + "inner": { + "properties": { + "bar": { "type": "string" } + } + } + }, + "$ref": "#/$defs/inner" + } + }, + "$ref": "schema-refs-absolute-uris-defs2.json" + }, + "tests": [ + { + "description": "invalid on inner field", + "data": { + "foo": { + "bar": 1 + }, + "bar": "a" + }, + "valid": false + }, + { + "description": "invalid on outer field", + "data": { + "foo": { + "bar": "a" + }, + "bar": 1 + }, + "valid": false + }, + { + "description": "valid on both fields", + "data": { + "foo": { + "bar": "a" + }, + "bar": "a" + }, + "valid": true + } + ] + }, + { + "description": "$id must be resolved against nearest parent, not just immediate parent", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://example.com/a.json", + "$defs": { + "x": { + "$id": "http://example.com/b/c.json", + "not": { + "$defs": { + "y": { + "$id": "d.json", + "type": "number" + } + } + } + } + }, + "allOf": [ + { + "$ref": "http://example.com/b/d.json" + } + ] + }, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "non-number is invalid", + "data": "a", + "valid": false + } + ] + }, + { + "description": "order of evaluation: $id and $ref", + "schema": { + "$comment": "$id must be evaluated before $ref to get the proper $ref destination", + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://example.com/draft2020-12/ref-and-id1/base.json", + "$ref": "int.json", + "$defs": { + "bigint": { + "$comment": "canonical uri: https://example.com/ref-and-id1/int.json", + "$id": "int.json", + "maximum": 10 + }, + "smallint": { + "$comment": "canonical uri: https://example.com/ref-and-id1-int.json", + "$id": "/draft2020-12/ref-and-id1-int.json", + "maximum": 2 + } + } + }, + "tests": [ + { + "description": "data is valid against first definition", + "data": 5, + "valid": true + }, + { + "description": "data is invalid against first definition", + "data": 50, + "valid": false + } + ] + }, + { + "description": "order of evaluation: $id and $anchor and $ref", + "schema": { + "$comment": "$id must be evaluated before $ref to get the proper $ref destination", + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://example.com/draft2020-12/ref-and-id2/base.json", + "$ref": "#bigint", + "$defs": { + "bigint": { + "$comment": "canonical uri: /ref-and-id2/base.json#/$defs/bigint; another valid uri for this location: /ref-and-id2/base.json#bigint", + "$anchor": "bigint", + "maximum": 10 + }, + "smallint": { + "$comment": "canonical uri: https://example.com/ref-and-id2#/$defs/smallint; another valid uri for this location: https://example.com/ref-and-id2/#bigint", + "$id": "https://example.com/draft2020-12/ref-and-id2/", + "$anchor": "bigint", + "maximum": 2 + } + } + }, + "tests": [ + { + "description": "data is valid against first definition", + "data": 5, + "valid": true + }, + { + "description": "data is invalid against first definition", + "data": 50, + "valid": false + } + ] + }, + { + "description": "order of evaluation: $id and $ref on nested schema", + "schema": { + "$comment": "$id must be evaluated before $ref to get the proper $ref destination", + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://example.com/draft2020-12/ref-and-id3/base.json", + "$ref": "nested/foo.json", + "$defs": { + "foo": { + "$comment": "canonical uri: https://example.com/draft2020-12/ref-and-id3/nested/foo.json", + "$id": "nested/foo.json", + "$ref": "./bar.json" + }, + "bar": { + "$comment": "canonical uri: https://example.com/draft2020-12/ref-and-id3/nested/bar.json", + "$id": "nested/bar.json", + "type": "number" + } + } + }, + "tests": [ + { + "description": "data is valid against nested sibling", + "data": 5, + "valid": true + }, + { + "description": "data is invalid against nested sibling", + "data": "a", + "valid": false + } + ] + }, + { + "description": "simple URN base URI with $ref via the URN", + "schema": { + "$comment": "URIs do not have to have HTTP(s) schemes", + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "urn:uuid:deadbeef-1234-ffff-ffff-4321feebdaed", + "minimum": 30, + "properties": { + "foo": {"$ref": "urn:uuid:deadbeef-1234-ffff-ffff-4321feebdaed"} + } + }, + "tests": [ + { + "description": "valid under the URN IDed schema", + "data": {"foo": 37}, + "valid": true + }, + { + "description": "invalid under the URN IDed schema", + "data": {"foo": 12}, + "valid": false + } + ] + }, + { + "description": "simple URN base URI with JSON pointer", + "schema": { + "$comment": "URIs do not have to have HTTP(s) schemes", + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "urn:uuid:deadbeef-1234-00ff-ff00-4321feebdaed", + "properties": { + "foo": {"$ref": "#/$defs/bar"} + }, + "$defs": { + "bar": {"type": "string"} + } + }, + "tests": [ + { + "description": "a string is valid", + "data": {"foo": "bar"}, + "valid": true + }, + { + "description": "a non-string is invalid", + "data": {"foo": 12}, + "valid": false + } + ] + }, + { + "description": "URN base URI with NSS", + "schema": { + "$comment": "RFC 8141 §2.2", + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "urn:example:1/406/47452/2", + "properties": { + "foo": {"$ref": "#/$defs/bar"} + }, + "$defs": { + "bar": {"type": "string"} + } + }, + "tests": [ + { + "description": "a string is valid", + "data": {"foo": "bar"}, + "valid": true + }, + { + "description": "a non-string is invalid", + "data": {"foo": 12}, + "valid": false + } + ] + }, + { + "description": "URN base URI with r-component", + "schema": { + "$comment": "RFC 8141 §2.3.1", + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "urn:example:foo-bar-baz-qux?+CCResolve:cc=uk", + "properties": { + "foo": {"$ref": "#/$defs/bar"} + }, + "$defs": { + "bar": {"type": "string"} + } + }, + "tests": [ + { + "description": "a string is valid", + "data": {"foo": "bar"}, + "valid": true + }, + { + "description": "a non-string is invalid", + "data": {"foo": 12}, + "valid": false + } + ] + }, + { + "description": "URN base URI with q-component", + "schema": { + "$comment": "RFC 8141 §2.3.2", + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "urn:example:weather?=op=map&lat=39.56&lon=-104.85&datetime=1969-07-21T02:56:15Z", + "properties": { + "foo": {"$ref": "#/$defs/bar"} + }, + "$defs": { + "bar": {"type": "string"} + } + }, + "tests": [ + { + "description": "a string is valid", + "data": {"foo": "bar"}, + "valid": true + }, + { + "description": "a non-string is invalid", + "data": {"foo": 12}, + "valid": false + } + ] + }, + { + "description": "URN base URI with URN and JSON pointer ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "urn:uuid:deadbeef-1234-0000-0000-4321feebdaed", + "properties": { + "foo": {"$ref": "urn:uuid:deadbeef-1234-0000-0000-4321feebdaed#/$defs/bar"} + }, + "$defs": { + "bar": {"type": "string"} + } + }, + "tests": [ + { + "description": "a string is valid", + "data": {"foo": "bar"}, + "valid": true + }, + { + "description": "a non-string is invalid", + "data": {"foo": 12}, + "valid": false + } + ] + }, + { + "description": "URN base URI with URN and anchor ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "urn:uuid:deadbeef-1234-ff00-00ff-4321feebdaed", + "properties": { + "foo": {"$ref": "urn:uuid:deadbeef-1234-ff00-00ff-4321feebdaed#something"} + }, + "$defs": { + "bar": { + "$anchor": "something", + "type": "string" + } + } + }, + "tests": [ + { + "description": "a string is valid", + "data": {"foo": "bar"}, + "valid": true + }, + { + "description": "a non-string is invalid", + "data": {"foo": 12}, + "valid": false + } + ] + }, + { + "description": "URN ref with nested pointer ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "urn:uuid:deadbeef-4321-ffff-ffff-1234feebdaed", + "$defs": { + "foo": { + "$id": "urn:uuid:deadbeef-4321-ffff-ffff-1234feebdaed", + "$defs": {"bar": {"type": "string"}}, + "$ref": "#/$defs/bar" + } + } + }, + "tests": [ + { + "description": "a string is valid", + "data": "bar", + "valid": true + }, + { + "description": "a non-string is invalid", + "data": 12, + "valid": false + } + ] + }, + { + "description": "ref to if", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "http://example.com/ref/if", + "if": { + "$id": "http://example.com/ref/if", + "type": "integer" + } + }, + "tests": [ + { + "description": "a non-integer is invalid due to the $ref", + "data": "foo", + "valid": false + }, + { + "description": "an integer is valid", + "data": 12, + "valid": true + } + ] + }, + { + "description": "ref to then", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "http://example.com/ref/then", + "then": { + "$id": "http://example.com/ref/then", + "type": "integer" + } + }, + "tests": [ + { + "description": "a non-integer is invalid due to the $ref", + "data": "foo", + "valid": false + }, + { + "description": "an integer is valid", + "data": 12, + "valid": true + } + ] + }, + { + "description": "ref to else", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "http://example.com/ref/else", + "else": { + "$id": "http://example.com/ref/else", + "type": "integer" + } + }, + "tests": [ + { + "description": "a non-integer is invalid due to the $ref", + "data": "foo", + "valid": false + }, + { + "description": "an integer is valid", + "data": 12, + "valid": true + } + ] + }, + { + "description": "ref with absolute-path-reference", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://example.com/ref/absref.json", + "$defs": { + "a": { + "$id": "http://example.com/ref/absref/foobar.json", + "type": "number" + }, + "b": { + "$id": "http://example.com/absref/foobar.json", + "type": "string" + } + }, + "$ref": "/absref/foobar.json" + }, + "tests": [ + { + "description": "a string is valid", + "data": "foo", + "valid": true + }, + { + "description": "an integer is invalid", + "data": 12, + "valid": false + } + ] + }, + { + "description": "$id with file URI still resolves pointers - *nix", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "file:///folder/file.json", + "$defs": { + "foo": { + "type": "number" + } + }, + "$ref": "#/$defs/foo" + }, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "non-number is invalid", + "data": "a", + "valid": false + } + ] + }, + { + "description": "$id with file URI still resolves pointers - windows", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "file:///c:/folder/file.json", + "$defs": { + "foo": { + "type": "number" + } + }, + "$ref": "#/$defs/foo" + }, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "non-number is invalid", + "data": "a", + "valid": false + } + ] + }, + { + "description": "empty tokens in $ref json-pointer", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "": { + "$defs": { + "": { "type": "number" } + } + } + }, + "allOf": [ + { + "$ref": "#/$defs//$defs/" + } + ] + }, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "non-number is invalid", + "data": "a", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/refRemote.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/refRemote.json new file mode 100644 index 000000000..6083aabb1 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/refRemote.json @@ -0,0 +1,342 @@ +[ + { + "description": "remote ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "http://localhost:1234/draft2020-12/integer.json" + }, + "tests": [ + { + "description": "remote ref valid", + "data": 1, + "valid": true + }, + { + "description": "remote ref invalid", + "data": "a", + "valid": false + } + ] + }, + { + "description": "fragment within remote ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "http://localhost:1234/draft2020-12/subSchemas.json#/$defs/integer" + }, + "tests": [ + { + "description": "remote fragment valid", + "data": 1, + "valid": true + }, + { + "description": "remote fragment invalid", + "data": "a", + "valid": false + } + ] + }, + { + "description": "anchor within remote ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "http://localhost:1234/draft2020-12/locationIndependentIdentifier.json#foo" + }, + "tests": [ + { + "description": "remote anchor valid", + "data": 1, + "valid": true + }, + { + "description": "remote anchor invalid", + "data": "a", + "valid": false + } + ] + }, + { + "description": "ref within remote ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "http://localhost:1234/draft2020-12/subSchemas.json#/$defs/refToInteger" + }, + "tests": [ + { + "description": "ref within ref valid", + "data": 1, + "valid": true + }, + { + "description": "ref within ref invalid", + "data": "a", + "valid": false + } + ] + }, + { + "description": "base URI change", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://localhost:1234/draft2020-12/", + "items": { + "$id": "baseUriChange/", + "items": {"$ref": "folderInteger.json"} + } + }, + "tests": [ + { + "description": "base URI change ref valid", + "data": [[1]], + "valid": true + }, + { + "description": "base URI change ref invalid", + "data": [["a"]], + "valid": false + } + ] + }, + { + "description": "base URI change - change folder", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://localhost:1234/draft2020-12/scope_change_defs1.json", + "type" : "object", + "properties": {"list": {"$ref": "baseUriChangeFolder/"}}, + "$defs": { + "baz": { + "$id": "baseUriChangeFolder/", + "type": "array", + "items": {"$ref": "folderInteger.json"} + } + } + }, + "tests": [ + { + "description": "number is valid", + "data": {"list": [1]}, + "valid": true + }, + { + "description": "string is invalid", + "data": {"list": ["a"]}, + "valid": false + } + ] + }, + { + "description": "base URI change - change folder in subschema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://localhost:1234/draft2020-12/scope_change_defs2.json", + "type" : "object", + "properties": {"list": {"$ref": "baseUriChangeFolderInSubschema/#/$defs/bar"}}, + "$defs": { + "baz": { + "$id": "baseUriChangeFolderInSubschema/", + "$defs": { + "bar": { + "type": "array", + "items": {"$ref": "folderInteger.json"} + } + } + } + } + }, + "tests": [ + { + "description": "number is valid", + "data": {"list": [1]}, + "valid": true + }, + { + "description": "string is invalid", + "data": {"list": ["a"]}, + "valid": false + } + ] + }, + { + "description": "root ref in remote ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://localhost:1234/draft2020-12/object", + "type": "object", + "properties": { + "name": {"$ref": "name-defs.json#/$defs/orNull"} + } + }, + "tests": [ + { + "description": "string is valid", + "data": { + "name": "foo" + }, + "valid": true + }, + { + "description": "null is valid", + "data": { + "name": null + }, + "valid": true + }, + { + "description": "object is invalid", + "data": { + "name": { + "name": null + } + }, + "valid": false + } + ] + }, + { + "description": "remote ref with ref to defs", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://localhost:1234/draft2020-12/schema-remote-ref-ref-defs1.json", + "$ref": "ref-and-defs.json" + }, + "tests": [ + { + "description": "invalid", + "data": { + "bar": 1 + }, + "valid": false + }, + { + "description": "valid", + "data": { + "bar": "a" + }, + "valid": true + } + ] + }, + { + "description": "Location-independent identifier in remote ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "http://localhost:1234/draft2020-12/locationIndependentIdentifier.json#/$defs/refToInteger" + }, + "tests": [ + { + "description": "integer is valid", + "data": 1, + "valid": true + }, + { + "description": "string is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "retrieved nested refs resolve relative to their URI not $id", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "http://localhost:1234/draft2020-12/some-id", + "properties": { + "name": {"$ref": "nested/foo-ref-string.json"} + } + }, + "tests": [ + { + "description": "number is invalid", + "data": { + "name": {"foo": 1} + }, + "valid": false + }, + { + "description": "string is valid", + "data": { + "name": {"foo": "a"} + }, + "valid": true + } + ] + }, + { + "description": "remote HTTP ref with different $id", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "http://localhost:1234/draft2020-12/different-id-ref-string.json" + }, + "tests": [ + { + "description": "number is invalid", + "data": 1, + "valid": false + }, + { + "description": "string is valid", + "data": "foo", + "valid": true + } + ] + }, + { + "description": "remote HTTP ref with different URN $id", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "http://localhost:1234/draft2020-12/urn-ref-string.json" + }, + "tests": [ + { + "description": "number is invalid", + "data": 1, + "valid": false + }, + { + "description": "string is valid", + "data": "foo", + "valid": true + } + ] + }, + { + "description": "remote HTTP ref with nested absolute ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "http://localhost:1234/draft2020-12/nested-absolute-ref-to-string.json" + }, + "tests": [ + { + "description": "number is invalid", + "data": 1, + "valid": false + }, + { + "description": "string is valid", + "data": "foo", + "valid": true + } + ] + }, + { + "description": "$ref to $ref finds detached $anchor", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "http://localhost:1234/draft2020-12/detached-ref.json#/$defs/foo" + }, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "non-number is invalid", + "data": "a", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/required.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/required.json new file mode 100644 index 000000000..17db8939e --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/required.json @@ -0,0 +1,169 @@ +[ + { + "description": "required validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": {}, + "bar": {} + }, + "required": ["foo"] + }, + "tests": [ + { + "description": "present required property is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "non-present required property is invalid", + "data": {"bar": 1}, + "valid": false + }, + { + "description": "ignores arrays", + "data": [], + "valid": true + }, + { + "description": "ignores strings", + "data": "", + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + }, + { + "description": "ignores null", + "data": null, + "valid": true + }, + { + "description":"ignores boolean", + "data": true, + "valid": true + + } + ] + }, + { + "description": "required default validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": {} + } + }, + "tests": [ + { + "description": "not required by default", + "data": {}, + "valid": true + } + ] + }, + { + "description": "required with empty array", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": {} + }, + "required": [] + }, + "tests": [ + { + "description": "property not required", + "data": {}, + "valid": true + } + ] + }, + { + "description": "required with escaped characters", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "required": [ + "foo\nbar", + "foo\"bar", + "foo\\bar", + "foo\rbar", + "foo\tbar", + "foo\fbar" + ] + }, + "tests": [ + { + "description": "object with all properties present is valid", + "data": { + "foo\nbar": 1, + "foo\"bar": 1, + "foo\\bar": 1, + "foo\rbar": 1, + "foo\tbar": 1, + "foo\fbar": 1 + }, + "valid": true + }, + { + "description": "object with some properties missing is invalid", + "data": { + "foo\nbar": "1", + "foo\"bar": "1" + }, + "valid": false + } + ] + }, + { + "description": "required properties whose names are Javascript object property names", + "comment": "Ensure JS implementations don't universally consider e.g. __proto__ to always be present in an object.", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "required": ["__proto__", "toString", "constructor"] + }, + "tests": [ + { + "description": "ignores arrays", + "data": [], + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + }, + { + "description": "none of the properties mentioned", + "data": {}, + "valid": false + }, + { + "description": "__proto__ present", + "data": { "__proto__": "foo" }, + "valid": false + }, + { + "description": "toString present", + "data": { "toString": { "length": 37 } }, + "valid": false + }, + { + "description": "constructor present", + "data": { "constructor": { "length": 37 } }, + "valid": false + }, + { + "description": "all present", + "data": { + "__proto__": 12, + "toString": { "length": "foo" }, + "constructor": 37 + }, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/type.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/type.json new file mode 100644 index 000000000..2123c408d --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/type.json @@ -0,0 +1,501 @@ +[ + { + "description": "integer type matches integers", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "integer" + }, + "tests": [ + { + "description": "an integer is an integer", + "data": 1, + "valid": true + }, + { + "description": "a float with zero fractional part is an integer", + "data": 1.0, + "valid": true + }, + { + "description": "a float is not an integer", + "data": 1.1, + "valid": false + }, + { + "description": "a string is not an integer", + "data": "foo", + "valid": false + }, + { + "description": "a string is still not an integer, even if it looks like one", + "data": "1", + "valid": false + }, + { + "description": "an object is not an integer", + "data": {}, + "valid": false + }, + { + "description": "an array is not an integer", + "data": [], + "valid": false + }, + { + "description": "a boolean is not an integer", + "data": true, + "valid": false + }, + { + "description": "null is not an integer", + "data": null, + "valid": false + } + ] + }, + { + "description": "number type matches numbers", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "number" + }, + "tests": [ + { + "description": "an integer is a number", + "data": 1, + "valid": true + }, + { + "description": "a float with zero fractional part is a number (and an integer)", + "data": 1.0, + "valid": true + }, + { + "description": "a float is a number", + "data": 1.1, + "valid": true + }, + { + "description": "a string is not a number", + "data": "foo", + "valid": false + }, + { + "description": "a string is still not a number, even if it looks like one", + "data": "1", + "valid": false + }, + { + "description": "an object is not a number", + "data": {}, + "valid": false + }, + { + "description": "an array is not a number", + "data": [], + "valid": false + }, + { + "description": "a boolean is not a number", + "data": true, + "valid": false + }, + { + "description": "null is not a number", + "data": null, + "valid": false + } + ] + }, + { + "description": "string type matches strings", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "string" + }, + "tests": [ + { + "description": "1 is not a string", + "data": 1, + "valid": false + }, + { + "description": "a float is not a string", + "data": 1.1, + "valid": false + }, + { + "description": "a string is a string", + "data": "foo", + "valid": true + }, + { + "description": "a string is still a string, even if it looks like a number", + "data": "1", + "valid": true + }, + { + "description": "an empty string is still a string", + "data": "", + "valid": true + }, + { + "description": "an object is not a string", + "data": {}, + "valid": false + }, + { + "description": "an array is not a string", + "data": [], + "valid": false + }, + { + "description": "a boolean is not a string", + "data": true, + "valid": false + }, + { + "description": "null is not a string", + "data": null, + "valid": false + } + ] + }, + { + "description": "object type matches objects", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object" + }, + "tests": [ + { + "description": "an integer is not an object", + "data": 1, + "valid": false + }, + { + "description": "a float is not an object", + "data": 1.1, + "valid": false + }, + { + "description": "a string is not an object", + "data": "foo", + "valid": false + }, + { + "description": "an object is an object", + "data": {}, + "valid": true + }, + { + "description": "an array is not an object", + "data": [], + "valid": false + }, + { + "description": "a boolean is not an object", + "data": true, + "valid": false + }, + { + "description": "null is not an object", + "data": null, + "valid": false + } + ] + }, + { + "description": "array type matches arrays", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "array" + }, + "tests": [ + { + "description": "an integer is not an array", + "data": 1, + "valid": false + }, + { + "description": "a float is not an array", + "data": 1.1, + "valid": false + }, + { + "description": "a string is not an array", + "data": "foo", + "valid": false + }, + { + "description": "an object is not an array", + "data": {}, + "valid": false + }, + { + "description": "an array is an array", + "data": [], + "valid": true + }, + { + "description": "a boolean is not an array", + "data": true, + "valid": false + }, + { + "description": "null is not an array", + "data": null, + "valid": false + } + ] + }, + { + "description": "boolean type matches booleans", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "boolean" + }, + "tests": [ + { + "description": "an integer is not a boolean", + "data": 1, + "valid": false + }, + { + "description": "zero is not a boolean", + "data": 0, + "valid": false + }, + { + "description": "a float is not a boolean", + "data": 1.1, + "valid": false + }, + { + "description": "a string is not a boolean", + "data": "foo", + "valid": false + }, + { + "description": "an empty string is not a boolean", + "data": "", + "valid": false + }, + { + "description": "an object is not a boolean", + "data": {}, + "valid": false + }, + { + "description": "an array is not a boolean", + "data": [], + "valid": false + }, + { + "description": "true is a boolean", + "data": true, + "valid": true + }, + { + "description": "false is a boolean", + "data": false, + "valid": true + }, + { + "description": "null is not a boolean", + "data": null, + "valid": false + } + ] + }, + { + "description": "null type matches only the null object", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "null" + }, + "tests": [ + { + "description": "an integer is not null", + "data": 1, + "valid": false + }, + { + "description": "a float is not null", + "data": 1.1, + "valid": false + }, + { + "description": "zero is not null", + "data": 0, + "valid": false + }, + { + "description": "a string is not null", + "data": "foo", + "valid": false + }, + { + "description": "an empty string is not null", + "data": "", + "valid": false + }, + { + "description": "an object is not null", + "data": {}, + "valid": false + }, + { + "description": "an array is not null", + "data": [], + "valid": false + }, + { + "description": "true is not null", + "data": true, + "valid": false + }, + { + "description": "false is not null", + "data": false, + "valid": false + }, + { + "description": "null is null", + "data": null, + "valid": true + } + ] + }, + { + "description": "multiple types can be specified in an array", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": ["integer", "string"] + }, + "tests": [ + { + "description": "an integer is valid", + "data": 1, + "valid": true + }, + { + "description": "a string is valid", + "data": "foo", + "valid": true + }, + { + "description": "a float is invalid", + "data": 1.1, + "valid": false + }, + { + "description": "an object is invalid", + "data": {}, + "valid": false + }, + { + "description": "an array is invalid", + "data": [], + "valid": false + }, + { + "description": "a boolean is invalid", + "data": true, + "valid": false + }, + { + "description": "null is invalid", + "data": null, + "valid": false + } + ] + }, + { + "description": "type as array with one item", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": ["string"] + }, + "tests": [ + { + "description": "string is valid", + "data": "foo", + "valid": true + }, + { + "description": "number is invalid", + "data": 123, + "valid": false + } + ] + }, + { + "description": "type: array or object", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": ["array", "object"] + }, + "tests": [ + { + "description": "array is valid", + "data": [1,2,3], + "valid": true + }, + { + "description": "object is valid", + "data": {"foo": 123}, + "valid": true + }, + { + "description": "number is invalid", + "data": 123, + "valid": false + }, + { + "description": "string is invalid", + "data": "foo", + "valid": false + }, + { + "description": "null is invalid", + "data": null, + "valid": false + } + ] + }, + { + "description": "type: array, object or null", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": ["array", "object", "null"] + }, + "tests": [ + { + "description": "array is valid", + "data": [1,2,3], + "valid": true + }, + { + "description": "object is valid", + "data": {"foo": 123}, + "valid": true + }, + { + "description": "null is valid", + "data": null, + "valid": true + }, + { + "description": "number is invalid", + "data": 123, + "valid": false + }, + { + "description": "string is invalid", + "data": "foo", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/unevaluatedItems.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/unevaluatedItems.json new file mode 100644 index 000000000..d8a7c1b5b --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/unevaluatedItems.json @@ -0,0 +1,854 @@ +[ + { + "description": "unevaluatedItems true", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "unevaluatedItems": true + }, + "tests": [ + { + "description": "with no unevaluated items", + "data": [], + "valid": true + }, + { + "description": "with unevaluated items", + "data": ["foo"], + "valid": true + } + ] + }, + { + "description": "unevaluatedItems false", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "unevaluatedItems": false + }, + "tests": [ + { + "description": "with no unevaluated items", + "data": [], + "valid": true + }, + { + "description": "with unevaluated items", + "data": ["foo"], + "valid": false + } + ] + }, + { + "description": "unevaluatedItems as schema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "unevaluatedItems": { "type": "string" } + }, + "tests": [ + { + "description": "with no unevaluated items", + "data": [], + "valid": true + }, + { + "description": "with valid unevaluated items", + "data": ["foo"], + "valid": true + }, + { + "description": "with invalid unevaluated items", + "data": [42], + "valid": false + } + ] + }, + { + "description": "unevaluatedItems with uniform items", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "items": { "type": "string" }, + "unevaluatedItems": false + }, + "tests": [ + { + "description": "unevaluatedItems doesn't apply", + "data": ["foo", "bar"], + "valid": true + } + ] + }, + { + "description": "unevaluatedItems with tuple", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [ + { "type": "string" } + ], + "unevaluatedItems": false + }, + "tests": [ + { + "description": "with no unevaluated items", + "data": ["foo"], + "valid": true + }, + { + "description": "with unevaluated items", + "data": ["foo", "bar"], + "valid": false + } + ] + }, + { + "description": "unevaluatedItems with items and prefixItems", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [ + { "type": "string" } + ], + "items": true, + "unevaluatedItems": false + }, + "tests": [ + { + "description": "unevaluatedItems doesn't apply", + "data": ["foo", 42], + "valid": true + } + ] + }, + { + "description": "unevaluatedItems with items", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "items": {"type": "number"}, + "unevaluatedItems": {"type": "string"} + }, + "tests": [ + { + "description": "valid under items", + "comment": "no elements are considered by unevaluatedItems", + "data": [5, 6, 7, 8], + "valid": true + }, + { + "description": "invalid under items", + "data": ["foo", "bar", "baz"], + "valid": false + } + ] + }, + { + "description": "unevaluatedItems with nested tuple", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [ + { "type": "string" } + ], + "allOf": [ + { + "prefixItems": [ + true, + { "type": "number" } + ] + } + ], + "unevaluatedItems": false + }, + "tests": [ + { + "description": "with no unevaluated items", + "data": ["foo", 42], + "valid": true + }, + { + "description": "with unevaluated items", + "data": ["foo", 42, true], + "valid": false + } + ] + }, + { + "description": "unevaluatedItems with nested items", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "unevaluatedItems": {"type": "boolean"}, + "anyOf": [ + { "items": {"type": "string"} }, + true + ] + }, + "tests": [ + { + "description": "with only (valid) additional items", + "data": [true, false], + "valid": true + }, + { + "description": "with no additional items", + "data": ["yes", "no"], + "valid": true + }, + { + "description": "with invalid additional item", + "data": ["yes", false], + "valid": false + } + ] + }, + { + "description": "unevaluatedItems with nested prefixItems and items", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + { + "prefixItems": [ + { "type": "string" } + ], + "items": true + } + ], + "unevaluatedItems": false + }, + "tests": [ + { + "description": "with no additional items", + "data": ["foo"], + "valid": true + }, + { + "description": "with additional items", + "data": ["foo", 42, true], + "valid": true + } + ] + }, + { + "description": "unevaluatedItems with nested unevaluatedItems", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + { + "prefixItems": [ + { "type": "string" } + ] + }, + { "unevaluatedItems": true } + ], + "unevaluatedItems": false + }, + "tests": [ + { + "description": "with no additional items", + "data": ["foo"], + "valid": true + }, + { + "description": "with additional items", + "data": ["foo", 42, true], + "valid": true + } + ] + }, + { + "description": "unevaluatedItems with anyOf", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [ + { "const": "foo" } + ], + "anyOf": [ + { + "prefixItems": [ + true, + { "const": "bar" } + ] + }, + { + "prefixItems": [ + true, + true, + { "const": "baz" } + ] + } + ], + "unevaluatedItems": false + }, + "tests": [ + { + "description": "when one schema matches and has no unevaluated items", + "data": ["foo", "bar"], + "valid": true + }, + { + "description": "when one schema matches and has unevaluated items", + "data": ["foo", "bar", 42], + "valid": false + }, + { + "description": "when two schemas match and has no unevaluated items", + "data": ["foo", "bar", "baz"], + "valid": true + }, + { + "description": "when two schemas match and has unevaluated items", + "data": ["foo", "bar", "baz", 42], + "valid": false + } + ] + }, + { + "description": "unevaluatedItems with oneOf", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [ + { "const": "foo" } + ], + "oneOf": [ + { + "prefixItems": [ + true, + { "const": "bar" } + ] + }, + { + "prefixItems": [ + true, + { "const": "baz" } + ] + } + ], + "unevaluatedItems": false + }, + "tests": [ + { + "description": "with no unevaluated items", + "data": ["foo", "bar"], + "valid": true + }, + { + "description": "with unevaluated items", + "data": ["foo", "bar", 42], + "valid": false + } + ] + }, + { + "description": "unevaluatedItems with not", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [ + { "const": "foo" } + ], + "not": { + "not": { + "prefixItems": [ + true, + { "const": "bar" } + ] + } + }, + "unevaluatedItems": false + }, + "tests": [ + { + "description": "with unevaluated items", + "data": ["foo", "bar"], + "valid": false + } + ] + }, + { + "description": "unevaluatedItems with if/then/else", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [ + { "const": "foo" } + ], + "if": { + "prefixItems": [ + true, + { "const": "bar" } + ] + }, + "then": { + "prefixItems": [ + true, + true, + { "const": "then" } + ] + }, + "else": { + "prefixItems": [ + true, + true, + true, + { "const": "else" } + ] + }, + "unevaluatedItems": false + }, + "tests": [ + { + "description": "when if matches and it has no unevaluated items", + "data": ["foo", "bar", "then"], + "valid": true + }, + { + "description": "when if matches and it has unevaluated items", + "data": ["foo", "bar", "then", "else"], + "valid": false + }, + { + "description": "when if doesn't match and it has no unevaluated items", + "data": ["foo", 42, 42, "else"], + "valid": true + }, + { + "description": "when if doesn't match and it has unevaluated items", + "data": ["foo", 42, 42, "else", 42], + "valid": false + } + ] + }, + { + "description": "unevaluatedItems with boolean schemas", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [true], + "unevaluatedItems": false + }, + "tests": [ + { + "description": "with no unevaluated items", + "data": [], + "valid": true + }, + { + "description": "with unevaluated items", + "data": ["foo"], + "valid": false + } + ] + }, + { + "description": "unevaluatedItems with $ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "#/$defs/bar", + "prefixItems": [ + { "type": "string" } + ], + "unevaluatedItems": false, + "$defs": { + "bar": { + "prefixItems": [ + true, + { "type": "string" } + ] + } + } + }, + "tests": [ + { + "description": "with no unevaluated items", + "data": ["foo", "bar"], + "valid": true + }, + { + "description": "with unevaluated items", + "data": ["foo", "bar", "baz"], + "valid": false + } + ] + }, + { + "description": "unevaluatedItems before $ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "unevaluatedItems": false, + "prefixItems": [ + { "type": "string" } + ], + "$ref": "#/$defs/bar", + "$defs": { + "bar": { + "prefixItems": [ + true, + { "type": "string" } + ] + } + } + }, + "tests": [ + { + "description": "with no unevaluated items", + "data": ["foo", "bar"], + "valid": true + }, + { + "description": "with unevaluated items", + "data": ["foo", "bar", "baz"], + "valid": false + } + ] + }, + { + "description": "unevaluatedItems with $dynamicRef", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://example.com/unevaluated-items-with-dynamic-ref/derived", + + "$ref": "./baseSchema", + + "$defs": { + "derived": { + "$dynamicAnchor": "addons", + "prefixItems": [ + true, + { "type": "string" } + ] + }, + "baseSchema": { + "$id": "./baseSchema", + + "$comment": "unevaluatedItems comes first so it's more likely to catch bugs with implementations that are sensitive to keyword ordering", + "unevaluatedItems": false, + "type": "array", + "prefixItems": [ + { "type": "string" } + ], + "$dynamicRef": "#addons", + + "$defs": { + "defaultAddons": { + "$comment": "Needed to satisfy the bookending requirement", + "$dynamicAnchor": "addons" + } + } + } + } + }, + "tests": [ + { + "description": "with no unevaluated items", + "data": ["foo", "bar"], + "valid": true + }, + { + "description": "with unevaluated items", + "data": ["foo", "bar", "baz"], + "valid": false + } + ] + }, + { + "description": "unevaluatedItems can't see inside cousins", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + { + "prefixItems": [ true ] + }, + { "unevaluatedItems": false } + ] + }, + "tests": [ + { + "description": "always fails", + "data": [ 1 ], + "valid": false + } + ] + }, + { + "description": "item is evaluated in an uncle schema to unevaluatedItems", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { + "prefixItems": [ + { "type": "string" } + ], + "unevaluatedItems": false + } + }, + "anyOf": [ + { + "properties": { + "foo": { + "prefixItems": [ + true, + { "type": "string" } + ] + } + } + } + ] + }, + "tests": [ + { + "description": "no extra items", + "data": { + "foo": [ + "test" + ] + }, + "valid": true + }, + { + "description": "uncle keyword evaluation is not significant", + "data": { + "foo": [ + "test", + "test" + ] + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedItems depends on adjacent contains", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [true], + "contains": {"type": "string"}, + "unevaluatedItems": false + }, + "tests": [ + { + "description": "second item is evaluated by contains", + "data": [ 1, "foo" ], + "valid": true + }, + { + "description": "contains fails, second item is not evaluated", + "data": [ 1, 2 ], + "valid": false + }, + { + "description": "contains passes, second item is not evaluated", + "data": [ 1, 2, "foo" ], + "valid": false + } + ] + }, + { + "description": "unevaluatedItems depends on multiple nested contains", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + { "contains": { "multipleOf": 2 } }, + { "contains": { "multipleOf": 3 } } + ], + "unevaluatedItems": { "multipleOf": 5 } + }, + "tests": [ + { + "description": "5 not evaluated, passes unevaluatedItems", + "data": [ 2, 3, 4, 5, 6 ], + "valid": true + }, + { + "description": "7 not evaluated, fails unevaluatedItems", + "data": [ 2, 3, 4, 7, 8 ], + "valid": false + } + ] + }, + { + "description": "unevaluatedItems and contains interact to control item dependency relationship", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "if": { + "contains": {"const": "a"} + }, + "then": { + "if": { + "contains": {"const": "b"} + }, + "then": { + "if": { + "contains": {"const": "c"} + } + } + }, + "unevaluatedItems": false + }, + "tests": [ + { + "description": "empty array is valid", + "data": [], + "valid": true + }, + { + "description": "only a's are valid", + "data": [ "a", "a" ], + "valid": true + }, + { + "description": "a's and b's are valid", + "data": [ "a", "b", "a", "b", "a" ], + "valid": true + }, + { + "description": "a's, b's and c's are valid", + "data": [ "c", "a", "c", "c", "b", "a" ], + "valid": true + }, + { + "description": "only b's are invalid", + "data": [ "b", "b" ], + "valid": false + }, + { + "description": "only c's are invalid", + "data": [ "c", "c" ], + "valid": false + }, + { + "description": "only b's and c's are invalid", + "data": [ "c", "b", "c", "b", "c" ], + "valid": false + }, + { + "description": "only a's and c's are invalid", + "data": [ "c", "a", "c", "a", "c" ], + "valid": false + } + ] + }, + { + "description" : "unevaluatedItems with minContains = 0", + "schema" : { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "contains": {"type": "string"}, + "minContains": 0, + "unevaluatedItems": false + }, + "tests" : [ + { + "description": "empty array is valid", + "data": [], + "valid": true + }, + { + "description": "no items evaluated by contains", + "data": [0], + "valid": false + }, + { + "description": "some but not all items evaluated by contains", + "data": ["foo", 0], + "valid": false + }, + { + "description": "all items evaluated by contains", + "data": ["foo", "bar"], + "valid": true + } + ] + }, + { + "description": "non-array instances are valid", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "unevaluatedItems": false + }, + "tests": [ + { + "description": "ignores booleans", + "data": true, + "valid": true + }, + { + "description": "ignores integers", + "data": 123, + "valid": true + }, + { + "description": "ignores floats", + "data": 1.0, + "valid": true + }, + { + "description": "ignores objects", + "data": {}, + "valid": true + }, + { + "description": "ignores strings", + "data": "foo", + "valid": true + }, + { + "description": "ignores null", + "data": null, + "valid": true + } + ] + }, + { + "description": "unevaluatedItems with null instance elements", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "unevaluatedItems": { + "type": "null" + } + }, + "tests": [ + { + "description": "allows null elements", + "data": [ null ], + "valid": true + } + ] + }, + { + "description": "unevaluatedItems can see annotations from if without then and else", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "if": { + "prefixItems": [{"const": "a"}] + }, + "unevaluatedItems": false + }, + "tests": [ + { + "description": "valid in case if is evaluated", + "data": [ "a" ], + "valid": true + }, + { + "description": "invalid in case if is evaluated", + "data": [ "b" ], + "valid": false + } + ] + }, + { + "description": "Evaluated items collection needs to consider instance location", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [ + { + "prefixItems": [ + true, + { "type": "string" } + ] + } + ], + "unevaluatedItems": false + }, + "tests": [ + { + "description": "with an unevaluated item that exists at another location", + "data": [ + ["foo", "bar"], + "bar" + ], + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/unevaluatedProperties.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/unevaluatedProperties.json new file mode 100644 index 000000000..1995d0ea9 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/unevaluatedProperties.json @@ -0,0 +1,1681 @@ +[ + { + "description": "unevaluatedProperties true", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "unevaluatedProperties": true + }, + "tests": [ + { + "description": "with no unevaluated properties", + "data": {}, + "valid": true + }, + { + "description": "with unevaluated properties", + "data": { + "foo": "foo" + }, + "valid": true + } + ] + }, + { + "description": "unevaluatedProperties schema", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "unevaluatedProperties": { + "type": "string", + "minLength": 3 + } + }, + "tests": [ + { + "description": "with no unevaluated properties", + "data": {}, + "valid": true + }, + { + "description": "with valid unevaluated properties", + "data": { + "foo": "foo" + }, + "valid": true + }, + { + "description": "with invalid unevaluated properties", + "data": { + "foo": "fo" + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties false", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "with no unevaluated properties", + "data": {}, + "valid": true + }, + { + "description": "with unevaluated properties", + "data": { + "foo": "foo" + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties with adjacent properties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { "type": "string" } + }, + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "with no unevaluated properties", + "data": { + "foo": "foo" + }, + "valid": true + }, + { + "description": "with unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties with adjacent patternProperties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "patternProperties": { + "^foo": { "type": "string" } + }, + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "with no unevaluated properties", + "data": { + "foo": "foo" + }, + "valid": true + }, + { + "description": "with unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties with adjacent bool additionalProperties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "foo": { "type": "string" } + }, + "additionalProperties": true, + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "with no additional properties", + "data": { + "foo": "foo" + }, + "valid": true + }, + { + "description": "with additional properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": true + } + ] + }, + { + "description": "unevaluatedProperties with adjacent non-bool additionalProperties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "foo": { "type": "string" } + }, + "additionalProperties": {"type": "string"}, + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "with no additional properties", + "data": { + "foo": "foo" + }, + "valid": true + }, + { + "description": "with additional properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": true + } + ] + }, + { + "description": "unevaluatedProperties with nested properties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { "type": "string" } + }, + "allOf": [ + { + "properties": { + "bar": { "type": "string" } + } + } + ], + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "with no additional properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": true + }, + { + "description": "with additional properties", + "data": { + "foo": "foo", + "bar": "bar", + "baz": "baz" + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties with nested patternProperties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { "type": "string" } + }, + "allOf": [ + { + "patternProperties": { + "^bar": { "type": "string" } + } + } + ], + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "with no additional properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": true + }, + { + "description": "with additional properties", + "data": { + "foo": "foo", + "bar": "bar", + "baz": "baz" + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties with nested additionalProperties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { "type": "string" } + }, + "allOf": [ + { + "additionalProperties": true + } + ], + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "with no additional properties", + "data": { + "foo": "foo" + }, + "valid": true + }, + { + "description": "with additional properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": true + } + ] + }, + { + "description": "unevaluatedProperties with nested unevaluatedProperties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { "type": "string" } + }, + "allOf": [ + { + "unevaluatedProperties": true + } + ], + "unevaluatedProperties": { + "type": "string", + "maxLength": 2 + } + }, + "tests": [ + { + "description": "with no nested unevaluated properties", + "data": { + "foo": "foo" + }, + "valid": true + }, + { + "description": "with nested unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": true + } + ] + }, + { + "description": "unevaluatedProperties with anyOf", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { "type": "string" } + }, + "anyOf": [ + { + "properties": { + "bar": { "const": "bar" } + }, + "required": ["bar"] + }, + { + "properties": { + "baz": { "const": "baz" } + }, + "required": ["baz"] + }, + { + "properties": { + "quux": { "const": "quux" } + }, + "required": ["quux"] + } + ], + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "when one matches and has no unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": true + }, + { + "description": "when one matches and has unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar", + "baz": "not-baz" + }, + "valid": false + }, + { + "description": "when two match and has no unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar", + "baz": "baz" + }, + "valid": true + }, + { + "description": "when two match and has unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar", + "baz": "baz", + "quux": "not-quux" + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties with oneOf", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { "type": "string" } + }, + "oneOf": [ + { + "properties": { + "bar": { "const": "bar" } + }, + "required": ["bar"] + }, + { + "properties": { + "baz": { "const": "baz" } + }, + "required": ["baz"] + } + ], + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "with no unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": true + }, + { + "description": "with unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar", + "quux": "quux" + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties with not", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { "type": "string" } + }, + "not": { + "not": { + "properties": { + "bar": { "const": "bar" } + }, + "required": ["bar"] + } + }, + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "with unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties with if/then/else", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "if": { + "properties": { + "foo": { "const": "then" } + }, + "required": ["foo"] + }, + "then": { + "properties": { + "bar": { "type": "string" } + }, + "required": ["bar"] + }, + "else": { + "properties": { + "baz": { "type": "string" } + }, + "required": ["baz"] + }, + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "when if is true and has no unevaluated properties", + "data": { + "foo": "then", + "bar": "bar" + }, + "valid": true + }, + { + "description": "when if is true and has unevaluated properties", + "data": { + "foo": "then", + "bar": "bar", + "baz": "baz" + }, + "valid": false + }, + { + "description": "when if is false and has no unevaluated properties", + "data": { + "baz": "baz" + }, + "valid": true + }, + { + "description": "when if is false and has unevaluated properties", + "data": { + "foo": "else", + "baz": "baz" + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties with if/then/else, then not defined", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "if": { + "properties": { + "foo": { "const": "then" } + }, + "required": ["foo"] + }, + "else": { + "properties": { + "baz": { "type": "string" } + }, + "required": ["baz"] + }, + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "when if is true and has no unevaluated properties", + "data": { + "foo": "then" + }, + "valid": true + }, + { + "description": "when if is true and has unevaluated properties", + "data": { + "foo": "then", + "bar": "bar" + }, + "valid": false + }, + { + "description": "when if is false and has no unevaluated properties", + "data": { + "baz": "baz" + }, + "valid": true + }, + { + "description": "when if is false and has unevaluated properties", + "data": { + "foo": "else", + "baz": "baz" + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties with if/then/else, else not defined", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "if": { + "properties": { + "foo": { "const": "then" } + }, + "required": ["foo"] + }, + "then": { + "properties": { + "bar": { "type": "string" } + }, + "required": ["bar"] + }, + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "when if is true and has no unevaluated properties", + "data": { + "foo": "then", + "bar": "bar" + }, + "valid": true + }, + { + "description": "when if is true and has unevaluated properties", + "data": { + "foo": "then", + "bar": "bar", + "baz": "baz" + }, + "valid": false + }, + { + "description": "when if is false and has no unevaluated properties", + "data": { + "baz": "baz" + }, + "valid": false + }, + { + "description": "when if is false and has unevaluated properties", + "data": { + "foo": "else", + "baz": "baz" + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties with dependentSchemas", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { "type": "string" } + }, + "dependentSchemas": { + "foo": { + "properties": { + "bar": { "const": "bar" } + }, + "required": ["bar"] + } + }, + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "with no unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": true + }, + { + "description": "with unevaluated properties", + "data": { + "bar": "bar" + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties with boolean schemas", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { "type": "string" } + }, + "allOf": [true], + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "with no unevaluated properties", + "data": { + "foo": "foo" + }, + "valid": true + }, + { + "description": "with unevaluated properties", + "data": { + "bar": "bar" + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties with $ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$ref": "#/$defs/bar", + "properties": { + "foo": { "type": "string" } + }, + "unevaluatedProperties": false, + "$defs": { + "bar": { + "properties": { + "bar": { "type": "string" } + } + } + } + }, + "tests": [ + { + "description": "with no unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": true + }, + { + "description": "with unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar", + "baz": "baz" + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties before $ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "unevaluatedProperties": false, + "properties": { + "foo": { "type": "string" } + }, + "$ref": "#/$defs/bar", + "$defs": { + "bar": { + "properties": { + "bar": { "type": "string" } + } + } + } + }, + "tests": [ + { + "description": "with no unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": true + }, + { + "description": "with unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar", + "baz": "baz" + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties with $dynamicRef", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://example.com/unevaluated-properties-with-dynamic-ref/derived", + + "$ref": "./baseSchema", + + "$defs": { + "derived": { + "$dynamicAnchor": "addons", + "properties": { + "bar": { "type": "string" } + } + }, + "baseSchema": { + "$id": "./baseSchema", + + "$comment": "unevaluatedProperties comes first so it's more likely to catch bugs with implementations that are sensitive to keyword ordering", + "unevaluatedProperties": false, + "properties": { + "foo": { "type": "string" } + }, + "$dynamicRef": "#addons", + + "$defs": { + "defaultAddons": { + "$comment": "Needed to satisfy the bookending requirement", + "$dynamicAnchor": "addons" + } + } + } + } + }, + "tests": [ + { + "description": "with no unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": true + }, + { + "description": "with unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar", + "baz": "baz" + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties can't see inside cousins", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + { + "properties": { + "foo": true + } + }, + { + "unevaluatedProperties": false + } + ] + }, + "tests": [ + { + "description": "always fails", + "data": { + "foo": 1 + }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties can't see inside cousins (reverse order)", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + { + "unevaluatedProperties": false + }, + { + "properties": { + "foo": true + } + } + ] + }, + "tests": [ + { + "description": "always fails", + "data": { + "foo": 1 + }, + "valid": false + } + ] + }, + { + "description": "nested unevaluatedProperties, outer false, inner true, properties outside", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { "type": "string" } + }, + "allOf": [ + { + "unevaluatedProperties": true + } + ], + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "with no nested unevaluated properties", + "data": { + "foo": "foo" + }, + "valid": true + }, + { + "description": "with nested unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": true + } + ] + }, + { + "description": "nested unevaluatedProperties, outer false, inner true, properties inside", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + { + "properties": { + "foo": { "type": "string" } + }, + "unevaluatedProperties": true + } + ], + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "with no nested unevaluated properties", + "data": { + "foo": "foo" + }, + "valid": true + }, + { + "description": "with nested unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": true + } + ] + }, + { + "description": "nested unevaluatedProperties, outer true, inner false, properties outside", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { "type": "string" } + }, + "allOf": [ + { + "unevaluatedProperties": false + } + ], + "unevaluatedProperties": true + }, + "tests": [ + { + "description": "with no nested unevaluated properties", + "data": { + "foo": "foo" + }, + "valid": false + }, + { + "description": "with nested unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": false + } + ] + }, + { + "description": "nested unevaluatedProperties, outer true, inner false, properties inside", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + { + "properties": { + "foo": { "type": "string" } + }, + "unevaluatedProperties": false + } + ], + "unevaluatedProperties": true + }, + "tests": [ + { + "description": "with no nested unevaluated properties", + "data": { + "foo": "foo" + }, + "valid": true + }, + { + "description": "with nested unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": false + } + ] + }, + { + "description": "cousin unevaluatedProperties, true and false, true with properties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + { + "properties": { + "foo": { "type": "string" } + }, + "unevaluatedProperties": true + }, + { + "unevaluatedProperties": false + } + ] + }, + "tests": [ + { + "description": "with no nested unevaluated properties", + "data": { + "foo": "foo" + }, + "valid": false + }, + { + "description": "with nested unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": false + } + ] + }, + { + "description": "cousin unevaluatedProperties, true and false, false with properties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + { + "unevaluatedProperties": true + }, + { + "properties": { + "foo": { "type": "string" } + }, + "unevaluatedProperties": false + } + ] + }, + "tests": [ + { + "description": "with no nested unevaluated properties", + "data": { + "foo": "foo" + }, + "valid": true + }, + { + "description": "with nested unevaluated properties", + "data": { + "foo": "foo", + "bar": "bar" + }, + "valid": false + } + ] + }, + { + "description": "property is evaluated in an uncle schema to unevaluatedProperties", + "comment": "see https://stackoverflow.com/questions/66936884/deeply-nested-unevaluatedproperties-and-their-expectations", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { + "properties": { + "bar": { + "type": "string" + } + }, + "unevaluatedProperties": false + } + }, + "anyOf": [ + { + "properties": { + "foo": { + "properties": { + "faz": { + "type": "string" + } + } + } + } + } + ] + }, + "tests": [ + { + "description": "no extra properties", + "data": { + "foo": { + "bar": "test" + } + }, + "valid": true + }, + { + "description": "uncle keyword evaluation is not significant", + "data": { + "foo": { + "bar": "test", + "faz": "test" + } + }, + "valid": false + } + ] + }, + { + "description": "in-place applicator siblings, allOf has unevaluated", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + { + "properties": { + "foo": true + }, + "unevaluatedProperties": false + } + ], + "anyOf": [ + { + "properties": { + "bar": true + } + } + ] + }, + "tests": [ + { + "description": "base case: both properties present", + "data": { + "foo": 1, + "bar": 1 + }, + "valid": false + }, + { + "description": "in place applicator siblings, bar is missing", + "data": { + "foo": 1 + }, + "valid": true + }, + { + "description": "in place applicator siblings, foo is missing", + "data": { + "bar": 1 + }, + "valid": false + } + ] + }, + { + "description": "in-place applicator siblings, anyOf has unevaluated", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "allOf": [ + { + "properties": { + "foo": true + } + } + ], + "anyOf": [ + { + "properties": { + "bar": true + }, + "unevaluatedProperties": false + } + ] + }, + "tests": [ + { + "description": "base case: both properties present", + "data": { + "foo": 1, + "bar": 1 + }, + "valid": false + }, + { + "description": "in place applicator siblings, bar is missing", + "data": { + "foo": 1 + }, + "valid": false + }, + { + "description": "in place applicator siblings, foo is missing", + "data": { + "bar": 1 + }, + "valid": true + } + ] + }, + { + "description": "unevaluatedProperties + single cyclic ref", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "x": { "$ref": "#" } + }, + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "Empty is valid", + "data": {}, + "valid": true + }, + { + "description": "Single is valid", + "data": { "x": {} }, + "valid": true + }, + { + "description": "Unevaluated on 1st level is invalid", + "data": { "x": {}, "y": {} }, + "valid": false + }, + { + "description": "Nested is valid", + "data": { "x": { "x": {} } }, + "valid": true + }, + { + "description": "Unevaluated on 2nd level is invalid", + "data": { "x": { "x": {}, "y": {} } }, + "valid": false + }, + { + "description": "Deep nested is valid", + "data": { "x": { "x": { "x": {} } } }, + "valid": true + }, + { + "description": "Unevaluated on 3rd level is invalid", + "data": { "x": { "x": { "x": {}, "y": {} } } }, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties + ref inside allOf / oneOf", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "one": { + "properties": { "a": true } + }, + "two": { + "required": ["x"], + "properties": { "x": true } + } + }, + "allOf": [ + { "$ref": "#/$defs/one" }, + { "properties": { "b": true } }, + { + "oneOf": [ + { "$ref": "#/$defs/two" }, + { + "required": ["y"], + "properties": { "y": true } + } + ] + } + ], + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "Empty is invalid (no x or y)", + "data": {}, + "valid": false + }, + { + "description": "a and b are invalid (no x or y)", + "data": { "a": 1, "b": 1 }, + "valid": false + }, + { + "description": "x and y are invalid", + "data": { "x": 1, "y": 1 }, + "valid": false + }, + { + "description": "a and x are valid", + "data": { "a": 1, "x": 1 }, + "valid": true + }, + { + "description": "a and y are valid", + "data": { "a": 1, "y": 1 }, + "valid": true + }, + { + "description": "a and b and x are valid", + "data": { "a": 1, "b": 1, "x": 1 }, + "valid": true + }, + { + "description": "a and b and y are valid", + "data": { "a": 1, "b": 1, "y": 1 }, + "valid": true + }, + { + "description": "a and b and x and y are invalid", + "data": { "a": 1, "b": 1, "x": 1, "y": 1 }, + "valid": false + } + ] + }, + { + "description": "dynamic evaluation inside nested refs", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$defs": { + "one": { + "oneOf": [ + { "$ref": "#/$defs/two" }, + { "required": ["b"], "properties": { "b": true } }, + { "required": ["xx"], "patternProperties": { "x": true } }, + { "required": ["all"], "unevaluatedProperties": true } + ] + }, + "two": { + "oneOf": [ + { "required": ["c"], "properties": { "c": true } }, + { "required": ["d"], "properties": { "d": true } } + ] + } + }, + "oneOf": [ + { "$ref": "#/$defs/one" }, + { "required": ["a"], "properties": { "a": true } } + ], + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "Empty is invalid", + "data": {}, + "valid": false + }, + { + "description": "a is valid", + "data": { "a": 1 }, + "valid": true + }, + { + "description": "b is valid", + "data": { "b": 1 }, + "valid": true + }, + { + "description": "c is valid", + "data": { "c": 1 }, + "valid": true + }, + { + "description": "d is valid", + "data": { "d": 1 }, + "valid": true + }, + { + "description": "a + b is invalid", + "data": { "a": 1, "b": 1 }, + "valid": false + }, + { + "description": "a + c is invalid", + "data": { "a": 1, "c": 1 }, + "valid": false + }, + { + "description": "a + d is invalid", + "data": { "a": 1, "d": 1 }, + "valid": false + }, + { + "description": "b + c is invalid", + "data": { "b": 1, "c": 1 }, + "valid": false + }, + { + "description": "b + d is invalid", + "data": { "b": 1, "d": 1 }, + "valid": false + }, + { + "description": "c + d is invalid", + "data": { "c": 1, "d": 1 }, + "valid": false + }, + { + "description": "xx is valid", + "data": { "xx": 1 }, + "valid": true + }, + { + "description": "xx + foox is valid", + "data": { "xx": 1, "foox": 1 }, + "valid": true + }, + { + "description": "xx + foo is invalid", + "data": { "xx": 1, "foo": 1 }, + "valid": false + }, + { + "description": "xx + a is invalid", + "data": { "xx": 1, "a": 1 }, + "valid": false + }, + { + "description": "xx + b is invalid", + "data": { "xx": 1, "b": 1 }, + "valid": false + }, + { + "description": "xx + c is invalid", + "data": { "xx": 1, "c": 1 }, + "valid": false + }, + { + "description": "xx + d is invalid", + "data": { "xx": 1, "d": 1 }, + "valid": false + }, + { + "description": "all is valid", + "data": { "all": 1 }, + "valid": true + }, + { + "description": "all + foo is valid", + "data": { "all": 1, "foo": 1 }, + "valid": true + }, + { + "description": "all + a is invalid", + "data": { "all": 1, "a": 1 }, + "valid": false + } + ] + }, + { + "description": "non-object instances are valid", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "ignores booleans", + "data": true, + "valid": true + }, + { + "description": "ignores integers", + "data": 123, + "valid": true + }, + { + "description": "ignores floats", + "data": 1.0, + "valid": true + }, + { + "description": "ignores arrays", + "data": [], + "valid": true + }, + { + "description": "ignores strings", + "data": "foo", + "valid": true + }, + { + "description": "ignores null", + "data": null, + "valid": true + } + ] + }, + { + "description": "unevaluatedProperties with null valued instance properties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "unevaluatedProperties": { + "type": "null" + } + }, + "tests": [ + { + "description": "allows null valued properties", + "data": {"foo": null}, + "valid": true + } + ] + }, + { + "description": "unevaluatedProperties not affected by propertyNames", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "propertyNames": {"maxLength": 1}, + "unevaluatedProperties": { + "type": "number" + } + }, + "tests": [ + { + "description": "allows only number properties", + "data": {"a": 1}, + "valid": true + }, + { + "description": "string property is invalid", + "data": {"a": "b"}, + "valid": false + } + ] + }, + { + "description": "unevaluatedProperties can see annotations from if without then and else", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "if": { + "patternProperties": { + "foo": { + "type": "string" + } + } + }, + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "valid in case if is evaluated", + "data": { + "foo": "a" + }, + "valid": true + }, + { + "description": "invalid in case if is evaluated", + "data": { + "bar": "a" + }, + "valid": false + } + ] + }, + { + "description": "dependentSchemas with unevaluatedProperties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": {"foo2": {}}, + "dependentSchemas": { + "foo" : {}, + "foo2": { + "properties": { + "bar":{} + } + } + }, + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "unevaluatedProperties doesn't consider dependentSchemas", + "data": {"foo": ""}, + "valid": false + }, + { + "description": "unevaluatedProperties doesn't see bar when foo2 is absent", + "data": {"bar": ""}, + "valid": false + }, + { + "description": "unevaluatedProperties sees bar when foo2 is present", + "data": { "foo2": "", "bar": ""}, + "valid": true + } + ] + }, + { + "description": "Evaluated properties collection needs to consider instance location", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { + "properties": { + "bar": { "type": "string" } + } + } + }, + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "with an unevaluated property that exists at another location", + "data": { + "foo": { "bar": "foo" }, + "bar": "bar" + }, + "valid": false + } + ] + }, + { + "description": "Evaluated properties collection needs to consider instance location with patternProperties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { + "patternProperties": { + "^bar$": { "type": "string" } + } + } + }, + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "with only the nested evaluated property", + "data": { + "foo": { "bar": "foo" } + }, + "valid": true + }, + { + "description": "with an unevaluated property that exists at another location", + "data": { + "foo": { "bar": "foo" }, + "bar": "bar" + }, + "valid": false + } + ] + }, + { + "description": "Evaluated properties collection needs to consider instance location with additionalProperties", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "properties": { + "foo": { + "additionalProperties": { "type": "string" } + } + }, + "unevaluatedProperties": false + }, + "tests": [ + { + "description": "with only the nested evaluated property", + "data": { + "foo": { "bar": "foo" } + }, + "valid": true + }, + { + "description": "with an unevaluated property that exists at another location", + "data": { + "foo": { "bar": "foo" }, + "bar": "bar" + }, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/uniqueItems.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/uniqueItems.json new file mode 100644 index 000000000..262114529 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/uniqueItems.json @@ -0,0 +1,399 @@ +[ + { + "description": "uniqueItems validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "uniqueItems": true + }, + "tests": [ + { + "description": "unique array of integers is valid", + "data": [1, 2], + "valid": true + }, + { + "description": "non-unique array of integers is invalid", + "data": [1, 1], + "valid": false + }, + { + "description": "non-unique array of more than two integers is invalid", + "data": [1, 2, 1], + "valid": false + }, + { + "description": "numbers are unique if mathematically unequal", + "data": [1.0, 1.00, 1], + "valid": false + }, + { + "description": "false is not equal to zero", + "data": [0, false], + "valid": true + }, + { + "description": "true is not equal to one", + "data": [1, true], + "valid": true + }, + { + "description": "unique array of strings is valid", + "data": ["foo", "bar", "baz"], + "valid": true + }, + { + "description": "non-unique array of strings is invalid", + "data": ["foo", "bar", "foo"], + "valid": false + }, + { + "description": "unique array of objects is valid", + "data": [{"foo": "bar"}, {"foo": "baz"}], + "valid": true + }, + { + "description": "non-unique array of objects is invalid", + "data": [{"foo": "bar"}, {"foo": "bar"}], + "valid": false + }, + { + "description": "property order of array of objects is ignored", + "data": [{"foo": "bar", "bar": "foo"}, {"bar": "foo", "foo": "bar"}], + "valid": false + }, + { + "description": "unique array of nested objects is valid", + "data": [ + {"foo": {"bar" : {"baz" : true}}}, + {"foo": {"bar" : {"baz" : false}}} + ], + "valid": true + }, + { + "description": "non-unique array of nested objects is invalid", + "data": [ + {"foo": {"bar" : {"baz" : true}}}, + {"foo": {"bar" : {"baz" : true}}} + ], + "valid": false + }, + { + "description": "unique array of arrays is valid", + "data": [["foo"], ["bar"]], + "valid": true + }, + { + "description": "non-unique array of arrays is invalid", + "data": [["foo"], ["foo"]], + "valid": false + }, + { + "description": "non-unique array of more than two arrays is invalid", + "data": [["foo"], ["bar"], ["foo"]], + "valid": false + }, + { + "description": "[1] and [true] are unique", + "data": [[1], [true]], + "valid": true + }, + { + "description": "[0] and [false] are unique", + "data": [[0], [false]], + "valid": true + }, + { + "description": "nested [1] and [true] are unique", + "data": [[[1], "foo"], [[true], "foo"]], + "valid": true + }, + { + "description": "nested [0] and [false] are unique", + "data": [[[0], "foo"], [[false], "foo"]], + "valid": true + }, + { + "description": "unique heterogeneous types are valid", + "data": [{}, [1], true, null, 1, "{}"], + "valid": true + }, + { + "description": "non-unique heterogeneous types are invalid", + "data": [{}, [1], true, null, {}, 1], + "valid": false + }, + { + "description": "different objects are unique", + "data": [{"a": 1, "b": 2}, {"a": 2, "b": 1}], + "valid": true + }, + { + "description": "objects are non-unique despite key order", + "data": [{"a": 1, "b": 2}, {"b": 2, "a": 1}], + "valid": false + }, + { + "description": "{\"a\": false} and {\"a\": 0} are unique", + "data": [{"a": false}, {"a": 0}], + "valid": true + }, + { + "description": "{\"a\": true} and {\"a\": 1} are unique", + "data": [{"a": true}, {"a": 1}], + "valid": true + } + ] + }, + { + "description": "uniqueItems with an array of items", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [{"type": "boolean"}, {"type": "boolean"}], + "uniqueItems": true + }, + "tests": [ + { + "description": "[false, true] from items array is valid", + "data": [false, true], + "valid": true + }, + { + "description": "[true, false] from items array is valid", + "data": [true, false], + "valid": true + }, + { + "description": "[false, false] from items array is not valid", + "data": [false, false], + "valid": false + }, + { + "description": "[true, true] from items array is not valid", + "data": [true, true], + "valid": false + }, + { + "description": "unique array extended from [false, true] is valid", + "data": [false, true, "foo", "bar"], + "valid": true + }, + { + "description": "unique array extended from [true, false] is valid", + "data": [true, false, "foo", "bar"], + "valid": true + }, + { + "description": "non-unique array extended from [false, true] is not valid", + "data": [false, true, "foo", "foo"], + "valid": false + }, + { + "description": "non-unique array extended from [true, false] is not valid", + "data": [true, false, "foo", "foo"], + "valid": false + } + ] + }, + { + "description": "uniqueItems with an array of items and additionalItems=false", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [{"type": "boolean"}, {"type": "boolean"}], + "uniqueItems": true, + "items": false + }, + "tests": [ + { + "description": "[false, true] from items array is valid", + "data": [false, true], + "valid": true + }, + { + "description": "[true, false] from items array is valid", + "data": [true, false], + "valid": true + }, + { + "description": "[false, false] from items array is not valid", + "data": [false, false], + "valid": false + }, + { + "description": "[true, true] from items array is not valid", + "data": [true, true], + "valid": false + }, + { + "description": "extra items are invalid even if unique", + "data": [false, true, null], + "valid": false + } + ] + }, + { + "description": "uniqueItems=false validation", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "uniqueItems": false + }, + "tests": [ + { + "description": "unique array of integers is valid", + "data": [1, 2], + "valid": true + }, + { + "description": "non-unique array of integers is valid", + "data": [1, 1], + "valid": true + }, + { + "description": "numbers are unique if mathematically unequal", + "data": [1.0, 1.00, 1], + "valid": true + }, + { + "description": "false is not equal to zero", + "data": [0, false], + "valid": true + }, + { + "description": "true is not equal to one", + "data": [1, true], + "valid": true + }, + { + "description": "unique array of objects is valid", + "data": [{"foo": "bar"}, {"foo": "baz"}], + "valid": true + }, + { + "description": "non-unique array of objects is valid", + "data": [{"foo": "bar"}, {"foo": "bar"}], + "valid": true + }, + { + "description": "unique array of nested objects is valid", + "data": [ + {"foo": {"bar" : {"baz" : true}}}, + {"foo": {"bar" : {"baz" : false}}} + ], + "valid": true + }, + { + "description": "non-unique array of nested objects is valid", + "data": [ + {"foo": {"bar" : {"baz" : true}}}, + {"foo": {"bar" : {"baz" : true}}} + ], + "valid": true + }, + { + "description": "unique array of arrays is valid", + "data": [["foo"], ["bar"]], + "valid": true + }, + { + "description": "non-unique array of arrays is valid", + "data": [["foo"], ["foo"]], + "valid": true + }, + { + "description": "unique heterogeneous types are valid", + "data": [{}, [1], true, null, 1], + "valid": true + }, + { + "description": "non-unique heterogeneous types are valid", + "data": [{}, [1], true, null, {}, 1], + "valid": true + } + ] + }, + { + "description": "uniqueItems=false with an array of items", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [{"type": "boolean"}, {"type": "boolean"}], + "uniqueItems": false + }, + "tests": [ + { + "description": "[false, true] from items array is valid", + "data": [false, true], + "valid": true + }, + { + "description": "[true, false] from items array is valid", + "data": [true, false], + "valid": true + }, + { + "description": "[false, false] from items array is valid", + "data": [false, false], + "valid": true + }, + { + "description": "[true, true] from items array is valid", + "data": [true, true], + "valid": true + }, + { + "description": "unique array extended from [false, true] is valid", + "data": [false, true, "foo", "bar"], + "valid": true + }, + { + "description": "unique array extended from [true, false] is valid", + "data": [true, false, "foo", "bar"], + "valid": true + }, + { + "description": "non-unique array extended from [false, true] is valid", + "data": [false, true, "foo", "foo"], + "valid": true + }, + { + "description": "non-unique array extended from [true, false] is valid", + "data": [true, false, "foo", "foo"], + "valid": true + } + ] + }, + { + "description": "uniqueItems=false with an array of items and additionalItems=false", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "prefixItems": [{"type": "boolean"}, {"type": "boolean"}], + "uniqueItems": false, + "items": false + }, + "tests": [ + { + "description": "[false, true] from items array is valid", + "data": [false, true], + "valid": true + }, + { + "description": "[true, false] from items array is valid", + "data": [true, false], + "valid": true + }, + { + "description": "[false, false] from items array is valid", + "data": [false, false], + "valid": true + }, + { + "description": "[true, true] from items array is valid", + "data": [true, true], + "valid": true + }, + { + "description": "extra items are invalid even if unique", + "data": [false, true, null], + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/vocabulary.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/vocabulary.json new file mode 100644 index 000000000..1acb96a93 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft2020-12/vocabulary.json @@ -0,0 +1,57 @@ +[ + { + "description": "schema that uses custom metaschema with with no validation vocabulary", + "schema": { + "$id": "https://schema/using/no/validation", + "$schema": "http://localhost:1234/draft2020-12/metaschema-no-validation.json", + "properties": { + "badProperty": false, + "numberProperty": { + "minimum": 10 + } + } + }, + "tests": [ + { + "description": "applicator vocabulary still works", + "data": { + "badProperty": "this property should not exist" + }, + "valid": false + }, + { + "description": "no validation: valid number", + "data": { + "numberProperty": 20 + }, + "valid": true + }, + { + "description": "no validation: invalid number, but it still validates", + "data": { + "numberProperty": 1 + }, + "valid": true + } + ] + }, + { + "description": "ignore unrecognized optional vocabulary", + "schema": { + "$schema": "http://localhost:1234/draft2020-12/metaschema-optional-vocabulary.json", + "type": "number" + }, + "tests": [ + { + "description": "string value", + "data": "foobar", + "valid": false + }, + { + "description": "number value", + "data": 20, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/additionalItems.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/additionalItems.json new file mode 100644 index 000000000..6ef4212b1 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/additionalItems.json @@ -0,0 +1,190 @@ +[ + { + "description": "additionalItems as schema", + "schema": { + "items": [{}], + "additionalItems": {"type": "integer"} + }, + "tests": [ + { + "description": "additional items match schema", + "data": [ null, 2, 3, 4 ], + "valid": true + }, + { + "description": "additional items do not match schema", + "data": [ null, 2, 3, "foo" ], + "valid": false + } + ] + }, + { + "description": "when items is schema, additionalItems does nothing", + "schema": { + "items": { + "type": "integer" + }, + "additionalItems": { + "type": "string" + } + }, + "tests": [ + { + "description": "valid with a array of type integers", + "data": [1,2,3], + "valid": true + }, + { + "description": "invalid with a array of mixed types", + "data": [1,"2","3"], + "valid": false + } + ] + }, + { + "description": "when items is schema, boolean additionalItems does nothing", + "schema": { + "items": {}, + "additionalItems": false + }, + "tests": [ + { + "description": "all items match schema", + "data": [ 1, 2, 3, 4, 5 ], + "valid": true + } + ] + }, + { + "description": "array of items with no additionalItems permitted", + "schema": { + "items": [{}, {}, {}], + "additionalItems": false + }, + "tests": [ + { + "description": "empty array", + "data": [ ], + "valid": true + }, + { + "description": "fewer number of items present (1)", + "data": [ 1 ], + "valid": true + }, + { + "description": "fewer number of items present (2)", + "data": [ 1, 2 ], + "valid": true + }, + { + "description": "equal number of items present", + "data": [ 1, 2, 3 ], + "valid": true + }, + { + "description": "additional items are not permitted", + "data": [ 1, 2, 3, 4 ], + "valid": false + } + ] + }, + { + "description": "additionalItems as false without items", + "schema": {"additionalItems": false}, + "tests": [ + { + "description": + "items defaults to empty schema so everything is valid", + "data": [ 1, 2, 3, 4, 5 ], + "valid": true + }, + { + "description": "ignores non-arrays", + "data": {"foo" : "bar"}, + "valid": true + } + ] + }, + { + "description": "additionalItems are allowed by default", + "schema": {"items": [{"type": "integer"}]}, + "tests": [ + { + "description": "only the first item is validated", + "data": [1, "foo", false], + "valid": true + } + ] + }, + { + "description": "additionalItems does not look in applicators, invalid case", + "schema": { + "allOf": [ + { "items": [ { "type": "integer" }, { "type": "string" } ] } + ], + "items": [ {"type": "integer" } ], + "additionalItems": { "type": "boolean" } + }, + "tests": [ + { + "description": "items defined in allOf are not examined", + "data": [ 1, "hello" ], + "valid": false + } + ] + }, + { + "description": "items validation adjusts the starting index for additionalItems", + "schema": { + "items": [ { "type": "string" } ], + "additionalItems": { "type": "integer" } + }, + "tests": [ + { + "description": "valid items", + "data": [ "x", 2, 3 ], + "valid": true + }, + { + "description": "wrong type of second item", + "data": [ "x", "y" ], + "valid": false + } + ] + }, + { + "description": "additionalItems with heterogeneous array", + "schema": { + "items": [{}], + "additionalItems": false + }, + "tests": [ + { + "description": "heterogeneous invalid instance", + "data": [ "foo", "bar", 37 ], + "valid": false + }, + { + "description": "valid instance", + "data": [ null ], + "valid": true + } + ] + }, + { + "description": "additionalItems with null instance elements", + "schema": { + "additionalItems": { + "type": "null" + } + }, + "tests": [ + { + "description": "allows null elements", + "data": [ null ], + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/additionalProperties.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/additionalProperties.json new file mode 100644 index 000000000..0f8e1627a --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/additionalProperties.json @@ -0,0 +1,147 @@ +[ + { + "description": + "additionalProperties being false does not allow other properties", + "schema": { + "properties": {"foo": {}, "bar": {}}, + "patternProperties": { "^v": {} }, + "additionalProperties": false + }, + "tests": [ + { + "description": "no additional properties is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "an additional property is invalid", + "data": {"foo" : 1, "bar" : 2, "quux" : "boom"}, + "valid": false + }, + { + "description": "ignores arrays", + "data": [1, 2, 3], + "valid": true + }, + { + "description": "ignores strings", + "data": "foobarbaz", + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + }, + { + "description": "patternProperties are not additional properties", + "data": {"foo":1, "vroom": 2}, + "valid": true + } + ] + }, + { + "description": "non-ASCII pattern with additionalProperties", + "schema": { + "patternProperties": {"^á": {}}, + "additionalProperties": false + }, + "tests": [ + { + "description": "matching the pattern is valid", + "data": {"ármányos": 2}, + "valid": true + }, + { + "description": "not matching the pattern is invalid", + "data": {"élmény": 2}, + "valid": false + } + ] + }, + { + "description": "additionalProperties with schema", + "schema": { + "properties": {"foo": {}, "bar": {}}, + "additionalProperties": {"type": "boolean"} + }, + "tests": [ + { + "description": "no additional properties is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "an additional valid property is valid", + "data": {"foo" : 1, "bar" : 2, "quux" : true}, + "valid": true + }, + { + "description": "an additional invalid property is invalid", + "data": {"foo" : 1, "bar" : 2, "quux" : 12}, + "valid": false + } + ] + }, + { + "description": + "additionalProperties can exist by itself", + "schema": { + "additionalProperties": {"type": "boolean"} + }, + "tests": [ + { + "description": "an additional valid property is valid", + "data": {"foo" : true}, + "valid": true + }, + { + "description": "an additional invalid property is invalid", + "data": {"foo" : 1}, + "valid": false + } + ] + }, + { + "description": "additionalProperties are allowed by default", + "schema": {"properties": {"foo": {}, "bar": {}}}, + "tests": [ + { + "description": "additional properties are allowed", + "data": {"foo": 1, "bar": 2, "quux": true}, + "valid": true + } + ] + }, + { + "description": "additionalProperties does not look in applicators", + "schema": { + "allOf": [ + {"properties": {"foo": {}}} + ], + "additionalProperties": {"type": "boolean"} + }, + "tests": [ + { + "description": "properties defined in allOf are not examined", + "data": {"foo": 1, "bar": true}, + "valid": false + } + ] + }, + { + "description": "additionalProperties with null valued instance properties", + "schema": { + "additionalProperties": { + "type": "null" + } + }, + "tests": [ + { + "description": "allows null values", + "data": {"foo": null}, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/allOf.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/allOf.json new file mode 100644 index 000000000..ec9319e14 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/allOf.json @@ -0,0 +1,294 @@ +[ + { + "description": "allOf", + "schema": { + "allOf": [ + { + "properties": { + "bar": {"type": "integer"} + }, + "required": ["bar"] + }, + { + "properties": { + "foo": {"type": "string"} + }, + "required": ["foo"] + } + ] + }, + "tests": [ + { + "description": "allOf", + "data": {"foo": "baz", "bar": 2}, + "valid": true + }, + { + "description": "mismatch second", + "data": {"foo": "baz"}, + "valid": false + }, + { + "description": "mismatch first", + "data": {"bar": 2}, + "valid": false + }, + { + "description": "wrong type", + "data": {"foo": "baz", "bar": "quux"}, + "valid": false + } + ] + }, + { + "description": "allOf with base schema", + "schema": { + "properties": {"bar": {"type": "integer"}}, + "required": ["bar"], + "allOf" : [ + { + "properties": { + "foo": {"type": "string"} + }, + "required": ["foo"] + }, + { + "properties": { + "baz": {"type": "null"} + }, + "required": ["baz"] + } + ] + }, + "tests": [ + { + "description": "valid", + "data": {"foo": "quux", "bar": 2, "baz": null}, + "valid": true + }, + { + "description": "mismatch base schema", + "data": {"foo": "quux", "baz": null}, + "valid": false + }, + { + "description": "mismatch first allOf", + "data": {"bar": 2, "baz": null}, + "valid": false + }, + { + "description": "mismatch second allOf", + "data": {"foo": "quux", "bar": 2}, + "valid": false + }, + { + "description": "mismatch both", + "data": {"bar": 2}, + "valid": false + } + ] + }, + { + "description": "allOf simple types", + "schema": { + "allOf": [ + {"maximum": 30}, + {"minimum": 20} + ] + }, + "tests": [ + { + "description": "valid", + "data": 25, + "valid": true + }, + { + "description": "mismatch one", + "data": 35, + "valid": false + } + ] + }, + { + "description": "allOf with boolean schemas, all true", + "schema": {"allOf": [true, true]}, + "tests": [ + { + "description": "any value is valid", + "data": "foo", + "valid": true + } + ] + }, + { + "description": "allOf with boolean schemas, some false", + "schema": {"allOf": [true, false]}, + "tests": [ + { + "description": "any value is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "allOf with boolean schemas, all false", + "schema": {"allOf": [false, false]}, + "tests": [ + { + "description": "any value is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "allOf with one empty schema", + "schema": { + "allOf": [ + {} + ] + }, + "tests": [ + { + "description": "any data is valid", + "data": 1, + "valid": true + } + ] + }, + { + "description": "allOf with two empty schemas", + "schema": { + "allOf": [ + {}, + {} + ] + }, + "tests": [ + { + "description": "any data is valid", + "data": 1, + "valid": true + } + ] + }, + { + "description": "allOf with the first empty schema", + "schema": { + "allOf": [ + {}, + { "type": "number" } + ] + }, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "string is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "allOf with the last empty schema", + "schema": { + "allOf": [ + { "type": "number" }, + {} + ] + }, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "string is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "nested allOf, to check validation semantics", + "schema": { + "allOf": [ + { + "allOf": [ + { + "type": "null" + } + ] + } + ] + }, + "tests": [ + { + "description": "null is valid", + "data": null, + "valid": true + }, + { + "description": "anything non-null is invalid", + "data": 123, + "valid": false + } + ] + }, + { + "description": "allOf combined with anyOf, oneOf", + "schema": { + "allOf": [ { "multipleOf": 2 } ], + "anyOf": [ { "multipleOf": 3 } ], + "oneOf": [ { "multipleOf": 5 } ] + }, + "tests": [ + { + "description": "allOf: false, anyOf: false, oneOf: false", + "data": 1, + "valid": false + }, + { + "description": "allOf: false, anyOf: false, oneOf: true", + "data": 5, + "valid": false + }, + { + "description": "allOf: false, anyOf: true, oneOf: false", + "data": 3, + "valid": false + }, + { + "description": "allOf: false, anyOf: true, oneOf: true", + "data": 15, + "valid": false + }, + { + "description": "allOf: true, anyOf: false, oneOf: false", + "data": 2, + "valid": false + }, + { + "description": "allOf: true, anyOf: false, oneOf: true", + "data": 10, + "valid": false + }, + { + "description": "allOf: true, anyOf: true, oneOf: false", + "data": 6, + "valid": false + }, + { + "description": "allOf: true, anyOf: true, oneOf: true", + "data": 30, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/anyOf.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/anyOf.json new file mode 100644 index 000000000..ab5eb386b --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/anyOf.json @@ -0,0 +1,189 @@ +[ + { + "description": "anyOf", + "schema": { + "anyOf": [ + { + "type": "integer" + }, + { + "minimum": 2 + } + ] + }, + "tests": [ + { + "description": "first anyOf valid", + "data": 1, + "valid": true + }, + { + "description": "second anyOf valid", + "data": 2.5, + "valid": true + }, + { + "description": "both anyOf valid", + "data": 3, + "valid": true + }, + { + "description": "neither anyOf valid", + "data": 1.5, + "valid": false + } + ] + }, + { + "description": "anyOf with base schema", + "schema": { + "type": "string", + "anyOf" : [ + { + "maxLength": 2 + }, + { + "minLength": 4 + } + ] + }, + "tests": [ + { + "description": "mismatch base schema", + "data": 3, + "valid": false + }, + { + "description": "one anyOf valid", + "data": "foobar", + "valid": true + }, + { + "description": "both anyOf invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "anyOf with boolean schemas, all true", + "schema": {"anyOf": [true, true]}, + "tests": [ + { + "description": "any value is valid", + "data": "foo", + "valid": true + } + ] + }, + { + "description": "anyOf with boolean schemas, some true", + "schema": {"anyOf": [true, false]}, + "tests": [ + { + "description": "any value is valid", + "data": "foo", + "valid": true + } + ] + }, + { + "description": "anyOf with boolean schemas, all false", + "schema": {"anyOf": [false, false]}, + "tests": [ + { + "description": "any value is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "anyOf complex types", + "schema": { + "anyOf": [ + { + "properties": { + "bar": {"type": "integer"} + }, + "required": ["bar"] + }, + { + "properties": { + "foo": {"type": "string"} + }, + "required": ["foo"] + } + ] + }, + "tests": [ + { + "description": "first anyOf valid (complex)", + "data": {"bar": 2}, + "valid": true + }, + { + "description": "second anyOf valid (complex)", + "data": {"foo": "baz"}, + "valid": true + }, + { + "description": "both anyOf valid (complex)", + "data": {"foo": "baz", "bar": 2}, + "valid": true + }, + { + "description": "neither anyOf valid (complex)", + "data": {"foo": 2, "bar": "quux"}, + "valid": false + } + ] + }, + { + "description": "anyOf with one empty schema", + "schema": { + "anyOf": [ + { "type": "number" }, + {} + ] + }, + "tests": [ + { + "description": "string is valid", + "data": "foo", + "valid": true + }, + { + "description": "number is valid", + "data": 123, + "valid": true + } + ] + }, + { + "description": "nested anyOf, to check validation semantics", + "schema": { + "anyOf": [ + { + "anyOf": [ + { + "type": "null" + } + ] + } + ] + }, + "tests": [ + { + "description": "null is valid", + "data": null, + "valid": true + }, + { + "description": "anything non-null is invalid", + "data": 123, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/boolean_schema.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/boolean_schema.json new file mode 100644 index 000000000..6d40f23f2 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/boolean_schema.json @@ -0,0 +1,104 @@ +[ + { + "description": "boolean schema 'true'", + "schema": true, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "string is valid", + "data": "foo", + "valid": true + }, + { + "description": "boolean true is valid", + "data": true, + "valid": true + }, + { + "description": "boolean false is valid", + "data": false, + "valid": true + }, + { + "description": "null is valid", + "data": null, + "valid": true + }, + { + "description": "object is valid", + "data": {"foo": "bar"}, + "valid": true + }, + { + "description": "empty object is valid", + "data": {}, + "valid": true + }, + { + "description": "array is valid", + "data": ["foo"], + "valid": true + }, + { + "description": "empty array is valid", + "data": [], + "valid": true + } + ] + }, + { + "description": "boolean schema 'false'", + "schema": false, + "tests": [ + { + "description": "number is invalid", + "data": 1, + "valid": false + }, + { + "description": "string is invalid", + "data": "foo", + "valid": false + }, + { + "description": "boolean true is invalid", + "data": true, + "valid": false + }, + { + "description": "boolean false is invalid", + "data": false, + "valid": false + }, + { + "description": "null is invalid", + "data": null, + "valid": false + }, + { + "description": "object is invalid", + "data": {"foo": "bar"}, + "valid": false + }, + { + "description": "empty object is invalid", + "data": {}, + "valid": false + }, + { + "description": "array is invalid", + "data": ["foo"], + "valid": false + }, + { + "description": "empty array is invalid", + "data": [], + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/const.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/const.json new file mode 100644 index 000000000..168fff4e3 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/const.json @@ -0,0 +1,384 @@ +[ + { + "description": "const validation", + "schema": {"const": 2}, + "tests": [ + { + "description": "same value is valid", + "data": 2, + "valid": true + }, + { + "description": "another value is invalid", + "data": 5, + "valid": false + }, + { + "description": "another type is invalid", + "data": "a", + "valid": false + } + ] + }, + { + "description": "const with object", + "schema": {"const": {"foo": "bar", "baz": "bax"}}, + "tests": [ + { + "description": "same object is valid", + "data": {"foo": "bar", "baz": "bax"}, + "valid": true + }, + { + "description": "same object with different property order is valid", + "data": {"baz": "bax", "foo": "bar"}, + "valid": true + }, + { + "description": "another object is invalid", + "data": {"foo": "bar"}, + "valid": false + }, + { + "description": "another type is invalid", + "data": [1, 2], + "valid": false + } + ] + }, + { + "description": "const with array", + "schema": {"const": [{ "foo": "bar" }]}, + "tests": [ + { + "description": "same array is valid", + "data": [{"foo": "bar"}], + "valid": true + }, + { + "description": "another array item is invalid", + "data": [2], + "valid": false + }, + { + "description": "array with additional items is invalid", + "data": [1, 2, 3], + "valid": false + } + ] + }, + { + "description": "const with null", + "schema": {"const": null}, + "tests": [ + { + "description": "null is valid", + "data": null, + "valid": true + }, + { + "description": "not null is invalid", + "data": 0, + "valid": false + } + ] + }, + { + "description": "const with false does not match 0", + "schema": {"const": false}, + "tests": [ + { + "description": "false is valid", + "data": false, + "valid": true + }, + { + "description": "integer zero is invalid", + "data": 0, + "valid": false + }, + { + "description": "float zero is invalid", + "data": 0.0, + "valid": false + } + ] + }, + { + "description": "const with true does not match 1", + "schema": {"const": true}, + "tests": [ + { + "description": "true is valid", + "data": true, + "valid": true + }, + { + "description": "integer one is invalid", + "data": 1, + "valid": false + }, + { + "description": "float one is invalid", + "data": 1.0, + "valid": false + } + ] + }, + { + "description": "const with [false] does not match [0]", + "schema": {"const": [false]}, + "tests": [ + { + "description": "[false] is valid", + "data": [false], + "valid": true + }, + { + "description": "[0] is invalid", + "data": [0], + "valid": false + }, + { + "description": "[0.0] is invalid", + "data": [0.0], + "valid": false + } + ] + }, + { + "description": "const with [true] does not match [1]", + "schema": {"const": [true]}, + "tests": [ + { + "description": "[true] is valid", + "data": [true], + "valid": true + }, + { + "description": "[1] is invalid", + "data": [1], + "valid": false + }, + { + "description": "[1.0] is invalid", + "data": [1.0], + "valid": false + } + ] + }, + { + "description": "const with {\"a\": false} does not match {\"a\": 0}", + "schema": {"const": {"a": false}}, + "tests": [ + { + "description": "{\"a\": false} is valid", + "data": {"a": false}, + "valid": true + }, + { + "description": "{\"a\": 0} is invalid", + "data": {"a": 0}, + "valid": false + }, + { + "description": "{\"a\": 0.0} is invalid", + "data": {"a": 0.0}, + "valid": false + } + ] + }, + { + "description": "const with {\"a\": true} does not match {\"a\": 1}", + "schema": {"const": {"a": true}}, + "tests": [ + { + "description": "{\"a\": true} is valid", + "data": {"a": true}, + "valid": true + }, + { + "description": "{\"a\": 1} is invalid", + "data": {"a": 1}, + "valid": false + }, + { + "description": "{\"a\": 1.0} is invalid", + "data": {"a": 1.0}, + "valid": false + } + ] + }, + { + "description": "const with 0 does not match other zero-like types", + "schema": {"const": 0}, + "tests": [ + { + "description": "false is invalid", + "data": false, + "valid": false + }, + { + "description": "integer zero is valid", + "data": 0, + "valid": true + }, + { + "description": "float zero is valid", + "data": 0.0, + "valid": true + }, + { + "description": "empty object is invalid", + "data": {}, + "valid": false + }, + { + "description": "empty array is invalid", + "data": [], + "valid": false + }, + { + "description": "empty string is invalid", + "data": "", + "valid": false + } + ] + }, + { + "description": "const with 1 does not match true", + "schema": {"const": 1}, + "tests": [ + { + "description": "true is invalid", + "data": true, + "valid": false + }, + { + "description": "integer one is valid", + "data": 1, + "valid": true + }, + { + "description": "float one is valid", + "data": 1.0, + "valid": true + } + ] + }, + { + "description": "const with -2.0 matches integer and float types", + "schema": {"const": -2.0}, + "tests": [ + { + "description": "integer -2 is valid", + "data": -2, + "valid": true + }, + { + "description": "integer 2 is invalid", + "data": 2, + "valid": false + }, + { + "description": "float -2.0 is valid", + "data": -2.0, + "valid": true + }, + { + "description": "float 2.0 is invalid", + "data": 2.0, + "valid": false + }, + { + "description": "float -2.00001 is invalid", + "data": -2.00001, + "valid": false + } + ] + }, + { + "description": "float and integers are equal up to 64-bit representation limits", + "schema": {"const": 9007199254740992}, + "tests": [ + { + "description": "integer is valid", + "data": 9007199254740992, + "valid": true + }, + { + "description": "integer minus one is invalid", + "data": 9007199254740991, + "valid": false + }, + { + "description": "float is valid", + "data": 9007199254740992.0, + "valid": true + }, + { + "description": "float minus one is invalid", + "data": 9007199254740991.0, + "valid": false + } + ] + }, + { + "description": "nul characters in strings", + "schema": { "const": "hello\u0000there" }, + "tests": [ + { + "description": "match string with nul", + "data": "hello\u0000there", + "valid": true + }, + { + "description": "do not match string lacking nul", + "data": "hellothere", + "valid": false + } + ] + }, + { + "description": "characters with the same visual representation but different codepoint", + "schema": { + "const": "μ", + "$comment": "U+03BC" + }, + "tests": [ + { + "description": "character uses the same codepoint", + "data": "μ", + "comment": "U+03BC", + "valid": true + }, + { + "description": "character looks the same but uses a different codepoint", + "data": "µ", + "comment": "U+00B5", + "valid": false + } + ] + }, + { + "description": "characters with the same visual representation, but different number of codepoints", + "schema": { + "const": "ä", + "$comment": "U+00E4" + }, + "tests": [ + { + "description": "character uses the same codepoint", + "data": "ä", + "comment": "U+00E4", + "valid": true + }, + { + "description": "character looks the same but uses combining marks", + "data": "ä", + "comment": "a, U+0308", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/contains.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/contains.json new file mode 100644 index 000000000..2b1a51520 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/contains.json @@ -0,0 +1,165 @@ +[ + { + "description": "contains keyword validation", + "schema": { + "contains": {"minimum": 5} + }, + "tests": [ + { + "description": "array with item matching schema (5) is valid", + "data": [3, 4, 5], + "valid": true + }, + { + "description": "array with item matching schema (6) is valid", + "data": [3, 4, 6], + "valid": true + }, + { + "description": "array with two items matching schema (5, 6) is valid", + "data": [3, 4, 5, 6], + "valid": true + }, + { + "description": "array without items matching schema is invalid", + "data": [2, 3, 4], + "valid": false + }, + { + "description": "empty array is invalid", + "data": [], + "valid": false + }, + { + "description": "not array is valid", + "data": {}, + "valid": true + } + ] + }, + { + "description": "contains keyword with const keyword", + "schema": { + "contains": { "const": 5 } + }, + "tests": [ + { + "description": "array with item 5 is valid", + "data": [3, 4, 5], + "valid": true + }, + { + "description": "array with two items 5 is valid", + "data": [3, 4, 5, 5], + "valid": true + }, + { + "description": "array without item 5 is invalid", + "data": [1, 2, 3, 4], + "valid": false + } + ] + }, + { + "description": "contains keyword with boolean schema true", + "schema": {"contains": true}, + "tests": [ + { + "description": "any non-empty array is valid", + "data": ["foo"], + "valid": true + }, + { + "description": "empty array is invalid", + "data": [], + "valid": false + } + ] + }, + { + "description": "contains keyword with boolean schema false", + "schema": {"contains": false}, + "tests": [ + { + "description": "any non-empty array is invalid", + "data": ["foo"], + "valid": false + }, + { + "description": "empty array is invalid", + "data": [], + "valid": false + }, + { + "description": "non-arrays are valid", + "data": "contains does not apply to strings", + "valid": true + } + ] + }, + { + "description": "items + contains", + "schema": { + "items": { "multipleOf": 2 }, + "contains": { "multipleOf": 3 } + }, + "tests": [ + { + "description": "matches items, does not match contains", + "data": [ 2, 4, 8 ], + "valid": false + }, + { + "description": "does not match items, matches contains", + "data": [ 3, 6, 9 ], + "valid": false + }, + { + "description": "matches both items and contains", + "data": [ 6, 12 ], + "valid": true + }, + { + "description": "matches neither items nor contains", + "data": [ 1, 5 ], + "valid": false + } + ] + }, + { + "description": "contains with false if subschema", + "schema": { + "contains": { + "if": false, + "else": true + } + }, + "tests": [ + { + "description": "any non-empty array is valid", + "data": ["foo"], + "valid": true + }, + { + "description": "empty array is invalid", + "data": [], + "valid": false + } + ] + }, + { + "description": "contains with null instance elements", + "schema": { + "contains": { + "type": "null" + } + }, + "tests": [ + { + "description": "allows null items", + "data": [ null ], + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/default.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/default.json new file mode 100644 index 000000000..289a9b66c --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/default.json @@ -0,0 +1,79 @@ +[ + { + "description": "invalid type for default", + "schema": { + "properties": { + "foo": { + "type": "integer", + "default": [] + } + } + }, + "tests": [ + { + "description": "valid when property is specified", + "data": {"foo": 13}, + "valid": true + }, + { + "description": "still valid when the invalid default is used", + "data": {}, + "valid": true + } + ] + }, + { + "description": "invalid string value for default", + "schema": { + "properties": { + "bar": { + "type": "string", + "minLength": 4, + "default": "bad" + } + } + }, + "tests": [ + { + "description": "valid when property is specified", + "data": {"bar": "good"}, + "valid": true + }, + { + "description": "still valid when the invalid default is used", + "data": {}, + "valid": true + } + ] + }, + { + "description": "the default keyword does not do anything if the property is missing", + "schema": { + "type": "object", + "properties": { + "alpha": { + "type": "number", + "maximum": 3, + "default": 5 + } + } + }, + "tests": [ + { + "description": "an explicit property value is checked against maximum (passing)", + "data": { "alpha": 1 }, + "valid": true + }, + { + "description": "an explicit property value is checked against maximum (failing)", + "data": { "alpha": 5 }, + "valid": false + }, + { + "description": "missing properties are not filled in with the default", + "data": {}, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/definitions.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/definitions.json new file mode 100644 index 000000000..afe396e42 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/definitions.json @@ -0,0 +1,26 @@ +[ + { + "description": "validate definition against metaschema", + "schema": {"$ref": "http://json-schema.org/draft-07/schema#"}, + "tests": [ + { + "description": "valid definition schema", + "data": { + "definitions": { + "foo": {"type": "integer"} + } + }, + "valid": true + }, + { + "description": "invalid definition schema", + "data": { + "definitions": { + "foo": {"type": 1} + } + }, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/dependencies.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/dependencies.json new file mode 100644 index 000000000..c0bd809f6 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/dependencies.json @@ -0,0 +1,286 @@ +[ + { + "description": "dependencies", + "schema": { + "dependencies": {"bar": ["foo"]} + }, + "tests": [ + { + "description": "neither", + "data": {}, + "valid": true + }, + { + "description": "nondependant", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "with dependency", + "data": {"foo": 1, "bar": 2}, + "valid": true + }, + { + "description": "missing dependency", + "data": {"bar": 2}, + "valid": false + }, + { + "description": "ignores arrays", + "data": ["bar"], + "valid": true + }, + { + "description": "ignores strings", + "data": "foobar", + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + } + ] + }, + { + "description": "dependencies with empty array", + "schema": { + "dependencies": {"bar": []} + }, + "tests": [ + { + "description": "empty object", + "data": {}, + "valid": true + }, + { + "description": "object with one property", + "data": {"bar": 2}, + "valid": true + }, + { + "description": "non-object is valid", + "data": 1, + "valid": true + } + ] + }, + { + "description": "multiple dependencies", + "schema": { + "dependencies": {"quux": ["foo", "bar"]} + }, + "tests": [ + { + "description": "neither", + "data": {}, + "valid": true + }, + { + "description": "nondependants", + "data": {"foo": 1, "bar": 2}, + "valid": true + }, + { + "description": "with dependencies", + "data": {"foo": 1, "bar": 2, "quux": 3}, + "valid": true + }, + { + "description": "missing dependency", + "data": {"foo": 1, "quux": 2}, + "valid": false + }, + { + "description": "missing other dependency", + "data": {"bar": 1, "quux": 2}, + "valid": false + }, + { + "description": "missing both dependencies", + "data": {"quux": 1}, + "valid": false + } + ] + }, + { + "description": "multiple dependencies subschema", + "schema": { + "dependencies": { + "bar": { + "properties": { + "foo": {"type": "integer"}, + "bar": {"type": "integer"} + } + } + } + }, + "tests": [ + { + "description": "valid", + "data": {"foo": 1, "bar": 2}, + "valid": true + }, + { + "description": "no dependency", + "data": {"foo": "quux"}, + "valid": true + }, + { + "description": "wrong type", + "data": {"foo": "quux", "bar": 2}, + "valid": false + }, + { + "description": "wrong type other", + "data": {"foo": 2, "bar": "quux"}, + "valid": false + }, + { + "description": "wrong type both", + "data": {"foo": "quux", "bar": "quux"}, + "valid": false + } + ] + }, + { + "description": "dependencies with boolean subschemas", + "schema": { + "dependencies": { + "foo": true, + "bar": false + } + }, + "tests": [ + { + "description": "object with property having schema true is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "object with property having schema false is invalid", + "data": {"bar": 2}, + "valid": false + }, + { + "description": "object with both properties is invalid", + "data": {"foo": 1, "bar": 2}, + "valid": false + }, + { + "description": "empty object is valid", + "data": {}, + "valid": true + } + ] + }, + { + "description": "dependencies with escaped characters", + "schema": { + "dependencies": { + "foo\nbar": ["foo\rbar"], + "foo\tbar": { + "minProperties": 4 + }, + "foo'bar": {"required": ["foo\"bar"]}, + "foo\"bar": ["foo'bar"] + } + }, + "tests": [ + { + "description": "valid object 1", + "data": { + "foo\nbar": 1, + "foo\rbar": 2 + }, + "valid": true + }, + { + "description": "valid object 2", + "data": { + "foo\tbar": 1, + "a": 2, + "b": 3, + "c": 4 + }, + "valid": true + }, + { + "description": "valid object 3", + "data": { + "foo'bar": 1, + "foo\"bar": 2 + }, + "valid": true + }, + { + "description": "invalid object 1", + "data": { + "foo\nbar": 1, + "foo": 2 + }, + "valid": false + }, + { + "description": "invalid object 2", + "data": { + "foo\tbar": 1, + "a": 2 + }, + "valid": false + }, + { + "description": "invalid object 3", + "data": { + "foo'bar": 1 + }, + "valid": false + }, + { + "description": "invalid object 4", + "data": { + "foo\"bar": 2 + }, + "valid": false + } + ] + }, + { + "description": "dependent subschema incompatible with root", + "schema": { + "properties": { + "foo": {} + }, + "dependencies": { + "foo": { + "properties": { + "bar": {} + }, + "additionalProperties": false + } + } + }, + "tests": [ + { + "description": "matches root", + "data": {"foo": 1}, + "valid": false + }, + { + "description": "matches dependency", + "data": {"bar": 1}, + "valid": true + }, + { + "description": "matches both", + "data": {"foo": 1, "bar": 2}, + "valid": false + }, + { + "description": "no dependency", + "data": {"baz": 1}, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/enum.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/enum.json new file mode 100644 index 000000000..bf1e96fc5 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/enum.json @@ -0,0 +1,341 @@ +[ + { + "description": "simple enum validation", + "schema": {"enum": [1, 2, 3]}, + "tests": [ + { + "description": "one of the enum is valid", + "data": 1, + "valid": true + }, + { + "description": "something else is invalid", + "data": 4, + "valid": false + } + ] + }, + { + "description": "heterogeneous enum validation", + "schema": {"enum": [6, "foo", [], true, {"foo": 12}]}, + "tests": [ + { + "description": "one of the enum is valid", + "data": [], + "valid": true + }, + { + "description": "something else is invalid", + "data": null, + "valid": false + }, + { + "description": "objects are deep compared", + "data": {"foo": false}, + "valid": false + }, + { + "description": "valid object matches", + "data": {"foo": 12}, + "valid": true + }, + { + "description": "extra properties in object is invalid", + "data": {"foo": 12, "boo": 42}, + "valid": false + } + ] + }, + { + "description": "heterogeneous enum-with-null validation", + "schema": { "enum": [6, null] }, + "tests": [ + { + "description": "null is valid", + "data": null, + "valid": true + }, + { + "description": "number is valid", + "data": 6, + "valid": true + }, + { + "description": "something else is invalid", + "data": "test", + "valid": false + } + ] + }, + { + "description": "enums in properties", + "schema": { + "type":"object", + "properties": { + "foo": {"enum":["foo"]}, + "bar": {"enum":["bar"]} + }, + "required": ["bar"] + }, + "tests": [ + { + "description": "both properties are valid", + "data": {"foo":"foo", "bar":"bar"}, + "valid": true + }, + { + "description": "wrong foo value", + "data": {"foo":"foot", "bar":"bar"}, + "valid": false + }, + { + "description": "wrong bar value", + "data": {"foo":"foo", "bar":"bart"}, + "valid": false + }, + { + "description": "missing optional property is valid", + "data": {"bar":"bar"}, + "valid": true + }, + { + "description": "missing required property is invalid", + "data": {"foo":"foo"}, + "valid": false + }, + { + "description": "missing all properties is invalid", + "data": {}, + "valid": false + } + ] + }, + { + "description": "enum with escaped characters", + "schema": { + "enum": ["foo\nbar", "foo\rbar"] + }, + "tests": [ + { + "description": "member 1 is valid", + "data": "foo\nbar", + "valid": true + }, + { + "description": "member 2 is valid", + "data": "foo\rbar", + "valid": true + }, + { + "description": "another string is invalid", + "data": "abc", + "valid": false + } + ] + }, + { + "description": "enum with false does not match 0", + "schema": {"enum": [false]}, + "tests": [ + { + "description": "false is valid", + "data": false, + "valid": true + }, + { + "description": "integer zero is invalid", + "data": 0, + "valid": false + }, + { + "description": "float zero is invalid", + "data": 0.0, + "valid": false + } + ] + }, + { + "description": "enum with [false] does not match [0]", + "schema": {"enum": [[false]]}, + "tests": [ + { + "description": "[false] is valid", + "data": [false], + "valid": true + }, + { + "description": "[0] is invalid", + "data": [0], + "valid": false + }, + { + "description": "[0.0] is invalid", + "data": [0.0], + "valid": false + } + ] + }, + { + "description": "enum with true does not match 1", + "schema": {"enum": [true]}, + "tests": [ + { + "description": "true is valid", + "data": true, + "valid": true + }, + { + "description": "integer one is invalid", + "data": 1, + "valid": false + }, + { + "description": "float one is invalid", + "data": 1.0, + "valid": false + } + ] + }, + { + "description": "enum with [true] does not match [1]", + "schema": {"enum": [[true]]}, + "tests": [ + { + "description": "[true] is valid", + "data": [true], + "valid": true + }, + { + "description": "[1] is invalid", + "data": [1], + "valid": false + }, + { + "description": "[1.0] is invalid", + "data": [1.0], + "valid": false + } + ] + }, + { + "description": "enum with 0 does not match false", + "schema": {"enum": [0]}, + "tests": [ + { + "description": "false is invalid", + "data": false, + "valid": false + }, + { + "description": "integer zero is valid", + "data": 0, + "valid": true + }, + { + "description": "float zero is valid", + "data": 0.0, + "valid": true + } + ] + }, + { + "description": "enum with [0] does not match [false]", + "schema": {"enum": [[0]]}, + "tests": [ + { + "description": "[false] is invalid", + "data": [false], + "valid": false + }, + { + "description": "[0] is valid", + "data": [0], + "valid": true + }, + { + "description": "[0.0] is valid", + "data": [0.0], + "valid": true + } + ] + }, + { + "description": "enum with 1 does not match true", + "schema": {"enum": [1]}, + "tests": [ + { + "description": "true is invalid", + "data": true, + "valid": false + }, + { + "description": "integer one is valid", + "data": 1, + "valid": true + }, + { + "description": "float one is valid", + "data": 1.0, + "valid": true + } + ] + }, + { + "description": "enum with [1] does not match [true]", + "schema": {"enum": [[1]]}, + "tests": [ + { + "description": "[true] is invalid", + "data": [true], + "valid": false + }, + { + "description": "[1] is valid", + "data": [1], + "valid": true + }, + { + "description": "[1.0] is valid", + "data": [1.0], + "valid": true + } + ] + }, + { + "description": "nul characters in strings", + "schema": { "enum": [ "hello\u0000there" ] }, + "tests": [ + { + "description": "match string with nul", + "data": "hello\u0000there", + "valid": true + }, + { + "description": "do not match string lacking nul", + "data": "hellothere", + "valid": false + } + ] + }, + { + "description": "characters with the same visual representation, but different number of codepoints", + "schema": { + "enum": ["ä"], + "$comment": "U+00E4" + }, + "tests": [ + { + "description": "character uses the same codepoint", + "data": "ä", + "comment": "U+00E4", + "valid": true + }, + { + "description": "character looks the same but uses combining marks", + "data": "ä", + "comment": "a, U+0308", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/exclusiveMaximum.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/exclusiveMaximum.json new file mode 100644 index 000000000..dc3cd709d --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/exclusiveMaximum.json @@ -0,0 +1,30 @@ +[ + { + "description": "exclusiveMaximum validation", + "schema": { + "exclusiveMaximum": 3.0 + }, + "tests": [ + { + "description": "below the exclusiveMaximum is valid", + "data": 2.2, + "valid": true + }, + { + "description": "boundary point is invalid", + "data": 3.0, + "valid": false + }, + { + "description": "above the exclusiveMaximum is invalid", + "data": 3.5, + "valid": false + }, + { + "description": "ignores non-numbers", + "data": "x", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/exclusiveMinimum.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/exclusiveMinimum.json new file mode 100644 index 000000000..b38d7ecec --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/exclusiveMinimum.json @@ -0,0 +1,30 @@ +[ + { + "description": "exclusiveMinimum validation", + "schema": { + "exclusiveMinimum": 1.1 + }, + "tests": [ + { + "description": "above the exclusiveMinimum is valid", + "data": 1.2, + "valid": true + }, + { + "description": "boundary point is invalid", + "data": 1.1, + "valid": false + }, + { + "description": "below the exclusiveMinimum is invalid", + "data": 0.6, + "valid": false + }, + { + "description": "ignores non-numbers", + "data": "x", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/format.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/format.json new file mode 100644 index 000000000..e2447d60f --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/format.json @@ -0,0 +1,614 @@ +[ + { + "description": "email format", + "schema": { "format": "email" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + } + ] + }, + { + "description": "idn-email format", + "schema": { "format": "idn-email" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + } + ] + }, + { + "description": "regex format", + "schema": { "format": "regex" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + } + ] + }, + { + "description": "ipv4 format", + "schema": { "format": "ipv4" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + } + ] + }, + { + "description": "ipv6 format", + "schema": { "format": "ipv6" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + } + ] + }, + { + "description": "idn-hostname format", + "schema": { "format": "idn-hostname" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + } + ] + }, + { + "description": "hostname format", + "schema": { "format": "hostname" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + } + ] + }, + { + "description": "date format", + "schema": { "format": "date" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + } + ] + }, + { + "description": "date-time format", + "schema": { "format": "date-time" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + } + ] + }, + { + "description": "time format", + "schema": { "format": "time" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + } + ] + }, + { + "description": "json-pointer format", + "schema": { "format": "json-pointer" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + } + ] + }, + { + "description": "relative-json-pointer format", + "schema": { "format": "relative-json-pointer" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + } + ] + }, + { + "description": "iri format", + "schema": { "format": "iri" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + } + ] + }, + { + "description": "iri-reference format", + "schema": { "format": "iri-reference" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + } + ] + }, + { + "description": "uri format", + "schema": { "format": "uri" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + } + ] + }, + { + "description": "uri-reference format", + "schema": { "format": "uri-reference" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + } + ] + }, + { + "description": "uri-template format", + "schema": { "format": "uri-template" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/if-then-else.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/if-then-else.json new file mode 100644 index 000000000..a3bab7c0b --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/if-then-else.json @@ -0,0 +1,296 @@ +[ + { + "description": "ignore if without then or else", + "schema": { + "if": { + "const": 0 + } + }, + "tests": [ + { + "description": "valid when valid against lone if", + "data": 0, + "valid": true + }, + { + "description": "valid when invalid against lone if", + "data": "hello", + "valid": true + } + ] + }, + { + "description": "ignore then without if", + "schema": { + "then": { + "const": 0 + } + }, + "tests": [ + { + "description": "valid when valid against lone then", + "data": 0, + "valid": true + }, + { + "description": "valid when invalid against lone then", + "data": "hello", + "valid": true + } + ] + }, + { + "description": "ignore else without if", + "schema": { + "else": { + "const": 0 + } + }, + "tests": [ + { + "description": "valid when valid against lone else", + "data": 0, + "valid": true + }, + { + "description": "valid when invalid against lone else", + "data": "hello", + "valid": true + } + ] + }, + { + "description": "if and then without else", + "schema": { + "if": { + "exclusiveMaximum": 0 + }, + "then": { + "minimum": -10 + } + }, + "tests": [ + { + "description": "valid through then", + "data": -1, + "valid": true + }, + { + "description": "invalid through then", + "data": -100, + "valid": false + }, + { + "description": "valid when if test fails", + "data": 3, + "valid": true + } + ] + }, + { + "description": "if and else without then", + "schema": { + "if": { + "exclusiveMaximum": 0 + }, + "else": { + "multipleOf": 2 + } + }, + "tests": [ + { + "description": "valid when if test passes", + "data": -1, + "valid": true + }, + { + "description": "valid through else", + "data": 4, + "valid": true + }, + { + "description": "invalid through else", + "data": 3, + "valid": false + } + ] + }, + { + "description": "validate against correct branch, then vs else", + "schema": { + "if": { + "exclusiveMaximum": 0 + }, + "then": { + "minimum": -10 + }, + "else": { + "multipleOf": 2 + } + }, + "tests": [ + { + "description": "valid through then", + "data": -1, + "valid": true + }, + { + "description": "invalid through then", + "data": -100, + "valid": false + }, + { + "description": "valid through else", + "data": 4, + "valid": true + }, + { + "description": "invalid through else", + "data": 3, + "valid": false + } + ] + }, + { + "description": "non-interference across combined schemas", + "schema": { + "allOf": [ + { + "if": { + "exclusiveMaximum": 0 + } + }, + { + "then": { + "minimum": -10 + } + }, + { + "else": { + "multipleOf": 2 + } + } + ] + }, + "tests": [ + { + "description": "valid, but would have been invalid through then", + "data": -100, + "valid": true + }, + { + "description": "valid, but would have been invalid through else", + "data": 3, + "valid": true + } + ] + }, + { + "description": "if with boolean schema true", + "schema": { + "if": true, + "then": { "const": "then" }, + "else": { "const": "else" } + }, + "tests": [ + { + "description": "boolean schema true in if always chooses the then path (valid)", + "data": "then", + "valid": true + }, + { + "description": "boolean schema true in if always chooses the then path (invalid)", + "data": "else", + "valid": false + } + ] + }, + { + "description": "if with boolean schema false", + "schema": { + "if": false, + "then": { "const": "then" }, + "else": { "const": "else" } + }, + "tests": [ + { + "description": "boolean schema false in if always chooses the else path (invalid)", + "data": "then", + "valid": false + }, + { + "description": "boolean schema false in if always chooses the else path (valid)", + "data": "else", + "valid": true + } + ] + }, + { + "description": "if appears at the end when serialized (keyword processing sequence)", + "schema": { + "then": { "const": "yes" }, + "else": { "const": "other" }, + "if": { "maxLength": 4 } + }, + "tests": [ + { + "description": "yes redirects to then and passes", + "data": "yes", + "valid": true + }, + { + "description": "other redirects to else and passes", + "data": "other", + "valid": true + }, + { + "description": "no redirects to then and fails", + "data": "no", + "valid": false + }, + { + "description": "invalid redirects to else and fails", + "data": "invalid", + "valid": false + } + ] + }, + { + "description": "then: false fails when condition matches", + "schema": { + "if": { "const": 1 }, + "then": false + }, + "tests": [ + { + "description": "matches if → then=false → invalid", + "data": 1, + "valid": false + }, + { + "description": "does not match if → then ignored → valid", + "data": 2, + "valid": true + } + ] + }, + { + "description": "else: false fails when condition does not match", + "schema": { + "if": { "const": 1 }, + "else": false + }, + "tests": [ + { + "description": "matches if → else ignored → valid", + "data": 1, + "valid": true + }, + { + "description": "does not match if → else executes → invalid", + "data": 2, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/infinite-loop-detection.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/infinite-loop-detection.json new file mode 100644 index 000000000..f98c74fc6 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/infinite-loop-detection.json @@ -0,0 +1,36 @@ +[ + { + "description": "evaluating the same schema location against the same data location twice is not a sign of an infinite loop", + "schema": { + "definitions": { + "int": { "type": "integer" } + }, + "allOf": [ + { + "properties": { + "foo": { + "$ref": "#/definitions/int" + } + } + }, + { + "additionalProperties": { + "$ref": "#/definitions/int" + } + } + ] + }, + "tests": [ + { + "description": "passing case", + "data": { "foo": 1 }, + "valid": true + }, + { + "description": "failing case", + "data": { "foo": "a string" }, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/items.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/items.json new file mode 100644 index 000000000..7ed6781b6 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/items.json @@ -0,0 +1,282 @@ +[ + { + "description": "a schema given for items", + "schema": { + "items": {"type": "integer"} + }, + "tests": [ + { + "description": "valid items", + "data": [ 1, 2, 3 ], + "valid": true + }, + { + "description": "wrong type of items", + "data": [1, "x"], + "valid": false + }, + { + "description": "ignores non-arrays", + "data": {"foo" : "bar"}, + "valid": true + }, + { + "description": "JavaScript pseudo-array is valid", + "data": { + "0": "invalid", + "length": 1 + }, + "valid": true + } + ] + }, + { + "description": "an array of schemas for items", + "schema": { + "items": [ + {"type": "integer"}, + {"type": "string"} + ] + }, + "tests": [ + { + "description": "correct types", + "data": [ 1, "foo" ], + "valid": true + }, + { + "description": "wrong types", + "data": [ "foo", 1 ], + "valid": false + }, + { + "description": "incomplete array of items", + "data": [ 1 ], + "valid": true + }, + { + "description": "array with additional items", + "data": [ 1, "foo", true ], + "valid": true + }, + { + "description": "empty array", + "data": [ ], + "valid": true + }, + { + "description": "JavaScript pseudo-array is valid", + "data": { + "0": "invalid", + "1": "valid", + "length": 2 + }, + "valid": true + } + ] + }, + { + "description": "items with boolean schema (true)", + "schema": {"items": true}, + "tests": [ + { + "description": "any array is valid", + "data": [ 1, "foo", true ], + "valid": true + }, + { + "description": "empty array is valid", + "data": [], + "valid": true + } + ] + }, + { + "description": "items with boolean schema (false)", + "schema": {"items": false}, + "tests": [ + { + "description": "any non-empty array is invalid", + "data": [ 1, "foo", true ], + "valid": false + }, + { + "description": "empty array is valid", + "data": [], + "valid": true + } + ] + }, + { + "description": "items with boolean schemas", + "schema": { + "items": [true, false] + }, + "tests": [ + { + "description": "array with one item is valid", + "data": [ 1 ], + "valid": true + }, + { + "description": "array with two items is invalid", + "data": [ 1, "foo" ], + "valid": false + }, + { + "description": "empty array is valid", + "data": [], + "valid": true + } + ] + }, + { + "description": "items and subitems", + "schema": { + "definitions": { + "item": { + "type": "array", + "additionalItems": false, + "items": [ + { "$ref": "#/definitions/sub-item" }, + { "$ref": "#/definitions/sub-item" } + ] + }, + "sub-item": { + "type": "object", + "required": ["foo"] + } + }, + "type": "array", + "additionalItems": false, + "items": [ + { "$ref": "#/definitions/item" }, + { "$ref": "#/definitions/item" }, + { "$ref": "#/definitions/item" } + ] + }, + "tests": [ + { + "description": "valid items", + "data": [ + [ {"foo": null}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ] + ], + "valid": true + }, + { + "description": "too many items", + "data": [ + [ {"foo": null}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ] + ], + "valid": false + }, + { + "description": "too many sub-items", + "data": [ + [ {"foo": null}, {"foo": null}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ] + ], + "valid": false + }, + { + "description": "wrong item", + "data": [ + {"foo": null}, + [ {"foo": null}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ] + ], + "valid": false + }, + { + "description": "wrong sub-item", + "data": [ + [ {}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ], + [ {"foo": null}, {"foo": null} ] + ], + "valid": false + }, + { + "description": "fewer items is valid", + "data": [ + [ {"foo": null} ], + [ {"foo": null} ] + ], + "valid": true + } + ] + }, + { + "description": "nested items", + "schema": { + "type": "array", + "items": { + "type": "array", + "items": { + "type": "array", + "items": { + "type": "array", + "items": { + "type": "number" + } + } + } + } + }, + "tests": [ + { + "description": "valid nested array", + "data": [[[[1]], [[2],[3]]], [[[4], [5], [6]]]], + "valid": true + }, + { + "description": "nested array with invalid type", + "data": [[[["1"]], [[2],[3]]], [[[4], [5], [6]]]], + "valid": false + }, + { + "description": "not deep enough", + "data": [[[1], [2],[3]], [[4], [5], [6]]], + "valid": false + } + ] + }, + { + "description": "single-form items with null instance elements", + "schema": { + "items": { + "type": "null" + } + }, + "tests": [ + { + "description": "allows null elements", + "data": [ null ], + "valid": true + } + ] + }, + { + "description": "array-form items with null instance elements", + "schema": { + "items": [ + { + "type": "null" + } + ] + }, + "tests": [ + { + "description": "allows null elements", + "data": [ null ], + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/maxItems.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/maxItems.json new file mode 100644 index 000000000..f0c36ab28 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/maxItems.json @@ -0,0 +1,44 @@ +[ + { + "description": "maxItems validation", + "schema": {"maxItems": 2}, + "tests": [ + { + "description": "shorter is valid", + "data": [1], + "valid": true + }, + { + "description": "exact length is valid", + "data": [1, 2], + "valid": true + }, + { + "description": "too long is invalid", + "data": [1, 2, 3], + "valid": false + }, + { + "description": "ignores non-arrays", + "data": "foobar", + "valid": true + } + ] + }, + { + "description": "maxItems validation with a decimal", + "schema": {"maxItems": 2.0}, + "tests": [ + { + "description": "shorter is valid", + "data": [1], + "valid": true + }, + { + "description": "too long is invalid", + "data": [1, 2, 3], + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/maxLength.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/maxLength.json new file mode 100644 index 000000000..be60c5407 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/maxLength.json @@ -0,0 +1,49 @@ +[ + { + "description": "maxLength validation", + "schema": {"maxLength": 2}, + "tests": [ + { + "description": "shorter is valid", + "data": "f", + "valid": true + }, + { + "description": "exact length is valid", + "data": "fo", + "valid": true + }, + { + "description": "too long is invalid", + "data": "foo", + "valid": false + }, + { + "description": "ignores non-strings", + "data": 100, + "valid": true + }, + { + "description": "two graphemes is long enough", + "data": "\uD83D\uDCA9\uD83D\uDCA9", + "valid": true + } + ] + }, + { + "description": "maxLength validation with a decimal", + "schema": {"maxLength": 2.0}, + "tests": [ + { + "description": "shorter is valid", + "data": "f", + "valid": true + }, + { + "description": "too long is invalid", + "data": "foo", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/maxProperties.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/maxProperties.json new file mode 100644 index 000000000..acec14209 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/maxProperties.json @@ -0,0 +1,70 @@ +[ + { + "description": "maxProperties validation", + "schema": {"maxProperties": 2}, + "tests": [ + { + "description": "shorter is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "exact length is valid", + "data": {"foo": 1, "bar": 2}, + "valid": true + }, + { + "description": "too long is invalid", + "data": {"foo": 1, "bar": 2, "baz": 3}, + "valid": false + }, + { + "description": "ignores arrays", + "data": [1, 2, 3], + "valid": true + }, + { + "description": "ignores strings", + "data": "foobar", + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + } + ] + }, + { + "description": "maxProperties validation with a decimal", + "schema": {"maxProperties": 2.0}, + "tests": [ + { + "description": "shorter is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "too long is invalid", + "data": {"foo": 1, "bar": 2, "baz": 3}, + "valid": false + } + ] + }, + { + "description": "maxProperties = 0 means the object is empty", + "schema": { "maxProperties": 0 }, + "tests": [ + { + "description": "no properties is valid", + "data": {}, + "valid": true + }, + { + "description": "one property is invalid", + "data": { "foo": 1 }, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/maximum.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/maximum.json new file mode 100644 index 000000000..c3e727069 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/maximum.json @@ -0,0 +1,54 @@ +[ + { + "description": "maximum validation", + "schema": {"maximum": 3.0}, + "tests": [ + { + "description": "below the maximum is valid", + "data": 2.6, + "valid": true + }, + { + "description": "boundary point is valid", + "data": 3.0, + "valid": true + }, + { + "description": "above the maximum is invalid", + "data": 3.5, + "valid": false + }, + { + "description": "ignores non-numbers", + "data": "x", + "valid": true + } + ] + }, + { + "description": "maximum validation with unsigned integer", + "schema": {"maximum": 300}, + "tests": [ + { + "description": "below the maximum is valid", + "data": 299.97, + "valid": true + }, + { + "description": "boundary point integer is valid", + "data": 300, + "valid": true + }, + { + "description": "boundary point float is valid", + "data": 300.00, + "valid": true + }, + { + "description": "above the maximum is invalid", + "data": 300.5, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/minItems.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/minItems.json new file mode 100644 index 000000000..d3b18720d --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/minItems.json @@ -0,0 +1,44 @@ +[ + { + "description": "minItems validation", + "schema": {"minItems": 1}, + "tests": [ + { + "description": "longer is valid", + "data": [1, 2], + "valid": true + }, + { + "description": "exact length is valid", + "data": [1], + "valid": true + }, + { + "description": "too short is invalid", + "data": [], + "valid": false + }, + { + "description": "ignores non-arrays", + "data": "", + "valid": true + } + ] + }, + { + "description": "minItems validation with a decimal", + "schema": {"minItems": 1.0}, + "tests": [ + { + "description": "longer is valid", + "data": [1, 2], + "valid": true + }, + { + "description": "too short is invalid", + "data": [], + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/minLength.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/minLength.json new file mode 100644 index 000000000..23c68fe3f --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/minLength.json @@ -0,0 +1,49 @@ +[ + { + "description": "minLength validation", + "schema": {"minLength": 2}, + "tests": [ + { + "description": "longer is valid", + "data": "foo", + "valid": true + }, + { + "description": "exact length is valid", + "data": "fo", + "valid": true + }, + { + "description": "too short is invalid", + "data": "f", + "valid": false + }, + { + "description": "ignores non-strings", + "data": 1, + "valid": true + }, + { + "description": "one grapheme is not long enough", + "data": "\uD83D\uDCA9", + "valid": false + } + ] + }, + { + "description": "minLength validation with a decimal", + "schema": {"minLength": 2.0}, + "tests": [ + { + "description": "longer is valid", + "data": "foo", + "valid": true + }, + { + "description": "too short is invalid", + "data": "f", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/minProperties.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/minProperties.json new file mode 100644 index 000000000..416e02010 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/minProperties.json @@ -0,0 +1,64 @@ +[ + { + "description": "minProperties validation", + "schema": {"minProperties": 1}, + "tests": [ + { + "description": "longer is valid", + "data": {"foo": 1, "bar": 2}, + "valid": true + }, + { + "description": "exact length is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "too short is invalid", + "data": {}, + "valid": false + }, + { + "description": "ignores arrays", + "data": [], + "valid": true + }, + { + "description": "ignores strings", + "data": "", + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + }, + { + "description": "ignores null", + "data": null, + "valid": true + }, + { + "description": "ignores booleans", + "data": true, + "valid": true + } + ] + }, + { + "description": "minProperties validation with a decimal", + "schema": {"minProperties": 1.0}, + "tests": [ + { + "description": "longer is valid", + "data": {"foo": 1, "bar": 2}, + "valid": true + }, + { + "description": "too short is invalid", + "data": {}, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/minimum.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/minimum.json new file mode 100644 index 000000000..21ae50e0e --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/minimum.json @@ -0,0 +1,69 @@ +[ + { + "description": "minimum validation", + "schema": {"minimum": 1.1}, + "tests": [ + { + "description": "above the minimum is valid", + "data": 2.6, + "valid": true + }, + { + "description": "boundary point is valid", + "data": 1.1, + "valid": true + }, + { + "description": "below the minimum is invalid", + "data": 0.6, + "valid": false + }, + { + "description": "ignores non-numbers", + "data": "x", + "valid": true + } + ] + }, + { + "description": "minimum validation with signed integer", + "schema": {"minimum": -2}, + "tests": [ + { + "description": "negative above the minimum is valid", + "data": -1, + "valid": true + }, + { + "description": "positive above the minimum is valid", + "data": 0, + "valid": true + }, + { + "description": "boundary point is valid", + "data": -2, + "valid": true + }, + { + "description": "boundary point with float is valid", + "data": -2.0, + "valid": true + }, + { + "description": "float below the minimum is invalid", + "data": -2.0001, + "valid": false + }, + { + "description": "int below the minimum is invalid", + "data": -3, + "valid": false + }, + { + "description": "ignores non-numbers", + "data": "x", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/multipleOf.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/multipleOf.json new file mode 100644 index 000000000..da0ce0830 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/multipleOf.json @@ -0,0 +1,87 @@ +[ + { + "description": "by int", + "schema": {"multipleOf": 2}, + "tests": [ + { + "description": "int by int", + "data": 10, + "valid": true + }, + { + "description": "int by int fail", + "data": 7, + "valid": false + }, + { + "description": "ignores non-numbers", + "data": "foo", + "valid": true + } + ] + }, + { + "description": "by number", + "schema": {"multipleOf": 1.5}, + "tests": [ + { + "description": "zero is multiple of anything", + "data": 0, + "valid": true + }, + { + "description": "4.5 is multiple of 1.5", + "data": 4.5, + "valid": true + }, + { + "description": "-4.5 is multiple of 1.5", + "data": -4.5, + "valid": true + }, + { + "description": "35 is not multiple of 1.5", + "data": 35, + "valid": false + } + ] + }, + { + "description": "by small number", + "schema": {"multipleOf": 0.0001}, + "tests": [ + { + "description": "0.0075 is multiple of 0.0001", + "data": 0.0075, + "valid": true + }, + { + "description": "0.00751 is not multiple of 0.0001", + "data": 0.00751, + "valid": false + } + ] + }, + { + "description": "float division = inf", + "schema": {"type": "integer", "multipleOf": 0.123456789}, + "tests": [ + { + "description": "always invalid, but naive implementations may raise an overflow error", + "data": 1e308, + "valid": false + } + ] + }, + { + "description": "small multiple of large integer", + "schema": {"type": "integer", "multipleOf": 1e-8}, + "tests": [ + { + "description": "any integer is a multiple of 1e-8", + "data": 12391239123, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/not.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/not.json new file mode 100644 index 000000000..b46c4ed05 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/not.json @@ -0,0 +1,259 @@ +[ + { + "description": "not", + "schema": { + "not": {"type": "integer"} + }, + "tests": [ + { + "description": "allowed", + "data": "foo", + "valid": true + }, + { + "description": "disallowed", + "data": 1, + "valid": false + } + ] + }, + { + "description": "not multiple types", + "schema": { + "not": {"type": ["integer", "boolean"]} + }, + "tests": [ + { + "description": "valid", + "data": "foo", + "valid": true + }, + { + "description": "mismatch", + "data": 1, + "valid": false + }, + { + "description": "other mismatch", + "data": true, + "valid": false + } + ] + }, + { + "description": "not more complex schema", + "schema": { + "not": { + "type": "object", + "properties": { + "foo": { + "type": "string" + } + } + } + }, + "tests": [ + { + "description": "match", + "data": 1, + "valid": true + }, + { + "description": "other match", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "mismatch", + "data": {"foo": "bar"}, + "valid": false + } + ] + }, + { + "description": "forbidden property", + "schema": { + "properties": { + "foo": { + "not": {} + } + } + }, + "tests": [ + { + "description": "property present", + "data": {"foo": 1, "bar": 2}, + "valid": false + }, + { + "description": "property absent", + "data": {"bar": 1, "baz": 2}, + "valid": true + } + ] + }, + { + "description": "forbid everything with empty schema", + "schema": { "not": {} }, + "tests": [ + { + "description": "number is invalid", + "data": 1, + "valid": false + }, + { + "description": "string is invalid", + "data": "foo", + "valid": false + }, + { + "description": "boolean true is invalid", + "data": true, + "valid": false + }, + { + "description": "boolean false is invalid", + "data": false, + "valid": false + }, + { + "description": "null is invalid", + "data": null, + "valid": false + }, + { + "description": "object is invalid", + "data": {"foo": "bar"}, + "valid": false + }, + { + "description": "empty object is invalid", + "data": {}, + "valid": false + }, + { + "description": "array is invalid", + "data": ["foo"], + "valid": false + }, + { + "description": "empty array is invalid", + "data": [], + "valid": false + } + ] + }, + { + "description": "forbid everything with boolean schema true", + "schema": { "not": true }, + "tests": [ + { + "description": "number is invalid", + "data": 1, + "valid": false + }, + { + "description": "string is invalid", + "data": "foo", + "valid": false + }, + { + "description": "boolean true is invalid", + "data": true, + "valid": false + }, + { + "description": "boolean false is invalid", + "data": false, + "valid": false + }, + { + "description": "null is invalid", + "data": null, + "valid": false + }, + { + "description": "object is invalid", + "data": {"foo": "bar"}, + "valid": false + }, + { + "description": "empty object is invalid", + "data": {}, + "valid": false + }, + { + "description": "array is invalid", + "data": ["foo"], + "valid": false + }, + { + "description": "empty array is invalid", + "data": [], + "valid": false + } + ] + }, + { + "description": "allow everything with boolean schema false", + "schema": { "not": false }, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "string is valid", + "data": "foo", + "valid": true + }, + { + "description": "boolean true is valid", + "data": true, + "valid": true + }, + { + "description": "boolean false is valid", + "data": false, + "valid": true + }, + { + "description": "null is valid", + "data": null, + "valid": true + }, + { + "description": "object is valid", + "data": {"foo": "bar"}, + "valid": true + }, + { + "description": "empty object is valid", + "data": {}, + "valid": true + }, + { + "description": "array is valid", + "data": ["foo"], + "valid": true + }, + { + "description": "empty array is valid", + "data": [], + "valid": true + } + ] + }, + { + "description": "double negation", + "schema": { "not": { "not": {} } }, + "tests": [ + { + "description": "any value is valid", + "data": "foo", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/oneOf.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/oneOf.json new file mode 100644 index 000000000..c30a65c0d --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/oneOf.json @@ -0,0 +1,274 @@ +[ + { + "description": "oneOf", + "schema": { + "oneOf": [ + { + "type": "integer" + }, + { + "minimum": 2 + } + ] + }, + "tests": [ + { + "description": "first oneOf valid", + "data": 1, + "valid": true + }, + { + "description": "second oneOf valid", + "data": 2.5, + "valid": true + }, + { + "description": "both oneOf valid", + "data": 3, + "valid": false + }, + { + "description": "neither oneOf valid", + "data": 1.5, + "valid": false + } + ] + }, + { + "description": "oneOf with base schema", + "schema": { + "type": "string", + "oneOf" : [ + { + "minLength": 2 + }, + { + "maxLength": 4 + } + ] + }, + "tests": [ + { + "description": "mismatch base schema", + "data": 3, + "valid": false + }, + { + "description": "one oneOf valid", + "data": "foobar", + "valid": true + }, + { + "description": "both oneOf valid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "oneOf with boolean schemas, all true", + "schema": {"oneOf": [true, true, true]}, + "tests": [ + { + "description": "any value is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "oneOf with boolean schemas, one true", + "schema": {"oneOf": [true, false, false]}, + "tests": [ + { + "description": "any value is valid", + "data": "foo", + "valid": true + } + ] + }, + { + "description": "oneOf with boolean schemas, more than one true", + "schema": {"oneOf": [true, true, false]}, + "tests": [ + { + "description": "any value is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "oneOf with boolean schemas, all false", + "schema": {"oneOf": [false, false, false]}, + "tests": [ + { + "description": "any value is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "oneOf complex types", + "schema": { + "oneOf": [ + { + "properties": { + "bar": {"type": "integer"} + }, + "required": ["bar"] + }, + { + "properties": { + "foo": {"type": "string"} + }, + "required": ["foo"] + } + ] + }, + "tests": [ + { + "description": "first oneOf valid (complex)", + "data": {"bar": 2}, + "valid": true + }, + { + "description": "second oneOf valid (complex)", + "data": {"foo": "baz"}, + "valid": true + }, + { + "description": "both oneOf valid (complex)", + "data": {"foo": "baz", "bar": 2}, + "valid": false + }, + { + "description": "neither oneOf valid (complex)", + "data": {"foo": 2, "bar": "quux"}, + "valid": false + } + ] + }, + { + "description": "oneOf with empty schema", + "schema": { + "oneOf": [ + { "type": "number" }, + {} + ] + }, + "tests": [ + { + "description": "one valid - valid", + "data": "foo", + "valid": true + }, + { + "description": "both valid - invalid", + "data": 123, + "valid": false + } + ] + }, + { + "description": "oneOf with required", + "schema": { + "type": "object", + "oneOf": [ + { "required": ["foo", "bar"] }, + { "required": ["foo", "baz"] } + ] + }, + "tests": [ + { + "description": "both invalid - invalid", + "data": {"bar": 2}, + "valid": false + }, + { + "description": "first valid - valid", + "data": {"foo": 1, "bar": 2}, + "valid": true + }, + { + "description": "second valid - valid", + "data": {"foo": 1, "baz": 3}, + "valid": true + }, + { + "description": "both valid - invalid", + "data": {"foo": 1, "bar": 2, "baz" : 3}, + "valid": false + } + ] + }, + { + "description": "oneOf with missing optional property", + "schema": { + "oneOf": [ + { + "properties": { + "bar": true, + "baz": true + }, + "required": ["bar"] + }, + { + "properties": { + "foo": true + }, + "required": ["foo"] + } + ] + }, + "tests": [ + { + "description": "first oneOf valid", + "data": {"bar": 8}, + "valid": true + }, + { + "description": "second oneOf valid", + "data": {"foo": "foo"}, + "valid": true + }, + { + "description": "both oneOf valid", + "data": {"foo": "foo", "bar": 8}, + "valid": false + }, + { + "description": "neither oneOf valid", + "data": {"baz": "quux"}, + "valid": false + } + ] + }, + { + "description": "nested oneOf, to check validation semantics", + "schema": { + "oneOf": [ + { + "oneOf": [ + { + "type": "null" + } + ] + } + ] + }, + "tests": [ + { + "description": "null is valid", + "data": null, + "valid": true + }, + { + "description": "anything non-null is invalid", + "data": 123, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/bignum.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/bignum.json new file mode 100644 index 000000000..94b4a4e62 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/bignum.json @@ -0,0 +1,93 @@ +[ + { + "description": "integer", + "schema": { "type": "integer" }, + "tests": [ + { + "description": "a bignum is an integer", + "data": 12345678910111213141516171819202122232425262728293031, + "valid": true + }, + { + "description": "a negative bignum is an integer", + "data": -12345678910111213141516171819202122232425262728293031, + "valid": true + } + ] + }, + { + "description": "number", + "schema": { "type": "number" }, + "tests": [ + { + "description": "a bignum is a number", + "data": 98249283749234923498293171823948729348710298301928331, + "valid": true + }, + { + "description": "a negative bignum is a number", + "data": -98249283749234923498293171823948729348710298301928331, + "valid": true + } + ] + }, + { + "description": "string", + "schema": { "type": "string" }, + "tests": [ + { + "description": "a bignum is not a string", + "data": 98249283749234923498293171823948729348710298301928331, + "valid": false + } + ] + }, + { + "description": "maximum integer comparison", + "schema": { "maximum": 18446744073709551615 }, + "tests": [ + { + "description": "comparison works for high numbers", + "data": 18446744073709551600, + "valid": true + } + ] + }, + { + "description": "float comparison with high precision", + "schema": { + "exclusiveMaximum": 972783798187987123879878123.18878137 + }, + "tests": [ + { + "description": "comparison works for high numbers", + "data": 972783798187987123879878123.188781371, + "valid": false + } + ] + }, + { + "description": "minimum integer comparison", + "schema": { "minimum": -18446744073709551615 }, + "tests": [ + { + "description": "comparison works for very negative numbers", + "data": -18446744073709551600, + "valid": true + } + ] + }, + { + "description": "float comparison with high precision on negative numbers", + "schema": { + "exclusiveMinimum": -972783798187987123879878123.18878137 + }, + "tests": [ + { + "description": "comparison works for very negative numbers", + "data": -972783798187987123879878123.188781371, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/content.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/content.json new file mode 100644 index 000000000..3f5a7430b --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/content.json @@ -0,0 +1,77 @@ +[ + { + "description": "validation of string-encoded content based on media type", + "schema": { + "contentMediaType": "application/json" + }, + "tests": [ + { + "description": "a valid JSON document", + "data": "{\"foo\": \"bar\"}", + "valid": true + }, + { + "description": "an invalid JSON document", + "data": "{:}", + "valid": false + }, + { + "description": "ignores non-strings", + "data": 100, + "valid": true + } + ] + }, + { + "description": "validation of binary string-encoding", + "schema": { + "contentEncoding": "base64" + }, + "tests": [ + { + "description": "a valid base64 string", + "data": "eyJmb28iOiAiYmFyIn0K", + "valid": true + }, + { + "description": "an invalid base64 string (% is not a valid character)", + "data": "eyJmb28iOi%iYmFyIn0K", + "valid": false + }, + { + "description": "ignores non-strings", + "data": 100, + "valid": true + } + ] + }, + { + "description": "validation of binary-encoded media type documents", + "schema": { + "contentMediaType": "application/json", + "contentEncoding": "base64" + }, + "tests": [ + { + "description": "a valid base64-encoded JSON document", + "data": "eyJmb28iOiAiYmFyIn0K", + "valid": true + }, + { + "description": "a validly-encoded invalid JSON document", + "data": "ezp9Cg==", + "valid": false + }, + { + "description": "an invalid base64 string that is valid JSON", + "data": "{}", + "valid": false + }, + { + "description": "ignores non-strings", + "data": 100, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/cross-draft.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/cross-draft.json new file mode 100644 index 000000000..8ff537362 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/cross-draft.json @@ -0,0 +1,25 @@ +[ + { + "description": "refs to future drafts are processed as future drafts", + "schema": { + "type": "object", + "allOf": [ + { "properties": { "foo": true } }, + { "$ref": "http://localhost:1234/draft2019-09/dependentRequired.json" } + ] + }, + "tests": [ + { + "description": "missing bar is invalid", + "comment": "if the implementation is not processing the $ref as a 2019-09 schema, this test will fail", + "data": {"foo": "any value"}, + "valid": false + }, + { + "description": "present bar is valid", + "data": {"foo": "any value", "bar": "also any value"}, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/ecmascript-regex.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/ecmascript-regex.json new file mode 100644 index 000000000..c4886aaa8 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/ecmascript-regex.json @@ -0,0 +1,552 @@ +[ + { + "description": "ECMA 262 regex $ does not match trailing newline", + "schema": { + "type": "string", + "pattern": "^abc$" + }, + "tests": [ + { + "description": "matches in Python, but not in ECMA 262", + "data": "abc\\n", + "valid": false + }, + { + "description": "matches", + "data": "abc", + "valid": true + } + ] + }, + { + "description": "ECMA 262 regex converts \\t to horizontal tab", + "schema": { + "type": "string", + "pattern": "^\\t$" + }, + "tests": [ + { + "description": "does not match", + "data": "\\t", + "valid": false + }, + { + "description": "matches", + "data": "\u0009", + "valid": true + } + ] + }, + { + "description": "ECMA 262 regex escapes control codes with \\c and upper letter", + "schema": { + "type": "string", + "pattern": "^\\cC$" + }, + "tests": [ + { + "description": "does not match", + "data": "\\cC", + "valid": false + }, + { + "description": "matches", + "data": "\u0003", + "valid": true + } + ] + }, + { + "description": "ECMA 262 regex escapes control codes with \\c and lower letter", + "schema": { + "type": "string", + "pattern": "^\\cc$" + }, + "tests": [ + { + "description": "does not match", + "data": "\\cc", + "valid": false + }, + { + "description": "matches", + "data": "\u0003", + "valid": true + } + ] + }, + { + "description": "ECMA 262 \\d matches ascii digits only", + "schema": { + "type": "string", + "pattern": "^\\d$" + }, + "tests": [ + { + "description": "ASCII zero matches", + "data": "0", + "valid": true + }, + { + "description": "NKO DIGIT ZERO does not match (unlike e.g. Python)", + "data": "߀", + "valid": false + }, + { + "description": "NKO DIGIT ZERO (as \\u escape) does not match", + "data": "\u07c0", + "valid": false + } + ] + }, + { + "description": "ECMA 262 \\D matches everything but ascii digits", + "schema": { + "type": "string", + "pattern": "^\\D$" + }, + "tests": [ + { + "description": "ASCII zero does not match", + "data": "0", + "valid": false + }, + { + "description": "NKO DIGIT ZERO matches (unlike e.g. Python)", + "data": "߀", + "valid": true + }, + { + "description": "NKO DIGIT ZERO (as \\u escape) matches", + "data": "\u07c0", + "valid": true + } + ] + }, + { + "description": "ECMA 262 \\w matches ascii letters only", + "schema": { + "type": "string", + "pattern": "^\\w$" + }, + "tests": [ + { + "description": "ASCII 'a' matches", + "data": "a", + "valid": true + }, + { + "description": "latin-1 e-acute does not match (unlike e.g. Python)", + "data": "é", + "valid": false + } + ] + }, + { + "description": "ECMA 262 \\W matches everything but ascii letters", + "schema": { + "type": "string", + "pattern": "^\\W$" + }, + "tests": [ + { + "description": "ASCII 'a' does not match", + "data": "a", + "valid": false + }, + { + "description": "latin-1 e-acute matches (unlike e.g. Python)", + "data": "é", + "valid": true + } + ] + }, + { + "description": "ECMA 262 \\s matches whitespace", + "schema": { + "type": "string", + "pattern": "^\\s$" + }, + "tests": [ + { + "description": "ASCII space matches", + "data": " ", + "valid": true + }, + { + "description": "Character tabulation matches", + "data": "\t", + "valid": true + }, + { + "description": "Line tabulation matches", + "data": "\u000b", + "valid": true + }, + { + "description": "Form feed matches", + "data": "\u000c", + "valid": true + }, + { + "description": "latin-1 non-breaking-space matches", + "data": "\u00a0", + "valid": true + }, + { + "description": "zero-width whitespace matches", + "data": "\ufeff", + "valid": true + }, + { + "description": "line feed matches (line terminator)", + "data": "\u000a", + "valid": true + }, + { + "description": "paragraph separator matches (line terminator)", + "data": "\u2029", + "valid": true + }, + { + "description": "EM SPACE matches (Space_Separator)", + "data": "\u2003", + "valid": true + }, + { + "description": "Non-whitespace control does not match", + "data": "\u0001", + "valid": false + }, + { + "description": "Non-whitespace does not match", + "data": "\u2013", + "valid": false + } + ] + }, + { + "description": "ECMA 262 \\S matches everything but whitespace", + "schema": { + "type": "string", + "pattern": "^\\S$" + }, + "tests": [ + { + "description": "ASCII space does not match", + "data": " ", + "valid": false + }, + { + "description": "Character tabulation does not match", + "data": "\t", + "valid": false + }, + { + "description": "Line tabulation does not match", + "data": "\u000b", + "valid": false + }, + { + "description": "Form feed does not match", + "data": "\u000c", + "valid": false + }, + { + "description": "latin-1 non-breaking-space does not match", + "data": "\u00a0", + "valid": false + }, + { + "description": "zero-width whitespace does not match", + "data": "\ufeff", + "valid": false + }, + { + "description": "line feed does not match (line terminator)", + "data": "\u000a", + "valid": false + }, + { + "description": "paragraph separator does not match (line terminator)", + "data": "\u2029", + "valid": false + }, + { + "description": "EM SPACE does not match (Space_Separator)", + "data": "\u2003", + "valid": false + }, + { + "description": "Non-whitespace control matches", + "data": "\u0001", + "valid": true + }, + { + "description": "Non-whitespace matches", + "data": "\u2013", + "valid": true + } + ] + }, + { + "description": "patterns always use unicode semantics with pattern", + "schema": { "pattern": "\\p{Letter}cole" }, + "tests": [ + { + "description": "ascii character in json string", + "data": "Les hivers de mon enfance etaient des saisons longues, longues. Nous vivions en trois lieux: l'ecole, l'eglise et la patinoire; mais la vraie vie etait sur la patinoire.", + "valid": true + }, + { + "description": "literal unicode character in json string", + "data": "Les hivers de mon enfance étaient des saisons longues, longues. Nous vivions en trois lieux: l'école, l'église et la patinoire; mais la vraie vie était sur la patinoire.", + "valid": true + }, + { + "description": "unicode character in hex format in string", + "data": "Les hivers de mon enfance étaient des saisons longues, longues. Nous vivions en trois lieux: l'\u00e9cole, l'église et la patinoire; mais la vraie vie était sur la patinoire.", + "valid": true + }, + { + "description": "unicode matching is case-sensitive", + "data": "LES HIVERS DE MON ENFANCE ÉTAIENT DES SAISONS LONGUES, LONGUES. NOUS VIVIONS EN TROIS LIEUX: L'ÉCOLE, L'ÉGLISE ET LA PATINOIRE; MAIS LA VRAIE VIE ÉTAIT SUR LA PATINOIRE.", + "valid": false + } + ] + }, + { + "description": "\\w in patterns matches [A-Za-z0-9_], not unicode letters", + "schema": { "pattern": "\\wcole" }, + "tests": [ + { + "description": "ascii character in json string", + "data": "Les hivers de mon enfance etaient des saisons longues, longues. Nous vivions en trois lieux: l'ecole, l'eglise et la patinoire; mais la vraie vie etait sur la patinoire.", + "valid": true + }, + { + "description": "literal unicode character in json string", + "data": "Les hivers de mon enfance étaient des saisons longues, longues. Nous vivions en trois lieux: l'école, l'église et la patinoire; mais la vraie vie était sur la patinoire.", + "valid": false + }, + { + "description": "unicode character in hex format in string", + "data": "Les hivers de mon enfance étaient des saisons longues, longues. Nous vivions en trois lieux: l'\u00e9cole, l'église et la patinoire; mais la vraie vie était sur la patinoire.", + "valid": false + }, + { + "description": "unicode matching is case-sensitive", + "data": "LES HIVERS DE MON ENFANCE ÉTAIENT DES SAISONS LONGUES, LONGUES. NOUS VIVIONS EN TROIS LIEUX: L'ÉCOLE, L'ÉGLISE ET LA PATINOIRE; MAIS LA VRAIE VIE ÉTAIT SUR LA PATINOIRE.", + "valid": false + } + ] + }, + { + "description": "pattern with ASCII ranges", + "schema": { "pattern": "[a-z]cole" }, + "tests": [ + { + "description": "literal unicode character in json string", + "data": "Les hivers de mon enfance étaient des saisons longues, longues. Nous vivions en trois lieux: l'école, l'église et la patinoire; mais la vraie vie était sur la patinoire.", + "valid": false + }, + { + "description": "unicode character in hex format in string", + "data": "Les hivers de mon enfance étaient des saisons longues, longues. Nous vivions en trois lieux: l'\u00e9cole, l'église et la patinoire; mais la vraie vie était sur la patinoire.", + "valid": false + }, + { + "description": "ascii characters match", + "data": "Les hivers de mon enfance etaient des saisons longues, longues. Nous vivions en trois lieux: l'ecole, l'eglise et la patinoire; mais la vraie vie etait sur la patinoire.", + "valid": true + } + ] + }, + { + "description": "\\d in pattern matches [0-9], not unicode digits", + "schema": { "pattern": "^\\d+$" }, + "tests": [ + { + "description": "ascii digits", + "data": "42", + "valid": true + }, + { + "description": "ascii non-digits", + "data": "-%#", + "valid": false + }, + { + "description": "non-ascii digits (BENGALI DIGIT FOUR, BENGALI DIGIT TWO)", + "data": "৪২", + "valid": false + } + ] + }, + { + "description": "pattern with non-ASCII digits", + "schema": { "pattern": "^\\p{digit}+$" }, + "tests": [ + { + "description": "ascii digits", + "data": "42", + "valid": true + }, + { + "description": "ascii non-digits", + "data": "-%#", + "valid": false + }, + { + "description": "non-ascii digits (BENGALI DIGIT FOUR, BENGALI DIGIT TWO)", + "data": "৪২", + "valid": true + } + ] + }, + { + "description": "patterns always use unicode semantics with patternProperties", + "schema": { + "type": "object", + "patternProperties": { + "\\p{Letter}cole": true + }, + "additionalProperties": false + }, + "tests": [ + { + "description": "ascii character in json string", + "data": { "l'ecole": "pas de vraie vie" }, + "valid": true + }, + { + "description": "literal unicode character in json string", + "data": { "l'école": "pas de vraie vie" }, + "valid": true + }, + { + "description": "unicode character in hex format in string", + "data": { "l'\u00e9cole": "pas de vraie vie" }, + "valid": true + }, + { + "description": "unicode matching is case-sensitive", + "data": { "L'ÉCOLE": "PAS DE VRAIE VIE" }, + "valid": false + } + ] + }, + { + "description": "\\w in patternProperties matches [A-Za-z0-9_], not unicode letters", + "schema": { + "type": "object", + "patternProperties": { + "\\wcole": true + }, + "additionalProperties": false + }, + "tests": [ + { + "description": "ascii character in json string", + "data": { "l'ecole": "pas de vraie vie" }, + "valid": true + }, + { + "description": "literal unicode character in json string", + "data": { "l'école": "pas de vraie vie" }, + "valid": false + }, + { + "description": "unicode character in hex format in string", + "data": { "l'\u00e9cole": "pas de vraie vie" }, + "valid": false + }, + { + "description": "unicode matching is case-sensitive", + "data": { "L'ÉCOLE": "PAS DE VRAIE VIE" }, + "valid": false + } + ] + }, + { + "description": "patternProperties with ASCII ranges", + "schema": { + "type": "object", + "patternProperties": { + "[a-z]cole": true + }, + "additionalProperties": false + }, + "tests": [ + { + "description": "literal unicode character in json string", + "data": { "l'école": "pas de vraie vie" }, + "valid": false + }, + { + "description": "unicode character in hex format in string", + "data": { "l'\u00e9cole": "pas de vraie vie" }, + "valid": false + }, + { + "description": "ascii characters match", + "data": { "l'ecole": "pas de vraie vie" }, + "valid": true + } + ] + }, + { + "description": "\\d in patternProperties matches [0-9], not unicode digits", + "schema": { + "type": "object", + "patternProperties": { + "^\\d+$": true + }, + "additionalProperties": false + }, + "tests": [ + { + "description": "ascii digits", + "data": { "42": "life, the universe, and everything" }, + "valid": true + }, + { + "description": "ascii non-digits", + "data": { "-%#": "spending the year dead for tax reasons" }, + "valid": false + }, + { + "description": "non-ascii digits (BENGALI DIGIT FOUR, BENGALI DIGIT TWO)", + "data": { "৪২": "khajit has wares if you have coin" }, + "valid": false + } + ] + }, + { + "description": "patternProperties with non-ASCII digits", + "schema": { + "type": "object", + "patternProperties": { + "^\\p{digit}+$": true + }, + "additionalProperties": false + }, + "tests": [ + { + "description": "ascii digits", + "data": { "42": "life, the universe, and everything" }, + "valid": true + }, + { + "description": "ascii non-digits", + "data": { "-%#": "spending the year dead for tax reasons" }, + "valid": false + }, + { + "description": "non-ascii digits (BENGALI DIGIT FOUR, BENGALI DIGIT TWO)", + "data": { "৪২": "khajit has wares if you have coin" }, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/float-overflow.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/float-overflow.json new file mode 100644 index 000000000..52ff9827c --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/float-overflow.json @@ -0,0 +1,13 @@ +[ + { + "description": "all integers are multiples of 0.5, if overflow is handled", + "schema": {"type": "integer", "multipleOf": 0.5}, + "tests": [ + { + "description": "valid if optional overflow handling is implemented", + "data": 1e308, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/date-time.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/date-time.json new file mode 100644 index 000000000..07a4e9195 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/date-time.json @@ -0,0 +1,247 @@ +[ + { + "description": "validation of date-time strings", + "schema": { "format": "date-time" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid date-time string", + "data": "1963-06-19T08:30:06.283185Z", + "valid": true + }, + { + "description": "a valid date-time string without second fraction", + "data": "1963-06-19T08:30:06Z", + "valid": true + }, + { + "description": "a valid date-time string with plus offset", + "data": "1937-01-01T12:00:27.87+00:20", + "valid": true + }, + { + "description": "a valid date-time string with minus offset", + "data": "1990-12-31T15:59:50.123-08:00", + "valid": true + }, + { + "description": "a valid date-time with a leap second, UTC", + "data": "1998-12-31T23:59:60Z", + "valid": true + }, + { + "description": "a valid date-time with a leap second, with minus offset", + "data": "1998-12-31T15:59:60.123-08:00", + "valid": true + }, + { + "description": "an invalid date-time past leap second, UTC", + "data": "1998-12-31T23:59:61Z", + "valid": false + }, + { + "description": "an invalid date-time with leap second on a wrong minute, UTC", + "data": "1998-12-31T23:58:60Z", + "valid": false + }, + { + "description": "an invalid date-time with leap second on a wrong hour, UTC", + "data": "1998-12-31T22:59:60Z", + "valid": false + }, + { + "description": "an invalid hour in date-time string", + "data": "1990-12-31T24:00:00Z", + "valid": false + }, + { + "description": "an invalid minute in date-time string", + "data": "1990-12-31T15:60:00Z", + "valid": false + }, + { + "description": "an invalid offset minute in date-time string", + "data": "1990-12-31T10:00:00+10:60", + "valid": false + }, + { + "description": "an invalid day in date-time string", + "data": "1990-02-31T15:59:59.123-08:00", + "valid": false + }, + { + "description": "an invalid offset in date-time string", + "data": "1990-12-31T15:59:59-24:00", + "valid": false + }, + { + "description": "an invalid closing Z after time-zone offset", + "data": "1963-06-19T08:30:06.28123+01:00Z", + "valid": false + }, + { + "description": "an invalid date-time string", + "data": "06/19/1963 08:30:06 PST", + "valid": false + }, + { + "description": "case-insensitive T and Z", + "data": "1963-06-19t08:30:06.283185z", + "valid": true + }, + { + "description": "only RFC3339 not all of ISO 8601 are valid", + "data": "2013-350T01:01:01", + "valid": false + }, + { + "description": "invalid non-padded month dates", + "data": "1963-6-19T08:30:06.283185Z", + "valid": false + }, + { + "description": "invalid non-padded day dates", + "data": "1963-06-1T08:30:06.283185Z", + "valid": false + }, + { + "description": "invalid non-ASCII '৪' (a Bengali 4) in date portion", + "data": "1963-06-1৪T00:00:00Z", + "valid": false + }, + { + "description": "invalid non-ASCII '৪' (a Bengali 4) in time portion", + "data": "1963-06-11T0৪:00:00Z", + "valid": false + }, + { + "description": "invalid extended year", + "data": "+11963-06-19T08:30:06.283185Z", + "valid": false + }, + { + "description": "a numeric offset without minutes is invalid", + "data": "1985-04-12T23:20:50+01", + "valid": false + }, + { + "description": "hour 24 is invalid even with a leap second", + "data": "2016-12-31T24:59:60+01:00", + "valid": false + }, + { + "description": "a second fraction of fifteen nines is valid", + "data": "1985-04-12T00:59:59.999999999999999Z", + "valid": true + }, + { + "description": "a trailing newline is invalid", + "data": "1985-04-12T23:20:50Z\n", + "valid": false + }, + { + "description": "an invalid date-time string without seconds", + "data": "1985-04-12T23:20Z", + "valid": false + }, + { + "description": "an invalid date-time string with trailing content after the offset", + "data": "1985-04-12T23:20:50Ztail", + "valid": false + }, + { + "description": "an invalid minute in date-time string, with a numeric offset", + "data": "1985-04-12T23:60:00+00:01", + "valid": false + }, + { + "description": "a valid date-time string with 28 days in February (normal)", + "data": "2021-02-28T00:00:00Z", + "valid": true + }, + { + "description": "an invalid date-time string with 30 days in February (leap)", + "data": "2020-02-30T00:00:00Z", + "valid": false + }, + { + "description": "2021 is not a leap year", + "data": "2021-02-29T00:00:00Z", + "valid": false + }, + { + "description": "2020 is a leap year", + "data": "2020-02-29T00:00:00Z", + "valid": true + }, + { + "description": "century year 0100 is not a leap year", + "data": "0100-02-29T00:00:00Z", + "valid": false + }, + { + "description": "century year 0400 is a leap year", + "data": "0400-02-29T00:00:00Z", + "valid": true + }, + { + "description": "century year 2100 is not a leap year", + "data": "2100-02-29T00:00:00Z", + "valid": false + }, + { + "description": "a second fraction with no digits", + "comment": "RFC 3339 section 5.6: time-secfrac = \".\" 1*DIGIT requires at least one digit", + "data": "1985-04-12T23:20:50.Z", + "valid": false + }, + { + "description": "a comma decimal separator with a time-offset", + "comment": "RFC 3339 section 5.6: the decimal separator in time-secfrac is \".\" - the ISO 8601 comma is not part of this profile", + "data": "1985-04-12T23:20:50,5Z", + "valid": false + }, + { + "description": "leading whitespace before a valid date-time", + "comment": "RFC 3339 section 5.6: no production of date-time emits whitespace", + "data": " 1985-04-12T23:20:50Z", + "valid": false + }, + { + "description": "trailing whitespace after a valid date-time", + "comment": "RFC 3339 section 5.6: no production of date-time emits whitespace", + "data": "1985-04-12T23:20:50Z ", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/date.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/date.json new file mode 100644 index 000000000..4a3847b4f --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/date.json @@ -0,0 +1,420 @@ +[ + { + "description": "validation of date strings", + "schema": { "format": "date" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid date string", + "data": "1963-06-19", + "valid": true + }, + { + "description": "a valid date string with 31 days in January", + "data": "2020-01-31", + "valid": true + }, + { + "description": "a invalid date string with 32 days in January", + "data": "2020-01-32", + "valid": false + }, + { + "description": "a valid date string with 28 days in February (normal)", + "data": "2021-02-28", + "valid": true + }, + { + "description": "a invalid date string with 30 days in February (leap)", + "data": "2020-02-30", + "valid": false + }, + { + "description": "a valid date string with 31 days in March", + "data": "2020-03-31", + "valid": true + }, + { + "description": "a invalid date string with 32 days in March", + "data": "2020-03-32", + "valid": false + }, + { + "description": "a valid date string with 30 days in April", + "data": "2020-04-30", + "valid": true + }, + { + "description": "a invalid date string with 31 days in April", + "data": "2020-04-31", + "valid": false + }, + { + "description": "a valid date string with 31 days in May", + "data": "2020-05-31", + "valid": true + }, + { + "description": "a invalid date string with 32 days in May", + "data": "2020-05-32", + "valid": false + }, + { + "description": "a valid date string with 30 days in June", + "data": "2020-06-30", + "valid": true + }, + { + "description": "a invalid date string with 31 days in June", + "data": "2020-06-31", + "valid": false + }, + { + "description": "a valid date string with 31 days in July", + "data": "2020-07-31", + "valid": true + }, + { + "description": "a invalid date string with 32 days in July", + "data": "2020-07-32", + "valid": false + }, + { + "description": "a valid date string with 31 days in August", + "data": "2020-08-31", + "valid": true + }, + { + "description": "a invalid date string with 32 days in August", + "data": "2020-08-32", + "valid": false + }, + { + "description": "a valid date string with 30 days in September", + "data": "2020-09-30", + "valid": true + }, + { + "description": "a invalid date string with 31 days in September", + "data": "2020-09-31", + "valid": false + }, + { + "description": "a valid date string with 31 days in October", + "data": "2020-10-31", + "valid": true + }, + { + "description": "a invalid date string with 32 days in October", + "data": "2020-10-32", + "valid": false + }, + { + "description": "a valid date string with 30 days in November", + "data": "2020-11-30", + "valid": true + }, + { + "description": "a invalid date string with 31 days in November", + "data": "2020-11-31", + "valid": false + }, + { + "description": "a valid date string with 31 days in December", + "data": "2020-12-31", + "valid": true + }, + { + "description": "a invalid date string with 32 days in December", + "data": "2020-12-32", + "valid": false + }, + { + "description": "an invalid date string", + "data": "06/19/1963", + "valid": false + }, + { + "description": "only RFC3339 not all of ISO 8601 are valid", + "data": "2013-350", + "valid": false + }, + { + "description": "non-padded month dates are not valid", + "data": "1998-1-20", + "valid": false + }, + { + "description": "non-padded day dates are not valid", + "data": "1998-01-1", + "valid": false + }, + { + "description": "invalid month", + "data": "1998-13-01", + "valid": false + }, + { + "description": "2021 is not a leap year", + "data": "2021-02-29", + "valid": false + }, + { + "description": "2020 is a leap year", + "data": "2020-02-29", + "valid": true + }, + { + "description": "invalid non-ASCII '৪' (a Bengali 4)", + "data": "1963-06-1৪", + "valid": false + }, + { + "description": "invalid: non-ASCII Bengali digit in month field", + "data": "2020-0৪-01", + "valid": false + }, + { + "description": "ISO8601 / non-RFC3339: YYYYMMDD without dashes (2023-03-28)", + "data": "20230328", + "valid": false + }, + { + "description": "ISO8601 / non-RFC3339: week number implicit day of week (2023-01-02)", + "data": "2023-W01", + "valid": false + }, + { + "description": "ISO8601 / non-RFC3339: week number with day of week (2023-03-28)", + "data": "2023-W13-2", + "valid": false + }, + { + "description": "ISO8601 / non-RFC3339: week number rollover to next year (2023-01-01)", + "data": "2022W527", + "valid": false + }, + { + "description": "an invalid time string in date-time format", + "data": "2020-11-28T23:55:45Z", + "valid": false + }, + { + "description": "century year 0100 is not a leap year", + "comment": "https://www.rfc-editor.org/rfc/rfc3339#appendix-C — 100 % 100 == 0 but 100 % 400 != 0", + "data": "0100-02-29", + "valid": false + }, + { + "description": "century year 0400 is a leap year", + "comment": "https://www.rfc-editor.org/rfc/rfc3339#appendix-C — 400-year cycle boundary", + "data": "0400-02-29", + "valid": true + }, + { + "description": "century year 2100 is not a leap year", + "comment": "https://www.rfc-editor.org/rfc/rfc3339#appendix-C — future century year", + "data": "2100-02-29", + "valid": false + }, + { + "description": "invalid: leading whitespace is not permitted", + "comment": "https://www.rfc-editor.org/rfc/rfc3339#section-5.6 — full-date grammar does not include whitespace", + "data": " 2024-01-15", + "valid": false + }, + { + "description": "invalid: trailing whitespace is not permitted", + "comment": "https://www.rfc-editor.org/rfc/rfc3339#section-5.6 — full-date grammar does not include whitespace", + "data": "2024-01-15 ", + "valid": false + }, + { + "description": "invalid: month 00 is not valid per date-month range 01-12", + "comment": "https://www.rfc-editor.org/rfc/rfc3339#section-5.6 — date-month = 2DIGIT ; 01-12", + "data": "2024-00-15", + "valid": false + }, + { + "description": "invalid: day 00 is not valid per date-mday minimum of 01", + "comment": "https://www.rfc-editor.org/rfc/rfc3339#section-5.6 — date-mday = 2DIGIT ; 01-28/29/30/31", + "data": "2024-01-00", + "valid": false + }, + { + "description": "invalid: empty string", + "data": "", + "valid": false + }, + { + "description": "invalid: embedded whitespace between year and month", + "data": "2020 -01-01", + "valid": false + }, + { + "description": "invalid: trailing character after valid full-date", + "data": "2020-01-01X", + "valid": false + }, + { + "description": "invalid: trailing Z after full-date", + "data": "2020-01-01Z", + "valid": false + }, + { + "description": "invalid: full-date followed by space and time component", + "data": "2020-01-01 00:00:00Z", + "valid": false + }, + { + "description": "valid: four-digit year 0001", + "data": "0001-01-01", + "valid": true + }, + { + "description": "invalid: two-digit year (N-2 digits)", + "data": "20-01-01", + "valid": false + }, + { + "description": "invalid: three-digit year (N-1 digits)", + "data": "998-01-01", + "valid": false + }, + { + "description": "invalid: five-digit year (N+1 digits)", + "data": "12020-01-01", + "valid": false + }, + { + "description": "invalid: positive sign prefix on year", + "data": "+2020-01-01", + "valid": false + }, + { + "description": "invalid: negative sign prefix on year", + "data": "-2020-01-01", + "valid": false + }, + { + "description": "invalid: non-ASCII Bengali digit in year field", + "data": "২020-01-01", + "valid": false + }, + { + "description": "invalid: alphabetic characters in year field", + "data": "YYYY-01-01", + "valid": false + }, + { + "description": "invalid: three-digit month (N+1 digits)", + "data": "2020-001-01", + "valid": false + }, + { + "description": "invalid: alphabetic characters in month field", + "data": "2020-MM-01", + "valid": false + }, + { + "description": "invalid: three-digit day (N+1 digits)", + "data": "2020-01-001", + "valid": false + }, + { + "description": "invalid: alphabetic characters in day field", + "data": "2020-01-DD", + "valid": false + }, + { + "description": "invalid: colon separators", + "data": "2020:01:01", + "valid": false + }, + { + "description": "invalid: dot separators", + "data": "2020.01.01", + "valid": false + }, + { + "description": "invalid: space separators", + "data": "2020 01 01", + "valid": false + }, + { + "description": "invalid: mixed slash and hyphen separators", + "data": "2020-01/01", + "valid": false + }, + { + "description": "invalid: duplicated first hyphen", + "data": "2020--01-01", + "valid": false + }, + { + "description": "invalid: duplicated second hyphen", + "data": "2020-01--01", + "valid": false + }, + { + "description": "valid: date inside the Julian to Gregorian reform gap", + "data": "1582-10-10", + "valid": true + }, + { + "description": "invalid: year field numerically larger than a 32-bit signed integer", + "data": "2147483648-01-01", + "valid": false + }, + { + "description": "invalid: character just above '9' as the second digit of the day", + "data": "2020-01-0:", + "valid": false + }, + { + "description": "invalid: non-ASCII digit leaving ten UTF-8 bytes but eight characters", + "data": "\u09e80-01-01", + "valid": false + }, + { + "description": "invalid: en dash separators instead of hyphen-minus", + "data": "2020\u201301\u201301", + "valid": false + }, + { + "description": "invalid: NUL character after an otherwise valid full-date", + "data": "2020-01-01\u0000", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/ecmascript-regex.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/ecmascript-regex.json new file mode 100644 index 000000000..bf3fae7f2 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/ecmascript-regex.json @@ -0,0 +1,98 @@ +[ + { + "description": "\\a is not an ECMA 262 control escape", + "schema": { "format": "regex" }, + "tests": [ + { + "description": "when used as a pattern", + "data": "\\a", + "valid": false + } + ] + }, + { + "description": "Python-specific regular expression syntax is not valid ECMA 262", + "schema": { "format": "regex" }, + "tests": [ + { + "description": "Python named group (?P...) is not ECMA 262", + "data": "(?Px)", + "valid": false + }, + { + "description": "Python named backreference (?P=name) is not ECMA 262", + "data": "(?Pa)(?P=n)", + "valid": false + }, + { + "description": "inline comment group (?#...) is not ECMA 262", + "data": "(?#comment)a", + "valid": false + } + ] + }, + { + "description": "global inline flag groups are not valid ECMA 262", + "schema": { "format": "regex" }, + "tests": [ + { + "description": "a single global inline flag (?i)", + "data": "(?i)abc", + "valid": false + }, + { + "description": "multiple global inline flags (?ims)", + "data": "(?ims)abc", + "valid": false + } + ] + }, + { + "description": "ECMA 262 named groups and backreferences are valid", + "schema": { "format": "regex" }, + "tests": [ + { + "description": "an ECMA 262 named group (?...)", + "data": "(?x)", + "valid": true + }, + { + "description": "an ECMA 262 named backreference \\k", + "data": "(?a)\\k", + "valid": true + } + ] + }, + { + "description": "ECMA 262 lookbehind is valid, including variable width", + "schema": { "format": "regex" }, + "tests": [ + { + "description": "a variable-width lookbehind (ES2018)", + "data": "(?<=a+)b", + "valid": true + } + ] + }, + { + "description": "ECMA 262 character classes and escapes", + "schema": { "format": "regex" }, + "tests": [ + { + "description": "an empty character class is valid ECMA 262", + "data": "[]", + "valid": true + }, + { + "description": "a negated empty character class is valid ECMA 262", + "data": "[^]", + "valid": true + }, + { + "description": "a control escape \\cA is valid ECMA 262", + "data": "\\cA", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/email.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/email.json new file mode 100644 index 000000000..6ef76f3c6 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/email.json @@ -0,0 +1,365 @@ +[ + { + "description": "validation of e-mail addresses", + "schema": { "format": "email" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid e-mail address", + "data": "joe.bloggs@example.com", + "valid": true + }, + { + "description": "an invalid e-mail address", + "data": "2962", + "valid": false + }, + { + "description": "tilde in local part is valid", + "data": "te~st@example.com", + "valid": true + }, + { + "description": "tilde before local part is valid", + "data": "~test@example.com", + "valid": true + }, + { + "description": "tilde after local part is valid", + "data": "test~@example.com", + "valid": true + }, + { + "description": "a quoted string with a space in the local part is valid", + "data": "\"joe bloggs\"@example.com", + "valid": true + }, + { + "description": "a quoted string with a double dot in the local part is valid", + "data": "\"joe..bloggs\"@example.com", + "valid": true + }, + { + "description": "a quoted string with a @ in the local part is valid", + "data": "\"joe@bloggs\"@example.com", + "valid": true + }, + { + "description": "an IPv4-address-literal after the @ is valid", + "data": "joe.bloggs@[127.0.0.1]", + "valid": true + }, + { + "description": "an IPv6-address-literal after the @ is valid", + "data": "joe.bloggs@[IPv6:::1]", + "valid": true + }, + { + "description": "dot before local part is not valid", + "data": ".test@example.com", + "valid": false + }, + { + "description": "dot after local part is not valid", + "data": "test.@example.com", + "valid": false + }, + { + "description": "two separated dots inside local part are valid", + "data": "te.s.t@example.com", + "valid": true + }, + { + "description": "two subsequent dots inside local part are not valid", + "data": "te..st@example.com", + "valid": false + }, + { + "description": "an invalid domain", + "data": "joe.bloggs@invalid=domain.com", + "valid": false + }, + { + "description": "an invalid IPv4-address-literal", + "data": "joe.bloggs@[127.0.0.300]", + "valid": false + }, + { + "description": "two email addresses is not valid", + "data": "user1@oceania.org, user2@oceania.org", + "valid": false + }, + { + "description": "full \"From\" header is invalid", + "data": "\"Winston Smith\" (Records Department)", + "valid": false + }, + { + "description": "local part is required", + "data": "@example.com", + "valid": false + }, + { + "description": "domain is required", + "data": "joe.bloggs@", + "valid": false + }, + { + "description": "unquoted space in local part is invalid", + "data": "joe bloggs@example.com", + "valid": false + }, + { + "description": "a domain that looks like an IPv4 address without brackets is valid", + "data": "test@255.255.255.255", + "valid": true + }, + { + "description": "an empty quoted string in the local part is valid", + "data": "\"\"@iana.org", + "valid": true + }, + { + "description": "a quoted string containing only a space in the local part is valid", + "data": "\" \"@iana.org", + "valid": true + }, + { + "description": "consecutive hyphens inside a domain label are valid", + "data": "test@c--n.com", + "valid": true + }, + { + "description": "a domain label starting with a digit is valid", + "data": "test@123.com", + "valid": true + }, + { + "description": "a non-ASCII character in the local part is not valid", + "data": "a\u00e9@iana.org", + "valid": false + }, + { + "description": "a non-ASCII character in the domain is not valid", + "data": "a@\u00e9.org", + "valid": false + }, + { + "description": "a fullwidth commercial at is not a valid separator", + "data": "a\uff20iana.org", + "valid": false + }, + { + "description": "a non-ASCII character in a quoted pair is not valid", + "data": "\"test\\\u00a9\"@iana.org", + "valid": false + }, + { + "description": "a trailing line feed is not valid", + "data": "test@iana.org\n", + "valid": false + }, + { + "description": "a trailing carriage return is not valid", + "data": "test@iana.org\r", + "valid": false + }, + { + "description": "a trailing space is not valid", + "data": "a@iana.org ", + "valid": false + }, + { + "description": "a leading space is not valid", + "data": " a@iana.org", + "valid": false + }, + { + "description": "a tab in the local part is not valid", + "data": "a\tb@iana.org", + "valid": false + }, + { + "description": "a delete character in the local part is not valid", + "data": "\u007f@iana.org", + "valid": false + }, + { + "description": "a quoted string with no special characters in the local part is valid", + "data": "\"test\"@iana.org", + "valid": true + }, + { + "description": "a quoted pair in the local part is valid", + "data": "\"\\a\"@iana.org", + "valid": true + }, + { + "description": "an escaped double quote in the local part is valid", + "data": "\"\\\"\"@iana.org", + "valid": true + }, + { + "description": "an escaped backslash in the local part is valid", + "data": "\"\\\\\"@iana.org", + "valid": true + }, + { + "description": "an unescaped double quote in an unquoted local part is not valid", + "data": "test\"@iana.org", + "valid": false + }, + { + "description": "text after the closing double quote in the local part is not valid", + "data": "\"test\"test@iana.org", + "valid": false + }, + { + "description": "an unclosed double quote in the local part is not valid", + "data": "\"test@iana.org", + "valid": false + }, + { + "description": "two unquoted @ signs are not valid", + "data": "a@b@c.org", + "valid": false + }, + { + "description": "a backslash-escaped @ in an unquoted local part is not valid", + "data": "test\\@test@iana.org", + "valid": false + }, + { + "description": "an address literal in the local part position is not valid", + "data": "[1.2.3.4]@iana.org", + "valid": false + }, + { + "description": "an empty string is not valid", + "data": "", + "valid": false + }, + { + "description": "a domain label starting with a hyphen is not valid", + "data": "test@-iana.org", + "valid": false + }, + { + "description": "a domain label ending with a hyphen is not valid", + "data": "test@iana-.com", + "valid": false + }, + { + "description": "a domain starting with a dot is not valid", + "data": "test@.iana.org", + "valid": false + }, + { + "description": "a domain ending with a dot is not valid", + "data": "test@iana.org.", + "valid": false + }, + { + "description": "two subsequent dots inside the domain are not valid", + "data": "test@iana..com", + "valid": false + }, + { + "description": "a single-label domain is valid", + "data": "test@io", + "valid": true + }, + { + "description": "an unclosed address literal is not valid", + "data": "test@[1.2.3.4", + "valid": false + }, + { + "description": "empty brackets after the @ are not valid", + "data": "a@[]", + "valid": false + }, + { + "description": "an IPv4-address-literal with three octets is not valid", + "data": "a@[1.2.3]", + "valid": false + }, + { + "description": "an IPv4-address-literal with five octets is not valid", + "data": "a@[1.2.3.4.5]", + "valid": false + }, + { + "description": "bracketed content that is not an address is not valid", + "data": "test@[RFC-5322-domain-literal]", + "valid": false + }, + { + "description": "a domain label before the opening bracket is not valid", + "data": "test@a[255.255.255.255]", + "valid": false + }, + { + "description": "a lowercase IPv6 tag in an address literal is valid", + "data": "a@[ipv6:::1]", + "valid": true + }, + { + "description": "a comma in an unquoted local part is not valid", + "data": "a,b@iana.org", + "valid": false + }, + { + "description": "a semicolon in an unquoted local part is not valid", + "data": "a;b@iana.org", + "valid": false + }, + { + "description": "a colon in an unquoted local part is not valid", + "data": "a:b@iana.org", + "valid": false + }, + { + "description": "a parenthesis in an unquoted local part is not valid", + "comment": "a comment is RFC 5322 syntax, which the RFC 5321 Mailbox grammar does not have", + "data": "test(comment)@iana.org", + "valid": false + }, + { + "description": "a parenthesis in the domain is not valid", + "comment": "a comment is RFC 5322 syntax, which the RFC 5321 Mailbox grammar does not have", + "data": "a@b(c).org", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/hostname.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/hostname.json new file mode 100644 index 000000000..bd5dcdea2 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/hostname.json @@ -0,0 +1,380 @@ +[ + { + "description": "validation of host names", + "schema": { "format": "hostname" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid host name", + "data": "www.example.com", + "valid": true + }, + { + "description": "single label", + "data": "hostname", + "valid": true + }, + { + "description": "single label with digits", + "data": "h0stn4me", + "valid": true + }, + { + "description": "single label starting with digit", + "data": "1host", + "valid": true + }, + { + "description": "single label ending with digit", + "data": "hostnam3", + "valid": true + }, + { + "description": "empty string", + "data": "", + "valid": false + }, + { + "description": "single dot", + "data": ".", + "valid": false + }, + { + "description": "leading dot", + "data": ".example", + "valid": false + }, + { + "description": "trailing dot", + "data": "example.", + "valid": false + }, + { + "description": "IDN label separator", + "data": "example\uff0ecom", + "valid": false + }, + { + "description": "single label with hyphen", + "data": "host-name", + "valid": true + }, + { + "description": "starts with hyphen", + "data": "-hostname", + "valid": false + }, + { + "description": "ends with hyphen", + "data": "hostname-", + "valid": false + }, + { + "description": "contains underscore", + "data": "host_name", + "valid": false + }, + { + "description": "exceeds maximum overall length (256)", + "data": "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk.abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk.abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk.abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk.com", + "valid": false + }, + { + "description": "maximum label length (63)", + "data": "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk.com", + "valid": true + }, + { + "description": "exceeds maximum label length (63)", + "data": "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijkl.com", + "valid": false + }, + { + "description": "trailing newline is invalid", + "data": "example.com\n", + "valid": false + }, + { + "description": "leading space is invalid", + "data": " example.com", + "valid": false + }, + { + "description": "space inside a label is invalid", + "data": "ex ample.com", + "valid": false + }, + { + "description": "invalid non-ASCII KELVIN SIGN (U+212A)", + "data": "\u212Aelvin.example.com", + "valid": false + }, + { + "description": "consecutive hyphens inside a label", + "data": "a--b.com", + "valid": true + } + ] + }, + { + "description": "validation of A-label (punycode) host names", + "schema": { "format": "hostname" }, + "tests": [ + { + "description": "invalid Punycode", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.4 https://tools.ietf.org/html/rfc5890#section-2.3.2.1", + "data": "xn--X", + "valid": false + }, + { + "description": "a valid host name (example.test in Hangul)", + "data": "xn--9n2bp8q.xn--9t4b11yi5a", + "valid": true + }, + { + "description": "contains illegal char U+302E Hangul single dot tone mark", + "data": "xn--07jt112bpxg.xn--9t4b11yi5a", + "valid": false + }, + { + "description": "Begins with a Spacing Combining Mark", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.2", + "data": "xn--hello-txk", + "valid": false + }, + { + "description": "Begins with a Nonspacing Mark", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.2", + "data": "xn--hello-zed", + "valid": false + }, + { + "description": "Begins with an Enclosing Mark", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.2", + "data": "xn--hello-6bf", + "valid": false + }, + { + "description": "Exceptions that are PVALID, left-to-right chars", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.2 https://tools.ietf.org/html/rfc5892#section-2.6", + "data": "xn--zca29lwxobi7a", + "valid": true + }, + { + "description": "Exceptions that are PVALID, right-to-left chars", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.2 https://tools.ietf.org/html/rfc5892#section-2.6", + "data": "xn--qmbc", + "valid": true + }, + { + "description": "Exceptions that are DISALLOWED, right-to-left chars", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.2 https://tools.ietf.org/html/rfc5892#section-2.6", + "data": "xn--chb89f", + "valid": false + }, + { + "description": "Exceptions that are DISALLOWED, left-to-right chars", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.2 https://tools.ietf.org/html/rfc5892#section-2.6 Note: The two combining marks (U+302E and U+302F) are in the middle and not at the start", + "data": "xn--07jceefgh4c", + "valid": false + }, + { + "description": "MIDDLE DOT with no preceding 'l'", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "xn--al-0ea", + "valid": false + }, + { + "description": "MIDDLE DOT with nothing preceding", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "xn--l-fda", + "valid": false + }, + { + "description": "MIDDLE DOT with no following 'l'", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "xn--la-0ea", + "valid": false + }, + { + "description": "MIDDLE DOT with nothing following", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "xn--l-gda", + "valid": false + }, + { + "description": "MIDDLE DOT with surrounding 'l's", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "xn--ll-0ea", + "valid": true + }, + { + "description": "Greek KERAIA not followed by Greek", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.4", + "data": "xn--S-jib3p", + "valid": false + }, + { + "description": "Greek KERAIA not followed by anything", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.4", + "data": "xn--wva3j", + "valid": false + }, + { + "description": "Greek KERAIA followed by Greek", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.4", + "data": "xn--wva3je", + "valid": true + }, + { + "description": "Hebrew GERESH not preceded by Hebrew", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.5", + "data": "xn--A-2hc5h", + "valid": false + }, + { + "description": "Hebrew GERESH not preceded by anything", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.5", + "data": "xn--5db1e", + "valid": false + }, + { + "description": "Hebrew GERESH preceded by Hebrew", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.5", + "data": "xn--4dbc5h", + "valid": true + }, + { + "description": "Hebrew GERSHAYIM not preceded by Hebrew", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.6", + "data": "xn--A-2hc8h", + "valid": false + }, + { + "description": "Hebrew GERSHAYIM not preceded by anything", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.6", + "data": "xn--5db3e", + "valid": false + }, + { + "description": "Hebrew GERSHAYIM preceded by Hebrew", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.6", + "data": "xn--4dbc8h", + "valid": true + }, + { + "description": "KATAKANA MIDDLE DOT with no Hiragana, Katakana, or Han", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "xn--defabc-k64e", + "valid": false + }, + { + "description": "KATAKANA MIDDLE DOT with no other characters", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "xn--vek", + "valid": false + }, + { + "description": "KATAKANA MIDDLE DOT with Hiragana", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "xn--k8j5u", + "valid": true + }, + { + "description": "KATAKANA MIDDLE DOT with Katakana", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "xn--bck0j", + "valid": true + }, + { + "description": "KATAKANA MIDDLE DOT with Han", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "xn--vek778f", + "valid": true + }, + { + "description": "Arabic-Indic digits mixed with Extended Arabic-Indic digits", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.8", + "data": "xn--ngb6iyr", + "valid": false + }, + { + "description": "Arabic-Indic digits not mixed with Extended Arabic-Indic digits", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.8", + "data": "xn--ngba1o", + "valid": true + }, + { + "description": "Extended Arabic-Indic digits not mixed with Arabic-Indic digits", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.9", + "data": "xn--0-gyc", + "valid": true + }, + { + "description": "ZERO WIDTH JOINER not preceded by Virama", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.2 https://www.unicode.org/review/pr-37.pdf", + "data": "xn--11b2er09f", + "valid": false + }, + { + "description": "ZERO WIDTH JOINER not preceded by anything", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.2 https://www.unicode.org/review/pr-37.pdf", + "data": "xn--02b508i", + "valid": false + }, + { + "description": "ZERO WIDTH JOINER preceded by Virama", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.2 https://www.unicode.org/review/pr-37.pdf", + "data": "xn--11b2ezcw70k", + "valid": true + }, + { + "description": "ZERO WIDTH NON-JOINER preceded by Virama", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.1", + "data": "xn--11b2ezcs70k", + "valid": true + }, + { + "description": "ZERO WIDTH NON-JOINER not preceded by Virama but matches regexp", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.1 https://www.w3.org/TR/alreq/#h_disjoining_enforcement", + "data": "xn--ngba5hb2804a", + "valid": true + }, + { + "description": "contains \"--\" in the 3rd and 4th position", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.1 https://tools.ietf.org/html/rfc5890#section-2.3.2.1", + "data": "XN--aa---o47jg78q", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/idn-email.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/idn-email.json new file mode 100644 index 000000000..c88bd9fc5 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/idn-email.json @@ -0,0 +1,137 @@ +[ + { + "description": "validation of an internationalized e-mail addresses", + "schema": { "format": "idn-email" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid idn e-mail (example@example.test in Hangul)", + "data": "실례@실례.테스트", + "valid": true + }, + { + "description": "an invalid e-mail/idn e-mail address", + "data": "2962", + "valid": false + }, + { + "description": "a valid e-mail address", + "data": "joe.bloggs@example.com", + "valid": true + }, + { + "description": "a non-ASCII local part with an ASCII domain is valid", + "data": "δοκιμή@example.com", + "valid": true + }, + { + "description": "a non-ASCII quoted local part is valid", + "data": "\"δοκιμή\"@example.com", + "valid": true + }, + { + "description": "a domain label that is not in Unicode NFC is valid", + "data": "user@cafe\u0301.com", + "valid": true + }, + { + "description": "a local part that is not in Unicode NFC is valid", + "data": "cafe\u0301@example.com", + "valid": true + }, + { + "description": "a C1 control in the local part is valid", + "comment": "https://www.rfc-editor.org/rfc/rfc6531#section-3.3 atext =/ UTF8-non-ascii; https://www.rfc-editor.org/rfc/rfc3629#section-4 UTF8-2 admits U+0085", + "data": "\u0085@example.com", + "valid": true + }, + { + "description": "a noncharacter in the local part is valid", + "comment": "https://www.rfc-editor.org/rfc/rfc6531#section-3.3 atext =/ UTF8-non-ascii; https://www.rfc-editor.org/rfc/rfc3629#section-4 UTF8-3 admits U+FFFF", + "data": "\uffff@example.com", + "valid": true + }, + { + "description": "a local part at the 64-octet limit is valid", + "comment": "https://www.rfc-editor.org/rfc/rfc5321#section-4.5.3.1 every implementation must be able to receive a local part of 64 octets", + "data": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa@example.com", + "valid": true + }, + { + "description": "a fullwidth commercial at is not a local-part separator", + "comment": "https://www.rfc-editor.org/rfc/rfc5321#section-4.1.2 Mailbox = Local-part \"@\" ( Domain / address-literal ) splits on U+0040, and any IDN mapping applies only to the domain that split produces, so U+FF20 cannot become the delimiter", + "data": "user\uff20example.com", + "valid": false + }, + { + "description": "a local part with a supplementary-plane character is valid", + "comment": "https://www.rfc-editor.org/rfc/rfc6531#section-3.3 atext =/ UTF8-non-ascii, and https://www.rfc-editor.org/rfc/rfc3629#section-4 UTF8-4 covers U+10000 and above", + "data": "\ud835\udd4f@example.com", + "valid": true + }, + { + "description": "a domain label starting with a hyphen is not valid", + "comment": "https://www.rfc-editor.org/rfc/rfc5321#section-4.1.2 sub-domain = Let-dig [Ldh-str], so a label starts with a letter or digit; RFC 6531 section 3.3 only adds sub-domain =/ U-label", + "data": "δοκιμή@-example.com", + "valid": false + }, + { + "description": "a domain label ending with a hyphen is not valid", + "comment": "https://www.rfc-editor.org/rfc/rfc5321#section-4.1.2 Ldh-str = *( ALPHA / DIGIT / \"-\" ) Let-dig, so a label ends with a letter or digit", + "data": "δοκιμή@example-.com", + "valid": false + }, + { + "description": "a domain starting with a dot is not valid", + "comment": "https://www.rfc-editor.org/rfc/rfc5321#section-4.1.2 Domain = sub-domain *(\".\" sub-domain) admits no empty sub-domain", + "data": "δοκιμή@.example.com", + "valid": false + }, + { + "description": "a domain ending with a dot is not valid", + "data": "δοκιμή@example.com.", + "valid": false + }, + { + "description": "two subsequent dots inside the domain are not valid", + "data": "δοκιμή@example..com", + "valid": false + }, + { + "description": "domain is required", + "comment": "https://www.rfc-editor.org/rfc/rfc5321#section-4.1.2 Mailbox = Local-part \"@\" ( Domain / address-literal )", + "data": "δοκιμή@", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/idn-hostname.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/idn-hostname.json new file mode 100644 index 000000000..13d223d24 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/idn-hostname.json @@ -0,0 +1,582 @@ +[ + { + "description": "validation of internationalized host names", + "schema": { "format": "idn-hostname" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid host name (example.test in Hangul)", + "data": "실례.테스트", + "valid": true + }, + { + "description": "illegal first char U+302E Hangul single dot tone mark", + "data": "〮실례.테스트", + "valid": false + }, + { + "description": "contains illegal char U+302E Hangul single dot tone mark", + "data": "실〮례.테스트", + "valid": false + }, + { + "description": "a single label of 63 characters is valid", + "data": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "valid": true + }, + { + "description": "a single label of 64 characters is too long", + "data": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "valid": false + }, + { + "description": "invalid label, correct Punycode", + "comment": "https://tools.ietf.org/html/rfc5890#section-2.3.2.1 https://tools.ietf.org/html/rfc5891#section-4.4 https://tools.ietf.org/html/rfc3492#section-7.1", + "data": "-> $1.00 <--", + "valid": false + }, + { + "description": "valid Chinese Punycode", + "comment": "https://tools.ietf.org/html/rfc5890#section-2.3.2.1 https://tools.ietf.org/html/rfc5891#section-4.4", + "data": "xn--ihqwcrb4cv8a8dqg056pqjye", + "valid": true + }, + { + "description": "invalid Punycode", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.4 https://tools.ietf.org/html/rfc5890#section-2.3.2.1", + "data": "xn--X", + "valid": false + }, + { + "description": "U-label contains \"--\" in the 3rd and 4th position", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.1 https://tools.ietf.org/html/rfc5890#section-2.3.2.1", + "data": "XN--aa---o47jg78q", + "valid": false + }, + { + "description": "U-label starts with a dash", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.1", + "data": "-hello", + "valid": false + }, + { + "description": "U-label ends with a dash", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.1", + "data": "hello-", + "valid": false + }, + { + "description": "U-label starts and ends with a dash", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.1", + "data": "-hello-", + "valid": false + }, + { + "description": "Begins with a Spacing Combining Mark", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.2", + "data": "\u0903hello", + "valid": false + }, + { + "description": "Begins with a Nonspacing Mark", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.2", + "data": "\u0300hello", + "valid": false + }, + { + "description": "Begins with an Enclosing Mark", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.2", + "data": "\u0488hello", + "valid": false + }, + { + "description": "Exceptions that are PVALID, left-to-right chars", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.2 https://tools.ietf.org/html/rfc5892#section-2.6", + "data": "\u00df\u03c2\u0f0b\u3007", + "valid": true + }, + { + "description": "Exceptions that are PVALID, right-to-left chars", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.2 https://tools.ietf.org/html/rfc5892#section-2.6", + "data": "\u06fd\u06fe", + "valid": true + }, + { + "description": "Exceptions that are DISALLOWED, right-to-left chars", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.2 https://tools.ietf.org/html/rfc5892#section-2.6", + "data": "\u0640\u07fa", + "valid": false + }, + { + "description": "Exceptions that are DISALLOWED, left-to-right chars", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.2 https://tools.ietf.org/html/rfc5892#section-2.6 Note: The two combining marks (U+302E and U+302F) are in the middle and not at the start", + "data": "\u3031\u3032\u3033\u3034\u3035\u302e\u302f\u303b", + "valid": false + }, + { + "description": "MIDDLE DOT with no preceding 'l'", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "a\u00b7l", + "valid": false + }, + { + "description": "MIDDLE DOT with nothing preceding", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "\u00b7l", + "valid": false + }, + { + "description": "MIDDLE DOT with no following 'l'", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "l\u00b7a", + "valid": false + }, + { + "description": "MIDDLE DOT with nothing following", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "l\u00b7", + "valid": false + }, + { + "description": "MIDDLE DOT with surrounding 'l's", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.3", + "data": "l\u00b7l", + "valid": true + }, + { + "description": "Greek KERAIA not followed by Greek", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.4", + "data": "\u03b1\u0375S", + "valid": false + }, + { + "description": "Greek KERAIA not followed by anything", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.4", + "data": "\u03b1\u0375", + "valid": false + }, + { + "description": "Greek KERAIA followed by Greek", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.4", + "data": "\u03b1\u0375\u03b2", + "valid": true + }, + { + "description": "Hebrew GERESH not preceded by Hebrew", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.5", + "data": "A\u05f3\u05d1", + "valid": false + }, + { + "description": "Hebrew GERESH not preceded by anything", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.5", + "data": "\u05f3\u05d1", + "valid": false + }, + { + "description": "Hebrew GERESH preceded by Hebrew", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.5", + "data": "\u05d0\u05f3\u05d1", + "valid": true + }, + { + "description": "Hebrew GERSHAYIM not preceded by Hebrew", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.6", + "data": "A\u05f4\u05d1", + "valid": false + }, + { + "description": "Hebrew GERSHAYIM not preceded by anything", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.6", + "data": "\u05f4\u05d1", + "valid": false + }, + { + "description": "Hebrew GERSHAYIM preceded by Hebrew", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.6", + "data": "\u05d0\u05f4\u05d1", + "valid": true + }, + { + "description": "KATAKANA MIDDLE DOT with no Hiragana, Katakana, or Han", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "def\u30fbabc", + "valid": false + }, + { + "description": "KATAKANA MIDDLE DOT with no other characters", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "\u30fb", + "valid": false + }, + { + "description": "KATAKANA MIDDLE DOT with Hiragana", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "\u30fb\u3041", + "valid": true + }, + { + "description": "KATAKANA MIDDLE DOT with Katakana", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "\u30fb\u30a1", + "valid": true + }, + { + "description": "KATAKANA MIDDLE DOT with Han", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.7", + "data": "\u30fb\u4e08", + "valid": true + }, + { + "description": "Arabic-Indic digits mixed with Extended Arabic-Indic digits", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.8", + "data": "\u0628\u0660\u06f0", + "valid": false + }, + { + "description": "Arabic-Indic digits not mixed with Extended Arabic-Indic digits", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.8", + "data": "\u0628\u0660\u0628", + "valid": true + }, + { + "description": "Extended Arabic-Indic digits not mixed with Arabic-Indic digits", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.9", + "data": "\u06f00", + "valid": true + }, + { + "description": "ZERO WIDTH JOINER not preceded by Virama", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.2 https://www.unicode.org/review/pr-37.pdf", + "data": "\u0915\u200d\u0937", + "valid": false + }, + { + "description": "ZERO WIDTH JOINER not preceded by anything", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.2 https://www.unicode.org/review/pr-37.pdf", + "data": "\u200d\u0937", + "valid": false + }, + { + "description": "ZERO WIDTH JOINER preceded by Virama", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.2 https://www.unicode.org/review/pr-37.pdf", + "data": "\u0915\u094d\u200d\u0937", + "valid": true + }, + { + "description": "ZERO WIDTH NON-JOINER preceded by Virama", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.1", + "data": "\u0915\u094d\u200c\u0937", + "valid": true + }, + { + "description": "ZERO WIDTH NON-JOINER not preceded by Virama but matches regexp", + "comment": "https://tools.ietf.org/html/rfc5891#section-4.2.3.3 https://tools.ietf.org/html/rfc5892#appendix-A.1 https://www.w3.org/TR/alreq/#h_disjoining_enforcement", + "data": "\u0628\u064a\u200c\u0628\u064a", + "valid": true + }, + { + "description": "single label", + "data": "hostname", + "valid": true + }, + { + "description": "single label with hyphen", + "data": "host-name", + "valid": true + }, + { + "description": "single label with digits", + "data": "h0stn4me", + "valid": true + }, + { + "description": "single label starting with digit", + "data": "1host", + "valid": true + }, + { + "description": "single label ending with digit", + "data": "hostnam3", + "valid": true + }, + { + "description": "empty string", + "data": "", + "valid": false + }, + { + "description": "trailing newline is invalid", + "data": "example.com\n", + "valid": false + }, + { + "description": "leading space is invalid", + "data": " example.com", + "valid": false + }, + { + "description": "space inside a label is invalid", + "data": "ex ample.com", + "valid": false + }, + { + "description": "fullwidth digits are mapped to ASCII digits", + "comment": "IdnaMappingTable.txt maps U+FF11 to 0031, so this label maps to 123", + "data": "\uff11\uff12\uff13", + "valid": true + }, + { + "description": "a zero width space is ignored by the mapping", + "comment": "IdnaMappingTable.txt lists U+200B as ignored, so this label maps to ab", + "data": "a\u200bb", + "valid": true + }, + { + "description": "an A-label with an uppercase Punycode body is valid", + "comment": "https://www.rfc-editor.org/rfc/rfc5891#section-5.3 lowercases the A-label before the round-trip test, and RFC 3492 section 5 gives A-Z and a-z the same digit-values", + "data": "xn--NXASMQ6B", + "valid": true + }, + { + "description": "the two Arabic-Indic digit blocks may appear in different labels", + "comment": "https://www.rfc-editor.org/rfc/rfc5892#appendix-A.8 scopes the rule to \"this label\", so one block per label is valid", + "data": "\u0628\u0660.\u0628\u06f0", + "valid": true + }, + { + "description": "a label of only Arabic-Indic digits is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5893#section-2 condition 1 allows only L, R or AL as the first character, and an Arabic-Indic digit is AN", + "data": "\u0660\u0661", + "valid": false + }, + { + "description": "a non-NFC U-label is valid after UTS 46 normalisation", + "comment": "UTS 46 section 4 normalises to NFC before validating, so the decomposed form is accepted", + "data": "cafe\u0301.com", + "valid": true + }, + { + "description": "a label of 63 fullwidth characters is valid after mapping", + "comment": "UTS 46 maps each U+FF41 to an ASCII letter, so this label is 63 octets in its mapped form", + "data": "\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41\uff41", + "valid": true + }, + { + "description": "an ACE prefix produced by the mapping step is decoded as an A-label", + "comment": "UTS 46 maps U+FF58 U+FF4E to \"xn\", so the mapped label is the A-label xn--nxasmq6b", + "data": "\uff58\uff4e--nxasmq6b", + "valid": true + }, + { + "description": "zero width non-joiner must pass at every occurrence", + "comment": "https://www.rfc-editor.org/rfc/rfc5892#appendix-A.1", + "data": "\u0915\u094d\u200c\u0937x\u200cy", + "valid": false + }, + { + "description": "Bidi domain name with a digit-first label is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5893#section-2 a label in a Bidi domain name must start with an L, R or AL character", + "data": "0a.\u05d0", + "valid": false + }, + { + "description": "label starting with a digit before a right-to-left letter is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5893#section-2", + "data": "0\u0627", + "valid": false + }, + { + "description": "left-to-right label containing a right-to-left letter is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5893#section-2", + "data": "a\u05d0", + "valid": false + }, + { + "description": "right-to-left label mixing both digit types is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5893#section-2", + "data": "\u05d00\u0660", + "valid": false + }, + { + "description": "A-label that decodes to a disallowed code point is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5890#section-2.3.2.1 https://www.rfc-editor.org/rfc/rfc5892#section-2.6", + "data": "xn--7a", + "valid": false + }, + { + "description": "A-label that decodes to a Bidi rule violation is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5890#section-2.3.2.1 https://www.rfc-editor.org/rfc/rfc5893", + "data": "xn--0ca24w", + "valid": false + }, + { + "description": "a U-label whose A-label form is longer than 63 octets is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5891#section-4.2.4 the 63-octet limit is on the A-label form, not the code point count", + "data": "\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc\u00fc", + "valid": false + }, + { + "description": "empty label between two dots is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc1034#section-3.1", + "data": "a..b", + "valid": false + }, + { + "description": "a name longer than 253 characters is invalid", + "data": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "valid": false + }, + { + "description": "A-label that decodes to only ASCII is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5890#section-2.3.2.1 a U-label must have at least one non-ASCII character", + "data": "xn--example-", + "valid": false + }, + { + "description": "non-canonical Punycode that does not re-encode to itself is invalid", + "comment": "https://www.rfc-editor.org/rfc/rfc5891#section-5.4 a label decoded from Punycode must be identical to the original A-label", + "data": "xn---9uc", + "valid": false + } + ] + }, + { + "description": "validation of separators in internationalized host names", + "specification": [ + {"rfc3490": "3.1", "quote": "Whenever dots are used as label separators, the following characters MUST be recognized as dots: U+002E (full stop), U+3002 (ideographic full stop), U+FF0E (fullwidth full stop), U+FF61(halfwidth ideographic full stop)"} + ], + "schema": { "format": "idn-hostname" }, + "tests": [ + { + "description": "single dot", + "data": ".", + "valid": false + }, + { + "description": "single ideographic full stop", + "data": "\u3002", + "valid": false + }, + { + "description": "single fullwidth full stop", + "data": "\uff0e", + "valid": false + }, + { + "description": "single halfwidth ideographic full stop", + "data": "\uff61", + "valid": false + }, + { + "description": "dot as label separator", + "data": "a.b", + "valid": true + }, + { + "description": "ideographic full stop as label separator", + "data": "a\u3002b", + "valid": true + }, + { + "description": "fullwidth full stop as label separator", + "data": "a\uff0eb", + "valid": true + }, + { + "description": "halfwidth ideographic full stop as label separator", + "data": "a\uff61b", + "valid": true + }, + { + "description": "leading dot", + "data": ".example", + "valid": false + }, + { + "description": "leading ideographic full stop", + "data": "\u3002example", + "valid": false + }, + { + "description": "leading fullwidth full stop", + "data": "\uff0eexample", + "valid": false + }, + { + "description": "leading halfwidth ideographic full stop", + "data": "\uff61example", + "valid": false + }, + { + "description": "trailing dot", + "data": "example.", + "valid": false + }, + { + "description": "trailing ideographic full stop", + "data": "example\u3002", + "valid": false + }, + { + "description": "trailing fullwidth full stop", + "data": "example\uff0e", + "valid": false + }, + { + "description": "trailing halfwidth ideographic full stop", + "data": "example\uff61", + "valid": false + }, + { + "description": "label too long if separator ignored (full stop)", + "data": "παράδειγμαπαράδειγμαπαράδειγμαπαράδειγμαπαράδειγμαπα.com", + "valid": true + }, + { + "description": "label too long if separator ignored (ideographic full stop)", + "data": "παράδειγμαπαράδειγμαπαράδειγμαπαράδειγμαπαράδειγμαπα\u3002com", + "valid": true + }, + { + "description": "label too long if separator ignored (fullwidth full stop)", + "data": "παράδειγμαπαράδειγμαπαράδειγμαπαράδειγμαπαράδειγμαπα\uff0ecom", + "valid": true + }, + { + "description": "label too long if separator ignored (halfwidth ideographic full stop)", + "data": "παράδειγμαπαράδειγμαπαράδειγμαπαράδειγμαπαράδειγμαπα\uff61com", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/ipv4.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/ipv4.json new file mode 100644 index 000000000..d56d62b33 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/ipv4.json @@ -0,0 +1,214 @@ +[ + { + "description": "validation of IP addresses", + "comment": "RFC 2673, Section 3.2: dotted-quad = decbyte \".\" decbyte \".\" decbyte \".\" decbyte. A 'decbyte' (1*3DIGIT) restricts semantic values to 0-255, allows leading zeros, and strictly forbids symbols, alpha/hex, whitespace, and non-ASCII characters.", + "schema": { "format": "ipv4" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid IP address", + "data": "192.168.0.1", + "valid": true + }, + { + "description": "an IP address with too many components", + "data": "127.0.0.0.1", + "valid": false + }, + { + "description": "an IP address with out-of-range values", + "data": "256.256.256.256", + "valid": false + }, + { + "description": "an IP address without 4 components", + "data": "127.0", + "valid": false + }, + { + "description": "a 2-part address resolving to a routable IP (inet_aton shorthand)", + "data": "127.1", + "valid": false + }, + { + "description": "a 3-part address resolving to a routable IP (inet_aton shorthand)", + "data": "127.0.1", + "valid": false + }, + { + "description": "an IP address as an integer", + "data": "0x7f000001", + "valid": false + }, + { + "description": "an IP address as an integer (decimal)", + "data": "2130706433", + "valid": false + }, + { + "description": "invalid non-ASCII '২' (a Bengali 2)", + "data": "1২7.0.0.1", + "valid": false + }, + { + "description": "invalid fullwidth digits (non-ASCII)", + "data": "１９２.１６８.１.１", + "valid": false + }, + { + "description": "invalid mathematical bold digits (non-ASCII)", + "data": "𝟏𝟗𝟐.𝟏𝟔𝟖.𝟏.𝟏", + "valid": false + }, + { + "description": "netmask is not a part of ipv4 address", + "data": "192.168.1.0/24", + "valid": false + }, + { + "description": "leading whitespace is invalid", + "data": " 192.168.0.1", + "valid": false + }, + { + "description": "trailing whitespace is invalid", + "data": "192.168.0.1 ", + "valid": false + }, + { + "description": "trailing newline is invalid", + "data": "192.168.0.1\n", + "valid": false + }, + { + "description": "hexadecimal notation is invalid", + "data": "0x7f.0.0.1", + "valid": false + }, + { + "description": "octal notation explicit is invalid", + "data": "0o10.0.0.1", + "valid": false + }, + { + "description": "empty part (double dot) is invalid", + "data": "192.168..1", + "valid": false + }, + { + "description": "leading dot is invalid", + "data": ".192.168.0.1", + "valid": false + }, + { + "description": "trailing dot is invalid", + "data": "192.168.0.1.", + "valid": false + }, + { + "description": "minimum valid IPv4 address", + "data": "0.0.0.0", + "valid": true + }, + { + "description": "maximum valid IPv4 address", + "data": "255.255.255.255", + "valid": true + }, + { + "description": "valid IPv4 with an octet in the 200 to 249 range", + "data": "200.0.0.0", + "valid": true + }, + { + "description": "valid IPv4 with two-digit octets", + "data": "10.20.30.40", + "valid": true + }, + { + "description": "empty string is invalid", + "data": "", + "valid": false + }, + { + "description": "plus sign is invalid", + "data": "+1.2.3.4", + "valid": false + }, + { + "description": "negative sign is invalid", + "data": "-1.2.3.4", + "valid": false + }, + { + "description": "exponential notation is invalid", + "data": "1e2.0.0.1", + "valid": false + }, + { + "description": "alpha characters are invalid", + "data": "192.168.a.1", + "valid": false + }, + { + "description": "internal whitespace is invalid", + "data": "192. 168.0.1", + "valid": false + }, + { + "description": "tab character is invalid", + "data": "192.168.0.1\t", + "valid": false + }, + { + "description": "additional content after an embedded NUL byte", + "data": "192.168.0.1\u0000.evil.com", + "valid": false + }, + { + "description": "with port number is invalid", + "data": "192.168.0.1:80", + "valid": false + }, + { + "description": "an IPv4-mapped IPv6 address is invalid", + "data": "::ffff:192.168.0.1", + "valid": false + }, + { + "description": "single octet out of range in last position", + "data": "192.168.0.256", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/ipv6.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/ipv6.json new file mode 100644 index 000000000..7f4674be4 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/ipv6.json @@ -0,0 +1,263 @@ +[ + { + "description": "validation of IPv6 addresses", + "schema": { "format": "ipv6" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid IPv6 address", + "data": "::1", + "valid": true + }, + { + "description": "a group with 5 hex digits is invalid", + "data": "12345::", + "valid": false + }, + { + "description": "trailing 4 hex symbols is valid", + "data": "::abef", + "valid": true + }, + { + "description": "trailing 5 hex symbols is invalid", + "data": "::abcef", + "valid": false + }, + { + "description": "an IPv6 address with too many components", + "data": "1:1:1:1:1:1:1:1:1:1:1:1:1:1:1:1", + "valid": false + }, + { + "description": "an IPv6 address containing illegal characters", + "data": "::laptop", + "valid": false + }, + { + "description": "no digits is valid", + "data": "::", + "valid": true + }, + { + "description": "leading colons is valid", + "data": "::42:ff:1", + "valid": true + }, + { + "description": "trailing colons is valid", + "data": "d6::", + "valid": true + }, + { + "description": "missing leading octet is invalid", + "data": ":2:3:4:5:6:7:8", + "valid": false + }, + { + "description": "missing trailing octet is invalid", + "data": "1:2:3:4:5:6:7:", + "valid": false + }, + { + "description": "missing leading octet with omitted octets later", + "data": ":2:3:4::8", + "valid": false + }, + { + "description": "single set of double colons in the middle is valid", + "data": "1:d6::42", + "valid": true + }, + { + "description": "two sets of double colons is invalid", + "data": "1::d6::42", + "valid": false + }, + { + "description": "mixed format with the ipv4 section as decimal octets", + "data": "1::d6:192.168.0.1", + "valid": true + }, + { + "description": "mixed format with double colons between the sections", + "data": "1:2::192.168.0.1", + "valid": true + }, + { + "description": "mixed format with ipv4 section with octet out of range", + "data": "1::2:192.168.256.1", + "valid": false + }, + { + "description": "mixed format with ipv4 section with a hex octet", + "data": "1::2:192.168.ff.1", + "valid": false + }, + { + "description": "mixed format with leading double colons (ipv4-mapped ipv6 address)", + "data": "::ffff:192.168.0.1", + "valid": true + }, + { + "description": "triple colons is invalid", + "data": "1:2:3:4:5:::8", + "valid": false + }, + { + "description": "8 octets", + "data": "1:2:3:4:5:6:7:8", + "valid": true + }, + { + "description": "insufficient octets without double colons", + "data": "1:2:3:4:5:6:7", + "valid": false + }, + { + "description": "no colons is invalid", + "data": "1", + "valid": false + }, + { + "description": "ipv4 is not ipv6", + "data": "127.0.0.1", + "valid": false + }, + { + "description": "ipv4 segment must have 4 octets", + "data": "1:2:3:4:1.2.3", + "valid": false + }, + { + "description": "leading whitespace is invalid", + "data": " ::1", + "valid": false + }, + { + "description": "trailing whitespace is invalid", + "data": "::1 ", + "valid": false + }, + { + "description": "netmask is not a part of ipv6 address", + "data": "fe80::/64", + "valid": false + }, + { + "description": "zone id is not a part of ipv6 address", + "data": "fe80::a%eth1", + "valid": false + }, + { + "description": "a long valid ipv6", + "data": "1000:1000:1000:1000:1000:1000:255.255.255.255", + "valid": true + }, + { + "description": "a long invalid ipv6, below length limit, first", + "data": "100:100:100:100:100:100:255.255.255.255.255", + "valid": false + }, + { + "description": "a long invalid ipv6, below length limit, second", + "data": "100:100:100:100:100:100:100:255.255.255.255", + "valid": false + }, + { + "description": "invalid non-ASCII '৪' (a Bengali 4)", + "data": "1:2:3:4:5:6:7:৪", + "valid": false + }, + { + "description": "invalid non-ASCII '৪' (a Bengali 4) in the IPv4 portion", + "data": "1:2::192.16৪.0.1", + "valid": false + }, + { + "description": "a bracketed address is invalid", + "data": "[::1]", + "valid": false + }, + { + "description": "a leading zero in the last IPv4 octet is invalid", + "data": "::ffff:192.168.0.01", + "valid": false + }, + { + "description": "uppercase hex digits are valid", + "data": "2001:DB8::1", + "valid": true + }, + { + "description": "leading zeros in every group are valid", + "data": "2001:0db8:0000:0000:0000:0000:0000:0001", + "valid": true + }, + { + "description": "two groups before '::' and four groups after is valid", + "data": "1:2::3:4:5:6", + "valid": true + }, + { + "description": "eight groups alongside '::' is invalid", + "data": "1:2:3:4:5:6:7:8::", + "valid": false + }, + { + "description": "trailing newline is invalid", + "data": "::1\n", + "valid": false + }, + { + "description": "an out-of-range first octet in the embedded IPv4 is invalid", + "data": "::ffff:256.1.1.1", + "valid": false + }, + { + "description": "one group before '::' and five groups after is valid", + "data": "1::2:3:4:5:6", + "valid": true + }, + { + "description": "'::' followed by six groups is valid", + "data": "::1:2:3:4:5:6", + "valid": true + }, + { + "description": "leading '::' before five groups and an embedded IPv4 is valid", + "data": "::1:2:3:4:5:1.2.3.4", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/iri-reference.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/iri-reference.json new file mode 100644 index 000000000..c4b6f1853 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/iri-reference.json @@ -0,0 +1,113 @@ +[ + { + "description": "validation of IRI References", + "schema": { "format": "iri-reference" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid IRI", + "data": "http://ƒøø.ßår/?∂éœ=πîx#πîüx", + "valid": true + }, + { + "description": "a valid protocol-relative IRI Reference", + "data": "//ƒøø.ßår/?∂éœ=πîx#πîüx", + "valid": true + }, + { + "description": "a valid relative IRI Reference", + "data": "/âππ", + "valid": true + }, + { + "description": "an invalid IRI Reference", + "data": "\\\\WINDOWS\\filëßåré", + "valid": false + }, + { + "description": "a valid IRI Reference", + "data": "âππ", + "valid": true + }, + { + "description": "a valid IRI fragment", + "data": "#ƒrägmênt", + "valid": true + }, + { + "description": "an invalid IRI fragment", + "data": "#ƒräg\\mênt", + "valid": false + }, + { + "description": "an embedded IPv4 with a leading zero is invalid", + "data": "//[::ffff:192.168.0.01]/p", + "valid": false + }, + { + "description": "a valid protocol-relative IRI Reference with a compressed IPv6 host", + "data": "//[2001:db8::1]/p", + "valid": true + }, + { + "description": "a valid relative IRI Reference with a supplementary-plane character", + "data": "/âππ/𐌀", + "valid": true + }, + { + "description": "a query-only IRI Reference with a supplementary private-use char", + "data": "?q=󰀀", + "valid": true + }, + { + "description": "a trailing newline after a valid IRI Reference is invalid", + "data": "/âππ\n", + "valid": false + }, + { + "description": "invalid percent-encoding with non-hex digits", + "data": "/%6G", + "valid": false + }, + { + "description": "incomplete percent-encoding triplet", + "data": "/%A", + "valid": false + }, + { + "description": "lone percent sign is invalid", + "data": "/%", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/iri.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/iri.json new file mode 100644 index 000000000..4025e85d5 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/iri.json @@ -0,0 +1,138 @@ +[ + { + "description": "validation of IRIs", + "schema": { "format": "iri" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid IRI with anchor tag", + "data": "http://ƒøø.ßår/?∂éœ=πîx#πîüx", + "valid": true + }, + { + "description": "a valid IRI with anchor tag and parentheses", + "data": "http://ƒøø.com/blah_(wîkïpédiå)_blah#ßité-1", + "valid": true + }, + { + "description": "a valid IRI with URL-encoded stuff", + "data": "http://ƒøø.ßår/?q=Test%20URL-encoded%20stuff", + "valid": true + }, + { + "description": "invalid percent-encoding with non-hex digits", + "data": "http://ƒøø.ßår/%6G", + "valid": false + }, + { + "description": "incomplete percent-encoding triplet", + "data": "http://ƒøø.ßår/%A", + "valid": false + }, + { + "description": "lone percent sign is invalid", + "data": "http://ƒøø.ßår/%", + "valid": false + }, + { + "description": "a valid IRI with many special characters", + "data": "http://-.~_!$&'()*+,;=:%40:80%2f::::::@example.com", + "valid": true + }, + { + "description": "a valid IRI based on IPv6", + "data": "http://[2001:0db8:85a3:0000:0000:8a2e:0370:7334]", + "valid": true + }, + { + "description": "an IPv6 address without enclosing brackets is invalid", + "data": "http://2001:0db8:85a3:0000:0000:8a2e:0370:7334", + "valid": false + }, + { + "description": "an invalid relative IRI Reference", + "data": "/abc", + "valid": false + }, + { + "description": "an invalid IRI", + "data": "\\\\WINDOWS\\filëßåré", + "valid": false + }, + { + "description": "an invalid IRI though valid IRI reference", + "data": "âππ", + "valid": false + }, + { + "description": "an IPv6 host whose embedded IPv4 has a leading zero is invalid", + "data": "http://[::ffff:192.168.0.01]", + "valid": false + }, + { + "description": "a valid IRI with a compressed IPv6 host", + "data": "http://[2001:db8::1]", + "valid": true + }, + { + "description": "a trailing newline after a valid IRI is invalid", + "data": "http://ƒøø.ßår/\n", + "valid": false + }, + { + "description": "a valid IRI with an IPv4 host", + "data": "http://192.168.0.1/p", + "valid": true + }, + { + "description": "a valid IRI with no authority and a rootless path", + "data": "urn:example:resource", + "valid": true + }, + { + "description": "a valid IRI with no authority and an absolute path", + "data": "file:/etc/hosts", + "valid": true + }, + { + "description": "a valid IRI with a supplementary-plane character in the path", + "data": "http://ƒøø.ßår/𐌀", + "valid": true + }, + { + "description": "a valid IRI with a supplementary-plane private-use char in query", + "data": "http://ƒøø.ßår/?q=󰀀", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/json-pointer.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/json-pointer.json new file mode 100644 index 000000000..134eca586 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/json-pointer.json @@ -0,0 +1,208 @@ +[ + { + "description": "validation of JSON-pointers (JSON String Representation)", + "schema": { "format": "json-pointer" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid JSON-pointer", + "data": "/foo/bar~0/baz~1/%a", + "valid": true + }, + { + "description": "not a valid JSON-pointer (~ not escaped)", + "data": "/foo/bar~", + "valid": false + }, + { + "description": "valid JSON-pointer with empty segment", + "data": "/foo//bar", + "valid": true + }, + { + "description": "valid JSON-pointer with the last empty segment", + "data": "/foo/bar/", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #1", + "data": "", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #2", + "data": "/foo", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #3", + "data": "/foo/0", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #4", + "data": "/", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #5", + "data": "/a~1b", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #6", + "data": "/c%d", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #7", + "data": "/e^f", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #8", + "data": "/g|h", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #9", + "data": "/i\\j", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #10", + "data": "/k\"l", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #11", + "data": "/ ", + "valid": true + }, + { + "description": "valid JSON-pointer as stated in RFC 6901 #12", + "data": "/m~0n", + "valid": true + }, + { + "description": "valid JSON-pointer used adding to the last array position", + "data": "/foo/-", + "valid": true + }, + { + "description": "valid JSON-pointer (- used as object member name)", + "data": "/foo/-/bar", + "valid": true + }, + { + "description": "valid JSON-pointer (multiple escaped characters)", + "data": "/~1~0~0~1~1", + "valid": true + }, + { + "description": "valid JSON-pointer (escaped with fraction part) #1", + "data": "/~1.1", + "valid": true + }, + { + "description": "valid JSON-pointer (escaped with fraction part) #2", + "data": "/~0.1", + "valid": true + }, + { + "description": "not a valid JSON-pointer (URI Fragment Identifier) #1", + "data": "#", + "valid": false + }, + { + "description": "not a valid JSON-pointer (URI Fragment Identifier) #2", + "data": "#/", + "valid": false + }, + { + "description": "not a valid JSON-pointer (URI Fragment Identifier) #3", + "data": "#a", + "valid": false + }, + { + "description": "not a valid JSON-pointer (some escaped, but not all) #1", + "data": "/~0~", + "valid": false + }, + { + "description": "not a valid JSON-pointer (some escaped, but not all) #2", + "data": "/~0/~", + "valid": false + }, + { + "description": "not a valid JSON-pointer (wrong escape character) #1", + "data": "/~2", + "valid": false + }, + { + "description": "not a valid JSON-pointer (wrong escape character) #2", + "data": "/~-1", + "valid": false + }, + { + "description": "not a valid JSON-pointer (multiple characters not escaped)", + "data": "/~~", + "valid": false + }, + { + "description": "not a valid JSON-pointer (isn't empty nor starts with /) #1", + "data": "a", + "valid": false + }, + { + "description": "not a valid JSON-pointer (isn't empty nor starts with /) #2", + "data": "0", + "valid": false + }, + { + "description": "not a valid JSON-pointer (isn't empty nor starts with /) #3", + "data": "a/a", + "valid": false + }, + { + "description": "valid JSON-pointer (Unicode characters allowed by RFC 6901)", + "data": "/foo/bar/😎", + "valid": true + }, + { + "description": "valid JSON-pointer (control characters allowed after JSON unescaping)", + "data": "/foo\u0000bar\n\tbaz", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/regex.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/regex.json new file mode 100644 index 000000000..ff831fa2e --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/regex.json @@ -0,0 +1,100 @@ +[ + { + "description": "validation of regular expressions", + "schema": { "format": "regex" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid regular expression", + "data": "([abc])+\\s+$", + "valid": true + }, + { + "description": "a regular expression with unclosed parens is invalid", + "data": "^(abc]", + "valid": false + }, + { + "description": "a quantifier with nothing to repeat is invalid", + "comment": "https://tc39.es/ecma262/#sec-patterns a Quantifier follows something (Term :: Atom Quantifier), so a pattern cannot begin with one", + "data": "*a", + "valid": false + }, + { + "description": "a character class range in reverse order is invalid", + "comment": "https://tc39.es/ecma262/#sec-patterns-static-semantics-early-errors it is a Syntax Error if the first ClassAtom of a range has a greater CharacterValue than the second", + "data": "[z-a]", + "valid": false + }, + { + "description": "a trailing backslash is invalid", + "data": "\\", + "valid": false + }, + { + "description": "an unmatched closing parenthesis is invalid", + "data": "a)", + "valid": false + }, + { + "description": "an unterminated character class is invalid", + "data": "[abc", + "valid": false + }, + { + "description": "an empty regular expression is valid", + "data": "", + "valid": true + }, + { + "description": "alternation is valid", + "data": "a|b", + "valid": true + }, + { + "description": "a bounded quantifier is valid", + "data": "a{2,3}", + "valid": true + }, + { + "description": "a negated range character class is valid", + "data": "[^a-z]", + "valid": true + }, + { + "description": "a lazy quantifier is valid", + "data": "a+?", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/relative-json-pointer.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/relative-json-pointer.json new file mode 100644 index 000000000..59299e7b3 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/relative-json-pointer.json @@ -0,0 +1,138 @@ +[ + { + "description": "validation of Relative JSON Pointers (RJP)", + "schema": { "format": "relative-json-pointer" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid upwards RJP", + "data": "1", + "valid": true + }, + { + "description": "a valid downwards RJP", + "data": "0/foo/bar", + "valid": true + }, + { + "description": "a valid up and then down RJP, with array index", + "data": "2/0/baz/1/zip", + "valid": true + }, + { + "description": "a valid RJP taking the member or index name", + "data": "0#", + "valid": true + }, + { + "description": "an invalid RJP that is a valid JSON Pointer", + "data": "/foo/bar", + "valid": false + }, + { + "description": "a json-pointer part that does not start with a slash is invalid", + "data": "1foo", + "valid": false + }, + { + "description": "negative prefix", + "data": "-1/foo/bar", + "valid": false + }, + { + "description": "explicit positive prefix", + "data": "+1/foo/bar", + "valid": false + }, + { + "description": "non-ASCII digit in the prefix is not allowed", + "data": "١/foo", + "valid": false + }, + { + "description": "## is not a valid json-pointer", + "data": "0##", + "valid": false + }, + { + "description": "zero cannot be followed by other digits, plus json-pointer", + "data": "01/a", + "valid": false + }, + { + "description": "zero cannot be followed by other digits, plus octothorpe", + "data": "01#", + "valid": false + }, + { + "description": "empty string", + "data": "", + "valid": false + }, + { + "description": "multi-digit integer prefix", + "data": "120/foo/bar", + "valid": true + }, + { + "description": "multi-digit prefix with a zero followed by another digit", + "data": "100", + "valid": true + }, + { + "description": "tilde not followed by 0 or 1 in the json-pointer part", + "data": "0/~2", + "valid": false + }, + { + "description": "unescaped tilde at the end of the json-pointer part", + "data": "0/foo/bar~", + "valid": false + }, + { + "description": "octothorpe followed by a json-pointer", + "data": "1#/foo/bar", + "valid": false + }, + { + "description": "empty reference tokens in the json-pointer part", + "data": "0//", + "valid": true + }, + { + "description": "trailing newline after the non-negative integer", + "data": "1\n", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/time.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/time.json new file mode 100644 index 000000000..3182c3ea9 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/time.json @@ -0,0 +1,300 @@ +[ + { + "description": "validation of time strings", + "schema": { "format": "time" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid time string", + "data": "08:30:06Z", + "valid": true + }, + { + "description": "invalid time string with extra leading zeros", + "data": "008:030:006Z", + "valid": false + }, + { + "description": "invalid time string with no leading zero for single digit", + "data": "8:3:6Z", + "valid": false + }, + { + "description": "hour, minute, second must be two digits", + "data": "8:0030:6Z", + "valid": false + }, + { + "description": "a valid time string with leap second, Zulu", + "data": "23:59:60Z", + "valid": true + }, + { + "description": "invalid leap second, Zulu (wrong hour)", + "data": "22:59:60Z", + "valid": false + }, + { + "description": "invalid leap second, Zulu (wrong minute)", + "data": "23:58:60Z", + "valid": false + }, + { + "description": "valid leap second, zero time-offset", + "data": "23:59:60+00:00", + "valid": true + }, + { + "description": "invalid leap second, zero time-offset (wrong hour)", + "data": "22:59:60+00:00", + "valid": false + }, + { + "description": "invalid leap second, zero time-offset (wrong minute)", + "data": "23:58:60+00:00", + "valid": false + }, + { + "description": "valid leap second, positive time-offset", + "data": "01:29:60+01:30", + "valid": true + }, + { + "description": "valid leap second, large positive time-offset", + "data": "23:29:60+23:30", + "valid": true + }, + { + "description": "invalid leap second, positive time-offset (wrong hour)", + "data": "23:59:60+01:00", + "valid": false + }, + { + "description": "invalid leap second, positive time-offset (wrong minute)", + "data": "23:59:60+00:30", + "valid": false + }, + { + "description": "valid leap second, negative time-offset", + "data": "15:59:60-08:00", + "valid": true + }, + { + "description": "valid leap second, large negative time-offset", + "data": "00:29:60-23:30", + "valid": true + }, + { + "description": "invalid leap second, negative time-offset (wrong hour)", + "data": "23:59:60-01:00", + "valid": false + }, + { + "description": "invalid leap second, negative time-offset (wrong minute)", + "data": "23:59:60-00:30", + "valid": false + }, + { + "description": "a valid time string with second fraction", + "data": "23:20:50.52Z", + "valid": true + }, + { + "description": "a valid time string with precise second fraction", + "data": "08:30:06.283185Z", + "valid": true + }, + { + "description": "a valid time string with plus offset", + "data": "08:30:06+00:20", + "valid": true + }, + { + "description": "a valid time string with minus offset", + "data": "08:30:06-08:00", + "valid": true + }, + { + "description": "time with unknown local offset is valid", + "comment": "RFC 3339 section 4.3 (unknown local offset)", + "data": "12:34:56-00:00", + "valid": true + }, + { + "description": "hour, minute in time-offset must be two digits", + "data": "08:30:06-8:000", + "valid": false + }, + { + "description": "a valid time string with case-insensitive Z", + "data": "08:30:06z", + "valid": true + }, + { + "description": "an invalid time string with invalid hour", + "data": "24:00:00Z", + "valid": false + }, + { + "description": "an invalid time string with invalid minute", + "data": "00:60:00Z", + "valid": false + }, + { + "description": "an invalid time string with invalid second", + "data": "00:00:61Z", + "valid": false + }, + { + "description": "an invalid time string with invalid time numoffset hour", + "data": "01:02:03+24:00", + "valid": false + }, + { + "description": "an invalid time string with invalid time numoffset minute", + "data": "01:02:03+00:60", + "valid": false + }, + { + "description": "an invalid time string with invalid time with both Z and numoffset", + "data": "01:02:03Z+00:30", + "valid": false + }, + { + "description": "an invalid offset indicator", + "data": "08:30:06 PST", + "valid": false + }, + { + "description": "only RFC3339 not all of ISO 8601 are valid", + "data": "01:01:01,1111", + "valid": false + }, + { + "description": "no time offset", + "data": "12:00:00", + "valid": false + }, + { + "description": "no time offset with second fraction", + "data": "12:00:00.52", + "valid": false + }, + { + "description": "invalid non-ASCII '২' (a Bengali 2)", + "data": "1২:00:00Z", + "valid": false + }, + { + "description": "offset not starting with plus or minus", + "data": "08:30:06#00:20", + "valid": false + }, + { + "description": "contains letters", + "data": "ab:cd:ef", + "valid": false + }, + { + "description": "an invalid time string in date-time format", + "data": "2020-11-28T23:55:45Z", + "valid": false + }, + { + "description": "a numeric time-offset requires the colon separator", + "comment": "RFC 3339 section 5.6: time-numoffset = (\"+\" / \"-\") time-hour \":\" time-minute", + "data": "08:30:06+0130", + "valid": false + }, + { + "description": "a numeric time-offset requires the minutes component", + "comment": "RFC 3339 section 5.6: time-numoffset = (\"+\" / \"-\") time-hour \":\" time-minute", + "data": "08:30:06+01", + "valid": false + }, + { + "description": "a trailing newline after a valid time", + "comment": "RFC 3339 section 5.6: no production of full-time emits a line feed", + "data": "08:30:06Z\n", + "valid": false + }, + { + "description": "hour 24 is invalid with a numeric time-offset", + "comment": "RFC 3339 section 5.6: time-hour = 2DIGIT ; 00-23, and section 5.7 only allows 00 to 23", + "data": "24:59:00+01:00", + "valid": false + }, + { + "description": "minute 60 is invalid with a numeric time-offset", + "comment": "RFC 3339 section 5.6: time-minute = 2DIGIT ; 00-59", + "data": "23:60:00+00:01", + "valid": false + }, + { + "description": "a valid time string with a 15 digit second fraction", + "comment": "RFC 3339 section 5.6: time-secfrac = \".\" 1*DIGIT, with no upper bound on the number of digits", + "data": "00:59:59.999999999999999Z", + "valid": true + }, + { + "description": "a second fraction with no digits", + "comment": "RFC 3339 section 5.6: time-secfrac = \".\" 1*DIGIT requires at least one digit", + "data": "08:30:06.Z", + "valid": false + }, + { + "description": "partial-time requires the seconds component", + "comment": "RFC 3339 section 5.6: partial-time = time-hour \":\" time-minute \":\" time-second [time-secfrac]", + "data": "12:00Z", + "valid": false + }, + { + "description": "a comma decimal separator with a time-offset", + "comment": "RFC 3339 section 5.6: the decimal separator in time-secfrac is \".\" - the ISO 8601 comma is not part of this profile", + "data": "08:30:06,5Z", + "valid": false + }, + { + "description": "leading whitespace before a valid time", + "comment": "RFC 3339 section 5.6: no production of full-time emits whitespace", + "data": " 08:30:06Z", + "valid": false + }, + { + "description": "trailing whitespace after a valid time", + "comment": "RFC 3339 section 5.6: no production of full-time emits whitespace", + "data": "08:30:06Z ", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/unknown.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/unknown.json new file mode 100644 index 000000000..12339ae57 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/unknown.json @@ -0,0 +1,43 @@ +[ + { + "description": "unknown format", + "schema": { "format": "unknown" }, + "tests": [ + { + "description": "unknown formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "unknown formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "unknown formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "unknown formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "unknown formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "unknown formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "unknown formats ignore strings", + "data": "string", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/uri-reference.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/uri-reference.json new file mode 100644 index 000000000..3741bad01 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/uri-reference.json @@ -0,0 +1,179 @@ +[ + { + "description": "validation of URI References", + "schema": { "format": "uri-reference" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid URI", + "data": "http://foo.bar/?baz=qux#quux", + "valid": true + }, + { + "description": "a valid protocol-relative URI Reference", + "data": "//foo.bar/?baz=qux#quux", + "valid": true + }, + { + "description": "a valid relative URI Reference", + "data": "/abc", + "valid": true + }, + { + "description": "an invalid URI Reference", + "data": "\\\\WINDOWS\\fileshare", + "valid": false + }, + { + "description": "a valid URI Reference", + "data": "abc", + "valid": true + }, + { + "description": "a valid URI fragment", + "data": "#fragment", + "valid": true + }, + { + "description": "an invalid URI fragment", + "data": "#frag\\ment", + "valid": false + }, + { + "description": "unescaped non US-ASCII characters", + "data": "/foobar®.txt", + "valid": false + }, + { + "description": "invalid backslash character", + "data": "https://example.org/foobar\\.txt", + "valid": false + }, + { + "description": "URI with leading-zero IPv4 is structurally valid as a reg-name", + "comment": "RFC 3986, Section 3.2.2: If an IP literal fails strict IPv4address parsing, it falls back to reg-name. A string of digits and dots is valid under unreserved characters. JSON Schema format asserts syntax, not scheme-specific DNS semantics.", + "data": "http://087.10.0.1/", + "valid": true + }, + { + "description": "URI with out-of-bounds IPv4 is structurally valid as a reg-name", + "comment": "RFC 3986, Section 3.2.2: Fallback to reg-name allows digits and dots.", + "data": "http://999.999.999.999/", + "valid": true + }, + { + "description": "an empty URI reference", + "comment": "RFC 3986, Section 4.2: relative-part includes path-empty, which is zero characters.", + "data": "", + "valid": true + }, + { + "description": "a query-only reference", + "comment": "RFC 3986, Section 4.2: relative-ref = relative-part [ \"?\" query ] [ \"#\" fragment ], so a query with an empty path is a reference.", + "data": "?query=1", + "valid": true + }, + { + "description": "a network-path reference with an empty authority", + "comment": "RFC 3986, Section 3.2.2: host = IP-literal / IPv4address / reg-name, and reg-name may be empty.", + "data": "//", + "valid": true + }, + { + "description": "an incomplete percent-encoding", + "comment": "RFC 3986, Section 2.1: pct-encoded = \"%\" HEXDIG HEXDIG.", + "data": "/%zz", + "valid": false + }, + { + "description": "incomplete percent-encoding triplet", + "comment": "RFC 3986, Section 2.1: pct-encoded = \"%\" HEXDIG HEXDIG.", + "data": "/%A", + "valid": false + }, + { + "description": "lone percent sign is invalid", + "comment": "RFC 3986, Section 2.1: pct-encoded = \"%\" HEXDIG HEXDIG.", + "data": "/%", + "valid": false + }, + { + "description": "a double quote in a path", + "comment": "RFC 3986, Appendix A: pchar = unreserved / pct-encoded / sub-delims / \":\" / \"@\", and the double quote is in none of them.", + "data": "/a\"b", + "valid": false + }, + { + "description": "square brackets outside an authority", + "comment": "RFC 3986, Section 3.2.2: square brackets are reserved for an IP-literal inside the authority. path-absolute is built from pchar, which excludes them.", + "data": "/[::1]", + "valid": false + }, + { + "description": "a non-numeric port in a network-path reference", + "comment": "RFC 3986, Section 3.2.3: port = *DIGIT. reg-name admits no colon, so the text after the colon can only be a port.", + "data": "//example.com:abc/p", + "valid": false + }, + { + "description": "more than one at-sign in the authority", + "comment": "RFC 3986, Section 3.2: authority = [ userinfo \"@\" ] host [ \":\" port ]. Neither userinfo nor host admits a literal at-sign.", + "data": "//a@b@example.com/", + "valid": false + }, + { + "description": "a leading zero in the IPv4 part of an IPv6 literal", + "comment": "RFC 3986, Section 3.2.2: inside an IP-literal the dotted quad must be an IPv4address, and dec-octet has no leading-zero alternative. There is no reg-name fallback inside brackets.", + "data": "//[::ffff:192.168.0.01]/p", + "valid": false + }, + { + "description": "a colon in the first segment of a relative-path reference", + "comment": "RFC 3986, Section 4.2: relative-part uses path-noscheme, whose first segment is segment-nz-nc - a non-zero-length segment without any colon. A scheme must start with ALPHA, so 1:b cannot be read as a URI either.", + "data": "1:b", + "valid": false + }, + { + "description": "a colon in the first segment preceded by a dot-segment", + "comment": "RFC 3986, Section 4.2: \"Such a segment must be preceded by a dot-segment (e.g., \\\"./this:that\\\") to make a relative-path reference.\"", + "data": "./this:that", + "valid": true + }, + { + "description": "a trailing line feed", + "comment": "RFC 3986, Appendix A: no production admits %x0A.", + "data": "/p\n", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/uri-template.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/uri-template.json new file mode 100644 index 000000000..f0aaa52f7 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/uri-template.json @@ -0,0 +1,248 @@ +[ + { + "description": "format: uri-template", + "schema": { "format": "uri-template" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid uri-template", + "data": "http://example.com/dictionary/{term:1}/{term}", + "valid": true + }, + { + "description": "an invalid uri-template", + "data": "http://example.com/dictionary/{term:1}/{term", + "valid": false + }, + { + "description": "a valid uri-template without variables", + "data": "http://example.com/dictionary", + "valid": true + }, + { + "description": "a valid relative uri-template", + "data": "dictionary/{term:1}/{term}", + "valid": true + }, + { + "description": "an empty expression is invalid", + "data": "{}", + "valid": false + }, + { + "description": "an empty varspec inside the variable list is invalid", + "data": "{a,,b}", + "valid": false + }, + { + "description": "a zero prefix max-length is invalid", + "data": "{v:0}", + "valid": false + }, + { + "description": "a five-digit prefix max-length is invalid", + "data": "{v:10000}", + "valid": false + }, + { + "description": "an empty string is a valid uri-template", + "data": "", + "valid": true + }, + { + "description": "an unmatched closing brace is invalid", + "data": "foo}bar", + "valid": false + }, + { + "description": "a percent-encoded triplet in a literal is valid", + "data": "a%41b", + "valid": true + }, + { + "description": "a literal with a lone percent sign is invalid", + "data": "a%", + "valid": false + }, + { + "description": "a literal with an incomplete percent-encoded triplet is invalid", + "data": "a%4", + "valid": false + }, + { + "description": "a literal with non-hex percent encoding is invalid", + "data": "a%GG", + "valid": false + }, + { + "description": "a supplementary plane character in a literal is valid", + "data": "a\ud83d\ude00b", + "valid": true + }, + { + "description": "a private-use character in a literal is valid", + "data": "a\ue000b", + "valid": true + }, + { + "description": "a supplementary private-use character in a literal is valid", + "data": "a\udb80\udc00b", + "valid": true + }, + { + "description": "reserved expansion operator", + "data": "{+var}", + "valid": true + }, + { + "description": "fragment expansion operator", + "data": "{#var}", + "valid": true + }, + { + "description": "label expansion operator", + "data": "{.var}", + "valid": true + }, + { + "description": "path segment expansion operator", + "data": "{/var}", + "valid": true + }, + { + "description": "path style parameter expansion operator", + "data": "{;var}", + "valid": true + }, + { + "description": "form style query expansion operator", + "data": "{?var}", + "valid": true + }, + { + "description": "form style query continuation operator", + "data": "{&var}", + "valid": true + }, + { + "description": "the explode modifier is valid", + "data": "{var*}", + "valid": true + }, + { + "description": "combined prefix and explode modifiers are invalid", + "data": "{var:1*}", + "valid": false + }, + { + "description": "a variable list with more than one varspec is valid", + "data": "{x,y}", + "valid": true + }, + { + "description": "a dotted variable name is valid", + "data": "{a.b}", + "valid": true + }, + { + "description": "a trailing dot in a variable name is invalid", + "data": "{a.}", + "valid": false + }, + { + "description": "a double dot in a variable name is invalid", + "data": "{a..b}", + "valid": false + }, + { + "description": "a variable name may start with a percent-encoded triplet", + "data": "{%41}", + "valid": true + }, + { + "description": "a variable name with a lone percent sign is invalid", + "data": "{%}", + "valid": false + }, + { + "description": "a variable name with an incomplete percent-encoded triplet is invalid", + "data": "{%4}", + "valid": false + }, + { + "description": "a variable name with non-hex percent encoding is invalid", + "data": "{%GG}", + "valid": false + }, + { + "description": "a four-digit prefix max-length is valid", + "data": "{v:1000}", + "valid": true + }, + { + "description": "a leading zero in the prefix max-length is invalid", + "data": "{v:01}", + "valid": false + }, + { + "description": "a space in a literal is invalid", + "data": "a b", + "valid": false + }, + { + "description": "a delete character in a literal is invalid", + "data": "a\u007fb", + "valid": false + }, + { + "description": "a trailing comma in the variable list is invalid", + "data": "{a,}", + "valid": false + }, + { + "description": "a default value in a varspec is invalid", + "data": "{var=def}", + "valid": false + }, + { + "description": "an operator in place of the first varspec is invalid", + "data": "{,+var}", + "valid": false + }, + { + "description": "an apostrophe in a literal is valid", + "data": "a'b", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/uri.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/uri.json new file mode 100644 index 000000000..b9d080027 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/format/uri.json @@ -0,0 +1,250 @@ +[ + { + "description": "validation of URIs", + "schema": { "format": "uri" }, + "tests": [ + { + "description": "all string formats ignore integers", + "data": 12, + "valid": true + }, + { + "description": "all string formats ignore floats", + "data": 13.7, + "valid": true + }, + { + "description": "all string formats ignore objects", + "data": {}, + "valid": true + }, + { + "description": "all string formats ignore arrays", + "data": [], + "valid": true + }, + { + "description": "all string formats ignore booleans", + "data": false, + "valid": true + }, + { + "description": "all string formats ignore nulls", + "data": null, + "valid": true + }, + { + "description": "a valid URL with anchor tag", + "data": "http://foo.bar/?baz=qux#quux", + "valid": true + }, + { + "description": "a valid URL with anchor tag and parentheses", + "data": "http://foo.com/blah_(wikipedia)_blah#cite-1", + "valid": true + }, + { + "description": "a valid URL with URL-encoded stuff", + "data": "http://foo.bar/?q=Test%20URL-encoded%20stuff", + "valid": true + }, + { + "description": "a valid puny-coded URL ", + "data": "http://xn--nw2a.xn--j6w193g/", + "valid": true + }, + { + "description": "a valid URL with many special characters", + "data": "http://-.~_!$&'()*+,;=:%40:80%2f::::::@example.com", + "valid": true + }, + { + "description": "a valid URL based on IPv4", + "data": "http://223.255.255.254", + "valid": true + }, + { + "description": "a valid URL with ftp scheme", + "data": "ftp://ftp.is.co.za/rfc/rfc1808.txt", + "valid": true + }, + { + "description": "a valid URL for a simple text file", + "data": "http://www.ietf.org/rfc/rfc2396.txt", + "valid": true + }, + { + "description": "a valid URL", + "data": "ldap://[2001:db8::7]/c=GB?objectClass?one", + "valid": true + }, + { + "description": "a valid mailto URI", + "data": "mailto:John.Doe@example.com", + "valid": true + }, + { + "description": "a valid newsgroup URI", + "data": "news:comp.infosystems.www.servers.unix", + "valid": true + }, + { + "description": "a valid tel URI", + "data": "tel:+1-816-555-1212", + "valid": true + }, + { + "description": "a valid URN", + "data": "urn:oasis:names:specification:docbook:dtd:xml:4.1.2", + "valid": true + }, + { + "description": "an invalid protocol-relative URI Reference", + "data": "//foo.bar/?baz=qux#quux", + "valid": false + }, + { + "description": "an invalid relative URI Reference", + "data": "/abc", + "valid": false + }, + { + "description": "an invalid URI", + "data": "\\\\WINDOWS\\fileshare", + "valid": false + }, + { + "description": "an invalid URI though valid URI reference", + "data": "abc", + "valid": false + }, + { + "description": "an invalid URI with spaces", + "data": "http:// shouldfail.com", + "valid": false + }, + { + "description": "an invalid URI with spaces and missing scheme", + "data": ":// should fail", + "valid": false + }, + { + "description": "an invalid URI with comma in scheme", + "data": "bar,baz:foo", + "valid": false + }, + { + "description": "invalid userinfo", + "data": "https://[@example.org/test.txt", + "valid": false + }, + { + "description": "unescaped non US-ASCII characters", + "data": "https://example.org/foobar®.txt", + "valid": false + }, + { + "description": "invalid backslash character", + "data": "https://example.org/foobar\\.txt", + "valid": false + }, + { + "description": "invalid \" character", + "data": "https://example.org/foobar\".txt", + "valid": false + }, + { + "description": "invalid <> characters", + "data": "https://example.org/foobar<>.txt", + "valid": false + }, + { + "description": "invalid {} characters", + "data": "https://example.org/foobar{}.txt", + "valid": false + }, + { + "description": "invalid ^ character", + "data": "https://example.org/foobar^.txt", + "valid": false + }, + { + "description": "invalid ` character", + "data": "https://example.org/foobar`.txt", + "valid": false + }, + { + "description": "invalid SPACE character", + "data": "https://example.org/foo bar.txt", + "valid": false + }, + { + "description": "invalid | character", + "data": "https://example.org/foobar|.txt", + "valid": false + }, + { + "description": "URI with leading-zero IPv4 is structurally valid as a reg-name", + "comment": "RFC 3986, Section 3.2.2: If an IP literal fails strict IPv4address parsing, it falls back to reg-name. A string of digits and dots is valid under unreserved characters. JSON Schema format asserts syntax, not scheme-specific DNS semantics.", + "data": "http://087.10.0.1/", + "valid": true + }, + { + "description": "URI with out-of-bounds IPv4 is structurally valid as a reg-name", + "comment": "RFC 3986, Section 3.2.2: Fallback to reg-name allows digits and dots.", + "data": "http://999.999.999.999/", + "valid": true + }, + { + "description": "invalid percent-encoding with non-hex digits", + "data": "http://example.com/%6G", + "valid": false + }, + { + "description": "incomplete percent-encoding triplet", + "data": "http://example.com/%A", + "valid": false + }, + { + "description": "lone percent sign is invalid", + "data": "http://example.com/%", + "valid": false + }, + { + "description": "scheme must start with a letter", + "data": "1http://example.com", + "valid": false + }, + { + "description": "invalid character in scheme", + "data": "ht_tp://example.com", + "valid": false + }, + { + "description": "non-numeric port is invalid", + "data": "http://example.com:abc/path", + "valid": false + }, + { + "description": "leading zero in an embedded IPv4 address is invalid", + "data": "http://[::ffff:01.2.3.4]", + "valid": false + }, + { + "description": "square brackets are not allowed in a path segment", + "data": "http:/[::1]", + "valid": false + }, + { + "description": "a trailing newline after a valid URI is invalid", + "data": "http://foo.bar/\n", + "valid": false + }, + { + "description": "an IPv6 address without enclosing brackets is invalid", + "data": "http://2001:0db8:85a3:0000:0000:8a2e:0370:7334", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/id.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/id.json new file mode 100644 index 000000000..6be81b8da --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/id.json @@ -0,0 +1,114 @@ +[ + { + "description": "id inside an enum is not a real identifier", + "comment": "the implementation must not be confused by an id buried in the enum", + "schema": { + "definitions": { + "id_in_enum": { + "enum": [ + { + "$id": "https://localhost:1234/id/my_identifier.json", + "type": "null" + } + ] + }, + "real_id_in_schema": { + "$id": "https://localhost:1234/id/my_identifier.json", + "type": "string" + }, + "zzz_id_in_const": { + "const": { + "$id": "https://localhost:1234/id/my_identifier.json", + "type": "null" + } + } + }, + "anyOf": [ + { "$ref": "#/definitions/id_in_enum" }, + { "$ref": "https://localhost:1234/id/my_identifier.json" } + ] + }, + "tests": [ + { + "description": "exact match to enum, and type matches", + "data": { + "$id": "https://localhost:1234/id/my_identifier.json", + "type": "null" + }, + "valid": true + }, + { + "description": "match $ref to id", + "data": "a string to match #/definitions/id_in_enum", + "valid": true + }, + { + "description": "no match on enum or $ref to id", + "data": 1, + "valid": false + } + ] + }, + { + "description": "non-schema object containing a plain-name $id property", + "schema": { + "definitions": { + "const_not_anchor": { + "const": { + "$id": "#not_a_real_anchor" + } + } + }, + "if": { + "const": "skip not_a_real_anchor" + }, + "then": true, + "else" : { + "$ref": "#/definitions/const_not_anchor" + } + }, + "tests": [ + { + "description": "skip traversing definition for a valid result", + "data": "skip not_a_real_anchor", + "valid": true + }, + { + "description": "const at const_not_anchor does not match", + "data": 1, + "valid": false + } + ] + }, + { + "description": "non-schema object containing an $id property", + "schema": { + "definitions": { + "const_not_id": { + "const": { + "$id": "not_a_real_id" + } + } + }, + "if": { + "const": "skip not_a_real_id" + }, + "then": true, + "else" : { + "$ref": "#/definitions/const_not_id" + } + }, + "tests": [ + { + "description": "skip traversing definition for a valid result", + "data": "skip not_a_real_id", + "valid": true + }, + { + "description": "const at const_not_id does not match", + "data": 1, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/non-bmp-regex.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/non-bmp-regex.json new file mode 100644 index 000000000..dd67af2b2 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/non-bmp-regex.json @@ -0,0 +1,82 @@ +[ + { + "description": "Proper UTF-16 surrogate pair handling: pattern", + "comment": "Optional because .Net doesn't correctly handle 32-bit Unicode characters", + "schema": { "pattern": "^🐲*$" }, + "tests": [ + { + "description": "matches empty", + "data": "", + "valid": true + }, + { + "description": "matches single", + "data": "🐲", + "valid": true + }, + { + "description": "matches two", + "data": "🐲🐲", + "valid": true + }, + { + "description": "doesn't match one", + "data": "🐉", + "valid": false + }, + { + "description": "doesn't match two", + "data": "🐉🐉", + "valid": false + }, + { + "description": "doesn't match one ASCII", + "data": "D", + "valid": false + }, + { + "description": "doesn't match two ASCII", + "data": "DD", + "valid": false + } + ] + }, + { + "description": "Proper UTF-16 surrogate pair handling: patternProperties", + "comment": "Optional because .Net doesn't correctly handle 32-bit Unicode characters", + "schema": { + "patternProperties": { + "^🐲*$": { + "type": "integer" + } + } + }, + "tests": [ + { + "description": "matches empty", + "data": { "": 1 }, + "valid": true + }, + { + "description": "matches single", + "data": { "🐲": 1 }, + "valid": true + }, + { + "description": "matches two", + "data": { "🐲🐲": 1 }, + "valid": true + }, + { + "description": "doesn't match one", + "data": { "🐲": "hello" }, + "valid": false + }, + { + "description": "doesn't match two", + "data": { "🐲🐲": "hello" }, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/unknownKeyword.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/unknownKeyword.json new file mode 100644 index 000000000..1f58d97e3 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/optional/unknownKeyword.json @@ -0,0 +1,56 @@ +[ + { + "description": "$id inside an unknown keyword is not a real identifier", + "comment": "the implementation must not be confused by an $id in locations we do not know how to parse", + "schema": { + "definitions": { + "id_in_unknown0": { + "not": { + "array_of_schemas": [ + { + "$id": "https://localhost:1234/unknownKeyword/my_identifier.json", + "type": "null" + } + ] + } + }, + "real_id_in_schema": { + "$id": "https://localhost:1234/unknownKeyword/my_identifier.json", + "type": "string" + }, + "id_in_unknown1": { + "not": { + "object_of_schemas": { + "foo": { + "$id": "https://localhost:1234/unknownKeyword/my_identifier.json", + "type": "integer" + } + } + } + } + }, + "anyOf": [ + { "$ref": "#/definitions/id_in_unknown0" }, + { "$ref": "#/definitions/id_in_unknown1" }, + { "$ref": "https://localhost:1234/unknownKeyword/my_identifier.json" } + ] + }, + "tests": [ + { + "description": "type matches second anyOf, which has a real schema in it", + "data": "a string", + "valid": true + }, + { + "description": "type matches non-schema in first anyOf", + "data": null, + "valid": false + }, + { + "description": "type matches non-schema in third anyOf", + "data": 1, + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/pattern.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/pattern.json new file mode 100644 index 000000000..92db0f971 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/pattern.json @@ -0,0 +1,59 @@ +[ + { + "description": "pattern validation", + "schema": {"pattern": "^a*$"}, + "tests": [ + { + "description": "a matching pattern is valid", + "data": "aaa", + "valid": true + }, + { + "description": "a non-matching pattern is invalid", + "data": "abc", + "valid": false + }, + { + "description": "ignores booleans", + "data": true, + "valid": true + }, + { + "description": "ignores integers", + "data": 123, + "valid": true + }, + { + "description": "ignores floats", + "data": 1.0, + "valid": true + }, + { + "description": "ignores objects", + "data": {}, + "valid": true + }, + { + "description": "ignores arrays", + "data": [], + "valid": true + }, + { + "description": "ignores null", + "data": null, + "valid": true + } + ] + }, + { + "description": "pattern is not anchored", + "schema": {"pattern": "a+"}, + "tests": [ + { + "description": "matches a substring", + "data": "xxaayy", + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/patternProperties.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/patternProperties.json new file mode 100644 index 000000000..c276e6479 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/patternProperties.json @@ -0,0 +1,171 @@ +[ + { + "description": + "patternProperties validates properties matching a regex", + "schema": { + "patternProperties": { + "f.*o": {"type": "integer"} + } + }, + "tests": [ + { + "description": "a single valid match is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "multiple valid matches is valid", + "data": {"foo": 1, "foooooo" : 2}, + "valid": true + }, + { + "description": "a single invalid match is invalid", + "data": {"foo": "bar", "fooooo": 2}, + "valid": false + }, + { + "description": "multiple invalid matches is invalid", + "data": {"foo": "bar", "foooooo" : "baz"}, + "valid": false + }, + { + "description": "ignores arrays", + "data": ["foo"], + "valid": true + }, + { + "description": "ignores strings", + "data": "foo", + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + } + ] + }, + { + "description": "multiple simultaneous patternProperties are validated", + "schema": { + "patternProperties": { + "a*": {"type": "integer"}, + "aaa*": {"maximum": 20} + } + }, + "tests": [ + { + "description": "a single valid match is valid", + "data": {"a": 21}, + "valid": true + }, + { + "description": "a simultaneous match is valid", + "data": {"aaaa": 18}, + "valid": true + }, + { + "description": "multiple matches is valid", + "data": {"a": 21, "aaaa": 18}, + "valid": true + }, + { + "description": "an invalid due to one is invalid", + "data": {"a": "bar"}, + "valid": false + }, + { + "description": "an invalid due to the other is invalid", + "data": {"aaaa": 31}, + "valid": false + }, + { + "description": "an invalid due to both is invalid", + "data": {"aaa": "foo", "aaaa": 31}, + "valid": false + } + ] + }, + { + "description": "regexes are not anchored by default and are case sensitive", + "schema": { + "patternProperties": { + "[0-9]{2,}": { "type": "boolean" }, + "X_": { "type": "string" } + } + }, + "tests": [ + { + "description": "non recognized members are ignored", + "data": { "answer 1": "42" }, + "valid": true + }, + { + "description": "recognized members are accounted for", + "data": { "a31b": null }, + "valid": false + }, + { + "description": "regexes are case sensitive", + "data": { "a_x_3": 3 }, + "valid": true + }, + { + "description": "regexes are case sensitive, 2", + "data": { "a_X_3": 3 }, + "valid": false + } + ] + }, + { + "description": "patternProperties with boolean schemas", + "schema": { + "patternProperties": { + "f.*": true, + "b.*": false + } + }, + "tests": [ + { + "description": "object with property matching schema true is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "object with property matching schema false is invalid", + "data": {"bar": 2}, + "valid": false + }, + { + "description": "object with both properties is invalid", + "data": {"foo": 1, "bar": 2}, + "valid": false + }, + { + "description": "object with a property matching both true and false is invalid", + "data": {"foobar":1}, + "valid": false + }, + { + "description": "empty object is valid", + "data": {}, + "valid": true + } + ] + }, + { + "description": "patternProperties with null valued instance properties", + "schema": { + "patternProperties": { + "^.*bar$": {"type": "null"} + } + }, + "tests": [ + { + "description": "allows null values", + "data": {"foobar": null}, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/properties.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/properties.json new file mode 100644 index 000000000..5b971ca0e --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/properties.json @@ -0,0 +1,236 @@ +[ + { + "description": "object properties validation", + "schema": { + "properties": { + "foo": {"type": "integer"}, + "bar": {"type": "string"} + } + }, + "tests": [ + { + "description": "both properties present and valid is valid", + "data": {"foo": 1, "bar": "baz"}, + "valid": true + }, + { + "description": "one property invalid is invalid", + "data": {"foo": 1, "bar": {}}, + "valid": false + }, + { + "description": "both properties invalid is invalid", + "data": {"foo": [], "bar": {}}, + "valid": false + }, + { + "description": "doesn't invalidate other properties", + "data": {"quux": []}, + "valid": true + }, + { + "description": "ignores arrays", + "data": [], + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + } + ] + }, + { + "description": + "properties, patternProperties, additionalProperties interaction", + "schema": { + "properties": { + "foo": {"type": "array", "maxItems": 3}, + "bar": {"type": "array"} + }, + "patternProperties": {"f.o": {"minItems": 2}}, + "additionalProperties": {"type": "integer"} + }, + "tests": [ + { + "description": "property validates property", + "data": {"foo": [1, 2]}, + "valid": true + }, + { + "description": "property invalidates property", + "data": {"foo": [1, 2, 3, 4]}, + "valid": false + }, + { + "description": "patternProperty invalidates property", + "data": {"foo": []}, + "valid": false + }, + { + "description": "patternProperty validates nonproperty", + "data": {"fxo": [1, 2]}, + "valid": true + }, + { + "description": "patternProperty invalidates nonproperty", + "data": {"fxo": []}, + "valid": false + }, + { + "description": "additionalProperty ignores property", + "data": {"bar": []}, + "valid": true + }, + { + "description": "additionalProperty validates others", + "data": {"quux": 3}, + "valid": true + }, + { + "description": "additionalProperty invalidates others", + "data": {"quux": "foo"}, + "valid": false + } + ] + }, + { + "description": "properties with boolean schema", + "schema": { + "properties": { + "foo": true, + "bar": false + } + }, + "tests": [ + { + "description": "no property present is valid", + "data": {}, + "valid": true + }, + { + "description": "only 'true' property present is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "only 'false' property present is invalid", + "data": {"bar": 2}, + "valid": false + }, + { + "description": "both properties present is invalid", + "data": {"foo": 1, "bar": 2}, + "valid": false + } + ] + }, + { + "description": "properties with escaped characters", + "schema": { + "properties": { + "foo\nbar": {"type": "number"}, + "foo\"bar": {"type": "number"}, + "foo\\bar": {"type": "number"}, + "foo\rbar": {"type": "number"}, + "foo\tbar": {"type": "number"}, + "foo\fbar": {"type": "number"} + } + }, + "tests": [ + { + "description": "object with all numbers is valid", + "data": { + "foo\nbar": 1, + "foo\"bar": 1, + "foo\\bar": 1, + "foo\rbar": 1, + "foo\tbar": 1, + "foo\fbar": 1 + }, + "valid": true + }, + { + "description": "object with strings is invalid", + "data": { + "foo\nbar": "1", + "foo\"bar": "1", + "foo\\bar": "1", + "foo\rbar": "1", + "foo\tbar": "1", + "foo\fbar": "1" + }, + "valid": false + } + ] + }, + { + "description": "properties with null valued instance properties", + "schema": { + "properties": { + "foo": {"type": "null"} + } + }, + "tests": [ + { + "description": "allows null values", + "data": {"foo": null}, + "valid": true + } + ] + }, + { + "description": "properties whose names are Javascript object property names", + "comment": "Ensure JS implementations don't universally consider e.g. __proto__ to always be present in an object.", + "schema": { + "properties": { + "__proto__": {"type": "number"}, + "toString": { + "properties": { "length": { "type": "string" } } + }, + "constructor": {"type": "number"} + } + }, + "tests": [ + { + "description": "ignores arrays", + "data": [], + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + }, + { + "description": "none of the properties mentioned", + "data": {}, + "valid": true + }, + { + "description": "__proto__ not valid", + "data": { "__proto__": "foo" }, + "valid": false + }, + { + "description": "toString not valid", + "data": { "toString": { "length": 37 } }, + "valid": false + }, + { + "description": "constructor not valid", + "data": { "constructor": { "length": 37 } }, + "valid": false + }, + { + "description": "all present and valid", + "data": { + "__proto__": 12, + "toString": { "length": "foo" }, + "constructor": 37 + }, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/propertyNames.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/propertyNames.json new file mode 100644 index 000000000..b86564310 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/propertyNames.json @@ -0,0 +1,164 @@ +[ + { + "description": "propertyNames validation", + "schema": { + "propertyNames": {"maxLength": 3} + }, + "tests": [ + { + "description": "all property names valid", + "data": { + "f": {}, + "foo": {} + }, + "valid": true + }, + { + "description": "some property names invalid", + "data": { + "foo": {}, + "foobar": {} + }, + "valid": false + }, + { + "description": "object without properties is valid", + "data": {}, + "valid": true + }, + { + "description": "ignores arrays", + "data": [1, 2, 3, 4], + "valid": true + }, + { + "description": "ignores strings", + "data": "foobar", + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + }, + { + "description": "ignores null", + "data": null, + "valid": true + }, + { + "description": "ignores booleans", + "data": true, + "valid": true + } + ] + }, + { + "description": "propertyNames validation with pattern", + "schema": { + "propertyNames": { "pattern": "^a+$" } + }, + "tests": [ + { + "description": "matching property names valid", + "data": { + "a": {}, + "aa": {}, + "aaa": {} + }, + "valid": true + }, + { + "description": "non-matching property name is invalid", + "data": { + "aaA": {} + }, + "valid": false + }, + { + "description": "object without properties is valid", + "data": {}, + "valid": true + } + ] + }, + { + "description": "propertyNames with boolean schema true", + "schema": {"propertyNames": true}, + "tests": [ + { + "description": "object with any properties is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "empty object is valid", + "data": {}, + "valid": true + } + ] + }, + { + "description": "propertyNames with boolean schema false", + "schema": {"propertyNames": false}, + "tests": [ + { + "description": "object with any properties is invalid", + "data": {"foo": 1}, + "valid": false + }, + { + "description": "empty object is valid", + "data": {}, + "valid": true + } + ] + }, + { + "description": "propertyNames with const", + "schema": {"propertyNames": {"const": "foo"}}, + "tests": [ + { + "description": "object with property foo is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "object with any other property is invalid", + "data": {"bar": 1}, + "valid": false + }, + { + "description": "empty object is valid", + "data": {}, + "valid": true + } + ] + }, + { + "description": "propertyNames with enum", + "schema": {"propertyNames": {"enum": ["foo", "bar"]}}, + "tests": [ + { + "description": "object with property foo is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "object with property foo and bar is valid", + "data": {"foo": 1, "bar": 1}, + "valid": true + }, + { + "description": "object with any other property is invalid", + "data": {"baz": 1}, + "valid": false + }, + { + "description": "empty object is valid", + "data": {}, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/ref.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/ref.json new file mode 100644 index 000000000..82e1e1672 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/ref.json @@ -0,0 +1,1043 @@ +[ + { + "description": "root pointer ref", + "schema": { + "properties": { + "foo": {"$ref": "#"} + }, + "additionalProperties": false + }, + "tests": [ + { + "description": "match", + "data": {"foo": false}, + "valid": true + }, + { + "description": "recursive match", + "data": {"foo": {"foo": false}}, + "valid": true + }, + { + "description": "mismatch", + "data": {"bar": false}, + "valid": false + }, + { + "description": "recursive mismatch", + "data": {"foo": {"bar": false}}, + "valid": false + } + ] + }, + { + "description": "relative pointer ref to object", + "schema": { + "properties": { + "foo": {"type": "integer"}, + "bar": {"$ref": "#/properties/foo"} + } + }, + "tests": [ + { + "description": "match", + "data": {"bar": 3}, + "valid": true + }, + { + "description": "mismatch", + "data": {"bar": true}, + "valid": false + } + ] + }, + { + "description": "relative pointer ref to array", + "schema": { + "items": [ + {"type": "integer"}, + {"$ref": "#/items/0"} + ] + }, + "tests": [ + { + "description": "match array", + "data": [1, 2], + "valid": true + }, + { + "description": "mismatch array", + "data": [1, "foo"], + "valid": false + } + ] + }, + { + "description": "escaped pointer ref", + "schema": { + "definitions": { + "tilde~field": {"type": "integer"}, + "slash/field": {"type": "integer"}, + "percent%field": {"type": "integer"} + }, + "properties": { + "tilde": {"$ref": "#/definitions/tilde~0field"}, + "slash": {"$ref": "#/definitions/slash~1field"}, + "percent": {"$ref": "#/definitions/percent%25field"} + } + }, + "tests": [ + { + "description": "slash invalid", + "data": {"slash": "aoeu"}, + "valid": false + }, + { + "description": "tilde invalid", + "data": {"tilde": "aoeu"}, + "valid": false + }, + { + "description": "percent invalid", + "data": {"percent": "aoeu"}, + "valid": false + }, + { + "description": "slash valid", + "data": {"slash": 123}, + "valid": true + }, + { + "description": "tilde valid", + "data": {"tilde": 123}, + "valid": true + }, + { + "description": "percent valid", + "data": {"percent": 123}, + "valid": true + } + ] + }, + { + "description": "nested refs", + "schema": { + "definitions": { + "a": {"type": "integer"}, + "b": {"$ref": "#/definitions/a"}, + "c": {"$ref": "#/definitions/b"} + }, + "allOf": [{ "$ref": "#/definitions/c" }] + }, + "tests": [ + { + "description": "nested ref valid", + "data": 5, + "valid": true + }, + { + "description": "nested ref invalid", + "data": "a", + "valid": false + } + ] + }, + { + "description": "ref overrides any sibling keywords", + "schema": { + "definitions": { + "reffed": { + "type": "array" + } + }, + "properties": { + "foo": { + "$ref": "#/definitions/reffed", + "maxItems": 2 + } + } + }, + "tests": [ + { + "description": "ref valid", + "data": { "foo": [] }, + "valid": true + }, + { + "description": "ref valid, maxItems ignored", + "data": { "foo": [ 1, 2, 3] }, + "valid": true + }, + { + "description": "ref invalid", + "data": { "foo": "string" }, + "valid": false + } + ] + }, + { + "description": "$ref prevents a sibling $id from changing the base uri", + "schema": { + "$id": "http://localhost:1234/sibling_id/base/", + "definitions": { + "foo": { + "$id": "http://localhost:1234/sibling_id/foo.json", + "type": "string" + }, + "base_foo": { + "$comment": "this canonical uri is http://localhost:1234/sibling_id/base/foo.json", + "$id": "foo.json", + "type": "number" + } + }, + "allOf": [ + { + "$comment": "$ref resolves to http://localhost:1234/sibling_id/base/foo.json, not http://localhost:1234/sibling_id/foo.json", + "$id": "http://localhost:1234/sibling_id/", + "$ref": "foo.json" + } + ] + }, + "tests": [ + { + "description": "$ref resolves to /definitions/base_foo, data does not validate", + "data": "a", + "valid": false + }, + { + "description": "$ref resolves to /definitions/base_foo, data validates", + "data": 1, + "valid": true + } + ] + }, + { + "description": "remote ref, containing refs itself", + "schema": {"$ref": "http://json-schema.org/draft-07/schema#"}, + "tests": [ + { + "description": "remote ref valid", + "data": {"minLength": 1}, + "valid": true + }, + { + "description": "remote ref invalid", + "data": {"minLength": -1}, + "valid": false + } + ] + }, + { + "description": "property named $ref that is not a reference", + "schema": { + "properties": { + "$ref": {"type": "string"} + } + }, + "tests": [ + { + "description": "property named $ref valid", + "data": {"$ref": "a"}, + "valid": true + }, + { + "description": "property named $ref invalid", + "data": {"$ref": 2}, + "valid": false + } + ] + }, + { + "description": "property named $ref, containing an actual $ref", + "schema": { + "properties": { + "$ref": {"$ref": "#/definitions/is-string"} + }, + "definitions": { + "is-string": { + "type": "string" + } + } + }, + "tests": [ + { + "description": "property named $ref valid", + "data": {"$ref": "a"}, + "valid": true + }, + { + "description": "property named $ref invalid", + "data": {"$ref": 2}, + "valid": false + } + ] + }, + { + "description": "$ref to boolean schema true", + "schema": { + "allOf": [{ "$ref": "#/definitions/bool" }], + "definitions": { + "bool": true + } + }, + "tests": [ + { + "description": "any value is valid", + "data": "foo", + "valid": true + } + ] + }, + { + "description": "$ref to boolean schema false", + "schema": { + "allOf": [{ "$ref": "#/definitions/bool" }], + "definitions": { + "bool": false + } + }, + "tests": [ + { + "description": "any value is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "Recursive references between schemas", + "schema": { + "$id": "http://localhost:1234/tree", + "description": "tree of nodes", + "type": "object", + "properties": { + "meta": {"type": "string"}, + "nodes": { + "type": "array", + "items": {"$ref": "node"} + } + }, + "required": ["meta", "nodes"], + "definitions": { + "node": { + "$id": "http://localhost:1234/node", + "description": "node", + "type": "object", + "properties": { + "value": {"type": "number"}, + "subtree": {"$ref": "tree"} + }, + "required": ["value"] + } + } + }, + "tests": [ + { + "description": "valid tree", + "data": { + "meta": "root", + "nodes": [ + { + "value": 1, + "subtree": { + "meta": "child", + "nodes": [ + {"value": 1.1}, + {"value": 1.2} + ] + } + }, + { + "value": 2, + "subtree": { + "meta": "child", + "nodes": [ + {"value": 2.1}, + {"value": 2.2} + ] + } + } + ] + }, + "valid": true + }, + { + "description": "invalid tree", + "data": { + "meta": "root", + "nodes": [ + { + "value": 1, + "subtree": { + "meta": "child", + "nodes": [ + {"value": "string is invalid"}, + {"value": 1.2} + ] + } + }, + { + "value": 2, + "subtree": { + "meta": "child", + "nodes": [ + {"value": 2.1}, + {"value": 2.2} + ] + } + } + ] + }, + "valid": false + } + ] + }, + { + "description": "refs with quote", + "schema": { + "properties": { + "foo\"bar": {"$ref": "#/definitions/foo%22bar"} + }, + "definitions": { + "foo\"bar": {"type": "number"} + } + }, + "tests": [ + { + "description": "object with numbers is valid", + "data": { + "foo\"bar": 1 + }, + "valid": true + }, + { + "description": "object with strings is invalid", + "data": { + "foo\"bar": "1" + }, + "valid": false + } + ] + }, + { + "description": "Location-independent identifier", + "schema": { + "allOf": [{ + "$ref": "#foo" + }], + "definitions": { + "A": { + "$id": "#foo", + "type": "integer" + } + } + }, + "tests": [ + { + "data": 1, + "description": "match", + "valid": true + }, + { + "data": "a", + "description": "mismatch", + "valid": false + } + ] + }, + { + "description": "Reference an anchor with a non-relative URI", + "schema": { + "$id": "https://example.com/schema-with-anchor", + "allOf": [{ + "$ref": "https://example.com/schema-with-anchor#foo" + }], + "definitions": { + "A": { + "$id": "#foo", + "type": "integer" + } + } + }, + "tests": [ + { + "data": 1, + "description": "match", + "valid": true + }, + { + "data": "a", + "description": "mismatch", + "valid": false + } + ] + }, + { + "description": "Location-independent identifier with base URI change in subschema", + "schema": { + "$id": "http://localhost:1234/root", + "allOf": [{ + "$ref": "http://localhost:1234/nested.json#foo" + }], + "definitions": { + "A": { + "$id": "nested.json", + "definitions": { + "B": { + "$id": "#foo", + "type": "integer" + } + } + } + } + }, + "tests": [ + { + "data": 1, + "description": "match", + "valid": true + }, + { + "data": "a", + "description": "mismatch", + "valid": false + } + ] + }, + { + "description": "naive replacement of $ref with its destination is not correct", + "schema": { + "definitions": { + "a_string": { "type": "string" } + }, + "enum": [ + { "$ref": "#/definitions/a_string" } + ] + }, + "tests": [ + { + "description": "do not evaluate the $ref inside the enum, matching any string", + "data": "this is a string", + "valid": false + }, + { + "description": "do not evaluate the $ref inside the enum, definition exact match", + "data": { "type": "string" }, + "valid": false + }, + { + "description": "match the enum exactly", + "data": { "$ref": "#/definitions/a_string" }, + "valid": true + } + ] + }, + { + "description": "refs with relative uris and defs", + "schema": { + "$id": "http://example.com/schema-relative-uri-defs1.json", + "properties": { + "foo": { + "$id": "schema-relative-uri-defs2.json", + "definitions": { + "inner": { + "properties": { + "bar": { "type": "string" } + } + } + }, + "allOf": [ { "$ref": "#/definitions/inner" } ] + } + }, + "allOf": [ { "$ref": "schema-relative-uri-defs2.json" } ] + }, + "tests": [ + { + "description": "invalid on inner field", + "data": { + "foo": { + "bar": 1 + }, + "bar": "a" + }, + "valid": false + }, + { + "description": "invalid on outer field", + "data": { + "foo": { + "bar": "a" + }, + "bar": 1 + }, + "valid": false + }, + { + "description": "valid on both fields", + "data": { + "foo": { + "bar": "a" + }, + "bar": "a" + }, + "valid": true + } + ] + }, + { + "description": "relative refs with absolute uris and defs", + "schema": { + "$id": "http://example.com/schema-refs-absolute-uris-defs1.json", + "properties": { + "foo": { + "$id": "http://example.com/schema-refs-absolute-uris-defs2.json", + "definitions": { + "inner": { + "properties": { + "bar": { "type": "string" } + } + } + }, + "allOf": [ { "$ref": "#/definitions/inner" } ] + } + }, + "allOf": [ { "$ref": "schema-refs-absolute-uris-defs2.json" } ] + }, + "tests": [ + { + "description": "invalid on inner field", + "data": { + "foo": { + "bar": 1 + }, + "bar": "a" + }, + "valid": false + }, + { + "description": "invalid on outer field", + "data": { + "foo": { + "bar": "a" + }, + "bar": 1 + }, + "valid": false + }, + { + "description": "valid on both fields", + "data": { + "foo": { + "bar": "a" + }, + "bar": "a" + }, + "valid": true + } + ] + }, + { + "description": "$id must be resolved against nearest parent, not just immediate parent", + "schema": { + "$id": "http://example.com/a.json", + "definitions": { + "x": { + "$id": "http://example.com/b/c.json", + "not": { + "definitions": { + "y": { + "$id": "d.json", + "type": "number" + } + } + } + } + }, + "allOf": [ + { + "$ref": "http://example.com/b/d.json" + } + ] + }, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "non-number is invalid", + "data": "a", + "valid": false + } + ] + }, + { + "description": "simple URN base URI with $ref via the URN", + "schema": { + "$comment": "URIs do not have to have HTTP(s) schemes", + "$id": "urn:uuid:deadbeef-1234-ffff-ffff-4321feebdaed", + "minimum": 30, + "properties": { + "foo": {"$ref": "urn:uuid:deadbeef-1234-ffff-ffff-4321feebdaed"} + } + }, + "tests": [ + { + "description": "valid under the URN IDed schema", + "data": {"foo": 37}, + "valid": true + }, + { + "description": "invalid under the URN IDed schema", + "data": {"foo": 12}, + "valid": false + } + ] + }, + { + "description": "simple URN base URI with JSON pointer", + "schema": { + "$comment": "URIs do not have to have HTTP(s) schemes", + "$id": "urn:uuid:deadbeef-1234-00ff-ff00-4321feebdaed", + "properties": { + "foo": {"$ref": "#/definitions/bar"} + }, + "definitions": { + "bar": {"type": "string"} + } + }, + "tests": [ + { + "description": "a string is valid", + "data": {"foo": "bar"}, + "valid": true + }, + { + "description": "a non-string is invalid", + "data": {"foo": 12}, + "valid": false + } + ] + }, + { + "description": "URN base URI with NSS", + "schema": { + "$comment": "RFC 8141 §2.2", + "$id": "urn:example:1/406/47452/2", + "properties": { + "foo": {"$ref": "#/definitions/bar"} + }, + "definitions": { + "bar": {"type": "string"} + } + }, + "tests": [ + { + "description": "a string is valid", + "data": {"foo": "bar"}, + "valid": true + }, + { + "description": "a non-string is invalid", + "data": {"foo": 12}, + "valid": false + } + ] + }, + { + "description": "URN base URI with r-component", + "schema": { + "$comment": "RFC 8141 §2.3.1", + "$id": "urn:example:foo-bar-baz-qux?+CCResolve:cc=uk", + "properties": { + "foo": {"$ref": "#/definitions/bar"} + }, + "definitions": { + "bar": {"type": "string"} + } + }, + "tests": [ + { + "description": "a string is valid", + "data": {"foo": "bar"}, + "valid": true + }, + { + "description": "a non-string is invalid", + "data": {"foo": 12}, + "valid": false + } + ] + }, + { + "description": "URN base URI with q-component", + "schema": { + "$comment": "RFC 8141 §2.3.2", + "$id": "urn:example:weather?=op=map&lat=39.56&lon=-104.85&datetime=1969-07-21T02:56:15Z", + "properties": { + "foo": {"$ref": "#/definitions/bar"} + }, + "definitions": { + "bar": {"type": "string"} + } + }, + "tests": [ + { + "description": "a string is valid", + "data": {"foo": "bar"}, + "valid": true + }, + { + "description": "a non-string is invalid", + "data": {"foo": 12}, + "valid": false + } + ] + }, + { + "description": "URN base URI with URN and JSON pointer ref", + "schema": { + "$id": "urn:uuid:deadbeef-1234-0000-0000-4321feebdaed", + "properties": { + "foo": {"$ref": "urn:uuid:deadbeef-1234-0000-0000-4321feebdaed#/definitions/bar"} + }, + "definitions": { + "bar": {"type": "string"} + } + }, + "tests": [ + { + "description": "a string is valid", + "data": {"foo": "bar"}, + "valid": true + }, + { + "description": "a non-string is invalid", + "data": {"foo": 12}, + "valid": false + } + ] + }, + { + "description": "URN base URI with URN and anchor ref", + "schema": { + "$id": "urn:uuid:deadbeef-1234-ff00-00ff-4321feebdaed", + "properties": { + "foo": {"$ref": "urn:uuid:deadbeef-1234-ff00-00ff-4321feebdaed#something"} + }, + "definitions": { + "bar": { + "$id": "#something", + "type": "string" + } + } + }, + "tests": [ + { + "description": "a string is valid", + "data": {"foo": "bar"}, + "valid": true + }, + { + "description": "a non-string is invalid", + "data": {"foo": 12}, + "valid": false + } + ] + }, + { + "description": "ref to if", + "schema": { + "allOf": [ + {"$ref": "http://example.com/ref/if"}, + { + "if": { + "$id": "http://example.com/ref/if", + "type": "integer" + } + } + ] + }, + "tests": [ + { + "description": "a non-integer is invalid due to the $ref", + "data": "foo", + "valid": false + }, + { + "description": "an integer is valid", + "data": 12, + "valid": true + } + ] + }, + { + "description": "ref to then", + "schema": { + "allOf": [ + {"$ref": "http://example.com/ref/then"}, + { + "then": { + "$id": "http://example.com/ref/then", + "type": "integer" + } + } + ] + }, + "tests": [ + { + "description": "a non-integer is invalid due to the $ref", + "data": "foo", + "valid": false + }, + { + "description": "an integer is valid", + "data": 12, + "valid": true + } + ] + }, + { + "description": "ref to else", + "schema": { + "allOf": [ + {"$ref": "http://example.com/ref/else"}, + { + "else": { + "$id": "http://example.com/ref/else", + "type": "integer" + } + } + ] + }, + "tests": [ + { + "description": "a non-integer is invalid due to the $ref", + "data": "foo", + "valid": false + }, + { + "description": "an integer is valid", + "data": 12, + "valid": true + } + ] + }, + { + "description": "ref with absolute-path-reference", + "schema": { + "$id": "http://example.com/ref/absref.json", + "definitions": { + "a": { + "$id": "http://example.com/ref/absref/foobar.json", + "type": "number" + }, + "b": { + "$id": "http://example.com/absref/foobar.json", + "type": "string" + } + }, + "allOf": [ + { "$ref": "/absref/foobar.json" } + ] + }, + "tests": [ + { + "description": "a string is valid", + "data": "foo", + "valid": true + }, + { + "description": "an integer is invalid", + "data": 12, + "valid": false + } + ] + }, + { + "description": "$id with file URI still resolves pointers - *nix", + "schema": { + "$id": "file:///folder/file.json", + "definitions": { + "foo": { + "type": "number" + } + }, + "allOf": [ + { + "$ref": "#/definitions/foo" + } + ] + }, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "non-number is invalid", + "data": "a", + "valid": false + } + ] + }, + { + "description": "$id with file URI still resolves pointers - windows", + "schema": { + "$id": "file:///c:/folder/file.json", + "definitions": { + "foo": { + "type": "number" + } + }, + "allOf": [ + { + "$ref": "#/definitions/foo" + } + ] + }, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "non-number is invalid", + "data": "a", + "valid": false + } + ] + }, + { + "description": "empty tokens in $ref json-pointer", + "schema": { + "definitions": { + "": { + "definitions": { + "": { "type": "number" } + } + } + }, + "allOf": [ + { + "$ref": "#/definitions//definitions/" + } + ] + }, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "non-number is invalid", + "data": "a", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/refRemote.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/refRemote.json new file mode 100644 index 000000000..450787af6 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/refRemote.json @@ -0,0 +1,257 @@ +[ + { + "description": "remote ref", + "schema": {"$ref": "http://localhost:1234/integer.json"}, + "tests": [ + { + "description": "remote ref valid", + "data": 1, + "valid": true + }, + { + "description": "remote ref invalid", + "data": "a", + "valid": false + } + ] + }, + { + "description": "fragment within remote ref", + "schema": {"$ref": "http://localhost:1234/draft7/subSchemas.json#/definitions/integer"}, + "tests": [ + { + "description": "remote fragment valid", + "data": 1, + "valid": true + }, + { + "description": "remote fragment invalid", + "data": "a", + "valid": false + } + ] + }, + { + "description": "ref within remote ref", + "schema": { + "$ref": "http://localhost:1234/draft7/subSchemas.json#/definitions/refToInteger" + }, + "tests": [ + { + "description": "ref within ref valid", + "data": 1, + "valid": true + }, + { + "description": "ref within ref invalid", + "data": "a", + "valid": false + } + ] + }, + { + "description": "base URI change", + "schema": { + "$id": "http://localhost:1234/", + "items": { + "$id": "baseUriChange/", + "items": {"$ref": "folderInteger.json"} + } + }, + "tests": [ + { + "description": "base URI change ref valid", + "data": [[1]], + "valid": true + }, + { + "description": "base URI change ref invalid", + "data": [["a"]], + "valid": false + } + ] + }, + { + "description": "base URI change - change folder", + "schema": { + "$id": "http://localhost:1234/scope_change_defs1.json", + "type" : "object", + "properties": { + "list": {"$ref": "#/definitions/baz"} + }, + "definitions": { + "baz": { + "$id": "baseUriChangeFolder/", + "type": "array", + "items": {"$ref": "folderInteger.json"} + } + } + }, + "tests": [ + { + "description": "number is valid", + "data": {"list": [1]}, + "valid": true + }, + { + "description": "string is invalid", + "data": {"list": ["a"]}, + "valid": false + } + ] + }, + { + "description": "base URI change - change folder in subschema", + "schema": { + "$id": "http://localhost:1234/scope_change_defs2.json", + "type" : "object", + "properties": { + "list": {"$ref": "#/definitions/baz/definitions/bar"} + }, + "definitions": { + "baz": { + "$id": "baseUriChangeFolderInSubschema/", + "definitions": { + "bar": { + "type": "array", + "items": {"$ref": "folderInteger.json"} + } + } + } + } + }, + "tests": [ + { + "description": "number is valid", + "data": {"list": [1]}, + "valid": true + }, + { + "description": "string is invalid", + "data": {"list": ["a"]}, + "valid": false + } + ] + }, + { + "description": "root ref in remote ref", + "schema": { + "$id": "http://localhost:1234/object", + "type": "object", + "properties": { + "name": {"$ref": "draft7/name.json#/definitions/orNull"} + } + }, + "tests": [ + { + "description": "string is valid", + "data": { + "name": "foo" + }, + "valid": true + }, + { + "description": "null is valid", + "data": { + "name": null + }, + "valid": true + }, + { + "description": "object is invalid", + "data": { + "name": { + "name": null + } + }, + "valid": false + } + ] + }, + { + "description": "remote ref with ref to definitions", + "schema": { + "$id": "http://localhost:1234/schema-remote-ref-ref-defs1.json", + "allOf": [ + { "$ref": "draft7/ref-and-definitions.json" } + ] + }, + "tests": [ + { + "description": "invalid", + "data": { + "bar": 1 + }, + "valid": false + }, + { + "description": "valid", + "data": { + "bar": "a" + }, + "valid": true + } + ] + }, + { + "description": "Location-independent identifier in remote ref", + "schema": { + "$ref": "http://localhost:1234/draft7/locationIndependentIdentifier.json#/definitions/refToInteger" + }, + "tests": [ + { + "description": "integer is valid", + "data": 1, + "valid": true + }, + { + "description": "string is invalid", + "data": "foo", + "valid": false + } + ] + }, + { + "description": "retrieved nested refs resolve relative to their URI not $id", + "schema": { + "$id": "http://localhost:1234/some-id", + "properties": { + "name": {"$ref": "nested/foo-ref-string.json"} + } + }, + "tests": [ + { + "description": "number is invalid", + "data": { + "name": {"foo": 1} + }, + "valid": false + }, + { + "description": "string is valid", + "data": { + "name": {"foo": "a"} + }, + "valid": true + } + ] + }, + { + "description": "$ref to $ref finds location-independent $id", + "schema": { + "$ref": "http://localhost:1234/draft7/detached-ref.json#/definitions/foo" + }, + "tests": [ + { + "description": "number is valid", + "data": 1, + "valid": true + }, + { + "description": "non-number is invalid", + "data": "a", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/required.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/required.json new file mode 100644 index 000000000..f98c64af8 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/required.json @@ -0,0 +1,161 @@ +[ + { + "description": "required validation", + "schema": { + "properties": { + "foo": {}, + "bar": {} + }, + "required": ["foo"] + }, + "tests": [ + { + "description": "present required property is valid", + "data": {"foo": 1}, + "valid": true + }, + { + "description": "non-present required property is invalid", + "data": {"bar": 1}, + "valid": false + }, + { + "description": "ignores arrays", + "data": [], + "valid": true + }, + { + "description": "ignores strings", + "data": "", + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + }, + { + "description": "ignores null", + "data": null, + "valid": true + }, + { + "description": "ignores boolean", + "data": true, + "valid": true + } + ] + }, + { + "description": "required default validation", + "schema": { + "properties": { + "foo": {} + } + }, + "tests": [ + { + "description": "not required by default", + "data": {}, + "valid": true + } + ] + }, + { + "description": "required with empty array", + "schema": { + "properties": { + "foo": {} + }, + "required": [] + }, + "tests": [ + { + "description": "property not required", + "data": {}, + "valid": true + } + ] + }, + { + "description": "required with escaped characters", + "schema": { + "required": [ + "foo\nbar", + "foo\"bar", + "foo\\bar", + "foo\rbar", + "foo\tbar", + "foo\fbar" + ] + }, + "tests": [ + { + "description": "object with all properties present is valid", + "data": { + "foo\nbar": 1, + "foo\"bar": 1, + "foo\\bar": 1, + "foo\rbar": 1, + "foo\tbar": 1, + "foo\fbar": 1 + }, + "valid": true + }, + { + "description": "object with some properties missing is invalid", + "data": { + "foo\nbar": "1", + "foo\"bar": "1" + }, + "valid": false + } + ] + }, + { + "description": "required properties whose names are Javascript object property names", + "comment": "Ensure JS implementations don't universally consider e.g. __proto__ to always be present in an object.", + "schema": { "required": ["__proto__", "toString", "constructor"] }, + "tests": [ + { + "description": "ignores arrays", + "data": [], + "valid": true + }, + { + "description": "ignores other non-objects", + "data": 12, + "valid": true + }, + { + "description": "none of the properties mentioned", + "data": {}, + "valid": false + }, + { + "description": "__proto__ present", + "data": { "__proto__": "foo" }, + "valid": false + }, + { + "description": "toString present", + "data": { "toString": { "length": 37 } }, + "valid": false + }, + { + "description": "constructor present", + "data": { "constructor": { "length": 37 } }, + "valid": false + }, + { + "description": "all present", + "data": { + "__proto__": 12, + "toString": { "length": "foo" }, + "constructor": 37 + }, + "valid": true + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/type.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/type.json new file mode 100644 index 000000000..830464702 --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/type.json @@ -0,0 +1,474 @@ +[ + { + "description": "integer type matches integers", + "schema": {"type": "integer"}, + "tests": [ + { + "description": "an integer is an integer", + "data": 1, + "valid": true + }, + { + "description": "a float with zero fractional part is an integer", + "data": 1.0, + "valid": true + }, + { + "description": "a float is not an integer", + "data": 1.1, + "valid": false + }, + { + "description": "a string is not an integer", + "data": "foo", + "valid": false + }, + { + "description": "a string is still not an integer, even if it looks like one", + "data": "1", + "valid": false + }, + { + "description": "an object is not an integer", + "data": {}, + "valid": false + }, + { + "description": "an array is not an integer", + "data": [], + "valid": false + }, + { + "description": "a boolean is not an integer", + "data": true, + "valid": false + }, + { + "description": "null is not an integer", + "data": null, + "valid": false + } + ] + }, + { + "description": "number type matches numbers", + "schema": {"type": "number"}, + "tests": [ + { + "description": "an integer is a number", + "data": 1, + "valid": true + }, + { + "description": "a float with zero fractional part is a number (and an integer)", + "data": 1.0, + "valid": true + }, + { + "description": "a float is a number", + "data": 1.1, + "valid": true + }, + { + "description": "a string is not a number", + "data": "foo", + "valid": false + }, + { + "description": "a string is still not a number, even if it looks like one", + "data": "1", + "valid": false + }, + { + "description": "an object is not a number", + "data": {}, + "valid": false + }, + { + "description": "an array is not a number", + "data": [], + "valid": false + }, + { + "description": "a boolean is not a number", + "data": true, + "valid": false + }, + { + "description": "null is not a number", + "data": null, + "valid": false + } + ] + }, + { + "description": "string type matches strings", + "schema": {"type": "string"}, + "tests": [ + { + "description": "1 is not a string", + "data": 1, + "valid": false + }, + { + "description": "a float is not a string", + "data": 1.1, + "valid": false + }, + { + "description": "a string is a string", + "data": "foo", + "valid": true + }, + { + "description": "a string is still a string, even if it looks like a number", + "data": "1", + "valid": true + }, + { + "description": "an empty string is still a string", + "data": "", + "valid": true + }, + { + "description": "an object is not a string", + "data": {}, + "valid": false + }, + { + "description": "an array is not a string", + "data": [], + "valid": false + }, + { + "description": "a boolean is not a string", + "data": true, + "valid": false + }, + { + "description": "null is not a string", + "data": null, + "valid": false + } + ] + }, + { + "description": "object type matches objects", + "schema": {"type": "object"}, + "tests": [ + { + "description": "an integer is not an object", + "data": 1, + "valid": false + }, + { + "description": "a float is not an object", + "data": 1.1, + "valid": false + }, + { + "description": "a string is not an object", + "data": "foo", + "valid": false + }, + { + "description": "an object is an object", + "data": {}, + "valid": true + }, + { + "description": "an array is not an object", + "data": [], + "valid": false + }, + { + "description": "a boolean is not an object", + "data": true, + "valid": false + }, + { + "description": "null is not an object", + "data": null, + "valid": false + } + ] + }, + { + "description": "array type matches arrays", + "schema": {"type": "array"}, + "tests": [ + { + "description": "an integer is not an array", + "data": 1, + "valid": false + }, + { + "description": "a float is not an array", + "data": 1.1, + "valid": false + }, + { + "description": "a string is not an array", + "data": "foo", + "valid": false + }, + { + "description": "an object is not an array", + "data": {}, + "valid": false + }, + { + "description": "an array is an array", + "data": [], + "valid": true + }, + { + "description": "a boolean is not an array", + "data": true, + "valid": false + }, + { + "description": "null is not an array", + "data": null, + "valid": false + } + ] + }, + { + "description": "boolean type matches booleans", + "schema": {"type": "boolean"}, + "tests": [ + { + "description": "an integer is not a boolean", + "data": 1, + "valid": false + }, + { + "description": "zero is not a boolean", + "data": 0, + "valid": false + }, + { + "description": "a float is not a boolean", + "data": 1.1, + "valid": false + }, + { + "description": "a string is not a boolean", + "data": "foo", + "valid": false + }, + { + "description": "an empty string is not a boolean", + "data": "", + "valid": false + }, + { + "description": "an object is not a boolean", + "data": {}, + "valid": false + }, + { + "description": "an array is not a boolean", + "data": [], + "valid": false + }, + { + "description": "true is a boolean", + "data": true, + "valid": true + }, + { + "description": "false is a boolean", + "data": false, + "valid": true + }, + { + "description": "null is not a boolean", + "data": null, + "valid": false + } + ] + }, + { + "description": "null type matches only the null object", + "schema": {"type": "null"}, + "tests": [ + { + "description": "an integer is not null", + "data": 1, + "valid": false + }, + { + "description": "a float is not null", + "data": 1.1, + "valid": false + }, + { + "description": "zero is not null", + "data": 0, + "valid": false + }, + { + "description": "a string is not null", + "data": "foo", + "valid": false + }, + { + "description": "an empty string is not null", + "data": "", + "valid": false + }, + { + "description": "an object is not null", + "data": {}, + "valid": false + }, + { + "description": "an array is not null", + "data": [], + "valid": false + }, + { + "description": "true is not null", + "data": true, + "valid": false + }, + { + "description": "false is not null", + "data": false, + "valid": false + }, + { + "description": "null is null", + "data": null, + "valid": true + } + ] + }, + { + "description": "multiple types can be specified in an array", + "schema": {"type": ["integer", "string"]}, + "tests": [ + { + "description": "an integer is valid", + "data": 1, + "valid": true + }, + { + "description": "a string is valid", + "data": "foo", + "valid": true + }, + { + "description": "a float is invalid", + "data": 1.1, + "valid": false + }, + { + "description": "an object is invalid", + "data": {}, + "valid": false + }, + { + "description": "an array is invalid", + "data": [], + "valid": false + }, + { + "description": "a boolean is invalid", + "data": true, + "valid": false + }, + { + "description": "null is invalid", + "data": null, + "valid": false + } + ] + }, + { + "description": "type as array with one item", + "schema": { + "type": ["string"] + }, + "tests": [ + { + "description": "string is valid", + "data": "foo", + "valid": true + }, + { + "description": "number is invalid", + "data": 123, + "valid": false + } + ] + }, + { + "description": "type: array or object", + "schema": { + "type": ["array", "object"] + }, + "tests": [ + { + "description": "array is valid", + "data": [1,2,3], + "valid": true + }, + { + "description": "object is valid", + "data": {"foo": 123}, + "valid": true + }, + { + "description": "number is invalid", + "data": 123, + "valid": false + }, + { + "description": "string is invalid", + "data": "foo", + "valid": false + }, + { + "description": "null is invalid", + "data": null, + "valid": false + } + ] + }, + { + "description": "type: array, object or null", + "schema": { + "type": ["array", "object", "null"] + }, + "tests": [ + { + "description": "array is valid", + "data": [1,2,3], + "valid": true + }, + { + "description": "object is valid", + "data": {"foo": 123}, + "valid": true + }, + { + "description": "null is valid", + "data": null, + "valid": true + }, + { + "description": "number is invalid", + "data": 123, + "valid": false + }, + { + "description": "string is invalid", + "data": "foo", + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/uniqueItems.json b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/uniqueItems.json new file mode 100644 index 000000000..a1da0096e --- /dev/null +++ b/packages/schema/test/fixtures/json-schema-test-suite/tests/draft7/uniqueItems.json @@ -0,0 +1,389 @@ +[ + { + "description": "uniqueItems validation", + "schema": {"uniqueItems": true}, + "tests": [ + { + "description": "unique array of integers is valid", + "data": [1, 2], + "valid": true + }, + { + "description": "non-unique array of integers is invalid", + "data": [1, 1], + "valid": false + }, + { + "description": "non-unique array of more than two integers is invalid", + "data": [1, 2, 1], + "valid": false + }, + { + "description": "numbers are unique if mathematically unequal", + "data": [1.0, 1.00, 1], + "valid": false + }, + { + "description": "false is not equal to zero", + "data": [0, false], + "valid": true + }, + { + "description": "true is not equal to one", + "data": [1, true], + "valid": true + }, + { + "description": "unique array of strings is valid", + "data": ["foo", "bar", "baz"], + "valid": true + }, + { + "description": "non-unique array of strings is invalid", + "data": ["foo", "bar", "foo"], + "valid": false + }, + { + "description": "unique array of objects is valid", + "data": [{"foo": "bar"}, {"foo": "baz"}], + "valid": true + }, + { + "description": "non-unique array of objects is invalid", + "data": [{"foo": "bar"}, {"foo": "bar"}], + "valid": false + }, + { + "description": "property order of array of objects is ignored", + "data": [{"foo": "bar", "bar": "foo"}, {"bar": "foo", "foo": "bar"}], + "valid": false + }, + { + "description": "unique array of nested objects is valid", + "data": [ + {"foo": {"bar" : {"baz" : true}}}, + {"foo": {"bar" : {"baz" : false}}} + ], + "valid": true + }, + { + "description": "non-unique array of nested objects is invalid", + "data": [ + {"foo": {"bar" : {"baz" : true}}}, + {"foo": {"bar" : {"baz" : true}}} + ], + "valid": false + }, + { + "description": "unique array of arrays is valid", + "data": [["foo"], ["bar"]], + "valid": true + }, + { + "description": "non-unique array of arrays is invalid", + "data": [["foo"], ["foo"]], + "valid": false + }, + { + "description": "non-unique array of more than two arrays is invalid", + "data": [["foo"], ["bar"], ["foo"]], + "valid": false + }, + { + "description": "[1] and [true] are unique", + "data": [[1], [true]], + "valid": true + }, + { + "description": "[0] and [false] are unique", + "data": [[0], [false]], + "valid": true + }, + { + "description": "nested [1] and [true] are unique", + "data": [[[1], "foo"], [[true], "foo"]], + "valid": true + }, + { + "description": "nested [0] and [false] are unique", + "data": [[[0], "foo"], [[false], "foo"]], + "valid": true + }, + { + "description": "unique heterogeneous types are valid", + "data": [{}, [1], true, null, 1, "{}"], + "valid": true + }, + { + "description": "non-unique heterogeneous types are invalid", + "data": [{}, [1], true, null, {}, 1], + "valid": false + }, + { + "description": "different objects are unique", + "data": [{"a": 1, "b": 2}, {"a": 2, "b": 1}], + "valid": true + }, + { + "description": "objects are non-unique despite key order", + "data": [{"a": 1, "b": 2}, {"b": 2, "a": 1}], + "valid": false + }, + { + "description": "{\"a\": false} and {\"a\": 0} are unique", + "data": [{"a": false}, {"a": 0}], + "valid": true + }, + { + "description": "{\"a\": true} and {\"a\": 1} are unique", + "data": [{"a": true}, {"a": 1}], + "valid": true + } + ] + }, + { + "description": "uniqueItems with an array of items", + "schema": { + "items": [{"type": "boolean"}, {"type": "boolean"}], + "uniqueItems": true + }, + "tests": [ + { + "description": "[false, true] from items array is valid", + "data": [false, true], + "valid": true + }, + { + "description": "[true, false] from items array is valid", + "data": [true, false], + "valid": true + }, + { + "description": "[false, false] from items array is not valid", + "data": [false, false], + "valid": false + }, + { + "description": "[true, true] from items array is not valid", + "data": [true, true], + "valid": false + }, + { + "description": "unique array extended from [false, true] is valid", + "data": [false, true, "foo", "bar"], + "valid": true + }, + { + "description": "unique array extended from [true, false] is valid", + "data": [true, false, "foo", "bar"], + "valid": true + }, + { + "description": "non-unique array extended from [false, true] is not valid", + "data": [false, true, "foo", "foo"], + "valid": false + }, + { + "description": "non-unique array extended from [true, false] is not valid", + "data": [true, false, "foo", "foo"], + "valid": false + } + ] + }, + { + "description": "uniqueItems with an array of items and additionalItems=false", + "schema": { + "items": [{"type": "boolean"}, {"type": "boolean"}], + "uniqueItems": true, + "additionalItems": false + }, + "tests": [ + { + "description": "[false, true] from items array is valid", + "data": [false, true], + "valid": true + }, + { + "description": "[true, false] from items array is valid", + "data": [true, false], + "valid": true + }, + { + "description": "[false, false] from items array is not valid", + "data": [false, false], + "valid": false + }, + { + "description": "[true, true] from items array is not valid", + "data": [true, true], + "valid": false + }, + { + "description": "extra items are invalid even if unique", + "data": [false, true, null], + "valid": false + } + ] + }, + { + "description": "uniqueItems=false validation", + "schema": { "uniqueItems": false }, + "tests": [ + { + "description": "unique array of integers is valid", + "data": [1, 2], + "valid": true + }, + { + "description": "non-unique array of integers is valid", + "data": [1, 1], + "valid": true + }, + { + "description": "numbers are unique if mathematically unequal", + "data": [1.0, 1.00, 1], + "valid": true + }, + { + "description": "false is not equal to zero", + "data": [0, false], + "valid": true + }, + { + "description": "true is not equal to one", + "data": [1, true], + "valid": true + }, + { + "description": "unique array of objects is valid", + "data": [{"foo": "bar"}, {"foo": "baz"}], + "valid": true + }, + { + "description": "non-unique array of objects is valid", + "data": [{"foo": "bar"}, {"foo": "bar"}], + "valid": true + }, + { + "description": "unique array of nested objects is valid", + "data": [ + {"foo": {"bar" : {"baz" : true}}}, + {"foo": {"bar" : {"baz" : false}}} + ], + "valid": true + }, + { + "description": "non-unique array of nested objects is valid", + "data": [ + {"foo": {"bar" : {"baz" : true}}}, + {"foo": {"bar" : {"baz" : true}}} + ], + "valid": true + }, + { + "description": "unique array of arrays is valid", + "data": [["foo"], ["bar"]], + "valid": true + }, + { + "description": "non-unique array of arrays is valid", + "data": [["foo"], ["foo"]], + "valid": true + }, + { + "description": "unique heterogeneous types are valid", + "data": [{}, [1], true, null, 1], + "valid": true + }, + { + "description": "non-unique heterogeneous types are valid", + "data": [{}, [1], true, null, {}, 1], + "valid": true + } + ] + }, + { + "description": "uniqueItems=false with an array of items", + "schema": { + "items": [{"type": "boolean"}, {"type": "boolean"}], + "uniqueItems": false + }, + "tests": [ + { + "description": "[false, true] from items array is valid", + "data": [false, true], + "valid": true + }, + { + "description": "[true, false] from items array is valid", + "data": [true, false], + "valid": true + }, + { + "description": "[false, false] from items array is valid", + "data": [false, false], + "valid": true + }, + { + "description": "[true, true] from items array is valid", + "data": [true, true], + "valid": true + }, + { + "description": "unique array extended from [false, true] is valid", + "data": [false, true, "foo", "bar"], + "valid": true + }, + { + "description": "unique array extended from [true, false] is valid", + "data": [true, false, "foo", "bar"], + "valid": true + }, + { + "description": "non-unique array extended from [false, true] is valid", + "data": [false, true, "foo", "foo"], + "valid": true + }, + { + "description": "non-unique array extended from [true, false] is valid", + "data": [true, false, "foo", "foo"], + "valid": true + } + ] + }, + { + "description": "uniqueItems=false with an array of items and additionalItems=false", + "schema": { + "items": [{"type": "boolean"}, {"type": "boolean"}], + "uniqueItems": false, + "additionalItems": false + }, + "tests": [ + { + "description": "[false, true] from items array is valid", + "data": [false, true], + "valid": true + }, + { + "description": "[true, false] from items array is valid", + "data": [true, false], + "valid": true + }, + { + "description": "[false, false] from items array is valid", + "data": [false, false], + "valid": true + }, + { + "description": "[true, true] from items array is valid", + "data": [true, true], + "valid": true + }, + { + "description": "extra items are invalid even if unique", + "data": [false, true, null], + "valid": false + } + ] + } +] diff --git a/packages/schema/test/fixtures/meta-schemas/README.md b/packages/schema/test/fixtures/meta-schemas/README.md new file mode 100644 index 000000000..46b120d97 --- /dev/null +++ b/packages/schema/test/fixtures/meta-schemas/README.md @@ -0,0 +1,10 @@ +# Meta-schemas + +The official meta-schemas some suite tests reference, unmodified: + +- `draft2020-12-schema.json`: https://json-schema.org/draft/2020-12/schema +- `draft2020-12-meta-*.json`: https://json-schema.org/draft/2020-12/meta/{core,applicator,unevaluated,validation,meta-data,format-annotation,content} +- `draft-07-schema.json`: http://json-schema.org/draft-07/schema + +Published by the JSON Schema organization (https://github.com/json-schema-org/json-schema-spec), licensed +AFL-3.0 or BSD-3-Clause. diff --git a/packages/schema/test/fixtures/meta-schemas/draft-07-schema.json b/packages/schema/test/fixtures/meta-schemas/draft-07-schema.json new file mode 100644 index 000000000..fb92c7f75 --- /dev/null +++ b/packages/schema/test/fixtures/meta-schemas/draft-07-schema.json @@ -0,0 +1,172 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "$id": "http://json-schema.org/draft-07/schema#", + "title": "Core schema meta-schema", + "definitions": { + "schemaArray": { + "type": "array", + "minItems": 1, + "items": { "$ref": "#" } + }, + "nonNegativeInteger": { + "type": "integer", + "minimum": 0 + }, + "nonNegativeIntegerDefault0": { + "allOf": [ + { "$ref": "#/definitions/nonNegativeInteger" }, + { "default": 0 } + ] + }, + "simpleTypes": { + "enum": [ + "array", + "boolean", + "integer", + "null", + "number", + "object", + "string" + ] + }, + "stringArray": { + "type": "array", + "items": { "type": "string" }, + "uniqueItems": true, + "default": [] + } + }, + "type": ["object", "boolean"], + "properties": { + "$id": { + "type": "string", + "format": "uri-reference" + }, + "$schema": { + "type": "string", + "format": "uri" + }, + "$ref": { + "type": "string", + "format": "uri-reference" + }, + "$comment": { + "type": "string" + }, + "title": { + "type": "string" + }, + "description": { + "type": "string" + }, + "default": true, + "readOnly": { + "type": "boolean", + "default": false + }, + "writeOnly": { + "type": "boolean", + "default": false + }, + "examples": { + "type": "array", + "items": true + }, + "multipleOf": { + "type": "number", + "exclusiveMinimum": 0 + }, + "maximum": { + "type": "number" + }, + "exclusiveMaximum": { + "type": "number" + }, + "minimum": { + "type": "number" + }, + "exclusiveMinimum": { + "type": "number" + }, + "maxLength": { "$ref": "#/definitions/nonNegativeInteger" }, + "minLength": { "$ref": "#/definitions/nonNegativeIntegerDefault0" }, + "pattern": { + "type": "string", + "format": "regex" + }, + "additionalItems": { "$ref": "#" }, + "items": { + "anyOf": [ + { "$ref": "#" }, + { "$ref": "#/definitions/schemaArray" } + ], + "default": true + }, + "maxItems": { "$ref": "#/definitions/nonNegativeInteger" }, + "minItems": { "$ref": "#/definitions/nonNegativeIntegerDefault0" }, + "uniqueItems": { + "type": "boolean", + "default": false + }, + "contains": { "$ref": "#" }, + "maxProperties": { "$ref": "#/definitions/nonNegativeInteger" }, + "minProperties": { "$ref": "#/definitions/nonNegativeIntegerDefault0" }, + "required": { "$ref": "#/definitions/stringArray" }, + "additionalProperties": { "$ref": "#" }, + "definitions": { + "type": "object", + "additionalProperties": { "$ref": "#" }, + "default": {} + }, + "properties": { + "type": "object", + "additionalProperties": { "$ref": "#" }, + "default": {} + }, + "patternProperties": { + "type": "object", + "additionalProperties": { "$ref": "#" }, + "propertyNames": { "format": "regex" }, + "default": {} + }, + "dependencies": { + "type": "object", + "additionalProperties": { + "anyOf": [ + { "$ref": "#" }, + { "$ref": "#/definitions/stringArray" } + ] + } + }, + "propertyNames": { "$ref": "#" }, + "const": true, + "enum": { + "type": "array", + "items": true, + "minItems": 1, + "uniqueItems": true + }, + "type": { + "anyOf": [ + { "$ref": "#/definitions/simpleTypes" }, + { + "type": "array", + "items": { "$ref": "#/definitions/simpleTypes" }, + "minItems": 1, + "uniqueItems": true + } + ] + }, + "format": { "type": "string" }, + "contentMediaType": { "type": "string" }, + "contentEncoding": { "type": "string" }, + "if": { "$ref": "#" }, + "then": { "$ref": "#" }, + "else": { "$ref": "#" }, + "allOf": { "$ref": "#/definitions/schemaArray" }, + "anyOf": { "$ref": "#/definitions/schemaArray" }, + "oneOf": { "$ref": "#/definitions/schemaArray" }, + "not": { "$ref": "#" } + }, + "default": true +} diff --git a/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-applicator.json b/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-applicator.json new file mode 100644 index 000000000..f4775974a --- /dev/null +++ b/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-applicator.json @@ -0,0 +1,45 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://json-schema.org/draft/2020-12/meta/applicator", + "$dynamicAnchor": "meta", + + "title": "Applicator vocabulary meta-schema", + "type": ["object", "boolean"], + "properties": { + "prefixItems": { "$ref": "#/$defs/schemaArray" }, + "items": { "$dynamicRef": "#meta" }, + "contains": { "$dynamicRef": "#meta" }, + "additionalProperties": { "$dynamicRef": "#meta" }, + "properties": { + "type": "object", + "additionalProperties": { "$dynamicRef": "#meta" }, + "default": {} + }, + "patternProperties": { + "type": "object", + "additionalProperties": { "$dynamicRef": "#meta" }, + "propertyNames": { "format": "regex" }, + "default": {} + }, + "dependentSchemas": { + "type": "object", + "additionalProperties": { "$dynamicRef": "#meta" }, + "default": {} + }, + "propertyNames": { "$dynamicRef": "#meta" }, + "if": { "$dynamicRef": "#meta" }, + "then": { "$dynamicRef": "#meta" }, + "else": { "$dynamicRef": "#meta" }, + "allOf": { "$ref": "#/$defs/schemaArray" }, + "anyOf": { "$ref": "#/$defs/schemaArray" }, + "oneOf": { "$ref": "#/$defs/schemaArray" }, + "not": { "$dynamicRef": "#meta" } + }, + "$defs": { + "schemaArray": { + "type": "array", + "minItems": 1, + "items": { "$dynamicRef": "#meta" } + } + } +} diff --git a/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-content.json b/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-content.json new file mode 100644 index 000000000..76e3760d2 --- /dev/null +++ b/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-content.json @@ -0,0 +1,14 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://json-schema.org/draft/2020-12/meta/content", + "$dynamicAnchor": "meta", + + "title": "Content vocabulary meta-schema", + + "type": ["object", "boolean"], + "properties": { + "contentEncoding": { "type": "string" }, + "contentMediaType": { "type": "string" }, + "contentSchema": { "$dynamicRef": "#meta" } + } +} diff --git a/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-core.json b/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-core.json new file mode 100644 index 000000000..691862289 --- /dev/null +++ b/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-core.json @@ -0,0 +1,48 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://json-schema.org/draft/2020-12/meta/core", + "$dynamicAnchor": "meta", + + "title": "Core vocabulary meta-schema", + "type": ["object", "boolean"], + "properties": { + "$id": { + "$ref": "#/$defs/uriReferenceString", + "$comment": "Non-empty fragments not allowed.", + "pattern": "^[^#]*#?$" + }, + "$schema": { "$ref": "#/$defs/uriString" }, + "$ref": { "$ref": "#/$defs/uriReferenceString" }, + "$anchor": { "$ref": "#/$defs/anchorString" }, + "$dynamicRef": { "$ref": "#/$defs/uriReferenceString" }, + "$dynamicAnchor": { "$ref": "#/$defs/anchorString" }, + "$vocabulary": { + "type": "object", + "propertyNames": { "$ref": "#/$defs/uriString" }, + "additionalProperties": { + "type": "boolean" + } + }, + "$comment": { + "type": "string" + }, + "$defs": { + "type": "object", + "additionalProperties": { "$dynamicRef": "#meta" } + } + }, + "$defs": { + "anchorString": { + "type": "string", + "pattern": "^[A-Za-z_][-A-Za-z0-9._]*$" + }, + "uriString": { + "type": "string", + "format": "uri" + }, + "uriReferenceString": { + "type": "string", + "format": "uri-reference" + } + } +} diff --git a/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-format-annotation.json b/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-format-annotation.json new file mode 100644 index 000000000..3479e6695 --- /dev/null +++ b/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-format-annotation.json @@ -0,0 +1,11 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://json-schema.org/draft/2020-12/meta/format-annotation", + "$dynamicAnchor": "meta", + + "title": "Format vocabulary meta-schema for annotation results", + "type": ["object", "boolean"], + "properties": { + "format": { "type": "string" } + } +} diff --git a/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-meta-data.json b/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-meta-data.json new file mode 100644 index 000000000..4049ab21b --- /dev/null +++ b/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-meta-data.json @@ -0,0 +1,34 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://json-schema.org/draft/2020-12/meta/meta-data", + "$dynamicAnchor": "meta", + + "title": "Meta-data vocabulary meta-schema", + + "type": ["object", "boolean"], + "properties": { + "title": { + "type": "string" + }, + "description": { + "type": "string" + }, + "default": true, + "deprecated": { + "type": "boolean", + "default": false + }, + "readOnly": { + "type": "boolean", + "default": false + }, + "writeOnly": { + "type": "boolean", + "default": false + }, + "examples": { + "type": "array", + "items": true + } + } +} diff --git a/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-unevaluated.json b/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-unevaluated.json new file mode 100644 index 000000000..93779e54e --- /dev/null +++ b/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-unevaluated.json @@ -0,0 +1,12 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://json-schema.org/draft/2020-12/meta/unevaluated", + "$dynamicAnchor": "meta", + + "title": "Unevaluated applicator vocabulary meta-schema", + "type": ["object", "boolean"], + "properties": { + "unevaluatedItems": { "$dynamicRef": "#meta" }, + "unevaluatedProperties": { "$dynamicRef": "#meta" } + } +} diff --git a/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-validation.json b/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-validation.json new file mode 100644 index 000000000..ebb75db77 --- /dev/null +++ b/packages/schema/test/fixtures/meta-schemas/draft2020-12-meta-validation.json @@ -0,0 +1,95 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://json-schema.org/draft/2020-12/meta/validation", + "$dynamicAnchor": "meta", + + "title": "Validation vocabulary meta-schema", + "type": ["object", "boolean"], + "properties": { + "type": { + "anyOf": [ + { "$ref": "#/$defs/simpleTypes" }, + { + "type": "array", + "items": { "$ref": "#/$defs/simpleTypes" }, + "minItems": 1, + "uniqueItems": true + } + ] + }, + "const": true, + "enum": { + "type": "array", + "items": true + }, + "multipleOf": { + "type": "number", + "exclusiveMinimum": 0 + }, + "maximum": { + "type": "number" + }, + "exclusiveMaximum": { + "type": "number" + }, + "minimum": { + "type": "number" + }, + "exclusiveMinimum": { + "type": "number" + }, + "maxLength": { "$ref": "#/$defs/nonNegativeInteger" }, + "minLength": { "$ref": "#/$defs/nonNegativeIntegerDefault0" }, + "pattern": { + "type": "string", + "format": "regex" + }, + "maxItems": { "$ref": "#/$defs/nonNegativeInteger" }, + "minItems": { "$ref": "#/$defs/nonNegativeIntegerDefault0" }, + "uniqueItems": { + "type": "boolean", + "default": false + }, + "maxContains": { "$ref": "#/$defs/nonNegativeInteger" }, + "minContains": { + "$ref": "#/$defs/nonNegativeInteger", + "default": 1 + }, + "maxProperties": { "$ref": "#/$defs/nonNegativeInteger" }, + "minProperties": { "$ref": "#/$defs/nonNegativeIntegerDefault0" }, + "required": { "$ref": "#/$defs/stringArray" }, + "dependentRequired": { + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/stringArray" + } + } + }, + "$defs": { + "nonNegativeInteger": { + "type": "integer", + "minimum": 0 + }, + "nonNegativeIntegerDefault0": { + "$ref": "#/$defs/nonNegativeInteger", + "default": 0 + }, + "simpleTypes": { + "enum": [ + "array", + "boolean", + "integer", + "null", + "number", + "object", + "string" + ] + }, + "stringArray": { + "type": "array", + "items": { "type": "string" }, + "uniqueItems": true, + "default": [] + } + } +} diff --git a/packages/schema/test/fixtures/meta-schemas/draft2020-12-schema.json b/packages/schema/test/fixtures/meta-schemas/draft2020-12-schema.json new file mode 100644 index 000000000..d5e2d31c3 --- /dev/null +++ b/packages/schema/test/fixtures/meta-schemas/draft2020-12-schema.json @@ -0,0 +1,58 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://json-schema.org/draft/2020-12/schema", + "$vocabulary": { + "https://json-schema.org/draft/2020-12/vocab/core": true, + "https://json-schema.org/draft/2020-12/vocab/applicator": true, + "https://json-schema.org/draft/2020-12/vocab/unevaluated": true, + "https://json-schema.org/draft/2020-12/vocab/validation": true, + "https://json-schema.org/draft/2020-12/vocab/meta-data": true, + "https://json-schema.org/draft/2020-12/vocab/format-annotation": true, + "https://json-schema.org/draft/2020-12/vocab/content": true + }, + "$dynamicAnchor": "meta", + + "title": "Core and Validation specifications meta-schema", + "allOf": [ + {"$ref": "meta/core"}, + {"$ref": "meta/applicator"}, + {"$ref": "meta/unevaluated"}, + {"$ref": "meta/validation"}, + {"$ref": "meta/meta-data"}, + {"$ref": "meta/format-annotation"}, + {"$ref": "meta/content"} + ], + "type": ["object", "boolean"], + "$comment": "This meta-schema also defines keywords that have appeared in previous drafts in order to prevent incompatible extensions as they remain in common use.", + "properties": { + "definitions": { + "$comment": "\"definitions\" has been replaced by \"$defs\".", + "type": "object", + "additionalProperties": { "$dynamicRef": "#meta" }, + "deprecated": true, + "default": {} + }, + "dependencies": { + "$comment": "\"dependencies\" has been split and replaced by \"dependentSchemas\" and \"dependentRequired\" in order to serve their differing semantics.", + "type": "object", + "additionalProperties": { + "anyOf": [ + { "$dynamicRef": "#meta" }, + { "$ref": "meta/validation#/$defs/stringArray" } + ] + }, + "deprecated": true, + "default": {} + }, + "$recursiveAnchor": { + "$comment": "\"$recursiveAnchor\" has been replaced by \"$dynamicAnchor\".", + "$ref": "meta/core#/$defs/anchorString", + "deprecated": true + }, + "$recursiveRef": { + "$comment": "\"$recursiveRef\" has been replaced by \"$dynamicRef\".", + "$ref": "meta/core#/$defs/uriReferenceString", + "deprecated": true + } + } +} diff --git a/packages/schema/test/shared.test.ts b/packages/schema/test/shared.test.ts new file mode 100644 index 000000000..cf3480fd1 --- /dev/null +++ b/packages/schema/test/shared.test.ts @@ -0,0 +1,41 @@ +// The caches shared across compiled schemas hold nothing alive: entries go once their schemas are collected. + +import { setFlagsFromString } from "node:v8"; +import { runInNewContext } from "node:vm"; +import { describe, expect, it } from "vitest"; +import { compile } from "../src/index.ts"; +import { sharedEntries } from "../src/shared.ts"; + +setFlagsFromString("--expose-gc"); +const gc = runInNewContext("gc") as () => void; + +async function collectGarbage(): Promise { + for (let round = 0; round < 6; round++) { + await new Promise((resolve) => setImmediate(resolve)); + gc(); + } +} + +describe("shared leaf checks", () => { + it("share one check between equal leaf schemas", () => { + const a = compile({ type: "integer", minimum: 3, description: "a" }); + const b = compile({ type: "integer", minimum: 3, description: "b" }); + expect(a.check).toBe(b.check); + expect(compile({ type: "integer", minimum: 4 }).check).not.toBe(a.check); + }); + + it("are collected with the last schema that uses them, and their cache entries removed", async () => { + await collectGarbage(); + const before = sharedEntries(); + let validators: unknown[] | undefined = Array.from({ length: 50 }, (_, i) => + compile({ type: "string", minLength: 1000 + i, pattern: `^unique-${i}$` }), + ); + const check = new WeakRef((validators[0] as { check: object }).check); + expect(sharedEntries().checks).toBe(before.checks + 50); + expect(sharedEntries().regexes).toBe(before.regexes + 50); + validators = undefined; + await collectGarbage(); + expect(check.deref()).toBeUndefined(); + expect(sharedEntries()).toEqual(before); + }); +}); diff --git a/packages/schema/test/suite.ts b/packages/schema/test/suite.ts new file mode 100644 index 000000000..fac5bebbd --- /dev/null +++ b/packages/schema/test/suite.ts @@ -0,0 +1,77 @@ +// Loads the vendored JSON-Schema-Test-Suite: its test files, the remote documents and the official meta-schemas. + +import { readdirSync, readFileSync, statSync } from "node:fs"; +import { join, relative } from "node:path"; +import type { JsonSchema } from "../src/index.ts"; + +/** One test file: its path below `tests/` and its groups. */ +export interface SuiteFile { + readonly name: string; + readonly draft: "2020-12" | "7"; + readonly groups: readonly SuiteGroup[]; +} + +/** A schema with values and whether each is valid. */ +export interface SuiteGroup { + readonly description: string; + readonly schema: JsonSchema; + readonly tests: readonly { readonly description: string; readonly data: unknown; readonly valid: boolean }[]; +} + +const fixtures = join(import.meta.dirname, "fixtures"); +const root = join(fixtures, "json-schema-test-suite"); + +/** Every test file of drafts 2020-12 and 7. */ +export function suiteFiles(): SuiteFile[] { + const files: SuiteFile[] = []; + for (const [dir, draft] of [ + ["draft2020-12", "2020-12"], + ["draft7", "7"], + ] as const) { + for (const path of jsonFiles(join(root, "tests", dir))) { + const name = relative(join(root, "tests"), path).replace(/\.json$/, ""); + files.push({ name, draft, groups: JSON.parse(readFileSync(path, "utf8")) }); + } + } + return files; +} + +/** The remote documents and meta-schemas the tests reference, by URI (fresh copies). */ +export function suiteSchemas(): Record { + const schemas: Record = {}; + const remotes = join(root, "remotes"); + for (const path of jsonFiles(remotes)) { + schemas[`http://localhost:1234/${relative(remotes, path).replaceAll("\\", "/")}`] = readJson(path); + } + const meta = join(fixtures, "meta-schemas"); + schemas["http://json-schema.org/draft-07/schema"] = readJson(join(meta, "draft-07-schema.json")); + schemas["https://json-schema.org/draft/2020-12/schema"] = readJson(join(meta, "draft2020-12-schema.json")); + for (const vocabulary of [ + "core", + "applicator", + "unevaluated", + "validation", + "meta-data", + "format-annotation", + "content", + ]) { + schemas[`https://json-schema.org/draft/2020-12/meta/${vocabulary}`] = readJson( + join(meta, `draft2020-12-meta-${vocabulary}.json`), + ); + } + return schemas; +} + +function readJson(path: string): JsonSchema { + return JSON.parse(readFileSync(path, "utf8")); +} + +function jsonFiles(dir: string): string[] { + const out: string[] = []; + for (const entry of readdirSync(dir).sort()) { + const path = join(dir, entry); + if (statSync(path).isDirectory()) out.push(...jsonFiles(path)); + else if (entry.endsWith(".json")) out.push(path); + } + return out; +} diff --git a/packages/schema/test/validator.test.ts b/packages/schema/test/validator.test.ts new file mode 100644 index 000000000..a62b38e31 --- /dev/null +++ b/packages/schema/test/validator.test.ts @@ -0,0 +1,190 @@ +// compile() and Validator: checks, errors, values, options and schemas from other sources. + +import Type from "typebox"; +import { describe, expect, it } from "vitest"; +import { formats } from "../src/formats/index.ts"; +import { compile } from "../src/index.ts"; + +const read = { + type: "object", + properties: { path: { type: "string" }, offset: { type: "integer", minimum: 1 } }, + required: ["path"], + additionalProperties: false, +}; + +const block = { + anyOf: [ + { + type: "object", + properties: { kind: { const: "text" }, text: { type: "string" } }, + required: ["kind", "text"], + additionalProperties: false, + }, + { + type: "object", + properties: { kind: { const: "image" }, width: { type: "integer", minimum: 0 } }, + required: ["kind", "width"], + additionalProperties: false, + }, + ], +}; + +describe("errors", () => { + it("reports a missing field and an unknown field at the field", () => { + expect(compile(read).errors({ offset: 0, extra: true })).toEqual([ + { path: "/path", keyword: "required", message: "missing" }, + { path: "/offset", keyword: "minimum", message: "expected >= 1" }, + { path: "/extra", keyword: "additionalProperties", message: "unknown field" }, + ]); + }); + + it("is empty for a valid value", () => { + expect(compile(read).errors({ path: "a.ts" })).toEqual([]); + }); + + it("reports the branch a tagged union's tag selects", () => { + const validator = compile({ type: "array", items: block }); + expect( + validator.errors([ + { kind: "text", text: "a" }, + { kind: "image", width: -1 }, + ]), + ).toEqual([{ path: "/1/width", keyword: "minimum", message: "expected >= 0" }]); + expect(validator.errors([{ kind: "video" }])).toEqual([ + { path: "/0/kind", keyword: "anyOf", message: 'expected one of ["text","image"]' }, + ]); + }); + + it("escapes ~ and / in paths as JSON Pointer does", () => { + const validator = compile({ type: "object", properties: { "a/b~c": { type: "string" } } }); + expect(validator.errors({ "a/b~c": 1 })).toEqual([ + { path: "/a~1b~0c", keyword: "type", message: "expected string, got integer" }, + ]); + }); + + it("follows $ref into the referenced schema", () => { + const validator = compile({ + $defs: { point: { type: "object", properties: { x: { type: "number" } }, required: ["x"] } }, + type: "array", + items: { $ref: "#/$defs/point" }, + }); + expect(validator.errors([{ x: 1 }, {}])).toEqual([{ path: "/1/x", keyword: "required", message: "missing" }]); + }); +}); + +describe("values", () => { + it("treats a property set to undefined as absent", () => { + const validator = compile(read); + expect(validator.check({ path: "a.ts", offset: undefined })).toBe(true); + expect(validator.check({ path: undefined })).toBe(false); + }); + + it("checks bytes as Uint8Array with length bounds", () => { + const validator = compile({ type: "object", properties: { data: { type: "bytes", maxLength: 4 } } }); + expect(validator.check({ data: new Uint8Array(4) })).toBe(true); + expect(validator.check({ data: new Uint8Array(5) })).toBe(false); + expect(validator.check({ data: [1, 2] })).toBe(false); + expect(validator.errors({ data: new Uint8Array(5) })).toEqual([ + { path: "/data", keyword: "maxLength", message: "expected at most 4 bytes" }, + ]); + }); + + it("does not treat bytes as an object", () => { + expect(compile({ type: "object" }).check(new Uint8Array(1))).toBe(false); + }); + + it("checks properties named like Object.prototype members as own properties", () => { + const validator = compile({ + type: "object", + properties: { constructor: { type: "string" }, toString: { type: "number" } }, + required: ["constructor"], + additionalProperties: false, + }); + expect(validator.check({})).toBe(false); + expect(validator.check({ constructor: "a" })).toBe(true); + expect(validator.check({ constructor: "a", toString: "b" })).toBe(false); + }); + + it("does not accept every value for a type named like an Object.prototype member", () => { + expect(compile({ type: "constructor" }).check(1)).toBe(false); + }); + + it("checks a detached check function", () => { + const { check } = compile(read); + expect([{ path: "a" }, { path: 1 }].filter(check)).toEqual([{ path: "a" }]); + }); +}); + +describe("options", () => { + it("treats format as an annotation unless formats are given", () => { + const schema = { type: "string", format: "email" }; + expect(compile(schema).check("not an email")).toBe(true); + expect(compile(schema, { formats }).check("not an email")).toBe(false); + expect(compile(schema, { formats }).check("a@example.com")).toBe(true); + }); + + it("resolves $ref to registered documents", () => { + const point = { type: "object", properties: { x: { type: "number" } }, required: ["x"] }; + const validator = compile( + { $ref: "https://example.com/point.json" }, + { schemas: { "https://example.com/point.json": point } }, + ); + expect(validator.check({ x: 1 })).toBe(true); + expect(validator.check({})).toBe(false); + }); + + it("throws for a $ref that names nothing", () => { + expect(() => compile({ $ref: "https://example.com/missing.json" })).toThrow(/Unresolved \$ref/); + expect(() => compile({ $ref: "#/$defs/missing" })).toThrow(/Unresolved \$ref/); + }); + + it("applies draft 7 semantics: $ref overrides its siblings", () => { + const schema = { definitions: { s: { type: "string" } }, $ref: "#/definitions/s", maxLength: 1 }; + expect(compile(schema, { draft: "7" }).check("ab")).toBe(true); + expect(compile(schema).check("ab")).toBe(false); + }); +}); + +describe("schemas", () => { + it("compiles TypeBox schemas as they are", () => { + const validator = compile(Type.Object({ name: Type.String(), tags: Type.Array(Type.String()) })); + expect(validator.check({ name: "a", tags: [] })).toBe(true); + expect(validator.errors({ name: "a", tags: [1] })).toEqual([ + { path: "/tags/0", keyword: "type", message: "expected string, got integer" }, + ]); + }); + + it("compiles deep-frozen schemas and never modifies a schema", () => { + const schema = deepFreeze(structuredClone({ ...block, $defs: { s: { type: "string", pattern: "^a" } } })); + const before = JSON.stringify(schema); + const validator = compile(schema); + expect(validator.check({ kind: "text", text: "x" })).toBe(true); + expect(validator.errors({ kind: "image", width: -1 })).toHaveLength(1); + expect(JSON.stringify(schema)).toBe(before); + }); + + it("compiles a recursive schema", () => { + const tree = { + type: "object", + properties: { value: { type: "number" }, children: { type: "array", items: { $ref: "#" } } }, + required: ["value"], + }; + const validator = compile(tree); + expect(validator.check({ value: 1, children: [{ value: 2, children: [{ value: 3 }] }] })).toBe(true); + expect(validator.errors({ value: 1, children: [{ value: 2, children: [{}] }] })).toEqual([ + { path: "/children/0/children/0/value", keyword: "required", message: "missing" }, + ]); + }); + + it("throws for a malformed schema", () => { + expect(() => compile({ properties: { a: 1 } })).toThrow(/must be an object or a boolean/); + }); +}); + +function deepFreeze(value: T): T { + if (typeof value === "object" && value !== null && !Object.isFrozen(value)) { + Object.freeze(value); + for (const child of Object.values(value)) deepFreeze(child); + } + return value; +} diff --git a/packages/schema/tsconfig.build.json b/packages/schema/tsconfig.build.json new file mode 100644 index 000000000..695dd9ad2 --- /dev/null +++ b/packages/schema/tsconfig.build.json @@ -0,0 +1,9 @@ +{ + "extends": "../../tsconfig.base.json", + "compilerOptions": { + "outDir": "./dist", + "rootDir": "./src" + }, + "include": ["src/**/*.ts"], + "exclude": ["node_modules", "dist", "**/*.d.ts", "src/**/*.d.ts"] +} diff --git a/packages/schema/vitest.config.ts b/packages/schema/vitest.config.ts new file mode 100644 index 000000000..18733b4d4 --- /dev/null +++ b/packages/schema/vitest.config.ts @@ -0,0 +1,11 @@ +import { defineConfig } from "vitest/config"; + +export default defineConfig({ + test: { + globals: true, + environment: "node", + reporters: process.env.GITHUB_ACTIONS ? ["dot", "github-actions"] : ["dot"], + }, + resolve: { conditions: ["source"] }, + ssr: { resolve: { conditions: ["source"] } }, +}); diff --git a/scripts/check-entry-graphs.mjs b/scripts/check-entry-graphs.mjs index 8ec4f337e..54dd30c5d 100644 --- a/scripts/check-entry-graphs.mjs +++ b/scripts/check-entry-graphs.mjs @@ -26,6 +26,7 @@ const WORKSPACE = { "@earendil-works/pi-codemode": "packages/codemode/src", "@earendil-works/pi-telemetry": "packages/telemetry/src", "@earendil-works/pi-mcp": "packages/mcp/src", + "@earendil-works/pi-schema": "packages/schema/src", "@earendil-works/pi-tui": "packages/tui/src", }; @@ -34,6 +35,10 @@ const WORKSPACE = { * graph it is allowed to reach. */ const BUDGETS = { + "packages/schema": { + // The format checks (vendored from TypeBox, about 40 KB) load only through ./formats. + ".": { maxFiles: 10, forbid: ["schema/src/formats/"] }, + }, "packages/ai": { "./models": { // Request telemetry reaches pi-ai's telemetry.ts and pi-telemetry's root (index, noop, metrics, memory). diff --git a/tsconfig.json b/tsconfig.json index 7a9cef85c..96c54539c 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -34,6 +34,8 @@ "@earendil-works/pi-codemode/portable": ["./packages/codemode/src/portable.ts"], "@earendil-works/pi-codemode/source": ["./packages/codemode/src/source.ts"], "@earendil-works/pi-codemode/worker": ["./packages/codemode/src/runtime/worker.ts"], + "@earendil-works/pi-schema": ["./packages/schema/src/index.ts"], + "@earendil-works/pi-schema/formats": ["./packages/schema/src/formats/index.ts"], "@earendil-works/pi-mcp": ["./packages/mcp/src/index.ts"], "@earendil-works/pi-mcp/oauth": ["./packages/mcp/src/oauth/index.ts"], "@earendil-works/pi-mcp/testing": ["./packages/mcp/src/testing/index.ts"], From 6d4b7443637356fe928118df3f1f6d922fcc191e Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 21:47:22 +0000 Subject: [PATCH 111/175] docs(chord): the schema package is built --- packages/chord/docs/next/decisions.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index e7aed524b..ca6ad2950 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2119,6 +2119,14 @@ validation and copies the v2 kernel added around Delta. is deleting that subpath). Not omptype (its validator compiles with `new Function`; its 10.7k lines are mostly ArkType's API). Open: which validator (measure `@cfworker/json-schema` and TypeBox against the official JSON-Schema-Test-Suite and Chord's argument checks first), and moving `Type` out of pi-ai's main export. + **Built** (owner, 2026-10-31: "clone pi add the package, done"; pi `fab2c53`): `packages/schema`, + `@earendil-works/pi-schema`: `compile(schema, { draft, formats, schemas })` with `check` (narrowing) and `errors` + (JSON Pointer paths), Chord-style builders typed with `Static`, formats as the opt-in `./formats` entry, leaf and regex + checks shared through weak caches (GC-tested); its own closure-compiled validator, chosen after measuring + (`/tmp/schema-measure.md`, Radius 01m69x3dz2fsg9k1r33vde7hv6): 2,222 of 2,223 core suite tests (TypeBox 2,200, + cfworker 2,167, omptype 1,522), Chord checks at 0.95-1.3x TypeBox's compiled speed and the same with `new Function` + blocked, 7.9 MB for 3,200 distinct schemas (cfworker 7.6, TypeBox 54.7), 8.5 KB gzip. Still open: switching Chord v2 + and pi-ai to it (`Type` out of pi-ai's main export), formats on by default in pi-ai. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 8ffea8a1bfdc32523ab056549b26141018c1e700 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 21:59:21 +0000 Subject: [PATCH 112/175] feat(schema): faster object checks and compile, less memory - Declared-object checks walk the value's own keys with for...in, whose property reads V8 serves from the enum cache, instead of reading each declared name from a list: nested Chord-style arguments 535 -> 409 ns (TypeBox 474), recursive $ref schemas 1.1 us -> 753 ns. - Schemas are indexed for references only on the first $ref; base URIs are derived from $id while compiling. Compiling a new schema 43 -> 21 us. - Drop the keyed cache of leaf checks: on distinct schemas its entries cost more than sharing saved. Bare types still share module-level checks and patterns share their RegExp. 3,200 distinct schemas 7.9 -> 7.1 MB (cfworker 7.6 MB). --- packages/schema/README.md | 22 +++---- packages/schema/src/checks.ts | 41 ++++++------ packages/schema/src/compiler.ts | 97 +++++++++++------------------ packages/schema/src/explain.ts | 2 +- packages/schema/src/resolver.ts | 39 ++++++++++-- packages/schema/src/shared.ts | 29 ++------- packages/schema/test/shared.test.ts | 31 ++++----- 7 files changed, 125 insertions(+), 136 deletions(-) diff --git a/packages/schema/README.md b/packages/schema/README.md index d31caa521..023ebdc02 100644 --- a/packages/schema/README.md +++ b/packages/schema/README.md @@ -134,22 +134,22 @@ Node 24.4.1 on one core of an i9-13900, against TypeBox 1.3.27, @cfworker/json-s | check, valid value | pi-schema | TypeBox | cfworker | omptype | |---|---:|---:|---:|---:| -| nested object with tagged unions | 535 ns | 464 ns | 43 µs | 713 ns | -| union of 5 messages | 96 ns | 94 ns | 10 µs | 94 ns | -| tool schema (read) | 34 ns | 9 ns | 2.0 µs | 8 ns | -| MCP tool schema with `$ref` | 207 ns | 88 ns | 13 µs | 1.7 µs | -| recursive `$ref` schema | 1.1 µs | 1.2 µs | 41 µs | 7.4 µs | +| nested object with tagged unions | 409 ns | 474 ns | 44 µs | 689 ns | +| union of 5 messages | 89 ns | 95 ns | 10 µs | 93 ns | +| tool schema (read) | 31 ns | 9 ns | 1.9 µs | 9 ns | +| MCP tool schema with `$ref` | 162 ns | 89 ns | 13 µs | 1.8 µs | +| recursive `$ref` schema | 753 ns | 1.2 µs | 41 µs | 7.4 µs | | same, without eval (strict CSP) | pi-schema | TypeBox | cfworker | omptype | |---|---:|---:|---:|---:| -| nested object with tagged unions | 527 ns | 42 µs | 44 µs | 2.7 µs | -| MCP tool schema with `$ref` | 209 ns | 38 µs | 13 µs | 2.1 µs | -| recursive `$ref` schema | 1.1 µs | 134 µs | 43 µs | 8.9 µs | +| nested object with tagged unions | 407 ns | 42 µs | 43 µs | 2.6 µs | +| MCP tool schema with `$ref` | 162 ns | 36 µs | 12 µs | 2.0 µs | +| recursive `$ref` schema | 750 ns | 124 µs | 41 µs | 9.0 µs | | cost | pi-schema | TypeBox | cfworker | omptype | |---|---:|---:|---:|---:| -| compile a new schema | 43 µs | 210 µs | 23 µs | 19 µs + 114 µs on its first check | -| memory, 3,200 distinct schemas | 7.9 MB | 54.7 MB | 7.6 MB | 20.0 MB | +| compile a new schema | 21 µs | 203 µs | 20 µs | 26 µs + 106 µs on its first check | +| memory, 3,200 distinct schemas | 7.1 MB | 54.7 MB | 7.6 MB | 20.0 MB | | bundle, minified + gzip | 8.5 KB | 35.2 KB | 5.8 KB | 38.8 KB | Reproduce with `npm run bench` in `packages/schema` (about 5 minutes). It runs every library through its fastest @@ -167,7 +167,7 @@ boolean path and keeps the object each returns. The scripts are in `test/bench/` Where code generation is unavailable, TypeBox falls back to its interpreter, and omptype throws `EvalError` once per schema, then validates through its interpreter. -Flat objects of simple types check in 15–90 ns, against 8–15 ns for TypeBox. Without code generation, reading a +Flat objects of simple types check in 13–80 ns, against 7–15 ns for TypeBox. Without code generation, reading a property whose name comes from a list costs a few nanoseconds. ## Development diff --git a/packages/schema/src/checks.ts b/packages/schema/src/checks.ts index b686c4c27..701707a28 100644 --- a/packages/schema/src/checks.ts +++ b/packages/schema/src/checks.ts @@ -206,36 +206,37 @@ export function guardArray(inner: Check, strict: boolean): Check { } /** - * An object with only declared properties (`additionalProperties: false`). `entries` is `[key, check, flags]` per - * property. Declared fields are checked, then the own keys are counted, so unknown keys are found without - * allocating. A property whose value is `undefined` counts as absent. + * An object with declared properties: `entries` is `[key, check, flags]` per property. The value's own keys are + * walked with for...in (whose `o[k]` V8 reads from the enum cache, unlike a load by a name from a list), each found + * among the declared names; `closed` rejects any other key. A property whose value is `undefined` counts as absent. */ -export function closedObject(entries: readonly unknown[], guard: Guard): Check { +export function declaredObject( + entries: readonly unknown[], + requiredCount: number, + closed: boolean, + guard: Guard, +): Check { + const n = entries.length; return (v) => { if (guard !== GUARD_NONE && !isObject(v)) return guard === GUARD_LOOSE; // Guarded: by the check above, or by the caller for GUARD_NONE. const o = v as Record; - let present = 0; - for (let i = 0; i < entries.length; i += 3) { - const k = entries[i] as string; + let required = 0; + for (const k in o) { + let i = 0; + while (i < n && entries[i] !== k) i += 3; + if (i === n) { + if (closed) return false; + continue; + } const x = o[k]; + if (x === undefined) continue; const f = entries[i + 2] as number; - if (x === undefined) { - if (f & REQUIRED) return false; - if (Object.hasOwn(o, k)) present++; - continue; - } - if (f & INHERITED && !Object.hasOwn(o, k)) { - if (f & REQUIRED) return false; - continue; - } - present++; const c = f & CODE; if (c !== 0 ? !leafOk(c, x) : !(entries[i + 1] as Check)(x)) return false; + if (f & REQUIRED) required++; } - let count = 0; - for (const _ in o) if (++count > present) return false; - return true; + return required === requiredCount; }; } diff --git a/packages/schema/src/compiler.ts b/packages/schema/src/compiler.ts index 01ec52bbf..8140e38d5 100644 --- a/packages/schema/src/compiler.ts +++ b/packages/schema/src/compiler.ts @@ -25,9 +25,9 @@ import { and, anyCheck, bytesCheck, - closedObject, constCheck, containsCheck, + declaredObject, dependentRequiredCheck, dependentSchemasCheck, enumCheck, @@ -56,12 +56,13 @@ import { REQUIRED, scopedCheck, stringCheck, + TYPE_CHECKS, taggedCheck, typeCheck, uniqueCheck, } from "./checks.ts"; import { documentOf, isSchemaObject, Resolver, resolveUri } from "./resolver.ts"; -import { sharedCheck, sharedRegex } from "./shared.ts"; +import { sharedRegex } from "./shared.ts"; import type { Check, CompileOptions, FormatCheck, JsonSchema, SchemaObject } from "./types.ts"; /** Thrown when a compile without dynamic scope tracking meets a bookended `$dynamicRef`: compile again with it. */ @@ -101,15 +102,11 @@ export class Compiler { throw new Error(`A schema must be an object or a boolean, got ${JSON.stringify(node)}`); const cached = this.cache.get(node); if (cached !== undefined) return cached; - const key = this.leafKey(node); - if (key !== undefined) { - const shared = sharedCheck(key, () => this.plain(node, base)); - this.cache.set(node, shared); - return shared; - } + const bare = this.bareType(node); + if (bare !== undefined) return bare; const slot: { check: Check | undefined } = { check: undefined }; this.cache.set(node, indirect(slot)); - const check = this.build(node, this.r.baseOf.get(node) ?? base); + const check = this.build(node, this.r.baseFor(node, base)); slot.check = check; this.cache.set(node, check); return check; @@ -125,18 +122,17 @@ export class Compiler { } /** - * The sharing key of a leaf schema (only assertion keywords; no subschemas, references or identifiers): equal leaves - * in all compiled schemas share one check. Undefined for every other schema. + * The module-level check of a schema that is only a `type` (with annotations), `ALWAYS` for one with no assertion + * at all; undefined for every other schema. These are the most common subschemas and cost nothing to share. */ - private leafKey(node: SchemaObject): string | undefined { - const picked: Record = {}; - for (const k of Object.keys(node)) { - if (LEAF_KEYWORDS.has(k)) picked[k] = node[k]; + private bareType(node: SchemaObject): Check | undefined { + let type: unknown; + for (const k in node) { + if (k === "type") type = node.type; else if (!NEUTRAL_KEYWORDS.has(k)) return undefined; } - // A format checks through this compile's format map, so a leaf with one is shared only while formats are off. - if (picked.format !== undefined && this.formats !== undefined) return undefined; - return this.draft + JSON.stringify(picked); + if (type === undefined) return ALWAYS; + return typeof type === "string" ? TYPE_CHECKS.get(type) : undefined; } /** Every keyword of `node`, not tracking annotations. */ @@ -149,20 +145,20 @@ export class Compiler { parts.push(bytesCheck(number(node.minLength) ?? 0, number(node.maxLength) ?? Number.POSITIVE_INFINITY)); typeInGroup = true; } else { - const groups: [Check | undefined, boolean][] = [ - [this.stringGroup(node, single === "string"), single === "string"], - [ - this.numberGroup(node, single === "number" ? 1 : single === "integer" ? 2 : 0), - single === "number" || single === "integer", - ], - [this.objectGroup(node, base, single === "object"), single === "object"], - [this.arrayGroup(node, base, single === "array"), single === "array"], - ]; - for (const [group, strict] of groups) { - if (group === undefined) continue; - parts.push(group); - if (strict) typeInGroup = true; - } + const string = this.stringGroup(node, single === "string"); + if (string !== undefined) parts.push(string); + const numeric = single === "number" || single === "integer"; + const number = this.numberGroup(node, single === "number" ? 1 : single === "integer" ? 2 : 0); + if (number !== undefined) parts.push(number); + const object = this.objectGroup(node, base, single === "object"); + if (object !== undefined) parts.push(object); + const array = this.arrayGroup(node, base, single === "array"); + if (array !== undefined) parts.push(array); + typeInGroup = + (single === "string" && string !== undefined) || + (numeric && number !== undefined) || + (single === "object" && object !== undefined) || + (single === "array" && array !== undefined); } if (types !== undefined && !typeInGroup) parts.unshift(typeCheck(types)); this.inPlace(node, base, parts); @@ -255,7 +251,7 @@ export class Compiler { for (const branch of branches) { if (!isSchemaObject(branch)) return undefined; const ref = typeof branch.$ref === "string" && Object.keys(branch).length === 1 ? branch.$ref : undefined; - const node = ref === undefined ? branch : this.r.resolve(ref, this.r.baseOf.get(branch) ?? base).node; + const node = ref === undefined ? branch : this.r.resolve(ref, this.r.baseFor(branch, base)).node; if (!isSchemaObject(node)) return undefined; resolved.push(node); } @@ -350,8 +346,11 @@ export class Compiler { } const extraRequired = required.filter((k) => !Object.hasOwn(properties, k)); const pats = patternKeys.map((p) => [sharedRegex(p), this.check(patterns[p], base)] as const); - if (additional === false && pats.length === 0 && extraRequired.length === 0) return closedObject(entries, guard); const extra = additional === undefined || additional === true ? undefined : this.check(additional, base); + if (pats.length === 0 && extraRequired.length === 0 && (extra === undefined || additional === false)) { + const requiredCount = keys.filter((k) => required.includes(k)).length; + return declaredObject(entries, requiredCount, additional === false, guard); + } return openObject(entries, keys, extraRequired, pats, extra, guard); } @@ -404,8 +403,7 @@ export class Compiler { if (cached !== undefined) return cached; const slot: { check: AnnotatingCheck | undefined } = { check: undefined }; this.annotatingCache.set(node, indirectAnnotating(slot)); - const b = this.r.baseOf.get(node) ?? base; - const check = this.buildAnnotating(node, b); + const check = this.buildAnnotating(node, this.r.baseFor(node, base)); slot.check = check; this.annotatingCache.set(node, check); return check; @@ -510,29 +508,6 @@ export class Compiler { } } -// Keywords whose check depends only on their own value, and keywords that never affect a check. -const LEAF_KEYWORDS = new Set([ - "type", - "enum", - "const", - "minLength", - "maxLength", - "pattern", - "format", - "minimum", - "maximum", - "exclusiveMinimum", - "exclusiveMaximum", - "multipleOf", - "minItems", - "maxItems", - "uniqueItems", - "minProperties", - "maxProperties", - "required", - "dependentRequired", -]); - /** Keywords that never affect a check. */ export const NEUTRAL_KEYWORDS = new Set([ "$schema", @@ -587,7 +562,7 @@ const ANNOTATING = new Set([ "unevaluatedItems", ]); -/** anyOf/oneOf whose branches are all `{ const: primitive }`: one shared enum (distinct values make oneOf the same). */ +/** anyOf/oneOf whose branches are all `{ const: primitive }`: an enum (distinct values make oneOf the same). */ function constUnion(branches: readonly unknown[], exactlyOne: boolean): Check | undefined { const values: unknown[] = []; for (const branch of branches) { @@ -597,7 +572,7 @@ function constUnion(branches: readonly unknown[], exactlyOne: boolean): Check | if (exactlyOne && values.includes(branch.const)) return undefined; values.push(branch.const); } - return sharedCheck(`enum${JSON.stringify(values)}`, () => enumCheck(values)); + return enumCheck(values); } // The tag of a tagged-union branch: a primitive const on a required property of an object schema. diff --git a/packages/schema/src/explain.ts b/packages/schema/src/explain.ts index fc8068a26..da55edb43 100644 --- a/packages/schema/src/explain.ts +++ b/packages/schema/src/explain.ts @@ -49,7 +49,7 @@ export class Explainer { return; } if (!isSchemaObject(node)) return; - const b = this.c.r.baseOf.get(node) ?? base; + const b = this.c.r.baseFor(node, base); if (this.ok(node, b, v)) return; const start = out.length; const push = (keyword: string, message: string, at = path) => { diff --git a/packages/schema/src/resolver.ts b/packages/schema/src/resolver.ts index a10d245b6..9541d2c33 100644 --- a/packages/schema/src/resolver.ts +++ b/packages/schema/src/resolver.ts @@ -12,7 +12,10 @@ export interface Resolved { /** The base URI of a root schema without `$id`; relative references resolve against it. */ export const DEFAULT_BASE = "https://json-schema.invalid/root.json"; -/** Indexes the root schema and resolves references in it and in the registered documents. */ +/** + * Resolves references in the root schema and the registered documents. The schema is indexed on the first + * reference, so a schema without references is never walked for identifiers. + */ export class Resolver { readonly draft: "2020-12" | "7"; readonly rootBase: string; @@ -20,22 +23,47 @@ export class Resolver { readonly baseOf = new Map(); /** URI (with a `#name` fragment) of every `$dynamicAnchor`, to its schema. */ readonly dynamicAnchors = new Map(); + private readonly root: JsonSchema; + private readonly options: CompileOptions; private readonly registry = new Map(); private readonly resources = new Map(); private readonly anchors = new Map(); + private indexed = false; constructor(root: JsonSchema, options: CompileOptions) { this.draft = options.draft === "7" ? "7" : "2020-12"; - for (const [uri, schema] of Object.entries(options.schemas ?? {})) { + this.root = root; + this.options = options; + this.rootBase = isSchemaObject(root) ? this.baseFor(root, DEFAULT_BASE) : DEFAULT_BASE; + } + + /** + * The base URI of `node`, whose parent has `base`: its `$id` changes it, unless draft 7 makes the `$id` inert. Once + * the schema is indexed, the indexed base (a resolved reference target already has its own). + */ + baseFor(node: SchemaObject, base: string): string { + const indexed = this.baseOf.get(node); + if (indexed !== undefined) return indexed; + const id = node.$id; + if (typeof id !== "string") return base; + if (this.draft === "7" && (typeof node.$ref === "string" || id.startsWith("#"))) return base; + return documentOf(resolveUri(id, base)); + } + + /** Indexes the root schema and the registered documents' URIs (once). */ + ensureIndexed(): void { + if (this.indexed) return; + this.indexed = true; + for (const [uri, schema] of Object.entries(this.options.schemas ?? {})) { this.registry.set(documentOf(resolveUri(uri, DEFAULT_BASE)), schema); } - this.resources.set(DEFAULT_BASE, root); - this.index(root, DEFAULT_BASE); - this.rootBase = (isSchemaObject(root) && this.baseOf.get(root)) || DEFAULT_BASE; + this.resources.set(DEFAULT_BASE, this.root); + this.index(this.root, DEFAULT_BASE); } /** The schema a reference names, with the base URI it compiles under. Throws if it names nothing. */ resolve(ref: string, base: string): Resolved { + this.ensureIndexed(); const uri = resolveUri(ref, base); const doc = documentOf(uri); const fragment = uri.length > doc.length ? decodeURIComponent(uri.slice(doc.length + 1)) : ""; @@ -50,6 +78,7 @@ export class Resolver { /** Whether `node` is the root of the schema resource at `base`. */ isResource(node: object, base: string): boolean { + this.ensureIndexed(); return this.resources.get(base) === node; } diff --git a/packages/schema/src/shared.ts b/packages/schema/src/shared.ts index bb3867d3d..42c30152f 100644 --- a/packages/schema/src/shared.ts +++ b/packages/schema/src/shared.ts @@ -1,23 +1,6 @@ -// Values shared across all compiled schemas while one still uses them: leaf checks and regular expressions. -// Both caches hold their values weakly and drop an entry once its value is collected, so they keep nothing alive. - -import type { Check } from "./types.ts"; - -/** The check shared under `key`, made by `make` when no live schema holds one. */ -export function sharedCheck(key: string, make: () => Check): Check { - const hit = checks.get(key)?.deref(); - if (hit !== undefined) return hit; - const check = make(); - checks.set(key, new WeakRef(check)); - checkCleanup.register(check, key); - return check; -} - -const checks = new Map>(); - -const checkCleanup = new FinalizationRegistry((key) => { - if (checks.get(key)?.deref() === undefined) checks.delete(key); -}); +// Regular expressions shared across all compiled schemas while one still uses them: tool schemas repeat a few +// patterns many times, and a RegExp is large once it has run. The cache holds its values weakly and drops an entry +// once its RegExp is collected, so it keeps nothing alive. /** * The regular expression of a `pattern` source, shared by source. Patterns are ECMAScript regular expressions with @@ -44,7 +27,7 @@ const regexCleanup = new FinalizationRegistry((source) => { if (regexes.get(source)?.deref() === undefined) regexes.delete(source); }); -/** The number of entries in both caches (for tests). */ -export function sharedEntries(): { checks: number; regexes: number } { - return { checks: checks.size, regexes: regexes.size }; +/** The number of cached regular expressions (for tests). */ +export function sharedRegexCount(): number { + return regexes.size; } diff --git a/packages/schema/test/shared.test.ts b/packages/schema/test/shared.test.ts index cf3480fd1..ba3e4ab30 100644 --- a/packages/schema/test/shared.test.ts +++ b/packages/schema/test/shared.test.ts @@ -1,10 +1,10 @@ -// The caches shared across compiled schemas hold nothing alive: entries go once their schemas are collected. +// Values shared across compiled schemas hold nothing alive: a cached pattern goes once its schemas are collected. import { setFlagsFromString } from "node:v8"; import { runInNewContext } from "node:vm"; import { describe, expect, it } from "vitest"; import { compile } from "../src/index.ts"; -import { sharedEntries } from "../src/shared.ts"; +import { sharedRegexCount } from "../src/shared.ts"; setFlagsFromString("--expose-gc"); const gc = runInNewContext("gc") as () => void; @@ -16,26 +16,27 @@ async function collectGarbage(): Promise { } } -describe("shared leaf checks", () => { - it("share one check between equal leaf schemas", () => { - const a = compile({ type: "integer", minimum: 3, description: "a" }); - const b = compile({ type: "integer", minimum: 3, description: "b" }); +describe("shared checks", () => { + it("share the module-level check of a bare type", () => { + const a = compile({ type: "string", description: "a" }); + const b = compile({ type: "string", title: "b" }); expect(a.check).toBe(b.check); - expect(compile({ type: "integer", minimum: 4 }).check).not.toBe(a.check); }); - it("are collected with the last schema that uses them, and their cache entries removed", async () => { + it("share a pattern's regular expression and drop it with the last schema that uses it", async () => { await collectGarbage(); - const before = sharedEntries(); - let validators: unknown[] | undefined = Array.from({ length: 50 }, (_, i) => - compile({ type: "string", minLength: 1000 + i, pattern: `^unique-${i}$` }), + const before = sharedRegexCount(); + let validators: { check: object }[] | undefined = Array.from({ length: 50 }, (_, i) => + compile({ type: "string", pattern: `^unique-${i}$` }), ); - const check = new WeakRef((validators[0] as { check: object }).check); - expect(sharedEntries().checks).toBe(before.checks + 50); - expect(sharedEntries().regexes).toBe(before.regexes + 50); + const again = compile({ type: "string", pattern: "^unique-0$" }); + expect(sharedRegexCount()).toBe(before + 50); + expect(again.check("unique-0")).toBe(true); + const check = new WeakRef(validators[0].check); validators = undefined; await collectGarbage(); + // `again` still uses ^unique-0$; the other 49 patterns are gone. expect(check.deref()).toBeUndefined(); - expect(sharedEntries()).toEqual(before); + expect(sharedRegexCount()).toBe(before + 1); }); }); From 2c8d1feb147f6b3d97e855da803bfe7b19f3c403 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sun, 1 Nov 2026 00:36:33 +0000 Subject: [PATCH 113/175] docs(chord): the schema switch decisions --- packages/chord/docs/next/decisions.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index ca6ad2950..7dd53c73b 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2127,6 +2127,19 @@ validation and copies the v2 kernel added around Delta. cfworker 2,167, omptype 1,522), Chord checks at 0.95-1.3x TypeBox's compiled speed and the same with `new Function` blocked, 7.9 MB for 3,200 distinct schemas (cfworker 7.6, TypeBox 54.7), 8.5 KB gzip. Still open: switching Chord v2 and pi-ai to it (`Type` out of pi-ai's main export), formats on by default in pi-ai. +- **The schema switch (owner, 2026-10-31):** TypeBox leaves Chord (v1 too: "remove typebox wherever we can"), + pi-ai, pi-agent-core and pi-durable; extensions must not break ("absolutely 100%"), proven by a guard over our own + extensions, built-in tools and generated arguments (verdicts, coerced arguments, provider payloads and inferred + parameter types identical; owner: "just use ours", no npm corpus). **Coercion lives in pi-schema** as an opt-in, + the superset of TypeBox's `Value.Convert` rules and pi-ai's own (TypeBox's rule first where they conflict): + `compile(schema, { coerce: true })` adds `coerce(value)` (a copy, never mutating) and `parse(value)` (`{ ok, value } + | { ok, errors }`); pi-ai uses it for every tool, whatever built the schema (so plain-JSON MCP tools gain the + TypeBox rules). **Error texts are pi-schema's own** (no mapping to TypeBox's wording). Extensions keep `Type`, + `Static`, `TSchema` from the coding agent's bundled TypeBox (the pi-ai they see is compat plus those); an extension + author type-checking against the published pi-ai no longer finds them there (accepted: "fine"). **`format` is an + annotation in Chord** ("no format check": no format module in Chord's or Durable's entries; pi-ai still checks + formats on tool arguments). **Chord's envelope check costs +10 to +100 ns per message** from untrusted links + (pi-schema reads fields by name from a list; no eval): accepted ("negligible"), no hand-written check. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 6536f80129a266c3619c044f8487c0279fc39cd9 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sun, 1 Nov 2026 10:08:10 +0000 Subject: [PATCH 114/175] feat(ai,agent,coding-agent): validate tool arguments with pi-schema, drop TypeBox pi-schema: opt-in coercion, compile(schema, { coerce: true }) with coerce() and parse(), the superset of TypeBox's Value.Convert rules and pi-ai's old coercion; unresolvedRefs option; url and json-pointer-uri-fragment formats. pi-ai: Tool takes any JSON Schema; validation through pi-schema; Type, Static and TSchema are no longer exported (breaking). pi-agent-core and pi-durable: tools on pi-schema, no TypeBox dependency. Coding agent: extensions keep Type, Static, TSchema and StringEnum through an extension view of pi-ai; built-in tools stay on TypeBox. --- package-lock.json | 11 +- package.json | 2 +- packages/agent/CHANGELOG.md | 4 + packages/agent/README.md | 14 +- packages/agent/examples/mcp-codemode/tools.ts | 17 +- packages/agent/package.json | 2 +- packages/agent/src/types.ts | 24 +- packages/agent/test/agent-loop.test.ts | 52 +- packages/agent/test/agent.test.ts | 22 +- packages/agent/test/utils/calculate.ts | 10 +- packages/agent/test/utils/get-current-time.ts | 12 +- packages/ai/CHANGELOG.md | 25 + packages/ai/README.md | 51 +- packages/ai/package.json | 5 +- packages/ai/src/index.ts | 5 +- packages/ai/src/providers/compat-schema.ts | 855 ++++++----- packages/ai/src/providers/model-schema.ts | 110 +- packages/ai/src/providers/schema-builders.ts | 34 + packages/ai/src/types.ts | 8 +- .../{typebox-helpers.ts => string-enum.ts} | 14 +- packages/ai/src/utils/transcript.ts | 2 +- packages/ai/src/utils/validation.ts | 340 +---- packages/ai/test/compat-schema.test.ts | 31 +- packages/ai/test/faux-provider.test.ts | 2 +- packages/ai/test/interleaved-thinking.test.ts | 2 +- packages/ai/test/stream.test.ts | 2 +- packages/ai/test/validation-probes.test.ts | 779 ++++++++++ packages/ai/test/validation.test.ts | 42 +- packages/ai/vitest.config.ts | 3 + packages/codemode/CHANGELOG.md | 4 + packages/codemode/README.md | 4 +- packages/coding-agent/CHANGELOG.md | 2 + packages/coding-agent/docs/extensions.md | 14 +- packages/coding-agent/examples/tsconfig.json | 180 +++ .../install-lock/package-lock.json | 14 +- .../coding-agent/schemas/models.schema.json | 72 +- .../src/core/extensions/loader.ts | 9 +- .../core/extensions/pi-ai-extension-view.ts | 28 + .../src/core/extensions/virtual-modules.ts | 15 +- .../coding-agent/src/core/model-config.ts | 19 +- .../coding-agent/src/core/pi-ai-schemas.ts | 40 + .../coding-agent/src/core/settings-schema.ts | 2 +- packages/coding-agent/src/core/tools/bash.ts | 5 +- packages/coding-agent/src/core/tools/edit.ts | 5 +- packages/coding-agent/src/core/tools/find.ts | 5 +- packages/coding-agent/src/core/tools/grep.ts | 5 +- packages/coding-agent/src/core/tools/ls.ts | 5 +- packages/coding-agent/src/core/tools/read.ts | 5 +- .../src/core/tools/tool-definition-wrapper.ts | 21 +- packages/coding-agent/src/core/tools/write.ts | 5 +- .../src/extensions/codemode/tool.ts | 4 +- .../test/sdk-codex-cache-probe-tool-loop.ts | 2 +- .../test/suite/agent-session-codemode.test.ts | 7 +- packages/durable/CHANGELOG.md | 1 + packages/durable/README.md | 16 +- packages/durable/package.json | 4 +- packages/durable/src/codemode/codemode.ts | 4 +- packages/durable/src/codemode/models.ts | 8 +- packages/durable/src/harness/define.ts | 3 +- packages/durable/src/harness/tool.ts | 18 +- packages/durable/src/harness/types.ts | 23 +- .../durable/src/tool-search/tool-search.ts | 15 +- packages/durable/src/tools/bash.ts | 40 +- packages/durable/src/tools/edit.ts | 36 +- packages/durable/src/tools/read.ts | 21 +- packages/durable/src/tools/write.ts | 13 +- packages/durable/test/codemode-bench.ts | 7 +- .../durable/test/codemode-describe.test.ts | 5 +- packages/durable/test/codemode.test.ts | 15 +- packages/durable/test/examples/05-harness.ts | 5 +- .../durable/test/examples/06-configuration.ts | 5 +- .../test/examples/09-registry-reload.ts | 5 +- .../test/examples/10-extension-state.ts | 5 +- .../durable/test/examples/20-late-join.ts | 5 +- .../test/examples/21-subagent-foreground.ts | 6 +- .../test/examples/22-subagent-background.ts | 14 +- .../durable/test/examples/26-plan-mode.ts | 5 +- .../test/examples/30-reload-and-restart.ts | 6 +- .../test/examples/32-tests-after-edits.ts | 5 +- .../examples/33-turn-limit-and-continue.ts | 5 +- .../test/examples/34-submit-in-commit.ts | 5 +- .../durable/test/harness-after-run.test.ts | 5 +- packages/durable/test/harness-callers.test.ts | 7 +- .../durable/test/harness-compaction.test.ts | 8 +- .../durable/test/harness-continue.test.ts | 7 +- .../test/harness-deferred-progress.test.ts | 8 +- .../durable/test/harness-host-inputs.test.ts | 4 +- packages/durable/test/harness-inbox.test.ts | 8 +- .../test/harness-input-entries.test.ts | 5 +- .../durable/test/harness-lifecycle.test.ts | 5 +- .../durable/test/harness-live-deltas.test.ts | 13 +- .../test/harness-nested-tools-restart.test.ts | 4 +- .../durable/test/harness-nested-tools.test.ts | 6 +- .../durable/test/harness-ownership.test.ts | 15 +- packages/durable/test/harness-prompt.test.ts | 5 +- .../durable/test/harness-registry.test.ts | 4 +- packages/durable/test/harness-reports.test.ts | 5 +- .../test/harness-restart-failure.test.ts | 9 +- .../durable/test/harness-run-failure.test.ts | 6 +- .../test/harness-structured-output.test.ts | 21 +- .../durable/test/harness-structured.test.ts | 14 +- .../test/harness-submit-in-commit.test.ts | 11 +- packages/durable/test/harness-support.ts | 5 +- .../test/harness-tasks-recovery.test.ts | 5 +- packages/durable/test/harness-tasks.test.ts | 5 +- .../durable/test/harness-telemetry.test.ts | 15 +- .../durable/test/harness-tool-hooks.test.ts | 7 +- .../test/harness-tools-recovery.test.ts | 4 +- packages/durable/test/harness-tools.test.ts | 4 +- .../durable/test/harness-turn-limit.test.ts | 6 +- .../durable/test/harness-view-run.test.ts | 11 +- packages/durable/test/nested-images-bench.ts | 19 +- packages/durable/test/nested-tools-bench.ts | 7 +- .../durable/test/scheduler-fanout-bench.ts | 7 +- .../test/session-document-retention.test.ts | 6 +- packages/durable/test/spec-usage.test.ts | 6 +- .../test/system-order-cache-e2e.test.ts | 4 +- packages/durable/test/tool-output-bench.ts | 5 +- packages/durable/test/tool-search.test.ts | 7 +- packages/mcp/CHANGELOG.md | 4 + packages/mcp/README.md | 3 +- packages/schema/CHANGELOG.md | 3 + packages/schema/README.md | 62 +- packages/schema/src/checks.ts | 6 +- packages/schema/src/coerce.ts | 556 +++++++ packages/schema/src/explain.ts | 89 +- packages/schema/src/formats/index.ts | 9 +- .../typebox/json_pointer_uri_fragment.ts | 38 + packages/schema/src/formats/typebox/url.ts | 36 + packages/schema/src/index.ts | 4 +- packages/schema/src/resolver.ts | 24 +- packages/schema/src/types.ts | 13 + packages/schema/src/validator.ts | 47 +- packages/schema/test/coerce-typebox.test.ts | 1344 +++++++++++++++++ packages/schema/test/coerce.test.ts | 458 ++++++ packages/schema/test/validator.test.ts | 133 ++ scripts/browser-smoke-entry.ts | 10 +- scripts/check-entry-graphs.mjs | 34 +- tsconfig.json | 2 +- 139 files changed, 5115 insertions(+), 1305 deletions(-) create mode 100644 packages/ai/src/providers/schema-builders.ts rename packages/ai/src/utils/{typebox-helpers.ts => string-enum.ts} (68%) create mode 100644 packages/ai/test/validation-probes.test.ts create mode 100644 packages/coding-agent/examples/tsconfig.json create mode 100644 packages/coding-agent/src/core/extensions/pi-ai-extension-view.ts create mode 100644 packages/coding-agent/src/core/pi-ai-schemas.ts create mode 100644 packages/schema/src/coerce.ts create mode 100644 packages/schema/src/formats/typebox/json_pointer_uri_fragment.ts create mode 100644 packages/schema/src/formats/typebox/url.ts create mode 100644 packages/schema/test/coerce-typebox.test.ts create mode 100644 packages/schema/test/coerce.test.ts diff --git a/package-lock.json b/package-lock.json index d26f79648..d37cfbbba 100644 --- a/package-lock.json +++ b/package-lock.json @@ -5442,7 +5442,7 @@ "license": "MIT", "dependencies": { "@earendil-works/pi-ai": "1.4.0", - "typebox": "1.3.27" + "@earendil-works/pi-schema": "1.4.0" }, "devDependencies": { "@types/node": "22.19.19", @@ -5466,14 +5466,14 @@ "dependencies": { "@anthropic-ai/sdk": "0.129.0", "@aws-sdk/client-bedrock-runtime": "3.1127.0", + "@earendil-works/pi-schema": "1.4.0", "@earendil-works/pi-telemetry": "1.4.0", "@google/genai": "2.21.0", "@smithy/node-http-handler": "4.12.1", "http-proxy-agent": "9.1.0", "https-proxy-agent": "9.1.0", "openai": "7.19.0", - "partial-json": "0.1.7", - "typebox": "1.3.27" + "partial-json": "0.1.7" }, "bin": { "pi-ai": "dist/cli.js" @@ -5481,6 +5481,7 @@ "devDependencies": { "@types/node": "22.19.19", "canvas": "3.2.3", + "typebox": "1.3.27", "vitest": "4.1.12" }, "engines": { @@ -5714,10 +5715,10 @@ "@earendil-works/chord": "1.4.0", "@earendil-works/pi-ai": "1.4.0", "@earendil-works/pi-codemode": "1.4.0", + "@earendil-works/pi-schema": "1.4.0", "@earendil-works/pi-telemetry": "1.4.0", "@silvia-odwyer/photon": "0.3.3", - "diff": "8.0.4", - "typebox": "1.3.27" + "diff": "8.0.4" }, "devDependencies": { "@cloudflare/vitest-plugin": "1.4.10", diff --git a/package.json b/package.json index 352767d97..daa608b30 100644 --- a/package.json +++ b/package.json @@ -17,7 +17,7 @@ "build:native:darwin": "npm --prefix packages/tui run build:native:darwin", "build:native:linux": "npm --prefix packages/tui run build:native:linux", "build:native:win32": "npm --prefix packages/tui run build:native:win32", - "check": "node scripts/biome-check.mjs --write && npm run check:pinned-deps && npm run check:runtime-deps && npm run check:ts-imports && npm run check:entry-graphs && npm run check:install-lock:coding-agent && tsc --noEmit && npm run check:browser-smoke", + "check": "node scripts/biome-check.mjs --write && npm run check:pinned-deps && npm run check:runtime-deps && npm run check:ts-imports && npm run check:entry-graphs && npm run check:install-lock:coding-agent && tsc --noEmit && tsc --noEmit -p packages/coding-agent/examples/tsconfig.json && npm run check:browser-smoke", "check:browser-smoke": "node scripts/check-browser-smoke.mjs", "check:pinned-deps": "node scripts/check-pinned-deps.mjs", "check:runtime-deps": "node scripts/check-runtime-deps.mjs", diff --git a/packages/agent/CHANGELOG.md b/packages/agent/CHANGELOG.md index 2ae23784d..f095f4f4f 100644 --- a/packages/agent/CHANGELOG.md +++ b/packages/agent/CHANGELOG.md @@ -2,6 +2,10 @@ ## [Unreleased] +### Breaking Changes + +- `AgentTool` takes any JSON Schema object as `parameters` (`AgentTool`) and types `params` and `prepareArguments` through `@earendil-works/pi-schema` (`AgentToolParams`): pi-schema's builders give their static type, other schemas (TypeBox included) give `unknown`, `AgentTool` stays `any`. `outputSchema` is any JSON Schema object. `typebox` is no longer a dependency. Migration: build parameters with pi-schema (`obj({ path: str() })` instead of `Type.Object({ path: Type.String() })`), or keep TypeBox schemas and narrow `params` yourself. + ### Fixed - Fixed `streamProxy()` re-parsing the accumulated tool-call arguments on every delta and keeping their raw text on the partial content block ([#9265](https://github.com/earendil-works/pi/issues/9265)) diff --git a/packages/agent/README.md b/packages/agent/README.md index 38625ce11..a6fa8256b 100644 --- a/packages/agent/README.md +++ b/packages/agent/README.md @@ -455,18 +455,20 @@ const agent = new Agent({ ## Tools -Define tools using `AgentTool`: +Define tools using `AgentTool`. `parameters` is any JSON Schema object; built with `@earendil-works/pi-schema`'s builders, it also types `params` (other schemas give `unknown`, `AgentTool` gives `any`): ```typescript -import { Type } from "typebox"; +import { obj, str } from "@earendil-works/pi-schema"; -const readFileTool: AgentTool = { +const readFileParameters = obj({ + path: str({ description: "File path" }), +}); + +const readFileTool: AgentTool = { name: "read_file", label: "Read File", // For UI display description: "Read a file's contents", - parameters: Type.Object({ - path: Type.String({ description: "File path" }), - }), + parameters: readFileParameters, // Override execution mode for this tool (optional). // "sequential" forces the entire batch to run one at a time. // "parallel" allows concurrent execution with other tool calls. diff --git a/packages/agent/examples/mcp-codemode/tools.ts b/packages/agent/examples/mcp-codemode/tools.ts index 37e358c65..06d678670 100644 --- a/packages/agent/examples/mcp-codemode/tools.ts +++ b/packages/agent/examples/mcp-codemode/tools.ts @@ -18,7 +18,7 @@ import { renderDeclarations, } from "@earendil-works/pi-codemode"; import { type McpClient, toLlmContent } from "@earendil-works/pi-mcp"; -import { Type } from "typebox"; +import { obj, str } from "@earendil-works/pi-schema"; /** Provider tool names are limited to 64 characters of `[A-Za-z0-9_-]`. */ function toToolName(prefix: string, name: string): string { @@ -45,12 +45,12 @@ export async function createMcpTools(client: McpClient, prefix: string): Promise label: tool.title ?? tool.name, description: tool.description ?? tool.title ?? `MCP tool ${tool.name}`, // Providers require an object schema, and some reject one without `properties`. - parameters: Type.Unsafe({ + parameters: { ...tool.inputSchema, type: "object", properties: tool.inputSchema.properties ?? {}, - }), - outputSchema: tool.outputSchema ? Type.Unsafe>(tool.outputSchema) : undefined, + }, + outputSchema: tool.outputSchema, execute: async (_toolCallId, params, signal, onUpdate) => { const result = await client.callTool(tool.name, params as Record, { signal, @@ -104,9 +104,11 @@ export function createNestedToolRunner( export const CODEMODE_TOOL_NAME = "codemode"; -const codemodeParameters = Type.Object({ - code: Type.String({ description: "JavaScript source. Top-level await and return work." }), -}); +const codemodeParameters = obj( + { code: str({ description: "JavaScript source. Top-level await and return work." }) }, + // Open, as the tool accepted extra arguments before it moved from TypeBox. + { open: true }, +); /** * A tool that runs model-written JavaScript in the codemode sandbox, where `tools.(args)` @@ -121,7 +123,6 @@ export function createCodemodeTool( const sandboxTools: CodemodeTool[] = callable.map((tool) => ({ name: tool.name, description: tool.description, - // TypeBox schemas are JSON Schema. inputSchema: tool.parameters as CodemodeJsonSchema, // What the script receives: the structured result if the tool declares one, else its text. outputSchema: (tool.outputSchema as CodemodeJsonSchema | undefined) ?? { type: "string" }, diff --git a/packages/agent/package.json b/packages/agent/package.json index a8f7de36f..5d7091e4a 100644 --- a/packages/agent/package.json +++ b/packages/agent/package.json @@ -24,7 +24,7 @@ }, "dependencies": { "@earendil-works/pi-ai": "1.4.0", - "typebox": "1.3.27" + "@earendil-works/pi-schema": "1.4.0" }, "keywords": [ "ai", diff --git a/packages/agent/src/types.ts b/packages/agent/src/types.ts index dbfba74a0..539fee1ae 100644 --- a/packages/agent/src/types.ts +++ b/packages/agent/src/types.ts @@ -14,7 +14,7 @@ import type { TranscriptContext, Usage, } from "@earendil-works/pi-ai"; -import type { Static, TSchema } from "typebox"; +import type { Static } from "@earendil-works/pi-schema"; /** * Stream function used by the agent loop. `Models.streamSimple` satisfies @@ -462,27 +462,39 @@ export interface AgentToolCallOutcome { */ export type AgentToolUpdateCallback = (partialResult: AgentToolResult) => void; -/** Tool definition used by the agent runtime. */ -export interface AgentTool extends Tool { +/** + * The arguments type of a tool with parameter schema `TParameters`: `Static` for schemas built with + * `@earendil-works/pi-schema`'s builders, `unknown` for other schemas, and `any` for `AgentTool`. + */ +export type AgentToolParams = IsAny extends true ? any : Static; + +// `0 extends 1 & T` is resolved too early for a `T` constrained to `object`; this form waits for the argument. +type IsAny = boolean extends (T extends never ? true : false) ? true : false; + +/** + * Tool definition used by the agent runtime. `parameters` is any JSON Schema object; see {@link AgentToolParams} for + * the type `execute` receives. + */ +export interface AgentTool extends Tool { /** Human-readable label for UI display. */ label: string; /** * Optional compatibility shim for raw tool-call arguments before schema validation. * Must return an object that matches `TParameters`. */ - prepareArguments?: (args: unknown) => Static; + prepareArguments?: (args: unknown) => AgentToolParams; /** * JSON Schema of `structuredContent` in successful results. Tools that declare it should always * set `structuredContent`. */ - outputSchema?: TSchema; + outputSchema?: object; /** * Execute the tool call. Throw on failure, or return a result with `isError: true`; do not only * describe the failure in `content`. */ execute: ( toolCallId: string, - params: Static, + params: AgentToolParams, signal?: AbortSignal, onUpdate?: AgentToolUpdateCallback, ) => Promise>; diff --git a/packages/agent/test/agent-loop.test.ts b/packages/agent/test/agent-loop.test.ts index 4a9386a13..b3ea8c259 100644 --- a/packages/agent/test/agent-loop.test.ts +++ b/packages/agent/test/agent-loop.test.ts @@ -6,7 +6,7 @@ import { type ToolResultMessage, type UserMessage, } from "@earendil-works/pi-ai"; -import { Type } from "typebox"; +import { arr, obj, str } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; import { agentLoop, agentLoopContinue, runAgentLoop, runToolCall } from "../src/agent-loop.ts"; import { setDefaultStreamFn } from "../src/index.ts"; @@ -303,7 +303,7 @@ describe("agentLoop with AgentMessage", () => { }); it("should handle tool calls and results", async () => { - const toolSchema = Type.Object({ value: Type.String() }); + const toolSchema = obj({ value: str() }); const executed: string[] = []; const toolUsage = { input: 1, @@ -400,7 +400,7 @@ describe("agentLoop with AgentMessage", () => { // #10549 it("records how long execute() took on the tool result, excluding hooks", async () => { - const toolSchema = Type.Object({ value: Type.String() }); + const toolSchema = obj({ value: str() }); const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); const tool: AgentTool = { name: "echo", @@ -454,7 +454,7 @@ describe("agentLoop with AgentMessage", () => { }); it("should not execute tool calls from a length-truncated assistant message", async () => { - const toolSchema = Type.Object({ value: Type.String() }); + const toolSchema = obj({ value: str() }); const executed: string[] = []; const tool: AgentTool = { name: "echo", @@ -526,7 +526,7 @@ describe("agentLoop with AgentMessage", () => { }); it("should execute mutated beforeToolCall args without revalidation", async () => { - const toolSchema = Type.Object({ value: Type.String() }); + const toolSchema = obj({ value: str() }); const executed: Array = []; const tool: AgentTool = { name: "echo", @@ -587,8 +587,8 @@ describe("agentLoop with AgentMessage", () => { }); it("should prepare tool arguments for validation", async () => { - const replaceSchema = Type.Object({ oldText: Type.String(), newText: Type.String() }); - const toolSchema = Type.Object({ edits: Type.Array(replaceSchema) }); + const replaceSchema = obj({ oldText: str(), newText: str() }); + const toolSchema = obj({ edits: arr(replaceSchema) }); const executed: Array> = []; const tool: AgentTool = { name: "edit", @@ -666,7 +666,7 @@ describe("agentLoop with AgentMessage", () => { }); it("should emit tool_execution_end in completion order but persist tool results in source order", async () => { - const toolSchema = Type.Object({ value: Type.String() }); + const toolSchema = obj({ value: str() }); let firstResolved = false; let parallelObserved = false; let releaseFirst: (() => void) | undefined; @@ -760,7 +760,7 @@ describe("agentLoop with AgentMessage", () => { }); it("should inject queued messages after all tool calls complete", async () => { - const toolSchema = Type.Object({ value: Type.String() }); + const toolSchema = obj({ value: str() }); const executed: string[] = []; const tool: AgentTool = { name: "echo", @@ -865,7 +865,7 @@ describe("agentLoop with AgentMessage", () => { }); it("should force sequential execution when a tool has executionMode=sequential even with default parallel config", async () => { - const toolSchema = Type.Object({ value: Type.String() }); + const toolSchema = obj({ value: str() }); let firstResolved = false; let parallelObserved = false; let releaseFirst: (() => void) | undefined; @@ -947,7 +947,7 @@ describe("agentLoop with AgentMessage", () => { }); it("should force sequential execution when one of multiple tools has executionMode=sequential", async () => { - const toolSchema = Type.Object({ value: Type.String() }); + const toolSchema = obj({ value: str() }); const executionOrder: string[] = []; let releaseSlow: (() => void) | undefined; const slowDone = new Promise((resolve) => { @@ -1033,7 +1033,7 @@ describe("agentLoop with AgentMessage", () => { }); it("should allow parallel execution when all tools have executionMode=parallel", async () => { - const toolSchema = Type.Object({ value: Type.String() }); + const toolSchema = obj({ value: str() }); let firstResolved = false; let parallelObserved = false; let releaseFirst: (() => void) | undefined; @@ -1106,7 +1106,7 @@ describe("agentLoop with AgentMessage", () => { }); it("runs finishTurn after tool-result messages and before turn_end", async () => { - const toolSchema = Type.Object({ value: Type.String() }); + const toolSchema = obj({ value: str() }); const tool: AgentTool = { name: "echo", label: "Echo", @@ -1215,7 +1215,7 @@ describe("agentLoop with AgentMessage", () => { ); it("action:end skips queue polling and next-turn preparation", async () => { - const toolSchema = Type.Object({}); + const toolSchema = obj({}); const tool: AgentTool = { name: "noop", label: "Noop", @@ -1305,7 +1305,7 @@ describe("agentLoop with AgentMessage", () => { }); it("lets a natural tool-result request satisfy continuation", async () => { - const toolSchema = Type.Object({}); + const toolSchema = obj({}); const tool: AgentTool = { name: "noop", label: "Noop", @@ -1501,7 +1501,7 @@ describe("agentLoop with AgentMessage", () => { }); it("should use prepareNextTurn snapshot before continuing", async () => { - const toolSchema = Type.Object({ value: Type.String() }); + const toolSchema = obj({ value: str() }); const tool: AgentTool = { name: "echo", label: "Echo", @@ -1578,7 +1578,7 @@ describe("agentLoop with AgentMessage", () => { }); it("picks up steering queued during prepareNextTurn before the next request", async () => { - const toolSchema = Type.Object({}); + const toolSchema = obj({}); const tool: AgentTool = { name: "noop", label: "Noop", @@ -1631,7 +1631,7 @@ describe("agentLoop with AgentMessage", () => { }); it("action:end receives finalized turn context and stops before queue polling", async () => { - const toolSchema = Type.Object({ value: Type.String() }); + const toolSchema = obj({ value: str() }); const executed: string[] = []; const tool: AgentTool = { name: "echo", @@ -1730,7 +1730,7 @@ describe("agentLoop with AgentMessage", () => { }); it("should stop after a tool batch when every tool result sets terminate=true", async () => { - const toolSchema = Type.Object({ value: Type.String() }); + const toolSchema = obj({ value: str() }); const tool: AgentTool = { name: "echo", label: "Echo", @@ -1781,7 +1781,7 @@ describe("agentLoop with AgentMessage", () => { }); it("should stop after a blocked tool call when beforeToolCall sets terminate=true", async () => { - const toolSchema = Type.Object({ value: Type.String() }); + const toolSchema = obj({ value: str() }); let executed = false; const tool: AgentTool = { name: "echo", @@ -1839,7 +1839,7 @@ describe("agentLoop with AgentMessage", () => { }); it("should continue after a mixed batch with one terminating blocked call", async () => { - const toolSchema = Type.Object({ value: Type.String() }); + const toolSchema = obj({ value: str() }); const executed: string[] = []; const tool: AgentTool = { name: "echo", @@ -1897,7 +1897,7 @@ describe("agentLoop with AgentMessage", () => { }); it("should continue after parallel tool calls when not all tool results terminate", async () => { - const toolSchema = Type.Object({ value: Type.String() }); + const toolSchema = obj({ value: str() }); const tool: AgentTool = { name: "echo", label: "Echo", @@ -1962,7 +1962,7 @@ describe("agentLoop with AgentMessage", () => { }); it("should allow afterToolCall to mark a tool batch as terminating", async () => { - const toolSchema = Type.Object({ value: Type.String() }); + const toolSchema = obj({ value: str() }); const tool: AgentTool = { name: "echo", label: "Echo", @@ -2131,13 +2131,13 @@ describe("agentLoopContinue with AgentMessage", () => { }); describe("runToolCall", () => { - const echoSchema = Type.Object({ value: Type.String() }); + const echoSchema = obj({ value: str() }); const echo: AgentTool = { name: "echo", label: "Echo", description: "Echo tool", parameters: echoSchema, - outputSchema: Type.Object({ value: Type.String() }), + outputSchema: obj({ value: str() }), async execute(_toolCallId, params, _signal, onUpdate) { onUpdate?.({ content: [{ type: "text", text: "partial" }], details: {} }); return { @@ -2151,7 +2151,7 @@ describe("runToolCall", () => { name: "failing", label: "Failing", description: "Returns an error result", - parameters: Type.Object({}), + parameters: obj({}), async execute() { return { content: [{ type: "text", text: "bad" }], details: { partial: true }, isError: true }; }, diff --git a/packages/agent/test/agent.test.ts b/packages/agent/test/agent.test.ts index 6fad6109f..9c8bddc39 100644 --- a/packages/agent/test/agent.test.ts +++ b/packages/agent/test/agent.test.ts @@ -7,7 +7,7 @@ import { toToolDeclaration, type UserMessage, } from "@earendil-works/pi-ai/compat"; -import { Type } from "typebox"; +import { obj } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; import { Agent, @@ -49,7 +49,7 @@ function createTool(name: string): AgentTool { name, label: name, description: `${name} tool`, - parameters: Type.Object({}), + parameters: obj({}), execute: async () => ({ content: [{ type: "text", text: name }], details: {} }), }; } @@ -146,7 +146,7 @@ describe("Agent", () => { name: "echo", label: "Echo", description: "Echo input", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => ({ content: [{ type: "text", text: "echo" }], details: {} }), }; const agent = new Agent({ @@ -200,7 +200,7 @@ describe("Agent", () => { expect(update).toEqual({ role: "system", content: "", - toolsAdded: [{ name: "second", description: "second tool", parameters: Type.Object({}) }], + toolsAdded: [{ name: "second", description: "second tool", parameters: obj({}) }], toolsRemoved: [{ name: "first" }], timestamp: expect.any(Number), }); @@ -214,7 +214,7 @@ describe("Agent", () => { name: "echo", label: "Echo", description: "Echo input", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => ({ content: [{ type: "text", text: "echo" }], details: {} }), }; const agent = new Agent({ @@ -239,7 +239,7 @@ describe("Agent", () => { role: "system", content: "", sections: { skills: "x" }, - toolsAdded: [{ name: "echo", description: "Echo input", parameters: Type.Object({}) }], + toolsAdded: [{ name: "echo", description: "Echo input", parameters: obj({}) }], timestamp: 1, }); }); @@ -284,7 +284,7 @@ describe("Agent", () => { name: "echo", label: "Echo", description: "Echo input", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => ({ content: [{ type: "text", text: "echo" }], details: {} }), }; const agent = new Agent({ @@ -471,7 +471,7 @@ describe("Agent", () => { }); it("should ignore tool updates after the tool execution settles", async () => { - const toolSchema = Type.Object({}); + const toolSchema = obj({}); let delayedUpdate: AgentToolUpdateCallback<{ status: string }> | undefined; const events: AgentEvent[] = []; const unhandledRejections: unknown[] = []; @@ -536,7 +536,7 @@ describe("Agent", () => { }); it("should ignore a settled parallel tool update while another tool is still running", async () => { - const toolSchema = Type.Object({}); + const toolSchema = obj({}); const slowStarted = createDeferred(); const settledToolEnded = createDeferred(); const releaseSlow = createDeferred(); @@ -858,7 +858,7 @@ describe("Agent", () => { }); it("keeps legacy prepareNextTurn signal callback behavior", async () => { - const schema = Type.Object({}); + const schema = obj({}); const tool: AgentTool = { name: "noop", label: "Noop", @@ -899,7 +899,7 @@ describe("Agent", () => { }); it("forwards finishTurn through AgentOptions with the active abort signal", async () => { - const schema = Type.Object({}); + const schema = obj({}); const tool: AgentTool = { name: "noop", label: "Noop", diff --git a/packages/agent/test/utils/calculate.ts b/packages/agent/test/utils/calculate.ts index 0a4fefc58..333cd5a0e 100644 --- a/packages/agent/test/utils/calculate.ts +++ b/packages/agent/test/utils/calculate.ts @@ -1,5 +1,5 @@ import type { Usage } from "@earendil-works/pi-ai"; -import { type Static, Type } from "typebox"; +import { obj, type Static, str } from "@earendil-works/pi-schema"; import type { AgentTool, AgentToolResult } from "../../src/types.ts"; export interface CalculateResult extends AgentToolResult { @@ -16,9 +16,11 @@ export function calculate(expression: string): CalculateResult { } } -const calculateSchema = Type.Object({ - expression: Type.String({ description: "The mathematical expression to evaluate" }), -}); +const calculateSchema = obj( + { expression: str({ description: "The mathematical expression to evaluate" }) }, + // Open, as the tool accepted extra arguments before it moved from TypeBox. + { open: true }, +); type CalculateParams = Static; diff --git a/packages/agent/test/utils/get-current-time.ts b/packages/agent/test/utils/get-current-time.ts index a83a82af3..11996cc41 100644 --- a/packages/agent/test/utils/get-current-time.ts +++ b/packages/agent/test/utils/get-current-time.ts @@ -1,4 +1,4 @@ -import { type Static, Type } from "typebox"; +import { obj, opt, type Static, str } from "@earendil-works/pi-schema"; import type { AgentTool, AgentToolResult } from "../../src/types.ts"; export interface GetCurrentTimeResult extends AgentToolResult<{ utcTimestamp: number }> {} @@ -27,11 +27,11 @@ export async function getCurrentTime(timezone?: string): Promise; diff --git a/packages/ai/CHANGELOG.md b/packages/ai/CHANGELOG.md index 996207fbd..6aae94ddb 100644 --- a/packages/ai/CHANGELOG.md +++ b/packages/ai/CHANGELOG.md @@ -4,6 +4,31 @@ ### Breaking Changes +- The main entry no longer re-exports TypeBox (`Type`, `Static`, `TSchema`), and `typebox` is no longer a dependency. `Tool.parameters` is any JSON Schema object (`Tool`): `@earendil-works/pi-schema`'s builders, which also type the arguments through pi-schema's `Static`, TypeBox schemas, or plain objects. `validateToolCall()`/`validateToolArguments()` validate with pi-schema (`compile(schema, { coerce: true }).parse()`): every schema gets the same coercion, the superset of TypeBox's `Value.Convert` rules (which only applied to TypeBox-built schemas) and pi-ai's own, so plain JSON Schema tools (MCP tools) now also get, for example, `null` to `"null"` for a required string, `"3.5"` to 3 for an integer and a single value wrapped into an array. The error text lists pi-schema's messages (`- edits.0.oldText: missing`, `- limit: expected number, got string`) instead of TypeBox's. `StringEnum()` returns a pi-schema `Schema` with the same JSON (`{ type: "string", enum }`). The model and compat schemas (`providers/model-schema`, `providers/compat-schema`) are built with pi-schema, with the same JSON Schema, and their types come from pi-schema's `Static`; `ProviderCompat` is exported. Migration: + + ```ts + // Before + import { Type, type Static, StringEnum } from "@earendil-works/pi-ai"; + const params = Type.Object({ path: Type.String(), mode: StringEnum(["a", "b"]) }); + type Params = Static; + + // After: pi-schema builders (closed objects; pass { open: true } to allow other properties) + import { StringEnum } from "@earendil-works/pi-ai"; + import { obj, type Static, str } from "@earendil-works/pi-schema"; + const params = obj({ path: str(), mode: StringEnum(["a", "b"]) }); + type Params = Static; + + // Or keep TypeBox, imported from typebox itself (its schemas are JSON Schema) + import { Type, type Static } from "typebox"; + ``` +- Tool arguments are validated as JSON Schema only; nothing runs through TypeBox's own validator any more, so TypeBox features outside JSON Schema no longer apply to tool arguments: + - `Type.Refine()` refinements are not run: arguments they would reject are accepted. + - `Type.BigInt()`, `Type.Undefined()` and `Type.Void()` reject every JSON argument (TypeBox converted to them). + - Codecs (`Type.Codec()`) do not decode: the tool receives the encoded JSON value. + - Formats registered with TypeBox's `Format.Set()` are not checked; the standard formats plus `url` and `json-pointer-uri-fragment` are. + - Property names are names, not regular expressions: `"a.b"` no longer also converts the argument `axb`. + - `additionalProperties` converts only undeclared properties; TypeBox also converted declared ones with it when an object had more than one property. +- A `$ref` that names nothing (a missing definition, an external document) rejects only arguments at its place (`- t: unresolved reference #/definitions/Missing`) instead of failing every call of the tool, as TypeBox's rejection did before. - `appendGrammarToolInputJsonDelta()` (`api/constrained-sampling`) takes the input delta instead of the whole input so far and no longer closes the input; close it with the new `closeGrammarToolInputJson()` ([#9265](https://github.com/earendil-works/pi/issues/9265)) ### Added diff --git a/packages/ai/README.md b/packages/ai/README.md index 583d55bc3..7af343eab 100644 --- a/packages/ai/README.md +++ b/packages/ai/README.md @@ -103,15 +103,16 @@ Unified LLM API with provider collections, automatic auth resolution, token and npm install @earendil-works/pi-ai ``` -TypeBox exports are re-exported from `@earendil-works/pi-ai`: `Type`, `Static`, and `TSchema`. +Tool parameters are JSON Schema. The examples build them with `@earendil-works/pi-schema`, which also types the arguments (`npm install @earendil-works/pi-schema`); TypeBox schemas and plain JSON Schema objects work as well. ## Quick Start You build a `Models` collection of providers and stream through it. The quickest start registers every built-in provider; apps that care about bundle size register individual providers instead (see [Provider Factories](#provider-factories) and [Bundling and Tree Shaking](#bundling-and-tree-shaking)). ```typescript -import { Type, type Context, type Tool } from '@earendil-works/pi-ai'; +import { type Context, type Tool } from '@earendil-works/pi-ai'; import { builtinModels } from '@earendil-works/pi-ai/providers/all'; +import { obj, opt, str } from '@earendil-works/pi-schema'; // A Models collection with every built-in provider registered const models = builtinModels(); @@ -119,12 +120,12 @@ const models = builtinModels(); // Sync lookup against the collection const model = models.getModel('openai', 'gpt-4o-mini')!; -// Define tools with TypeBox schemas for type safety and validation +// Define tools with JSON Schema; pi-schema's builders also type the arguments const tools: Tool[] = [{ name: 'get_time', description: 'Get the current time', - parameters: Type.Object({ - timezone: Type.Optional(Type.String({ description: 'Optional timezone (e.g., America/New_York)' })) + parameters: obj({ + timezone: opt(str({ description: 'Optional timezone (e.g., America/New_York)' })) }) }]; @@ -538,34 +539,35 @@ Amazon Bedrock resolves ambient AWS credentials (`AWS_PROFILE`, access key pairs ## Tools -Tools enable LLMs to interact with external systems. This library uses TypeBox schemas for type-safe tool definitions with automatic validation using TypeBox's built-in validator and value conversion utilities. TypeBox schemas can be serialized and deserialized as plain JSON, making them ideal for distributed systems. +Tools enable LLMs to interact with external systems. A tool's `parameters` is a JSON Schema object: built with `@earendil-works/pi-schema`'s builders (which also type the arguments through its `Static`), with TypeBox, or written by hand. `validateToolCall()` and `validateToolArguments()` validate with pi-schema, coercing loosely typed arguments first (numeric strings, `"true"`, a single value for an array, `null` for an omitted optional property). Schemas are plain JSON, so they serialize with the rest of a context. ### Defining Tools ```typescript -import { Type, type Tool, StringEnum } from '@earendil-works/pi-ai'; +import { type Tool, StringEnum } from '@earendil-works/pi-ai'; +import { arr, obj, str } from '@earendil-works/pi-schema'; -// Define tool parameters with TypeBox +// Define tool parameters with pi-schema's builders const weatherTool: Tool = { name: 'get_weather', description: 'Get current weather for a location', - parameters: Type.Object({ - location: Type.String({ description: 'City name or coordinates' }), + parameters: obj({ + location: str({ description: 'City name or coordinates' }), units: StringEnum(['celsius', 'fahrenheit'], { default: 'celsius' }) }) }; -// Note: For Google API compatibility, use StringEnum helper instead of Type.Enum -// Type.Enum generates anyOf/const patterns that Google doesn't support +// Note: StringEnum gives { type: "string", enum: [...] }, which every provider accepts, +// including Google's, which rejects anyOf/const patterns const bookMeetingTool: Tool = { name: 'book_meeting', description: 'Schedule a meeting', - parameters: Type.Object({ - title: Type.String({ minLength: 1 }), - startTime: Type.String({ format: 'date-time' }), - endTime: Type.String({ format: 'date-time' }), - attendees: Type.Array(Type.String({ format: 'email' }), { minItems: 1 }) + parameters: obj({ + title: str({ minLength: 1 }), + startTime: str({ format: 'date-time' }), + endTime: str({ format: 'date-time' }), + attendees: arr(str({ format: 'email' }), { minItems: 1 }) }) }; ``` @@ -578,10 +580,7 @@ Tools can opt in to provider-side constrained sampling. For JSON-schema tools, ` const strictTool: Tool = { name: 'edit_file', description: 'Edit a file', - parameters: Type.Object({ - path: Type.String(), - content: Type.String() - }, { additionalProperties: false }), + parameters: obj({ path: str(), content: str() }), // obj() is closed: additionalProperties: false constrainedSampling: { type: 'json_schema', strict: 'prefer' } }; ``` @@ -592,9 +591,7 @@ Strict JSON-schema constrained sampling is supported for OpenAI, Anthropic, supp const patchTool: Tool = { name: 'apply_patch', description: 'Apply a patch', - parameters: Type.Object({ - input: Type.String() - }, { additionalProperties: false }), + parameters: obj({ input: str() }), constrainedSampling: { type: 'grammar', variants: { @@ -698,7 +695,7 @@ for await (const event of s) { ### Validating Tool Arguments -When implementing your own tool execution loop, use `validateToolCall` to validate arguments before passing them to your tools: +When implementing your own tool execution loop, use `validateToolCall` to validate arguments before passing them to your tools. It coerces loosely typed arguments with pi-schema (`compile(schema, { coerce: true }).parse()`), returns the coerced arguments, and throws an error listing pi-schema's errors (`path: message`) followed by the arguments it received: ```typescript import { validateToolCall, type Tool } from '@earendil-works/pi-ai'; @@ -1726,8 +1723,8 @@ models.setProvider(openaiProvider()); Rules: -- `@earendil-works/pi-ai/models` exports the model runtime (`createModels`, `createProvider`, model helpers, and their types) without TypeBox, built-in catalogs, or SDK implementations. Other types can still use `import type` from the root. -- `@earendil-works/pi-ai` is the core entrypoint and does not import built-in catalogs, real provider factories, or SDK implementations, but it eagerly imports TypeBox and schema validation. Unbundled Node scripts do not tree-shake its unused exports; prefer `./models`, `./providers/faux`, and specific `./utils/*` subpaths when those are all you need. +- `@earendil-works/pi-ai/models` exports the model runtime (`createModels`, `createProvider`, model helpers, and their types) without schema validation, built-in catalogs, or SDK implementations. Other types can still use `import type` from the root. +- `@earendil-works/pi-ai` is the core entrypoint and does not import built-in catalogs, real provider factories, or SDK implementations, but it eagerly imports schema validation (pi-schema). Unbundled Node scripts do not tree-shake its unused exports; prefer `./models`, `./providers/faux`, and specific `./utils/*` subpaths when those are all you need. - `@earendil-works/pi-ai/providers/` imports that provider's catalog and lazy API wrapper only. - `@earendil-works/pi-ai/providers/all` imports every built-in provider factory and all catalogs. Use it only when you want the full built-in set. - With code splitting, provider SDKs stay in lazy chunks and load on first request. diff --git a/packages/ai/package.json b/packages/ai/package.json index 4e9d5d5a2..8de0434f8 100644 --- a/packages/ai/package.json +++ b/packages/ai/package.json @@ -69,14 +69,14 @@ "dependencies": { "@anthropic-ai/sdk": "0.129.0", "@aws-sdk/client-bedrock-runtime": "3.1127.0", + "@earendil-works/pi-schema": "1.4.0", "@earendil-works/pi-telemetry": "1.4.0", "@google/genai": "2.21.0", "@smithy/node-http-handler": "4.12.1", "http-proxy-agent": "9.1.0", "https-proxy-agent": "9.1.0", "openai": "7.19.0", - "partial-json": "0.1.7", - "typebox": "1.3.27" + "partial-json": "0.1.7" }, "keywords": [ "ai", @@ -101,6 +101,7 @@ "devDependencies": { "@types/node": "22.19.19", "canvas": "3.2.3", + "typebox": "1.3.27", "vitest": "4.1.12" } } diff --git a/packages/ai/src/index.ts b/packages/ai/src/index.ts index 54a0345f1..0c9197ae6 100644 --- a/packages/ai/src/index.ts +++ b/packages/ai/src/index.ts @@ -1,6 +1,3 @@ -export type { Static, TSchema } from "typebox"; -export { Type } from "typebox"; - // Core only, side-effect free: no generated catalogs, no provider factories, // no api-registry, no OAuth implementations, no compat. Provider factories // live under "@earendil-works/pi-ai/providers/*", API implementations under @@ -42,9 +39,9 @@ export * from "./utils/json-parse.ts"; export * from "./utils/overflow.ts"; export * from "./utils/retry.ts"; export * from "./utils/streaming-json.ts"; +export * from "./utils/string-enum.ts"; export { contentText, getSystemMessageText, renderSystemMessageUpdate } from "./utils/text.ts"; export * from "./utils/transcript.ts"; -export * from "./utils/typebox-helpers.ts"; export { uuidv7 } from "./utils/uuid.ts"; export * from "./utils/validation.ts"; export { PI_AI_VERSION } from "./utils/version.ts"; diff --git a/packages/ai/src/providers/compat-schema.ts b/packages/ai/src/providers/compat-schema.ts index b3ba8198e..9327da45c 100644 --- a/packages/ai/src/providers/compat-schema.ts +++ b/packages/ai/src/providers/compat-schema.ts @@ -1,17 +1,35 @@ -import { type Static, type TProperties, type TSchemaOptions, Type } from "typebox"; +import { + type Annotations, + arr, + bool, + nul, + num, + opt, + type Properties, + type Static, + str, + union, +} from "@earendil-works/pi-schema"; import { ModelCostSchema } from "./model-schema.ts"; +import { openObject, stringLiteral, stringRecord } from "./schema-builders.ts"; -function sessionAffinityFormat(options?: TSchemaOptions) { - return Type.Union([Type.Literal("openai"), Type.Literal("openai-nosession"), Type.Literal("openrouter")], options); +function sessionAffinityFormat(options?: Annotations<"openai" | "openai-nosession" | "openrouter">) { + return union([stringLiteral("openai"), stringLiteral("openai-nosession"), stringLiteral("openrouter")], options); } export const SessionAffinityFormatSchema = sessionAffinityFormat({ description: "Session-affinity header format used to route related requests consistently.", }); -function thinkingTokenBudgetField(options?: TSchemaOptions) { - return Type.Union( - [Type.Literal("thinking_token_budget"), Type.Literal("thinking_budget"), Type.Literal("thinking_budget_tokens")], +function thinkingTokenBudgetField( + options?: Annotations<"thinking_token_budget" | "thinking_budget" | "thinking_budget_tokens">, +) { + return union( + [ + stringLiteral("thinking_token_budget"), + stringLiteral("thinking_budget"), + stringLiteral("thinking_budget_tokens"), + ], options, ); } @@ -21,84 +39,78 @@ export const ThinkingTokenBudgetFieldSchema = thinkingTokenBudgetField({ 'Top-level request field name used by OpenAI-compatible endpoints to cap reasoning tokens. "thinking_token_budget" is used by vLLM, "thinking_budget" by Qwen, DashScope, or SGLang, and "thinking_budget_tokens" by llama.cpp.', }); -export const ChatTemplateKwargValueSchema = Type.Union([ - Type.String(), - Type.Number(), - Type.Boolean(), - Type.Null(), - Type.Object({ - $var: Type.Union([ - Type.Literal("thinking.enabled"), - Type.Literal("thinking.effort"), - Type.Literal("thinking.budget"), +export const ChatTemplateKwargValueSchema = union([ + str(), + num(), + bool(), + nul(), + openObject({ + $var: union([ + stringLiteral("thinking.enabled"), + stringLiteral("thinking.effort"), + stringLiteral("thinking.budget"), ]), - omitWhenOff: Type.Optional(Type.Boolean()), + omitWhenOff: opt(bool()), }), ]); function percentileCutoffs(metric: string) { - return Type.Object({ - p50: Type.Optional(Type.Number({ description: `${metric} at the 50th percentile.` })), - p75: Type.Optional(Type.Number({ description: `${metric} at the 75th percentile.` })), - p90: Type.Optional(Type.Number({ description: `${metric} at the 90th percentile.` })), - p99: Type.Optional(Type.Number({ description: `${metric} at the 99th percentile.` })), + return openObject({ + p50: opt(num({ description: `${metric} at the 50th percentile.` })), + p75: opt(num({ description: `${metric} at the 75th percentile.` })), + p90: opt(num({ description: `${metric} at the 90th percentile.` })), + p99: opt(num({ description: `${metric} at the 99th percentile.` })), }); } -export const OpenRouterRoutingSchema = Type.Object( +export const OpenRouterRoutingSchema = openObject( { - allow_fallbacks: Type.Optional( - Type.Boolean({ description: "Whether to allow backup providers to serve requests.", default: true }), + allow_fallbacks: opt( + bool({ description: "Whether to allow backup providers to serve requests.", default: true }), ), - require_parameters: Type.Optional( - Type.Boolean({ + require_parameters: opt( + bool({ description: "Whether to filter providers to only those that support all parameters in the request.", default: false, }), ), - data_collection: Type.Optional( - Type.Union([Type.Literal("deny"), Type.Literal("allow")], { + data_collection: opt( + union([stringLiteral("deny"), stringLiteral("allow")], { description: 'Data collection setting. "allow": allow providers that may store or train on data. "deny": only use providers that do not collect user data.', default: "allow", }), ), - zdr: Type.Optional( - Type.Boolean({ description: "Whether to restrict routing to only ZDR (Zero Data Retention) endpoints." }), + zdr: opt(bool({ description: "Whether to restrict routing to only ZDR (Zero Data Retention) endpoints." })), + enforce_distillable_text: opt( + bool({ description: "Whether to restrict routing to only models that allow text distillation." }), ), - enforce_distillable_text: Type.Optional( - Type.Boolean({ description: "Whether to restrict routing to only models that allow text distillation." }), - ), - order: Type.Optional( - Type.Array(Type.String(), { + order: opt( + arr(str(), { description: "An ordered list of provider names or slugs to try in sequence, falling back to the next if unavailable.", }), ), - only: Type.Optional( - Type.Array(Type.String(), { + only: opt( + arr(str(), { description: "List of provider names or slugs to exclusively allow for this request.", }), ), - ignore: Type.Optional( - Type.Array(Type.String(), { description: "List of provider names or slugs to skip for this request." }), - ), - quantizations: Type.Optional( - Type.Array(Type.String(), { + ignore: opt(arr(str(), { description: "List of provider names or slugs to skip for this request." })), + quantizations: opt( + arr(str(), { description: 'A list of quantization levels to filter providers by, for example ["fp16", "bf16", "fp8", "fp6", "int8", "int4", "fp4", "fp32"].', }), ), - sort: Type.Optional( - Type.Union( + sort: opt( + union( [ - Type.String(), - Type.Object({ - by: Type.Optional( - Type.String({ description: 'The sorting metric, such as "price", "throughput", or "latency".' }), - ), - partition: Type.Optional( - Type.Union([Type.String(), Type.Null()], { + str(), + openObject({ + by: opt(str({ description: 'The sorting metric, such as "price", "throughput", or "latency".' })), + partition: opt( + union([str(), nul()], { description: 'Partitioning strategy: "model" or "none".', default: "model", }), @@ -111,33 +123,25 @@ export const OpenRouterRoutingSchema = Type.Object( }, ), ), - max_price: Type.Optional( - Type.Object( + max_price: opt( + openObject( { - prompt: Type.Optional( - Type.Union([Type.Number(), Type.String()], { description: "Price per million prompt tokens." }), - ), - completion: Type.Optional( - Type.Union([Type.Number(), Type.String()], { description: "Price per million completion tokens." }), - ), - image: Type.Optional(Type.Union([Type.Number(), Type.String()], { description: "Price per image." })), - audio: Type.Optional( - Type.Union([Type.Number(), Type.String()], { description: "Price per audio unit." }), - ), - request: Type.Optional( - Type.Union([Type.Number(), Type.String()], { description: "Price per request." }), - ), + prompt: opt(union([num(), str()], { description: "Price per million prompt tokens." })), + completion: opt(union([num(), str()], { description: "Price per million completion tokens." })), + image: opt(union([num(), str()], { description: "Price per image." })), + audio: opt(union([num(), str()], { description: "Price per audio unit." })), + request: opt(union([num(), str()], { description: "Price per request." })), }, { description: "Maximum price per million tokens in USD." }, ), ), - preferred_min_throughput: Type.Optional( - Type.Union([Type.Number(), percentileCutoffs("Minimum tokens per second")], { + preferred_min_throughput: opt( + union([num(), percentileCutoffs("Minimum tokens per second")], { description: "Preferred minimum throughput in tokens per second. A number applies to p50.", }), ), - preferred_max_latency: Type.Optional( - Type.Union([Type.Number(), percentileCutoffs("Maximum latency in seconds")], { + preferred_max_latency: opt( + union([num(), percentileCutoffs("Maximum latency in seconds")], { description: "Preferred maximum latency in seconds. A number applies to p50.", }), ), @@ -148,16 +152,16 @@ export const OpenRouterRoutingSchema = Type.Object( }, ); -export const VercelGatewayRoutingSchema = Type.Object( +export const VercelGatewayRoutingSchema = openObject( { - only: Type.Optional( - Type.Array(Type.String(), { + only: opt( + arr(str(), { description: 'List of provider slugs to exclusively use for this request, for example ["bedrock", "anthropic"].', }), ), - order: Type.Optional( - Type.Array(Type.String(), { + order: opt( + arr(str(), { description: 'List of provider slugs to try in order, for example ["anthropic", "openai"].', }), ), @@ -168,374 +172,382 @@ export const VercelGatewayRoutingSchema = Type.Object( }, ); -export const AnthropicAllowedFallbackModelSchema = Type.Object( +export const AnthropicAllowedFallbackModelSchema = openObject( { - provider: Type.String({ minLength: 1 }), - model: Type.String({ minLength: 1 }), + provider: str({ minLength: 1 }), + model: str({ minLength: 1 }), cost: ModelCostSchema, }, { description: "An Anthropic server-side refusal fallback model with local pricing metadata." }, ); -function optionalCompatBoolean(options?: TSchemaOptions) { - return Type.Optional(Type.Boolean(options)); +function optionalCompatBoolean(options?: Annotations) { + return opt(bool(options)); } -function optionalSessionAffinityFormat(options?: TSchemaOptions) { - return Type.Optional(sessionAffinityFormat(options)); +function optionalSessionAffinityFormat(options?: Annotations<"openai" | "openai-nosession" | "openrouter">) { + return opt(sessionAffinityFormat(options)); } -export const OpenAICompletionsCompatSchema = Type.Object( - { - supportsStore: Type.Optional( - Type.Boolean({ - description: "Whether the provider supports the store field. Default: auto-detected from URL.", - }), - ), - supportsDeveloperRole: optionalCompatBoolean({ - description: - "Whether the provider supports the developer role instead of system. Default: auto-detected from URL.", +const OpenAICompletionsCompatProperties = { + supportsStore: opt( + bool({ + description: "Whether the provider supports the store field. Default: auto-detected from URL.", }), - supportsReasoningEffort: Type.Optional( - Type.Boolean({ - description: "Whether the provider supports reasoning_effort. Default: auto-detected from URL.", - }), - ), - supportsUsageInStreaming: Type.Optional( - Type.Boolean({ - description: - "Whether the provider supports stream_options.include_usage for token usage in streaming responses.", - default: true, - }), - ), - supportsFinishReason: Type.Optional( - Type.Boolean({ - description: - "Whether streamed responses include finish_reason. When false, pi infers stop or toolUse when the stream ends.", - default: true, - }), - ), - maxTokensField: Type.Optional( - Type.Union([Type.Literal("max_completion_tokens"), Type.Literal("max_tokens")], { - description: "Which field to use for max tokens. Default: auto-detected from URL.", - }), - ), - requiresToolResultName: Type.Optional( - Type.Boolean({ description: "Whether tool results require the name field. Default: auto-detected from URL." }), - ), - requiresAssistantAfterToolResult: Type.Optional( - Type.Boolean({ - description: - "Whether a user message after tool results requires an assistant message in between. Default: auto-detected from URL.", - }), - ), - requiresThinkingAsText: Type.Optional( - Type.Boolean({ - description: - "Whether thinking blocks must be converted to text blocks with delimiters. Default: auto-detected from URL.", - }), - ), - requiresReasoningContentOnAssistantMessages: Type.Optional( - Type.Boolean({ - description: - "Whether all replayed assistant messages must include an empty reasoning_content field when reasoning is enabled. Default: auto-detected from URL.", - }), - ), - thinkingFormat: Type.Optional( - Type.Union( - [ - Type.Literal("openai"), - Type.Literal("openrouter"), - Type.Literal("deepseek"), - Type.Literal("together"), - Type.Literal("baseten"), - Type.Literal("zai"), - Type.Literal("qwen"), - Type.Literal("chat-template"), - Type.Literal("qwen-chat-template"), - Type.Literal("string-thinking"), - Type.Literal("ant-ling"), - ], - { - description: - 'Format for reasoning or thinking parameters. When omitted, Pi auto-detects the format from the provider URL. "openai" uses reasoning_effort, "openrouter" uses reasoning.effort, "deepseek" uses thinking.type plus reasoning_effort when supported, "together" uses reasoning.enabled plus reasoning_effort when supported, "baseten" uses configurable chat_template_args plus reasoning_effort when supported, "zai" uses thinking.type, "qwen" uses top-level enable_thinking, "qwen-chat-template" uses chat_template_kwargs.enable_thinking and preserve_thinking, "chat-template" uses configurable chat_template_kwargs, "string-thinking" uses top-level thinking, and "ant-ling" uses reasoning.effort only when the mapped effort is non-null.', - }, - ), - ), - chatTemplateKwargs: Type.Optional( - Type.Record(Type.String(), ChatTemplateKwargValueSchema, { - description: - 'Kwargs sent as chat_template_kwargs when thinkingFormat is "chat-template". Use $var with "thinking.enabled", "thinking.effort", or "thinking.budget" for pi-controlled values.', - }), - ), - chatTemplateArgs: Type.Optional( - Type.Record(Type.String(), ChatTemplateKwargValueSchema, { - description: - 'Arguments sent as chat_template_args when thinkingFormat is "baseten". Use $var with "thinking.enabled", "thinking.effort", or "thinking.budget" for pi-controlled values.', - }), - ), - openRouterRouting: Type.Optional(OpenRouterRoutingSchema), - vercelGatewayRouting: Type.Optional(VercelGatewayRoutingSchema), - zaiToolStream: Type.Optional( - Type.Boolean({ - description: "Whether z.ai supports top-level tool_stream for streaming tool call deltas.", - default: false, - }), - ), - thinkingTokenBudgetField: Type.Optional( - thinkingTokenBudgetField({ - description: - 'Top-level request field used to cap reasoning tokens from thinkingBudgets. Reasoning and the answer share max_tokens on these endpoints. "thinking_token_budget" is vLLM, "thinking_budget" is Qwen, DashScope, or SGLang, and "thinking_budget_tokens" is llama.cpp. Off by default and not set on the generated catalog.', - }), - ), - supportsThinkingTokenBudget: Type.Optional( - Type.Boolean({ - description: - 'Alias for thinkingTokenBudgetField: "thinking_token_budget" (vLLM). Prefer thinkingTokenBudgetField.', - default: false, - }), - ), - supportsOpenAIGrammarTools: optionalCompatBoolean({ - description: - "Whether the provider supports OpenAI custom tools with Lark or regex grammar formats. When false, grammar-constrained tools fall back to normal function tools. The generated catalog enables this for capable models.", - default: false, - }), - supportsMidConvoSystemMessages: optionalCompatBoolean({ - description: - "Whether the exact model accepts system or developer messages after the conversation has started. When false, later system messages are folded into the leading system message. The generated catalog enables this for verified models.", - default: false, - }), - supportsMidConvoToolAdditions: Type.Optional( - Type.Boolean({ - description: - "Whether system messages can introduce additional tools mid-conversation. Requires supportsMidConvoSystemMessages. The generated catalog enables this for capable models.", - default: false, - }), - ), - supportsStrictMode: optionalCompatBoolean({ - description: - "Whether the provider supports the strict field in tool definitions. Generated capable models enable it explicitly.", - default: false, - }), - cacheControlFormat: Type.Optional( - Type.Literal("anthropic", { - description: - "Cache control convention for prompt caching. Anthropic applies cache_control markers to the system prompt, last tool definition, and last user, assistant, or tool-result text content.", - }), - ), - sendSessionAffinityHeaders: optionalCompatBoolean({ - description: - "Whether to send session-affinity data from options.sessionId. Default: true for OpenRouter endpoints, false otherwise.", - }), - sessionAffinityFormat: optionalSessionAffinityFormat({ - description: - "Session-affinity header format. openai sends session_id, x-client-request-id, and x-session-affinity; openai-nosession sends x-client-request-id and x-session-affinity; openrouter sends x-session-id. Does not affect prompt_cache_key. Default: auto-detected.", - }), - supportsLongCacheRetention: optionalCompatBoolean({ - description: - 'Whether the provider supports long prompt cache retention (prompt_cache_retention: "24h" or Anthropic-style cache_control.ttl: "1h", depending on format). Default: auto-detected from provider and URL.', - }), - vllmPriority: Type.Optional( - Type.Number({ - description: - "vLLM scheduler priority sent as the top-level priority request field. Lower values are handled earlier and the server default is 0. Only meaningful with --scheduling-policy priority. Off by default and not set on the generated catalog.", - }), - ), - }, - { + ), + supportsDeveloperRole: optionalCompatBoolean({ description: - "Compatibility settings for OpenAI-compatible completions APIs. Use this to override URL-based auto-detection for custom providers.", - additionalProperties: true, - }, -); - -export const OpenAIResponsesCompatSchema = Type.Object( - { - supportsDeveloperRole: optionalCompatBoolean({ - description: "Whether the provider supports the developer role instead of system.", + "Whether the provider supports the developer role instead of system. Default: auto-detected from URL.", + }), + supportsReasoningEffort: opt( + bool({ + description: "Whether the provider supports reasoning_effort. Default: auto-detected from URL.", + }), + ), + supportsUsageInStreaming: opt( + bool({ + description: + "Whether the provider supports stream_options.include_usage for token usage in streaming responses.", default: true, }), - supportsMidConvoSystemMessages: optionalCompatBoolean({ + ), + supportsFinishReason: opt( + bool({ description: - "Whether the exact model accepts developer or system messages after the conversation has started. When false, later system messages are folded into the leading system message. The generated catalog enables this for verified models.", - default: false, - }), - sessionAffinityFormat: optionalSessionAffinityFormat({ - description: - "Session-affinity header format. openai sends session_id and x-client-request-id; openai-nosession sends x-client-request-id; openrouter sends x-session-id. Does not affect prompt_cache_key. Default: auto-detected.", - }), - supportsLongCacheRetention: optionalCompatBoolean({ - description: - 'Whether the provider supports long prompt cache retention. This uses prompt_cache_options.ttl: "30m" on GPT-5.6+ and prompt_cache_retention: "24h" on earlier models.', + "Whether streamed responses include finish_reason. When false, pi infers stop or toolUse when the stream ends.", default: true, }), - supportsStrictMode: optionalCompatBoolean({ - description: - "Whether the provider supports strict JSON-schema function tools. Defaults are API-specific; generated OpenAI models enable it explicitly.", + ), + maxTokensField: opt( + union([stringLiteral("max_completion_tokens"), stringLiteral("max_tokens")], { + description: "Which field to use for max tokens. Default: auto-detected from URL.", }), - supportsOpenAIGrammarTools: optionalCompatBoolean({ + ), + requiresToolResultName: opt( + bool({ description: "Whether tool results require the name field. Default: auto-detected from URL." }), + ), + requiresAssistantAfterToolResult: opt( + bool({ description: - "Whether to emit OpenAI custom tools with Lark or regex grammar formats. When false, grammar-constrained tools fall back to normal function tools. The generated catalog enables this for capable models.", + "Whether a user message after tool results requires an assistant message in between. Default: auto-detected from URL.", + }), + ), + requiresThinkingAsText: opt( + bool({ + description: + "Whether thinking blocks must be converted to text blocks with delimiters. Default: auto-detected from URL.", + }), + ), + requiresReasoningContentOnAssistantMessages: opt( + bool({ + description: + "Whether all replayed assistant messages must include an empty reasoning_content field when reasoning is enabled. Default: auto-detected from URL.", + }), + ), + thinkingFormat: opt( + union( + [ + stringLiteral("openai"), + stringLiteral("openrouter"), + stringLiteral("deepseek"), + stringLiteral("together"), + stringLiteral("baseten"), + stringLiteral("zai"), + stringLiteral("qwen"), + stringLiteral("chat-template"), + stringLiteral("qwen-chat-template"), + stringLiteral("string-thinking"), + stringLiteral("ant-ling"), + ], + { + description: + 'Format for reasoning or thinking parameters. When omitted, Pi auto-detects the format from the provider URL. "openai" uses reasoning_effort, "openrouter" uses reasoning.effort, "deepseek" uses thinking.type plus reasoning_effort when supported, "together" uses reasoning.enabled plus reasoning_effort when supported, "baseten" uses configurable chat_template_args plus reasoning_effort when supported, "zai" uses thinking.type, "qwen" uses top-level enable_thinking, "qwen-chat-template" uses chat_template_kwargs.enable_thinking and preserve_thinking, "chat-template" uses configurable chat_template_kwargs, "string-thinking" uses top-level thinking, and "ant-ling" uses reasoning.effort only when the mapped effort is non-null.', + }, + ), + ), + chatTemplateKwargs: opt( + stringRecord(ChatTemplateKwargValueSchema, { + description: + 'Kwargs sent as chat_template_kwargs when thinkingFormat is "chat-template". Use $var with "thinking.enabled", "thinking.effort", or "thinking.budget" for pi-controlled values.', + }), + ), + chatTemplateArgs: opt( + stringRecord(ChatTemplateKwargValueSchema, { + description: + 'Arguments sent as chat_template_args when thinkingFormat is "baseten". Use $var with "thinking.enabled", "thinking.effort", or "thinking.budget" for pi-controlled values.', + }), + ), + openRouterRouting: opt(OpenRouterRoutingSchema), + vercelGatewayRouting: opt(VercelGatewayRoutingSchema), + zaiToolStream: opt( + bool({ + description: "Whether z.ai supports top-level tool_stream for streaming tool call deltas.", default: false, }), - supportsAdditionalTools: Type.Optional( - Type.Boolean({ - description: "Whether the model supports message-anchored additional_tools input items.", - default: false, - }), - ), - supportsToolSearch: Type.Optional( - Type.Boolean({ - description: "Whether the model supports client-executed tool search for transcript-anchored additions.", - default: false, - }), - ), - supportsExplicitPromptCacheMode: Type.Optional( - Type.Boolean({ - description: "Whether the model accepts prompt_cache_options. Older OpenAI models reject the parameter.", - default: false, - }), - ), - supportsMidConvoEffort: Type.Optional( - Type.Boolean({ - description: - "Whether thinking level changes are sent as configuration_update input items instead of changing reasoning.effort, so a change keeps the prompt cache. Only used by the openai-responses API, and ignored with Sign in with ChatGPT on the openai provider.", - default: false, - }), - ), - supportsMaxOutputTokens: Type.Optional( - Type.Boolean({ - description: "Whether the provider accepts max_output_tokens. Some Codex-protocol gateways reject it.", - default: true, - }), - ), - }, - { description: "Compatibility settings for OpenAI Responses APIs.", additionalProperties: true }, -); + ), + thinkingTokenBudgetField: opt( + thinkingTokenBudgetField({ + description: + 'Top-level request field used to cap reasoning tokens from thinkingBudgets. Reasoning and the answer share max_tokens on these endpoints. "thinking_token_budget" is vLLM, "thinking_budget" is Qwen, DashScope, or SGLang, and "thinking_budget_tokens" is llama.cpp. Off by default and not set on the generated catalog.', + }), + ), + supportsThinkingTokenBudget: opt( + bool({ + description: + 'Alias for thinkingTokenBudgetField: "thinking_token_budget" (vLLM). Prefer thinkingTokenBudgetField.', + default: false, + }), + ), + supportsOpenAIGrammarTools: optionalCompatBoolean({ + description: + "Whether the provider supports OpenAI custom tools with Lark or regex grammar formats. When false, grammar-constrained tools fall back to normal function tools. The generated catalog enables this for capable models.", + default: false, + }), + supportsMidConvoSystemMessages: optionalCompatBoolean({ + description: + "Whether the exact model accepts system or developer messages after the conversation has started. When false, later system messages are folded into the leading system message. The generated catalog enables this for verified models.", + default: false, + }), + supportsMidConvoToolAdditions: opt( + bool({ + description: + "Whether system messages can introduce additional tools mid-conversation. Requires supportsMidConvoSystemMessages. The generated catalog enables this for capable models.", + default: false, + }), + ), + supportsStrictMode: optionalCompatBoolean({ + description: + "Whether the provider supports the strict field in tool definitions. Generated capable models enable it explicitly.", + default: false, + }), + cacheControlFormat: opt( + stringLiteral("anthropic", { + description: + "Cache control convention for prompt caching. Anthropic applies cache_control markers to the system prompt, last tool definition, and last user, assistant, or tool-result text content.", + }), + ), + sendSessionAffinityHeaders: optionalCompatBoolean({ + description: + "Whether to send session-affinity data from options.sessionId. Default: true for OpenRouter endpoints, false otherwise.", + }), + sessionAffinityFormat: optionalSessionAffinityFormat({ + description: + "Session-affinity header format. openai sends session_id, x-client-request-id, and x-session-affinity; openai-nosession sends x-client-request-id and x-session-affinity; openrouter sends x-session-id. Does not affect prompt_cache_key. Default: auto-detected.", + }), + supportsLongCacheRetention: optionalCompatBoolean({ + description: + 'Whether the provider supports long prompt cache retention (prompt_cache_retention: "24h" or Anthropic-style cache_control.ttl: "1h", depending on format). Default: auto-detected from provider and URL.', + }), + vllmPriority: opt( + num({ + description: + "vLLM scheduler priority sent as the top-level priority request field. Lower values are handled earlier and the server default is 0. Only meaningful with --scheduling-policy priority. Off by default and not set on the generated catalog.", + }), + ), +}; -export const AnthropicMessagesCompatSchema = Type.Object( - { - supportsEagerToolInputStreaming: Type.Optional( - Type.Boolean({ - description: - "Whether the provider accepts per-tool eager_input_streaming. When false, the Anthropic provider omits tools[].eager_input_streaming and sends the legacy fine-grained-tool-streaming-2025-05-14 beta header for tool-enabled requests.", - default: true, - }), - ), - supportsLongCacheRetention: optionalCompatBoolean({ - description: "Whether the provider supports Anthropic long cache retention through cache_control.ttl.", +export const OpenAICompletionsCompatSchema = openObject(OpenAICompletionsCompatProperties, { + description: + "Compatibility settings for OpenAI-compatible completions APIs. Use this to override URL-based auto-detection for custom providers.", + additionalProperties: true, +}); + +const OpenAIResponsesCompatProperties = { + supportsDeveloperRole: optionalCompatBoolean({ + description: "Whether the provider supports the developer role instead of system.", + default: true, + }), + supportsMidConvoSystemMessages: optionalCompatBoolean({ + description: + "Whether the exact model accepts developer or system messages after the conversation has started. When false, later system messages are folded into the leading system message. The generated catalog enables this for verified models.", + default: false, + }), + sessionAffinityFormat: optionalSessionAffinityFormat({ + description: + "Session-affinity header format. openai sends session_id and x-client-request-id; openai-nosession sends x-client-request-id; openrouter sends x-session-id. Does not affect prompt_cache_key. Default: auto-detected.", + }), + supportsLongCacheRetention: optionalCompatBoolean({ + description: + 'Whether the provider supports long prompt cache retention. This uses prompt_cache_options.ttl: "30m" on GPT-5.6+ and prompt_cache_retention: "24h" on earlier models.', + default: true, + }), + supportsStrictMode: optionalCompatBoolean({ + description: + "Whether the provider supports strict JSON-schema function tools. Defaults are API-specific; generated OpenAI models enable it explicitly.", + }), + supportsOpenAIGrammarTools: optionalCompatBoolean({ + description: + "Whether to emit OpenAI custom tools with Lark or regex grammar formats. When false, grammar-constrained tools fall back to normal function tools. The generated catalog enables this for capable models.", + default: false, + }), + supportsAdditionalTools: opt( + bool({ + description: "Whether the model supports message-anchored additional_tools input items.", + default: false, + }), + ), + supportsToolSearch: opt( + bool({ + description: "Whether the model supports client-executed tool search for transcript-anchored additions.", + default: false, + }), + ), + supportsExplicitPromptCacheMode: opt( + bool({ + description: "Whether the model accepts prompt_cache_options. Older OpenAI models reject the parameter.", + default: false, + }), + ), + supportsMidConvoEffort: opt( + bool({ + description: + "Whether thinking level changes are sent as configuration_update input items instead of changing reasoning.effort, so a change keeps the prompt cache. Only used by the openai-responses API, and ignored with Sign in with ChatGPT on the openai provider.", + default: false, + }), + ), + supportsMaxOutputTokens: opt( + bool({ + description: "Whether the provider accepts max_output_tokens. Some Codex-protocol gateways reject it.", default: true, }), - sendSessionAffinityHeaders: optionalCompatBoolean({ + ), +}; + +export const OpenAIResponsesCompatSchema = openObject(OpenAIResponsesCompatProperties, { + description: "Compatibility settings for OpenAI Responses APIs.", + additionalProperties: true, +}); + +const AnthropicMessagesCompatProperties = { + supportsEagerToolInputStreaming: opt( + bool({ description: - "Whether to send x-session-affinity from options.sessionId when caching is enabled. Required for providers like Fireworks that use session affinity for prompt cache routing; requests to the same replica maximize cache hits. Default: true for OpenRouter endpoints, false otherwise.", + "Whether the provider accepts per-tool eager_input_streaming. When false, the Anthropic provider omits tools[].eager_input_streaming and sends the legacy fine-grained-tool-streaming-2025-05-14 beta header for tool-enabled requests.", + default: true, }), - sessionAffinityFormat: Type.Optional( - Type.Literal("openrouter", { - description: - "Session-affinity format. openrouter sends x-session-id; when unset, sends x-session-affinity.", - }), - ), - supportsCacheControlOnTools: Type.Optional( - Type.Boolean({ - description: - "Whether the provider supports Anthropic-style cache_control markers on tool definitions. When false, cache_control is omitted from tool parameters. Some Anthropic-compatible providers, such as Fireworks, do not support this field on tools and may reject or ignore it.", - default: true, - }), - ), - supportsTemperature: Type.Optional( - Type.Boolean({ - description: - "Whether the model accepts the Anthropic temperature request field. Claude Opus 4.7+ rejects non-default values.", - default: true, - }), - ), - forceAdaptiveThinking: Type.Optional( - Type.Boolean({ - description: - "Whether to force adaptive thinking (thinking.type: adaptive plus output_config.effort) regardless of model ID. Built-in models that require adaptive thinking set this in generated metadata. Custom Anthropic-compatible providers can set this to true for any model whose upstream requires the adaptive format. Set false to opt out on overridden built-in models.", - default: false, - }), - ), - allowEmptySignature: Type.Optional( - Type.Boolean({ - description: "Whether to replay empty thinking signatures instead of converting thinking to text.", - default: false, - }), - ), - supportsStrictTools: Type.Optional( - Type.Boolean({ - description: - "Whether the provider supports Anthropic strict tool schemas. Generated Anthropic models enable it explicitly.", - default: false, - }), - ), - supportsMidConvoEffort: Type.Optional( - Type.Boolean({ - description: - "Whether the exact model transport supports effort-only system messages and thinking binding controls.", - default: false, - }), - ), - supportsMidConvoSystemMessages: optionalCompatBoolean({ + ), + supportsLongCacheRetention: optionalCompatBoolean({ + description: "Whether the provider supports Anthropic long cache retention through cache_control.ttl.", + default: true, + }), + sendSessionAffinityHeaders: optionalCompatBoolean({ + description: + "Whether to send x-session-affinity from options.sessionId when caching is enabled. Required for providers like Fireworks that use session affinity for prompt cache routing; requests to the same replica maximize cache hits. Default: true for OpenRouter endpoints, false otherwise.", + }), + sessionAffinityFormat: opt( + stringLiteral("openrouter", { + description: "Session-affinity format. openrouter sends x-session-id; when unset, sends x-session-affinity.", + }), + ), + supportsCacheControlOnTools: opt( + bool({ description: - "Whether the exact model accepts system-role messages inside the conversation. When false, later system messages are folded into the top-level system prompt.", + "Whether the provider supports Anthropic-style cache_control markers on tool definitions. When false, cache_control is omitted from tool parameters. Some Anthropic-compatible providers, such as Fireworks, do not support this field on tools and may reject or ignore it.", + default: true, + }), + ), + supportsTemperature: opt( + bool({ + description: + "Whether the model accepts the Anthropic temperature request field. Claude Opus 4.7+ rejects non-default values.", + default: true, + }), + ), + forceAdaptiveThinking: opt( + bool({ + description: + "Whether to force adaptive thinking (thinking.type: adaptive plus output_config.effort) regardless of model ID. Built-in models that require adaptive thinking set this in generated metadata. Custom Anthropic-compatible providers can set this to true for any model whose upstream requires the adaptive format. Set false to opt out on overridden built-in models.", default: false, }), - supportsMidConvoToolChanges: Type.Optional( - Type.Boolean({ - description: - "Whether the exact model accepts mid-conversation tool_addition and tool_removal blocks. Requires supportsMidConvoSystemMessages.", - default: false, - }), - ), - allowedFallbackModels: Type.Optional( - Type.Array(AnthropicAllowedFallbackModelSchema, { - maxItems: 3, - description: - "Models Anthropic accepts for server-side refusal fallback, with local pricing metadata for returned fallback responses. When absent or empty, callers must omit fallbacks; Anthropic rejects the field for models with no permitted fallback targets.", - }), - ), - }, - { description: "Compatibility settings for Anthropic Messages-compatible APIs.", additionalProperties: true }, -); - -export const BedrockCompatSchema = Type.Object( - { - supportsStrictMode: optionalCompatBoolean({ - description: "Whether the model supports Bedrock strict tool schemas.", + ), + allowEmptySignature: opt( + bool({ + description: "Whether to replay empty thinking signatures instead of converting thinking to text.", default: false, }), - }, - { description: "Compatibility settings for Amazon Bedrock models.", additionalProperties: true }, -); - -export const MistralConversationsCompatSchema = Type.Object( - { - supportsMidConvoSystemMessages: optionalCompatBoolean({ + ), + supportsStrictTools: opt( + bool({ description: - "Whether the exact model accepts system messages after the conversation has started. When false, later system messages are folded into the leading system message.", + "Whether the provider supports Anthropic strict tool schemas. Generated Anthropic models enable it explicitly.", default: false, }), - }, - { description: "Compatibility settings for the Mistral chat API.", additionalProperties: true }, -); - -export const SystemOneCompatSchema = Type.Object( - { - imageInput: Type.Optional( - Type.Union([Type.Literal("assets"), Type.Literal("images"), Type.Literal("state-image-urls")], { - description: - 'How the service takes classifier images. "assets" sends TypeSafe\'s assets map (PNG and JPEG) and references the images from the state, as TypeSafe and Jev on Workers AI expect. "images" sends Clef\'s images list of data URLs (PNG, JPEG, and WebP, at most four), as Clef on Workers AI expects. "state-image-urls" places image_url parts before the state, as OpenRouter expects; OpenRouter drops assets without an error. Auto-detected from the provider, baseUrl, and model ID. Classifier models read it from the provider-level compat; models and modelOverrides entries apply to chat models only.', - }), - ), - sendImageOptInHeader: optionalCompatBoolean({ + ), + supportsMidConvoEffort: opt( + bool({ description: - "Whether requests with images send the TypeSafe-Danger-Zone-Images: true header, which TypeSafe's own API requires for its experimental image input. Defaults to true for the typesafe-system-one API unless the model is detected as OpenRouter.", + "Whether the exact model transport supports effort-only system messages and thinking binding controls.", + default: false, }), - }, - { description: "Compatibility settings for System One classifier APIs.", additionalProperties: true }, -); + ), + supportsMidConvoSystemMessages: optionalCompatBoolean({ + description: + "Whether the exact model accepts system-role messages inside the conversation. When false, later system messages are folded into the top-level system prompt.", + default: false, + }), + supportsMidConvoToolChanges: opt( + bool({ + description: + "Whether the exact model accepts mid-conversation tool_addition and tool_removal blocks. Requires supportsMidConvoSystemMessages.", + default: false, + }), + ), + allowedFallbackModels: opt( + arr(AnthropicAllowedFallbackModelSchema, { + maxItems: 3, + description: + "Models Anthropic accepts for server-side refusal fallback, with local pricing metadata for returned fallback responses. When absent or empty, callers must omit fallbacks; Anthropic rejects the field for models with no permitted fallback targets.", + }), + ), +}; + +export const AnthropicMessagesCompatSchema = openObject(AnthropicMessagesCompatProperties, { + description: "Compatibility settings for Anthropic Messages-compatible APIs.", + additionalProperties: true, +}); + +const BedrockCompatProperties = { + supportsStrictMode: optionalCompatBoolean({ + description: "Whether the model supports Bedrock strict tool schemas.", + default: false, + }), +}; + +export const BedrockCompatSchema = openObject(BedrockCompatProperties, { + description: "Compatibility settings for Amazon Bedrock models.", + additionalProperties: true, +}); + +const MistralConversationsCompatProperties = { + supportsMidConvoSystemMessages: optionalCompatBoolean({ + description: + "Whether the exact model accepts system messages after the conversation has started. When false, later system messages are folded into the leading system message.", + default: false, + }), +}; + +export const MistralConversationsCompatSchema = openObject(MistralConversationsCompatProperties, { + description: "Compatibility settings for the Mistral chat API.", + additionalProperties: true, +}); + +const SystemOneCompatProperties = { + imageInput: opt( + union([stringLiteral("assets"), stringLiteral("images"), stringLiteral("state-image-urls")], { + description: + 'How the service takes classifier images. "assets" sends TypeSafe\'s assets map (PNG and JPEG) and references the images from the state, as TypeSafe and Jev on Workers AI expect. "images" sends Clef\'s images list of data URLs (PNG, JPEG, and WebP, at most four), as Clef on Workers AI expects. "state-image-urls" places image_url parts before the state, as OpenRouter expects; OpenRouter drops assets without an error. Auto-detected from the provider, baseUrl, and model ID. Classifier models read it from the provider-level compat; models and modelOverrides entries apply to chat models only.', + }), + ), + sendImageOptInHeader: optionalCompatBoolean({ + description: + "Whether requests with images send the TypeSafe-Danger-Zone-Images: true header, which TypeSafe's own API requires for its experimental image input. Defaults to true for the typesafe-system-one API unless the model is detected as OpenRouter.", + }), +}; + +export const SystemOneCompatSchema = openObject(SystemOneCompatProperties, { + description: "Compatibility settings for System One classifier APIs.", + additionalProperties: true, +}); const ProviderCompatPropertyOverrides = { supportsDeveloperRole: optionalCompatBoolean({ @@ -569,7 +581,7 @@ const ProviderCompatPropertyOverrides = { sendSessionAffinityHeaders: optionalCompatBoolean({ description: "Whether to send session-affinity data from options.sessionId. Defaults are API-specific.", }), -} satisfies TProperties; +} satisfies Properties; type UnionToIntersection = (Union extends unknown ? (value: Union) => void : never) extends ( value: infer Intersection, @@ -577,7 +589,7 @@ type UnionToIntersection = (Union extends unknown ? (value: Union) => voi ? Intersection : never; -type MergedCompatProperties = Omit< +type MergedCompatProperties = Omit< UnionToIntersection, keyof Overrides > & @@ -588,11 +600,11 @@ type MergedCompatProperties(propertyGroups: PropertyGroups, overrides: Overrides): MergedCompatProperties { - const merged: TProperties = {}; +function mergeCompatProperties( + propertyGroups: PropertyGroups, + overrides: Overrides, +): MergedCompatProperties { + const merged: Record = {}; const duplicateNames = new Set(); for (const properties of propertyGroups) { @@ -630,16 +642,25 @@ export const CompatSchemasByApi = { } as const; const ProviderCompatProperties = mergeCompatProperties( - [...new Set(Object.values(CompatSchemasByApi))].map((schema) => schema.properties), + [ + OpenAICompletionsCompatProperties, + OpenAIResponsesCompatProperties, + AnthropicMessagesCompatProperties, + BedrockCompatProperties, + MistralConversationsCompatProperties, + SystemOneCompatProperties, + ], ProviderCompatPropertyOverrides, ); -export const ProviderCompatSchema = Type.Object(ProviderCompatProperties, { +export const ProviderCompatSchema = openObject(ProviderCompatProperties, { description: "Provider and model compatibility overrides.", additionalProperties: true, }); export type ChatTemplateKwargValue = Static; +/** Provider and model compatibility overrides in models.json: every API's compat fields. */ +export type ProviderCompat = Static; /** Top-level request field used to cap reasoning tokens on OpenAI-compatible servers. */ export type ThinkingTokenBudgetField = Static; export type SessionAffinityFormat = Static; diff --git a/packages/ai/src/providers/model-schema.ts b/packages/ai/src/providers/model-schema.ts index 453ceabc7..2eb0a5a52 100644 --- a/packages/ai/src/providers/model-schema.ts +++ b/packages/ai/src/providers/model-schema.ts @@ -1,4 +1,17 @@ -import { type Static, type TSchemaOptions, Type } from "typebox"; +import { + type Annotations, + arr, + int, + nul, + num, + oneOf, + opt, + type Static, + type StaticObject, + str, + union, +} from "@earendil-works/pi-schema"; +import { openObject as object } from "./schema-builders.ts"; export const THINKING_LEVELS = ["minimal", "low", "medium", "high", "xhigh", "max"] as const; export const MODEL_THINKING_LEVELS = ["off", ...THINKING_LEVELS] as const; @@ -6,19 +19,31 @@ export const MODEL_INPUT_MODALITIES = ["text", "image"] as const; export const CACHE_RETENTIONS = ["none", "short", "long"] as const; export const MODEL_PROMPT_CACHE_RETENTIONS = ["short", "long"] as const; -export const ThinkingLevelSchema = Type.Enum(THINKING_LEVELS); -export const ModelThinkingLevelSchema = Type.Enum(MODEL_THINKING_LEVELS); -export const ModelInputModalitySchema = Type.Enum(MODEL_INPUT_MODALITIES); -export const CacheRetentionSchema = Type.Enum(CACHE_RETENTIONS); +export const ThinkingLevelSchema = oneOf(THINKING_LEVELS); +export const ModelThinkingLevelSchema = oneOf(MODEL_THINKING_LEVELS); +export const ModelInputModalitySchema = oneOf(MODEL_INPUT_MODALITIES); +export const CacheRetentionSchema = oneOf(CACHE_RETENTIONS); -const ThinkingLevelMapValueSchema = Type.Union([Type.String(), Type.Null()]); -export const ThinkingLevelMapSchema = Type.Partial(Type.Record(ModelThinkingLevelSchema, ThinkingLevelMapValueSchema), { - description: - "Provider value sent for each thinking level. null marks a level unsupported. Levels the model does not map use the provider's default value through high; xhigh and max are unsupported.", -}); +const ThinkingLevelMapValueSchema = union([str(), nul()]); +export const ThinkingLevelMapSchema = object( + { + off: opt(ThinkingLevelMapValueSchema), + minimal: opt(ThinkingLevelMapValueSchema), + low: opt(ThinkingLevelMapValueSchema), + medium: opt(ThinkingLevelMapValueSchema), + high: opt(ThinkingLevelMapValueSchema), + xhigh: opt(ThinkingLevelMapValueSchema), + max: opt(ThinkingLevelMapValueSchema), + }, + { + description: + "Provider value sent for each thinking level. null marks a level unsupported. Levels the model does not map use the provider's default value through high; xhigh and max are unsupported.", + }, +); -export const ModelPromptCacheSchema = Type.Partial( - Type.Record(Type.Enum(MODEL_PROMPT_CACHE_RETENTIONS), Type.Number({ exclusiveMinimum: 0 })), +const PromptCacheLifetimeSchema = num({ exclusiveMinimum: 0 }); +export const ModelPromptCacheSchema = object( + { short: opt(PromptCacheLifetimeSchema), long: opt(PromptCacheLifetimeSchema) }, { description: "Best-effort prompt cache lifetime in seconds for the short and long retention tiers. Pi warms only tiers with a known lifetime.", @@ -26,63 +51,54 @@ export const ModelPromptCacheSchema = Type.Partial( ); const ModelCostRatesProperties = { - input: Type.Number({ description: "Input cost in USD per million tokens." }), - output: Type.Number({ description: "Output cost in USD per million tokens." }), - cacheRead: Type.Number({ description: "Cache-read cost in USD per million tokens." }), - cacheWrite: Type.Number({ description: "Cache-write cost in USD per million tokens." }), + input: num({ description: "Input cost in USD per million tokens." }), + output: num({ description: "Output cost in USD per million tokens." }), + cacheRead: num({ description: "Cache-read cost in USD per million tokens." }), + cacheWrite: num({ description: "Cache-write cost in USD per million tokens." }), }; -export const ModelCostRatesSchema = Type.Object(ModelCostRatesProperties); -export const ModelCostTierSchema = Type.Object({ - inputTokensAbove: Type.Number({ +export const ModelCostRatesSchema = object(ModelCostRatesProperties); +export const ModelCostTierSchema = object({ + inputTokensAbove: num({ description: "Use this tier when total request input exceeds this token count.", }), ...ModelCostRatesProperties, }); -export const ModelCostSchema = Type.Object({ +export const ModelCostSchema = object({ ...ModelCostRatesProperties, - tiers: Type.Optional( - Type.Array(ModelCostTierSchema, { + tiers: opt( + arr(ModelCostTierSchema, { description: "Request-wide pricing tiers. The highest matching input threshold applies to the full request.", }), ), }); -function modelImageResizeOptions(options?: TSchemaOptions) { - return Type.Object( - { - maxWidth: Type.Optional(Type.Integer({ minimum: 1 })), - maxHeight: Type.Optional(Type.Integer({ minimum: 1 })), - maxBytes: Type.Optional( - Type.Integer({ minimum: 1, description: "Maximum base64-encoded payload size in bytes." }), - ), - jpegQuality: Type.Optional(Type.Integer({ minimum: 1, maximum: 100 })), - }, - options, - ); +const ModelImageResizeProperties = { + maxWidth: opt(int({ minimum: 1 })), + maxHeight: opt(int({ minimum: 1 })), + maxBytes: opt(int({ minimum: 1, description: "Maximum base64-encoded payload size in bytes." })), + jpegQuality: opt(int({ minimum: 1, maximum: 100 })), +}; + +function modelImageResizeOptions(options?: Annotations>) { + return object(ModelImageResizeProperties, options); } export const ModelImageResizeOptionsSchema = modelImageResizeOptions(); -export const ModelImageInputLimitsSchema = Type.Object({ - resize: Type.Optional( +export const ModelImageInputLimitsSchema = object({ + resize: opt( modelImageResizeOptions({ description: "Cache-safe resize profile applied before a new image enters conversation history.", }), ), - maxPerMessage: Type.Optional( - Type.Integer({ minimum: 1, description: "Maximum images accepted in one provider message." }), - ), - maxPerRequest: Type.Optional( - Type.Integer({ minimum: 1, description: "Maximum images accepted across one provider request." }), - ), + maxPerMessage: opt(int({ minimum: 1, description: "Maximum images accepted in one provider message." })), + maxPerRequest: opt(int({ minimum: 1, description: "Maximum images accepted across one provider request." })), }); -export const ModelInputLimitsSchema = Type.Object({ - maxRequestBytes: Type.Optional( - Type.Integer({ minimum: 1, description: "Maximum serialized provider request size in bytes." }), - ), - images: Type.Optional(ModelImageInputLimitsSchema), +export const ModelInputLimitsSchema = object({ + maxRequestBytes: opt(int({ minimum: 1, description: "Maximum serialized provider request size in bytes." })), + images: opt(ModelImageInputLimitsSchema), }); export type ThinkingLevel = Static; diff --git a/packages/ai/src/providers/schema-builders.ts b/packages/ai/src/providers/schema-builders.ts new file mode 100644 index 000000000..2df034ca4 --- /dev/null +++ b/packages/ai/src/providers/schema-builders.ts @@ -0,0 +1,34 @@ +// Builders for pi-ai's own configuration schemas (model metadata, compat settings): pi-schema's builders, shaped like the +// TypeBox schemas they replace, so models.json and the catalog validate exactly as before. + +import { + type Annotations, + obj, + type Properties, + type Schema, + type Static, + type StaticObject, +} from "@earendil-works/pi-schema"; + +/** An object that allows other properties; `additionalProperties: true` says so explicitly. */ +export function openObject( + properties: P, + options: Annotations> & { readonly additionalProperties?: true } = {}, +): Schema> { + const { additionalProperties, ...annotations } = options; + const schema = obj(properties, { ...annotations, open: true }); + return additionalProperties ? { ...schema, additionalProperties } : schema; +} + +/** One string: `{ type: "string", const: value }`. */ +export function stringLiteral(value: V, options: Annotations = {}): Schema { + return { type: "string", const: value, ...options }; +} + +/** An object with any string keys and values of `value`: `{ type: "object", patternProperties: { "^.*$": value } }`. */ +export function stringRecord( + value: S, + options: Annotations>> = {}, +): Schema>> { + return { type: "object", patternProperties: { "^.*$": value }, ...options }; +} diff --git a/packages/ai/src/types.ts b/packages/ai/src/types.ts index 32b5a5f31..e435126ea 100644 --- a/packages/ai/src/types.ts +++ b/packages/ai/src/types.ts @@ -742,8 +742,6 @@ export interface ClassifierResult { timestamp: number; // Unix timestamp in milliseconds } -import type { TSchema } from "typebox"; - /** OpenAI grammar variants for constrained sampling. */ export type GrammarFormat = "openai_lark" | "openai_regex"; @@ -766,7 +764,11 @@ export type ConstrainedSamplingConfig = variants: GrammarVariants; }; -export interface Tool { +/** + * A tool the model may call. `parameters` is the JSON Schema of its arguments: pi-schema's builders (which also type + * the arguments, see `Static` in `@earendil-works/pi-schema`), TypeBox schemas, or any JSON Schema object. + */ +export interface Tool { name: string; description: string; parameters: TParameters; diff --git a/packages/ai/src/utils/typebox-helpers.ts b/packages/ai/src/utils/string-enum.ts similarity index 68% rename from packages/ai/src/utils/typebox-helpers.ts rename to packages/ai/src/utils/string-enum.ts index 85a53555c..b660178f8 100644 --- a/packages/ai/src/utils/typebox-helpers.ts +++ b/packages/ai/src/utils/string-enum.ts @@ -1,24 +1,24 @@ -import { type TUnsafe, Type } from "typebox"; +import type { Schema } from "@earendil-works/pi-schema"; /** * Creates a string enum schema compatible with Google's API and other providers - * that don't support anyOf/const patterns. + * that don't support anyOf/const patterns: `{ type: "string", enum: [...] }`. * * @example * const OperationSchema = StringEnum(["add", "subtract", "multiply", "divide"], { * description: "The operation to perform" * }); * - * type Operation = Static; // "add" | "subtract" | "multiply" | "divide" + * type Operation = Static; // "add" | "subtract" | "multiply" | "divide" (Static from @earendil-works/pi-schema) */ export function StringEnum( values: T, options?: { description?: string; default?: T[number] }, -): TUnsafe { - return Type.Unsafe({ +): Schema { + return { type: "string", - enum: values as any, + enum: values, ...(options?.description && { description: options.description }), ...(options?.default && { default: options.default }), - }); + }; } diff --git a/packages/ai/src/utils/transcript.ts b/packages/ai/src/utils/transcript.ts index 6f8a2691f..e9bec79f8 100644 --- a/packages/ai/src/utils/transcript.ts +++ b/packages/ai/src/utils/transcript.ts @@ -133,7 +133,7 @@ export function toToolDeclaration(tool: Tool): Tool { * Whether two tools declare the same interface to the model. * * Both sides go through {@link toToolDeclaration} first: its JSON round-trip drops the - * typebox symbol keys and `undefined` fields that a structural comparison would see, and + * schema metadata keys (TypeBox's non-enumerable kind marks) and `undefined` fields that a structural comparison would see, and * builds both objects with the same key order, so comparing the serialized declarations * is exact. This avoids a deep-equal dependency in a browser-safe package. */ diff --git a/packages/ai/src/utils/validation.ts b/packages/ai/src/utils/validation.ts index c9e818834..586765bf0 100644 --- a/packages/ai/src/utils/validation.ts +++ b/packages/ai/src/utils/validation.ts @@ -1,299 +1,34 @@ -import { Compile } from "typebox/compile"; -import type { TLocalizedValidationError } from "typebox/error"; -import { Value } from "typebox/value"; +import { type CoercingValidator, compile } from "@earendil-works/pi-schema"; +import { formats } from "@earendil-works/pi-schema/formats"; import type { Tool, ToolCall } from "../types.ts"; -const validatorCache = new WeakMap>(); -const TYPEBOX_KIND = Symbol.for("TypeBox.Kind"); +// One compiled validator per schema object: tool schemas are long-lived, so the WeakMap drops entries with their tools. +const validatorCache = new WeakMap(); -interface JsonSchemaObject { - type?: string | string[]; - properties?: Record; - required?: string[]; - items?: JsonSchemaObject | JsonSchemaObject[]; - additionalProperties?: boolean | JsonSchemaObject; - allOf?: JsonSchemaObject[]; - anyOf?: JsonSchemaObject[]; - oneOf?: JsonSchemaObject[]; -} - -function getSchemaTypes(schema: JsonSchemaObject): string[] { - if (typeof schema.type === "string") { - return [schema.type]; - } - if (Array.isArray(schema.type)) { - return schema.type.filter((type): type is string => typeof type === "string"); - } - return []; -} - -function matchesJsonType(value: unknown, type: string): boolean { - switch (type) { - case "number": - return typeof value === "number"; - case "integer": - return typeof value === "number" && Number.isInteger(value); - case "boolean": - return typeof value === "boolean"; - case "string": - return typeof value === "string"; - case "null": - return value === null; - case "array": - return Array.isArray(value); - case "object": - return typeof value === "object" && value !== null && !Array.isArray(value); - default: - return false; - } -} - -function getSubSchemaValidator(schema: JsonSchemaObject): ReturnType | undefined { - try { - return getValidator(schema as Tool["parameters"]); - } catch { - return undefined; - } -} - -function coercePrimitiveByType(value: unknown, type: string): unknown { - switch (type) { - case "number": { - if (value === null) { - return 0; - } - if (typeof value === "string" && value.trim() !== "") { - const parsed = Number(value); - if (Number.isFinite(parsed)) { - return parsed; - } - } - if (typeof value === "boolean") { - return value ? 1 : 0; - } - return value; - } - case "integer": { - if (value === null) { - return 0; - } - if (typeof value === "string" && value.trim() !== "") { - const parsed = Number(value); - if (Number.isInteger(parsed)) { - return parsed; - } - } - if (typeof value === "boolean") { - return value ? 1 : 0; - } - return value; - } - case "boolean": { - if (value === null) { - return false; - } - if (typeof value === "string") { - if (value === "true") { - return true; - } - if (value === "false") { - return false; - } - } - if (typeof value === "number") { - if (value === 1) { - return true; - } - if (value === 0) { - return false; - } - } - return value; - } - case "string": { - if (value === null) { - return ""; - } - if (typeof value === "number" || typeof value === "boolean") { - return String(value); - } - return value; - } - case "null": { - if (value === "" || value === 0 || value === false) { - return null; - } - return value; - } - default: - return value; - } -} - -function applySchemaObjectCoercion(value: Record, schema: JsonSchemaObject): void { - const properties = schema.properties; - const definedKeys = new Set(properties ? Object.keys(properties) : []); - - if (properties) { - for (const [key, propertySchema] of Object.entries(properties)) { - if (!(key in value)) { - continue; - } - value[key] = coerceWithJsonSchema(value[key], propertySchema); - } - } - - if (schema.additionalProperties && typeof schema.additionalProperties === "object") { - for (const [key, propertyValue] of Object.entries(value)) { - if (definedKeys.has(key)) { - continue; - } - value[key] = coerceWithJsonSchema(propertyValue, schema.additionalProperties); - } - } -} - -function applySchemaArrayCoercion(value: unknown[], schema: JsonSchemaObject): void { - if (Array.isArray(schema.items)) { - for (let index = 0; index < value.length; index++) { - const itemSchema = schema.items[index]; - if (!itemSchema) { - continue; - } - value[index] = coerceWithJsonSchema(value[index], itemSchema); - } - return; - } - - if (schema.items && typeof schema.items === "object") { - for (let index = 0; index < value.length; index++) { - value[index] = coerceWithJsonSchema(value[index], schema.items); - } - } -} - -function coerceWithUnionSchema(value: unknown, schemas: JsonSchemaObject[]): unknown { - for (const schema of schemas) { - const validator = getSubSchemaValidator(schema); - if (validator?.Check(value)) { - return value; - } - } - - for (const schema of schemas) { - const candidate = structuredClone(value); - const coerced = coerceWithJsonSchema(candidate, schema); - const validator = getSubSchemaValidator(schema); - if (validator?.Check(coerced)) { - return coerced; - } - } - return value; -} - -function coerceWithJsonSchema(value: unknown, schema: JsonSchemaObject): unknown { - let nextValue = value; - - if (Array.isArray(schema.allOf)) { - for (const nested of schema.allOf) { - nextValue = coerceWithJsonSchema(nextValue, nested); - } - } - - if (Array.isArray(schema.anyOf)) { - nextValue = coerceWithUnionSchema(nextValue, schema.anyOf); - } - - if (Array.isArray(schema.oneOf)) { - nextValue = coerceWithUnionSchema(nextValue, schema.oneOf); - } - - const schemaTypes = getSchemaTypes(schema); - const matchesUnionMember = - schemaTypes.length > 1 && schemaTypes.some((schemaType) => matchesJsonType(nextValue, schemaType)); - if (schemaTypes.length > 0 && !matchesUnionMember) { - for (const schemaType of schemaTypes) { - const candidate = coercePrimitiveByType(nextValue, schemaType); - if (candidate !== nextValue) { - nextValue = candidate; - break; - } - } - } - - if ( - schemaTypes.includes("object") && - typeof nextValue === "object" && - nextValue !== null && - !Array.isArray(nextValue) - ) { - applySchemaObjectCoercion(nextValue as Record, schema); - } - - if (schemaTypes.includes("array") && Array.isArray(nextValue)) { - applySchemaArrayCoercion(nextValue, schema); - } - - return nextValue; -} - -function normalizeOptionalNulls(value: unknown, schema: JsonSchemaObject): void { - if (Array.isArray(value)) { - if (Array.isArray(schema.items)) { - for (let index = 0; index < value.length; index++) { - const itemSchema = schema.items[index]; - if (itemSchema) normalizeOptionalNulls(value[index], itemSchema); - } - } else if (schema.items) { - for (const item of value) normalizeOptionalNulls(item, schema.items); - } - return; - } - if (typeof value !== "object" || value === null || !schema.properties) return; - - const object = value as Record; - const required = new Set(schema.required ?? []); - for (const [key, propertySchema] of Object.entries(schema.properties)) { - if (!(key in object)) continue; - if ( - object[key] === null && - !required.has(key) && - typeof (propertySchema as { $ref?: unknown }).$ref !== "string" && - getSubSchemaValidator(propertySchema)?.Check(null) === false - ) { - delete object[key]; - } else { - normalizeOptionalNulls(object[key], propertySchema); - } - } -} - -function getValidator(schema: Tool["parameters"]): ReturnType { - const key = schema as object; - const cached = validatorCache.get(key); +function getValidator(schema: Tool["parameters"]): CoercingValidator { + const cached = validatorCache.get(schema); if (cached) { return cached; } - const validator = Compile(schema); - validatorCache.set(key, validator); + // A $ref that names nothing (a dangling definition in an MCP server's schema) rejects only values at its place, + // instead of making every call of the tool fail. + const validator = compile(schema, { formats, coerce: true, unresolvedRefs: "reject" }); + validatorCache.set(schema, validator); return validator; } -function formatValidationPath(error: TLocalizedValidationError): string { - if (error.keyword === "required") { - const requiredProperties = (error.params as { requiredProperties?: string[] }).requiredProperties; - const requiredProperty = requiredProperties?.[0]; - if (requiredProperty) { - const basePath = error.instancePath.replace(/^\//, "").replace(/\//g, "."); - return basePath ? `${basePath}.${requiredProperty}` : requiredProperty; - } - } - const path = error.instancePath.replace(/^\//, "").replace(/\//g, "."); - return path || "root"; +/** A pi-schema error path (a JSON Pointer) as the dotted path the model sees: `/edits/0/oldText` -> `edits.0.oldText`. */ +function formatValidationPath(pointer: string): string { + if (pointer === "") return "root"; + return pointer + .slice(1) + .split("/") + .map((segment) => segment.replace(/~1/g, "/").replace(/~0/g, "~")) + .join("."); } /** - * Finds a tool by name and validates the tool call arguments against its TypeBox schema + * Finds a tool by name and validates the tool call arguments against its JSON Schema * @param tools Array of tool definitions * @param toolCall The tool call from the LLM * @returns The validated arguments @@ -308,8 +43,10 @@ export function validateToolCall(tools: Tool[], toolCall: ToolCall): any { } /** - * Validates tool call arguments against the tool's TypeBox schema - * @param tool The tool definition with TypeBox schema + * Validates tool call arguments against the tool's JSON Schema (any JSON Schema: pi-schema's builders, TypeBox, plain + * objects). The arguments are coerced first (pi-schema's `parse()`): models often send `"42"` for a number, a single + * value for an array, or `null` for an omitted optional property. + * @param tool The tool definition with its JSON Schema * @param toolCall The tool call from the LLM * @returns The validated (and potentially coerced) arguments * @throws Error with formatted message if validation fails @@ -318,34 +55,13 @@ export function validateToolArguments(tool: Tool, toolCall: ToolCall): any { if (toolCall.argumentsError) { throw new Error(`Invalid arguments for tool "${toolCall.name}": ${toolCall.argumentsError}`); } - const args = structuredClone(toolCall.arguments); - normalizeOptionalNulls(args, tool.parameters as JsonSchemaObject); - Value.Convert(tool.parameters, args); - - const validator = getValidator(tool.parameters); - if (!Object.getOwnPropertySymbols(tool.parameters).includes(TYPEBOX_KIND)) { - const coerced = coerceWithJsonSchema(args, tool.parameters as JsonSchemaObject); - if (coerced !== args) { - if (typeof args === "object" && args !== null && typeof coerced === "object" && coerced !== null) { - for (const key of Object.keys(args)) { - delete args[key]; - } - Object.assign(args, coerced); - } else { - return validator.Check(coerced) ? coerced : args; - } - } + // A copy, so tools that modify their arguments never modify the tool call in the transcript. + const result = getValidator(tool.parameters).parse(structuredClone(toolCall.arguments)); + if (result.ok) { + return result.value; } - if (validator.Check(args)) { - return args; - } - - const errors = - validator - .Errors(args) - .map((error) => ` - ${formatValidationPath(error)}: ${error.message}`) - .join("\n") || "Unknown validation error"; + const errors = result.errors.map((error) => ` - ${formatValidationPath(error.path)}: ${error.message}`).join("\n"); const errorMessage = `Validation failed for tool "${toolCall.name}":\n${errors}\n\nReceived arguments:\n${JSON.stringify(toolCall.arguments, null, 2)}`; diff --git a/packages/ai/test/compat-schema.test.ts b/packages/ai/test/compat-schema.test.ts index 4ef40713e..d1a6d66ca 100644 --- a/packages/ai/test/compat-schema.test.ts +++ b/packages/ai/test/compat-schema.test.ts @@ -1,5 +1,4 @@ -import type { Static } from "typebox"; -import { Compile } from "typebox/compile"; +import type { Static } from "@earendil-works/pi-schema"; import { describe, expect, expectTypeOf, it } from "vitest"; import { AnthropicMessagesCompatSchema, @@ -11,6 +10,16 @@ import { type ProviderCompat = Static; +/** The `default` annotation of each property of an object schema. */ +function defaults(schema: { readonly [keyword: string]: unknown }): Record { + const properties = schema.properties as Record; + return Object.fromEntries( + Object.entries(properties) + .filter(([, property]) => property.default !== undefined) + .map(([name, property]) => [name, property.default]), + ); +} + describe("compatibility schemas", () => { it("preserves property types in the provider superset", () => { expectTypeOf().toEqualTypeOf(); @@ -20,21 +29,21 @@ describe("compatibility schemas", () => { }); it("preserves API-specific defaults", () => { - const completionsDefaults = Compile(OpenAICompletionsCompatSchema).Default({}); + const completionsDefaults = defaults(OpenAICompletionsCompatSchema); expect(completionsDefaults).not.toHaveProperty("thinkingFormat"); expect(completionsDefaults).not.toHaveProperty("supportsLongCacheRetention"); - expect(Compile(OpenAIResponsesCompatSchema).Default({})).toMatchObject({ supportsDeveloperRole: true }); - const anthropicDefaults = Compile(AnthropicMessagesCompatSchema).Default({}); + expect(defaults(OpenAIResponsesCompatSchema)).toMatchObject({ supportsDeveloperRole: true }); + const anthropicDefaults = defaults(AnthropicMessagesCompatSchema); expect(anthropicDefaults).not.toHaveProperty("sendSessionAffinityHeaders"); expect(anthropicDefaults).toMatchObject({ supportsLongCacheRetention: true }); - expect(Compile(BedrockCompatSchema).Default({})).toMatchObject({ supportsStrictMode: false }); + expect(defaults(BedrockCompatSchema)).toMatchObject({ supportsStrictMode: false }); }); it("does not assign API-specific defaults to the provider superset", () => { - const defaults = Compile(ProviderCompatSchema).Default({}); - expect(defaults).not.toHaveProperty("supportsDeveloperRole"); - expect(defaults).not.toHaveProperty("sendSessionAffinityHeaders"); - expect(defaults).not.toHaveProperty("supportsLongCacheRetention"); - expect(defaults).not.toHaveProperty("supportsStrictMode"); + const providerDefaults = defaults(ProviderCompatSchema); + expect(providerDefaults).not.toHaveProperty("supportsDeveloperRole"); + expect(providerDefaults).not.toHaveProperty("sendSessionAffinityHeaders"); + expect(providerDefaults).not.toHaveProperty("supportsLongCacheRetention"); + expect(providerDefaults).not.toHaveProperty("supportsStrictMode"); }); }); diff --git a/packages/ai/test/faux-provider.test.ts b/packages/ai/test/faux-provider.test.ts index ad23ed938..afdf685b6 100644 --- a/packages/ai/test/faux-provider.test.ts +++ b/packages/ai/test/faux-provider.test.ts @@ -1,3 +1,4 @@ +import { Type } from "typebox"; import { afterEach, describe, expect, it } from "vitest"; import { complete, @@ -7,7 +8,6 @@ import { fauxToolCall, registerFauxProvider, stream, - Type, } from "../src/compat.ts"; import type { AssistantMessageEvent, Context } from "../src/types.ts"; diff --git a/packages/ai/test/interleaved-thinking.test.ts b/packages/ai/test/interleaved-thinking.test.ts index b4da12835..b787a2cf7 100644 --- a/packages/ai/test/interleaved-thinking.test.ts +++ b/packages/ai/test/interleaved-thinking.test.ts @@ -3,7 +3,7 @@ import { describe, expect, it } from "vitest"; import { completeSimple, getModel } from "../src/compat.ts"; import { getEnvApiKey } from "../src/env-api-keys.ts"; import type { Api, Context, Model, StopReason, Tool, ToolCall, ToolResultMessage } from "../src/types.ts"; -import { StringEnum } from "../src/utils/typebox-helpers.ts"; +import { StringEnum } from "../src/utils/string-enum.ts"; import { hasBedrockCredentials } from "./bedrock-utils.ts"; const calculatorSchema = Type.Object({ diff --git a/packages/ai/test/stream.test.ts b/packages/ai/test/stream.test.ts index 6cac98a15..3cc02268b 100644 --- a/packages/ai/test/stream.test.ts +++ b/packages/ai/test/stream.test.ts @@ -9,7 +9,7 @@ import type { Api, Context, ImageContent, Model, StreamOptions, Tool, ToolResult type StreamOptionsWithExtras = StreamOptions & Record; -import { StringEnum } from "../src/utils/typebox-helpers.ts"; +import { StringEnum } from "../src/utils/string-enum.ts"; import { hasAzureOpenAICredentials, resolveAzureDeploymentName } from "./azure-utils.ts"; import { hasBedrockCredentials } from "./bedrock-utils.ts"; import { hasCloudflareAiGatewayCredentials, hasCloudflareWorkersAICredentials } from "./cloudflare-utils.ts"; diff --git a/packages/ai/test/validation-probes.test.ts b/packages/ai/test/validation-probes.test.ts new file mode 100644 index 000000000..0ddfe8d12 --- /dev/null +++ b/packages/ai/test/validation-probes.test.ts @@ -0,0 +1,779 @@ +// The reviewers' probes of the schema switch (round 1) as regression tests: TypeBox-built and plain JSON +// (MCP-shaped) tool schemas with arguments through validateToolArguments. Each case's outcome is "ok ", "REJECT " or "THROW ". Cases marked as TypeBox-only record the documented +// differences from TypeBox (see the CHANGELOG): Refine and BigInt. + +// biome-ignore-all lint/suspicious/noTemplateCurlyInString: TypeBox template literal syntax. +// biome-ignore-all lint/suspicious/noThenProperty: JSON Schema's `then` keyword. +import { Type } from "typebox"; +import { describe, expect, it } from "vitest"; +import type { Tool, ToolCall } from "../src/types.ts"; +import { validateToolArguments } from "../src/utils/validation.ts"; + +/** The outcome of one call, in the probes' format. */ +function outcome(name: string, schema: object, args: unknown): string { + const tool = { name, description: "", parameters: schema } as Tool; + const call = { type: "toolCall", id: "1", name, arguments: args } as ToolCall; + try { + return `ok ${JSON.stringify(validateToolArguments(tool, call), (_key, value) => (typeof value === "bigint" ? `${value}n` : value))}`; + } catch (error) { + const message = (error as Error).message; + return message.startsWith("Validation failed") + ? `REJECT ${message.split("\n")[1]?.trim()}` + : `THROW ${message.split("\n")[0]}`; + } +} + +type Case = { name: string; schema: object; args: unknown[] }; + +const Node = Type.Cyclic( + { Node: Type.Object({ name: Type.String(), children: Type.Array(Type.Ref("Node")) }) }, + "Node", +); + +const cases: Case[] = [ + { + name: "cyclic", + schema: Type.Object({ tree: Node }), + args: [{ tree: { name: "a", children: [{ name: "b", children: [] }] } }, { tree: { name: 1, children: [] } }], + }, + { name: "ref-with-$id", schema: Type.Object({ a: Type.Ref("A") }), args: [{ a: { x: "1" } }] }, + { + name: "function", + schema: Type.Object({ f: Type.Optional(Type.Function([], Type.Void())) }), + args: [{}, { f: "x" }], + }, + { + name: "codec", + schema: Type.Object({ + c: Type.Codec(Type.String()) + .Decode((s) => Number(s)) + .Encode((n) => String(n)), + }), + args: [{ c: "12" }, { c: 12 }], + }, + { name: "undefined-optional", schema: Type.Object({ u: Type.Optional(Type.Undefined()) }), args: [{}, { u: null }] }, + { name: "bigint", schema: Type.Object({ n: Type.BigInt() }), args: [{ n: 1 }, { n: "12" }] }, + { + name: "refine", + schema: Type.Object({ n: Type.Refine(Type.Number(), (v) => v > 0) }), + args: [{ n: 5 }, { n: -5 }], + }, + { + name: "template-literal", + schema: Type.Object({ t: Type.TemplateLiteral("id-${number}") }), + args: [{ t: "id-12" }, { t: "id-x" }, { t: 3 }], + }, + { + name: "record-string", + schema: Type.Object({ r: Type.Record(Type.String(), Type.Number()) }), + args: [{ r: { a: "1" } }, { r: { a: 1, b: "x" } }], + }, + { + name: "record-literal-keys", + schema: Type.Object({ r: Type.Record(Type.Union([Type.Literal("a"), Type.Literal("b")]), Type.Number()) }), + args: [{ r: { a: 1, b: 2 } }, { r: { a: 1 } }, { r: { a: 1, b: 2, c: 3 } }], + }, + { + name: "intersect", + schema: Type.Intersect([Type.Object({ a: Type.Number() }), Type.Object({ b: Type.Boolean() })]), + args: [ + { a: "1", b: "true" }, + { a: 1, b: true, c: 1 }, + ], + }, + { + name: "union-objects", + schema: Type.Object({ + v: Type.Union([ + Type.Object({ k: Type.Literal("x"), n: Type.Number() }), + Type.Object({ k: Type.Literal("y"), s: Type.String() }), + ]), + }), + args: [{ v: { k: "x", n: "2" } }, { v: { k: "y", s: 2 } }], + }, + { name: "minLength-astral", schema: Type.Object({ s: Type.String({ minLength: 2 }) }), args: [{ s: "😀" }] }, + { name: "maxLength-astral", schema: Type.Object({ s: Type.String({ maxLength: 1 }) }), args: [{ s: "😀" }] }, + { name: "pattern-astral", schema: Type.Object({ s: Type.String({ pattern: "^.$" }) }), args: [{ s: "😀" }] }, + { + name: "pattern-class-dash", + schema: Type.Object({ s: Type.String({ pattern: "^[\\w-]+$" }) }), + args: [{ s: "a-b" }], + }, + { + name: "pattern-unicode-prop", + schema: Type.Object({ s: Type.String({ pattern: "^\\p{L}+$" }) }), + args: [{ s: "äb" }, { s: "p{L}" }], + }, + { + name: "multipleOf-float", + schema: Type.Object({ n: Type.Number({ multipleOf: 0.01 }) }), + args: [{ n: 0.07 }, { n: 1.1 }], + }, + { + name: "uniqueItems-keyorder", + schema: Type.Object({ a: Type.Array(Type.Unknown(), { uniqueItems: true }) }), + args: [ + { + a: [ + { x: 1, y: 2 }, + { y: 2, x: 1 }, + ], + }, + { a: [1, 1.0] }, + ], + }, + { name: "const-object", schema: Type.Object({ c: Type.Literal("a") }), args: [{ c: "a" }] }, + { + name: "formats", + schema: Type.Object({ + e: Type.Optional(Type.String({ format: "email" })), + u: Type.Optional(Type.String({ format: "uri" })), + d: Type.Optional(Type.String({ format: "date-time" })), + x: Type.Optional(Type.String({ format: "made-up" })), + h: Type.Optional(Type.String({ format: "hostname" })), + r: Type.Optional(Type.String({ format: "regex" })), + i: Type.Optional(Type.String({ format: "uuid" })), + ip: Type.Optional(Type.String({ format: "ipv4" })), + }), + args: [ + { e: "a@b.co" }, + { e: "not-an-email" }, + { u: "relative/path" }, + { u: "https://x.y/z" }, + { d: "2024-01-01T00:00:00Z" }, + { d: "2024-01-01" }, + { x: "anything" }, + { h: "under_score.example" }, + { r: "[" }, + { i: "00000000-0000-0000-0000-000000000000" }, + { i: "not" }, + { ip: "999.1.1.1" }, + ], + }, + { name: "integer-big", schema: Type.Object({ n: Type.Integer() }), args: [{ n: 1e21 }, { n: 2 ** 53 + 2 }] }, + { + name: "additional-false", + schema: Type.Object({ a: Type.String() }, { additionalProperties: false }), + args: [{ a: "x", b: 1 }], + }, + { + name: "prototype-key", + schema: Type.Object({ constructor: Type.Optional(Type.String()), toString: Type.Optional(Type.Number()) }), + args: [{}, { constructor: 1 }], + }, + { + name: "proto-key-value", + schema: Type.Object({}, { additionalProperties: Type.Array(Type.String()) }), + args: [JSON.parse('{"__proto__": "x", "a": "y"}')], + }, + // Plain JSON (MCP-shaped) + { + name: "mcp-draft7-ref-definitions", + schema: { + $schema: "http://json-schema.org/draft-07/schema#", + type: "object", + properties: { t: { $ref: "#/definitions/T" } }, + required: ["t"], + definitions: { T: { type: "object", properties: { path: { type: "string" } }, required: ["path"] } }, + }, + args: [{ t: { path: "a" } }, { t: { path: 42 } }], + }, + { + name: "mcp-ref-unresolved", + schema: { type: "object", properties: { t: { $ref: "#/definitions/Missing" } } }, + args: [{ t: 1 }, {}], + }, + { + name: "mcp-ref-external", + schema: { type: "object", properties: { t: { $ref: "https://example.com/schemas/t.json" } } }, + args: [{ t: 1 }], + }, + { + name: "mcp-nullable-openapi", + schema: { type: "object", properties: { s: { type: "string", nullable: true } }, required: ["s"] }, + args: [{ s: null }, { s: "x" }], + }, + { + name: "mcp-draft7-tuple", + schema: { + $schema: "http://json-schema.org/draft-07/schema#", + type: "object", + properties: { p: { type: "array", items: [{ type: "number" }, { type: "string" }], additionalItems: false } }, + }, + args: [{ p: [1, "a"] }, { p: [1, "a", 3] }, { p: ["1", 2] }], + }, + { + name: "mcp-draft7-ref-sibling", + schema: { + $schema: "http://json-schema.org/draft-07/schema#", + type: "object", + properties: { s: { $ref: "#/definitions/S", maxLength: 1 } }, + definitions: { S: { type: "string" } }, + }, + args: [{ s: "abc" }], + }, + { + name: "mcp-draft4-exclusive-bool", + schema: { + $schema: "http://json-schema.org/draft-04/schema#", + type: "object", + properties: { n: { type: "number", minimum: 0, exclusiveMinimum: true } }, + }, + args: [{ n: 0 }, { n: 1 }], + }, + { + name: "mcp-id-draft4", + schema: { + $schema: "http://json-schema.org/draft-04/schema#", + id: "http://x/y", + type: "object", + properties: { a: { type: "string" } }, + }, + args: [{ a: "x" }], + }, + { + name: "mcp-bad-pattern", + schema: { type: "object", properties: { s: { type: "string", pattern: "(? `v${i}`) } }, + }, + args: [{ e: "v4999" }, { e: "v5000" }], + }, +]; + +const o = (properties: Record, extra: Record = {}) => ({ + type: "object", + properties, + ...extra, +}); +const keywordCases: [string, object, unknown[]][] = [ + [ + "oneOf-overlap", + o({ v: { oneOf: [{ type: "string" }, { type: "string", maxLength: 5 }] } }), + [{ v: "abc" }, { v: "abcdefgh" }], + ], + ["oneOf-number-integer", o({ v: { oneOf: [{ type: "number" }, { type: "integer" }] } }), [{ v: 2 }, { v: 2.5 }]], + [ + "oneOf-objects-open", + o({ + v: { + oneOf: [ + { type: "object", properties: { a: { type: "string" } } }, + { type: "object", properties: { b: { type: "string" } } }, + ], + }, + }), + [{ v: { a: "x" } }, { v: {} }], + ], + [ + "if-then-else", + o( + { kind: { type: "string" }, n: { type: "number" } }, + { if: { properties: { kind: { const: "a" } } }, then: { required: ["n"] } }, + ), + [{ kind: "a" }, { kind: "b" }], + ], + [ + "dependentRequired", + o({ a: { type: "string" }, b: { type: "string" } }, { dependentRequired: { a: ["b"] } }), + [{ a: "x" }], + ], + [ + "dependencies-d7", + o({ a: { type: "string" }, b: { type: "string" } }, { dependencies: { a: ["b"] } }), + [{ a: "x" }], + ], + ["propertyNames", o({}, { propertyNames: { pattern: "^[a-z]+$" } }), [{ abc: 1 }, { ABC: 1 }]], + ["contains", o({ l: { type: "array", contains: { type: "number" } } }), [{ l: ["a"] }, { l: ["a", 1] }]], + ["not", o({ s: { not: { type: "null" } } }), [{ s: "x" }, { s: null }]], + ["minProperties", o({ m: { type: "object", minProperties: 1 } }), [{ m: {} }]], + [ + "unevaluatedProperties", + o({ a: { type: "string" } }, { unevaluatedProperties: false }), + [{ a: "x" }, { a: "x", b: 1 }], + ], + [ + "allOf-additional-false", + { allOf: [o({ a: { type: "string" } }, { additionalProperties: false }), o({ b: { type: "string" } })] }, + [{ a: "x" }, { a: "x", b: "y" }], + ], + ["const-object", o({ c: { const: { x: 1, y: 2 } } }), [{ c: { y: 2, x: 1 } }]], + ["enum-object", o({ e: { enum: [{ x: 1, y: 2 }] } }), [{ e: { y: 2, x: 1 } }]], + ["type-array-multi", o({ v: { type: ["string", "number"] } }), [{ v: true }, { v: null }]], + ["nullable-type-array", o({ v: { type: ["string", "null"] } }, { required: ["v"] }), [{ v: null }, { v: 5 }]], + ["exclusiveMinimum-number", o({ n: { type: "number", exclusiveMinimum: 0 } }), [{ n: 0 }, { n: "0" }]], + ["maxItems", o({ l: { type: "array", maxItems: 1 } }), [{ l: [1, 2] }, { l: "x" }]], + [ + "discriminator-openapi", + o({ + p: { + oneOf: [ + o({ kind: { type: "string", enum: ["a"] }, x: { type: "number" } }, { required: ["kind"] }), + o({ kind: { type: "string", enum: ["b"] } }, { required: ["kind"] }), + ], + discriminator: { propertyName: "kind" }, + }, + }), + [{ p: { kind: "a", x: "1" } }, { p: { kind: "b" } }], + ], + [ + "ref-root-defs", + { $ref: "#/$defs/Root", $defs: { Root: o({ a: { type: "string" } }, { required: ["a"] }) } }, + [{ a: "x" }, { a: 1 }, {}], + ], + [ + "ref-recursive", + { + $defs: { N: o({ c: { type: "array", items: { $ref: "#/$defs/N" } }, v: { type: "integer" } }) }, + ...o({ root: { $ref: "#/$defs/N" } }), + }, + [{ root: { v: "2", c: [{ v: "3", c: [] }] } }], + ], + [ + "ref-percent", + { ...o({ a: { $ref: "#/$defs/a%20b" } }), $defs: { "a b": { type: "string" } } }, + [{ a: "x" }, { a: 1 }], + ], + ["ref-tilde", { ...o({ a: { $ref: "#/$defs/a~1b" } }), $defs: { "a/b": { type: "string" } } }, [{ a: "x" }]], + [ + "defs-with-$id", + { ...o({ a: { $ref: "item.json" } }), $defs: { item: { $id: "item.json", type: "string" } } }, + [{ a: "x" }, { a: 2 }], + ], + [ + "anyOf-null-default", + o({ v: { anyOf: [{ type: "string" }, { type: "null" }], default: null } }), + [{ v: null }, {}], + ], + [ + "format-date-time-lower", + o({ d: { type: "string", format: "date-time" } }), + [{ d: "2024-01-01t00:00:00z" }, { d: "2024-01-01 00:00:00Z" }], + ], + ["format-url", o({ u: { type: "string", format: "url" } }), [{ u: "not a url" }]], + ["integer-as-float-1e0", o({ n: { type: "integer" } }), [{ n: 1e3 }]], + ["required-missing-props", { type: "object", required: ["a"] }, [{}, { a: null }]], + ["string-to-number-trim", o({ n: { type: "number" } }), [{ n: " 12 " }, { n: "0x10" }, { n: "1e3" }, { n: "" }]], +]; + +const tree = { + type: "object", + properties: { t: { $ref: "#/$defs/Tree" } }, + $defs: { Tree: { anyOf: [{ type: "string" }, { type: "array", items: { $ref: "#/$defs/Tree" } }] } }, +}; +const json = { + type: "object", + properties: { data: { $ref: "#/$defs/Json" } }, + required: ["data"], + $defs: { + Json: { + anyOf: [ + { type: "string" }, + { type: "number" }, + { type: "boolean" }, + { type: "array", items: { $ref: "#/$defs/Json" } }, + { type: "object", additionalProperties: { $ref: "#/$defs/Json" } }, + ], + }, + }, +}; +const filter = { + type: "object", + properties: { where: { $ref: "#/$defs/F" } }, + $defs: { + F: { + oneOf: [ + { + type: "object", + properties: { field: { type: "string" }, eq: { type: "string" } }, + required: ["field", "eq"], + additionalProperties: false, + }, + { + type: "object", + properties: { and: { type: "array", items: { $ref: "#/$defs/F" } } }, + required: ["and"], + additionalProperties: false, + }, + ], + }, + }, +}; +const recursionCases: [string, object, unknown][] = [ + ["tree valid", tree, { t: ["a", ["b"]] }], + ["tree number", tree, { t: 5 }], + ["tree [5]", tree, { t: [5] }], + ["json valid", json, { data: { a: [1, "x", { b: true }] } }], + ["json null", json, { data: null }], + ["json nested null", json, { data: { a: [1, null] } }], + ["filter valid", filter, { where: { and: [{ field: "a", eq: "b" }] } }], + ["filter bad leaf", filter, { where: { and: [{ field: "a", eq: 1, x: 2 }] } }], + ["filter deep bad", filter, { where: { and: [{ and: [{ and: [{ and: [{ and: [{ field: 1 }] }] }] }] }] } }], +]; + +const expected: Record = { + 'cyclic {"tree":{"name":"a","children":[{"name":"b","children":[]}]}}': + 'ok {"tree":{"name":"a","children":[{"name":"b","children":[]}]}}', + 'cyclic {"tree":{"name":1,"children":[]}}': 'ok {"tree":{"name":"1","children":[]}}', + 'ref-with-$id {"a":{"x":"1"}}': "REJECT - a: unresolved reference A", + "function {}": "ok {}", + 'function {"f":"x"}': "REJECT - f: expected function, got string", + 'codec {"c":"12"}': 'ok {"c":"12"}', + 'codec {"c":12}': 'ok {"c":"12"}', + "undefined-optional {}": "ok {}", + 'undefined-optional {"u":null}': "ok {}", + 'bigint {"n":1}': "REJECT - n: expected bigint, got integer", + 'bigint {"n":"12"}': "REJECT - n: expected bigint, got string", + 'refine {"n":5}': 'ok {"n":5}', + 'refine {"n":-5}': 'ok {"n":-5}', + 'template-literal {"t":"id-12"}': 'ok {"t":"id-12"}', + 'template-literal {"t":"id-x"}': "REJECT - t: expected to match ^id--?(?:0|[1-9][0-9]*)(?:\\.[0-9]+)?$", + 'template-literal {"t":3}': "REJECT - t: expected to match ^id--?(?:0|[1-9][0-9]*)(?:\\.[0-9]+)?$", + 'record-string {"r":{"a":"1"}}': 'ok {"r":{"a":1}}', + 'record-string {"r":{"a":1,"b":"x"}}': "REJECT - r.b: expected number, got string", + 'record-literal-keys {"r":{"a":1,"b":2}}': 'ok {"r":{"a":1,"b":2}}', + 'record-literal-keys {"r":{"a":1}}': "REJECT - r.b: missing", + 'record-literal-keys {"r":{"a":1,"b":2,"c":3}}': 'ok {"r":{"a":1,"b":2,"c":3}}', + 'intersect {"a":"1","b":"true"}': 'ok {"a":1,"b":true}', + 'intersect {"a":1,"b":true,"c":1}': 'ok {"a":1,"b":true,"c":1}', + 'union-objects {"v":{"k":"x","n":"2"}}': 'ok {"v":{"k":"x","n":2}}', + 'union-objects {"v":{"k":"y","s":2}}': 'ok {"v":{"k":"y","s":"2"}}', + 'minLength-astral {"s":"😀"}': "REJECT - s: expected at least 2 characters", + 'maxLength-astral {"s":"😀"}': 'ok {"s":"😀"}', + 'pattern-astral {"s":"😀"}': 'ok {"s":"😀"}', + 'pattern-class-dash {"s":"a-b"}': 'ok {"s":"a-b"}', + 'pattern-unicode-prop {"s":"äb"}': 'ok {"s":"äb"}', + 'pattern-unicode-prop {"s":"p{L}"}': "REJECT - s: expected to match ^\\p{L}+$", + 'multipleOf-float {"n":0.07}': 'ok {"n":0.07}', + 'multipleOf-float {"n":1.1}': 'ok {"n":1.1}', + 'uniqueItems-keyorder {"a":[{"x":1,"y":2},{"y":2,"x":1}]}': "REJECT - a: expected unique items", + 'uniqueItems-keyorder {"a":[1,1]}': "REJECT - a: expected unique items", + 'const-object {"c":"a"}': 'ok {"c":"a"}', + 'formats {"e":"a@b.co"}': 'ok {"e":"a@b.co"}', + 'formats {"e":"not-an-email"}': "REJECT - e: expected format email", + 'formats {"u":"relative/path"}': "REJECT - u: expected format uri", + 'formats {"u":"https://x.y/z"}': 'ok {"u":"https://x.y/z"}', + 'formats {"d":"2024-01-01T00:00:00Z"}': 'ok {"d":"2024-01-01T00:00:00Z"}', + 'formats {"d":"2024-01-01"}': "REJECT - d: expected format date-time", + 'formats {"x":"anything"}': 'ok {"x":"anything"}', + 'formats {"h":"under_score.example"}': "REJECT - h: expected format hostname", + 'formats {"r":"["}': "REJECT - r: expected format regex", + 'formats {"i":"00000000-0000-0000-0000-000000000000"}': 'ok {"i":"00000000-0000-0000-0000-000000000000"}', + 'formats {"i":"not"}': "REJECT - i: expected format uuid", + 'formats {"ip":"999.1.1.1"}': "REJECT - ip: expected format ipv4", + 'integer-big {"n":1e+21}': 'ok {"n":1e+21}', + 'integer-big {"n":9007199254740994}': 'ok {"n":9007199254740994}', + 'additional-false {"a":"x","b":1}': "REJECT - b: unknown field", + "prototype-key {}": "ok {}", + 'prototype-key {"constructor":1}': 'ok {"constructor":"1"}', + 'proto-key-value {"__proto__":"x","a":"y"}': 'ok {"__proto__":["x"],"a":["y"]}', + 'mcp-draft7-ref-definitions {"t":{"path":"a"}}': 'ok {"t":{"path":"a"}}', + 'mcp-draft7-ref-definitions {"t":{"path":42}}': 'ok {"t":{"path":"42"}}', + 'mcp-ref-unresolved {"t":1}': "REJECT - t: unresolved reference #/definitions/Missing", + "mcp-ref-unresolved {}": "ok {}", + 'mcp-ref-external {"t":1}': "REJECT - t: unresolved reference https://example.com/schemas/t.json", + 'mcp-nullable-openapi {"s":null}': 'ok {"s":"null"}', + 'mcp-nullable-openapi {"s":"x"}': 'ok {"s":"x"}', + 'mcp-draft7-tuple {"p":[1,"a"]}': 'ok {"p":[1,"a"]}', + 'mcp-draft7-tuple {"p":[1,"a",3]}': "REJECT - p.2: no value is allowed here", + 'mcp-draft7-tuple {"p":["1",2]}': 'ok {"p":[1,"2"]}', + 'mcp-draft7-ref-sibling {"s":"abc"}': "REJECT - s: expected at most 1 characters", + 'mcp-draft4-exclusive-bool {"n":0}': 'ok {"n":0}', + 'mcp-draft4-exclusive-bool {"n":1}': 'ok {"n":1}', + 'mcp-id-draft4 {"a":"x"}': 'ok {"a":"x"}', + 'mcp-bad-pattern {"s":"y"}': 'ok {"s":"y"}', + 'mcp-invalid-regex {"s":"y"}': "THROW Invalid regular expression: /[/: Unterminated character class", + 'mcp-type-integer-float-0 {"n":1}': 'ok {"n":1}', + 'mcp-type-integer-float-0 {"n":1.5}': 'ok {"n":1}', + 'mcp-format-unknown {"s":"xx"}': 'ok {"s":"xx"}', + 'mcp-required-null-string {"q":null}': 'ok {"q":"null"}', + 'mcp-array-json-string {"p":"[\\"a\\",\\"b\\"]"}': 'ok {"p":["[\\"a\\",\\"b\\"]"]}', + 'mcp-integer-float {"n":3.7}': 'ok {"n":3}', + 'mcp-integer-float {"n":"3.7"}': 'ok {"n":3}', + 'mcp-empty-schema {"a":1}': 'ok {"a":1}', + 'mcp-type-object-no-props {"a":1}': 'ok {"a":1}', + 'mcp-boolean-true-prop {"a":1}': 'ok {"a":1}', + 'mcp-boolean-true-prop {"b":1}': "REJECT - b: no value is allowed here", + 'mcp-huge-enum {"e":"v4999"}': 'ok {"e":"v4999"}', + 'mcp-huge-enum {"e":"v5000"}': + 'REJECT - e: expected one of ["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19", ... and 4980 more]', + 'oneOf-overlap {"v":"abc"}': "REJECT - v: matches more than one alternative", + 'oneOf-overlap {"v":"abcdefgh"}': 'ok {"v":"abcdefgh"}', + 'oneOf-number-integer {"v":2}': "REJECT - v: matches more than one alternative", + 'oneOf-number-integer {"v":2.5}': 'ok {"v":2.5}', + 'oneOf-objects-open {"v":{"a":"x"}}': "REJECT - v: matches more than one alternative", + 'oneOf-objects-open {"v":{}}': "REJECT - v: matches more than one alternative", + 'if-then-else {"kind":"a"}': "REJECT - n: missing", + 'if-then-else {"kind":"b"}': 'ok {"kind":"b"}', + 'dependentRequired {"a":"x"}': "REJECT - b: required when a is present", + 'dependencies-d7 {"a":"x"}': "REJECT - b: required when a is present", + 'propertyNames {"abc":1}': 'ok {"abc":1}', + 'propertyNames {"ABC":1}': "REJECT - ABC: invalid property name", + 'contains {"l":["a"]}': "REJECT - l: expected at least 1 matching item", + 'contains {"l":["a",1]}': 'ok {"l":["a",1]}', + 'not {"s":"x"}': 'ok {"s":"x"}', + 'not {"s":null}': "ok {}", + 'minProperties {"m":{}}': "REJECT - m: expected at least 1 properties", + 'unevaluatedProperties {"a":"x"}': 'ok {"a":"x"}', + 'unevaluatedProperties {"a":"x","b":1}': "REJECT - b: unknown field", + 'allOf-additional-false {"a":"x"}': 'ok {"a":"x"}', + 'allOf-additional-false {"a":"x","b":"y"}': "REJECT - b: unknown field", + 'const-object {"c":{"y":2,"x":1}}': 'ok {"c":{"y":2,"x":1}}', + 'enum-object {"e":{"y":2,"x":1}}': 'ok {"e":{"y":2,"x":1}}', + 'type-array-multi {"v":true}': 'ok {"v":"true"}', + 'type-array-multi {"v":null}': "ok {}", + 'nullable-type-array {"v":null}': 'ok {"v":null}', + 'nullable-type-array {"v":5}': 'ok {"v":"5"}', + 'exclusiveMinimum-number {"n":0}': "REJECT - n: expected > 0", + 'exclusiveMinimum-number {"n":"0"}': "REJECT - n: expected > 0", + 'maxItems {"l":[1,2]}': "REJECT - l: expected at most 1 items", + 'maxItems {"l":"x"}': 'ok {"l":["x"]}', + 'discriminator-openapi {"p":{"kind":"a","x":"1"}}': 'ok {"p":{"kind":"a","x":1}}', + 'discriminator-openapi {"p":{"kind":"b"}}': 'ok {"p":{"kind":"b"}}', + 'ref-root-defs {"a":"x"}': 'ok {"a":"x"}', + 'ref-root-defs {"a":1}': 'ok {"a":"1"}', + "ref-root-defs {}": "REJECT - a: missing", + 'ref-recursive {"root":{"v":"2","c":[{"v":"3","c":[]}]}}': 'ok {"root":{"v":2,"c":[{"v":3,"c":[]}]}}', + 'ref-percent {"a":"x"}': 'ok {"a":"x"}', + 'ref-percent {"a":1}': 'ok {"a":"1"}', + 'ref-tilde {"a":"x"}': 'ok {"a":"x"}', + 'defs-with-$id {"a":"x"}': 'ok {"a":"x"}', + 'defs-with-$id {"a":2}': 'ok {"a":"2"}', + 'anyOf-null-default {"v":null}': 'ok {"v":null}', + "anyOf-null-default {}": "ok {}", + 'format-date-time-lower {"d":"2024-01-01t00:00:00z"}': 'ok {"d":"2024-01-01t00:00:00z"}', + 'format-date-time-lower {"d":"2024-01-01 00:00:00Z"}': "REJECT - d: expected format date-time", + 'format-url {"u":"not a url"}': "REJECT - u: expected format url", + 'integer-as-float-1e0 {"n":1000}': 'ok {"n":1000}', + "required-missing-props {}": "REJECT - a: missing", + 'required-missing-props {"a":null}': 'ok {"a":null}', + 'string-to-number-trim {"n":" 12 "}': 'ok {"n":12}', + 'string-to-number-trim {"n":"0x10"}': 'ok {"n":16}', + 'string-to-number-trim {"n":"1e3"}': 'ok {"n":1000}', + 'string-to-number-trim {"n":""}': 'ok {"n":0}', + "tree valid": 'ok {"t":["a",["b"]]}', + "tree number": 'ok {"t":"5"}', + "tree [5]": 'ok {"t":["5"]}', + "json valid": 'ok {"data":{"a":[1,"x",{"b":true}]}}', + "json null": 'ok {"data":"null"}', + "json nested null": 'ok {"data":{"a":[1,"null"]}}', + "filter valid": 'ok {"where":{"and":[{"field":"a","eq":"b"}]}}', + "filter bad leaf": "REJECT - where.and.0.eq: expected string, got integer", + "filter deep bad": "REJECT - where.and.0.and.0.and.0.and.0.and.0.eq: missing", +}; + +/** Every case with the key of its expected outcome: the name and the arguments, or the name alone for recursion. */ +const all: Array<[string, string, object, unknown]> = [ + ...cases.flatMap(({ name, schema, args }) => + args.map((arg): [string, string, object, unknown] => [`${name} ${JSON.stringify(arg)}`, name, schema, arg]), + ), + ...keywordCases.flatMap(([name, schema, args]) => + args.map((arg): [string, string, object, unknown] => [`${name} ${JSON.stringify(arg)}`, name, schema, arg]), + ), + ...recursionCases.map(([name, schema, arg]): [string, string, object, unknown] => [name, name, schema, arg]), +]; + +describe("reviewer probes", () => { + for (const [key, name, schema, arg] of all) { + it(key, () => { + expect(outcome(name, schema, arg)).toBe(expected[key]); + }); + } +}); + +describe("reviewer probes (Sol)", () => { + const tree = { + type: "object", + properties: { t: { $ref: "#/$defs/Tree" } }, + $defs: { Tree: { anyOf: [{ type: "string" }, { type: "array", items: { $ref: "#/$defs/Tree" } }] } }, + }; + const solCases: Array<[string, object, unknown, string]> = [ + // TypeBox-only: Type.BigInt and Type.Undefined have no JSON form; TypeBox converted to them. + [ + "BigInt from a JSON string", + Type.Object({ n: Type.BigInt() }), + { n: "42" }, + "REJECT - n: expected bigint, got string", + ], + [ + "Undefined from JSON null", + Type.Object({ n: Type.Undefined() }), + { n: null }, + "REJECT - n: expected undefined, got null", + ], + // TypeBox-only: its additionalProperties also converted declared properties; pi-schema converts only undeclared ones. + [ + "additionalProperties on declared properties", + Type.Object({ amount: Type.Number(), label: Type.String() }, { additionalProperties: Type.Integer() }), + { amount: 3.5, label: "7" }, + 'ok {"amount":3.5,"label":"7"}', + ], + // TypeBox-only: property names are names, not regular expressions. + [ + "property names are not patterns", + Type.Object({ "a.b": Type.Integer(), axb: Type.String() }), + { "a.b": "4", axb: "5.8" }, + 'ok {"a.b":4,"axb":"5.8"}', + ], + // TypeBox-only: Refine runs only in TypeBox's own validator. + [ + "Refine is not run", + Type.Object({ mode: Type.Refine(Type.String(), (value) => value === "safe") }), + { mode: "unsafe" }, + 'ok {"mode":"unsafe"}', + ], + [ + "scalar-or-array that refers to itself", + { + type: "object", + properties: { + value: { anyOf: [{ type: "number" }, { type: "array", items: { $ref: "#/properties/value" } }] }, + }, + required: ["value"], + }, + { value: "42" }, + 'ok {"value":42}', + ], + [ + "StringEnum with an empty member from null", + Type.Object({ mode: Type.Unsafe({ type: "string", enum: ["", "x"] }) }), + { mode: null }, + 'ok {"mode":""}', + ], + [ + "Cyclic with named local definitions", + Type.Object({ + node: Type.Cyclic( + { Node: Type.Object({ n: Type.Number(), next: Type.Optional(Type.Ref("Node")) }) }, + "Node", + ), + }), + { node: { n: 1 } }, + 'ok {"node":{"n":1}}', + ], + [ + "unpaired high surrogate, minLength 2", + Type.Object({ s: Type.String({ minLength: 2 }) }), + { s: "\ud800A" }, + 'ok {"s":"\\ud800A"}', + ], + [ + "unpaired high surrogate, maxLength 1", + Type.Object({ s: Type.String({ maxLength: 1 }) }), + { s: "\ud800A" }, + "REJECT - s: expected at most 1 characters", + ], + ["integer from a string", Type.Object({ n: Type.Integer() }), { n: "42" }, 'ok {"n":42}'], + ["recursive union with a value no branch converts", tree, { t: {} }, "REJECT - t: expected string, got object"], + ]; + for (const [name, schema, arg, result] of solCases) { + it(name, () => { + expect(outcome(name, schema, arg)).toBe(result); + }); + } + + it("accepts the last member of a 10,000-value enum", () => { + const values = Array.from({ length: 10_000 }, (_, i) => `member${i}`); + const schema = { type: "object", properties: { choice: { type: "string", enum: values } }, required: ["choice"] }; + expect(outcome("enum", schema, { choice: "member9999" })).toBe('ok {"choice":"member9999"}'); + expect(outcome("enum", schema, { choice: "x" })).toBe( + `REJECT - choice: expected one of ${JSON.stringify(values.slice(0, 20)).slice(0, -1)}, ... and 9980 more]`, + ); + }); + + it("keeps the validator of a schema object, as before, also when the schema is changed later", () => { + const parameters = Type.Object({ n: Type.Number({ minimum: 0 }) }); + expect(outcome("late", parameters, { n: "4" })).toBe('ok {"n":4}'); + Object.assign(parameters.properties.n, { minimum: 10 }); + expect(outcome("late", parameters, { n: "4" })).toBe('ok {"n":4}'); + }); +}); + +describe("reviewer probes (round 2)", () => { + const item = { type: "object", properties: { n: { type: "integer", minimum: 1 } } }; + const nested: Case[] = [ + { name: "declared", schema: { type: "object", properties: { entry: item } }, args: [{ entry: { n: null } }] }, + { name: "additional", schema: { type: "object", additionalProperties: item }, args: [{ entry: { n: null } }] }, + { + name: "pattern", + schema: { type: "object", patternProperties: { "^entry$": item }, additionalProperties: false }, + args: [{ entry: { n: null } }], + }, + ]; + for (const { name, schema, args } of nested) { + it(`drops an optional null inside ${name} property values`, () => { + expect(outcome(name, schema, args[0])).toBe('ok {"entry":{}}'); + }); + } + + for (const keyword of ["anyOf", "oneOf"]) { + it(`converts a ${keyword} branch together with the node's sibling properties`, () => { + const schema = { + type: "object", + [keyword]: [ + { type: "object", properties: { a: { type: "array", items: { type: "integer" } } }, required: ["a"] }, + { type: "object", properties: { b: { type: "boolean" } }, required: ["b"] }, + ], + properties: { c: { type: "integer" } }, + required: ["c"], + }; + expect(outcome(keyword, schema, { a: "3", c: "2" })).toBe('ok {"a":[3],"c":2}'); + }); + } + + it("rejects a deep value every branch of a recursive object union rejects, without exponential work", () => { + const S = { + anyOf: [ + { type: "object", additionalProperties: { $ref: "#/$defs/S" }, required: ["a"] }, + { type: "object", additionalProperties: { $ref: "#/$defs/S" }, required: ["b"] }, + ], + }; + const schema = { type: "object", properties: { s: { $ref: "#/$defs/S" } }, $defs: { S } }; + let value: unknown = { x: 1 }; + for (let i = 0; i < 40; i++) value = { c: value }; + expect(outcome("deep", schema, { s: value })).toBe("REJECT - s.a: missing"); + }); +}); diff --git a/packages/ai/test/validation.test.ts b/packages/ai/test/validation.test.ts index 7cebbe541..474d0b6cf 100644 --- a/packages/ai/test/validation.test.ts +++ b/packages/ai/test/validation.test.ts @@ -1,5 +1,4 @@ import { Type } from "typebox"; -import { Compile } from "typebox/compile"; import { describe, expect, it } from "vitest"; import type { JsonValue, Tool, ToolCall } from "../src/types.ts"; import { validateToolArguments } from "../src/utils/validation.ts"; @@ -61,7 +60,7 @@ describe("validateToolArguments", () => { } }); - it("coerces serialized plain JSON schemas with AJV-compatible primitive rules", () => { + it("coerces serialized plain JSON schemas like TypeBox schemas", () => { const passingCases: Array<{ schema: Tool["parameters"]; input: unknown; @@ -75,7 +74,8 @@ describe("validateToolArguments", () => { { schema: { type: "boolean" } as Tool["parameters"], input: "false", expected: false }, { schema: { type: "boolean" } as Tool["parameters"], input: 1, expected: true }, { schema: { type: "boolean" } as Tool["parameters"], input: 0, expected: false }, - { schema: { type: "string" } as Tool["parameters"], input: null, expected: "" }, + // TypeBox's conversion rules come first: null is "null" for a required string. + { schema: { type: "string" } as Tool["parameters"], input: null, expected: "null" }, { schema: { type: "string" } as Tool["parameters"], input: true, expected: "true" }, { schema: { type: "null" } as Tool["parameters"], input: "", expected: null }, { schema: { type: "null" } as Tool["parameters"], input: 0, expected: null }, @@ -90,6 +90,11 @@ describe("validateToolArguments", () => { input: "1", expected: 1, }, + { schema: { type: "boolean" } as Tool["parameters"], input: "1", expected: true }, + { schema: { type: "boolean" } as Tool["parameters"], input: "0", expected: false }, + { schema: { type: "null" } as Tool["parameters"], input: "null", expected: null }, + { schema: { type: "integer" } as Tool["parameters"], input: "42.1", expected: 42 }, + { schema: { type: "array", items: { type: "number" } } as Tool["parameters"], input: "7", expected: [7] }, ]; for (const testCase of passingCases) { @@ -184,10 +189,6 @@ describe("validateToolArguments", () => { { type: ["array", "null"], items: { type: "string" } } as Tool["parameters"], null, ); - // The CSP test above selects TypeBox's process-wide interpreted fallback, so exercise the generated validator explicitly. - const generatedCheck = new Function(Compile(tool.parameters).Code())() as (value: unknown) => boolean; - - expect(generatedCheck(toolCall.arguments)).toBe(true); expect(validateToolArguments(tool, toolCall)).toEqual({ value: null }); }); @@ -196,10 +197,10 @@ describe("validateToolArguments", () => { schema: Tool["parameters"]; input: unknown; }> = [ - { schema: { type: "boolean" } as Tool["parameters"], input: "1" }, - { schema: { type: "boolean" } as Tool["parameters"], input: "0" }, - { schema: { type: "null" } as Tool["parameters"], input: "null" }, - { schema: { type: "integer" } as Tool["parameters"], input: "42.1" }, + { schema: { type: "boolean" } as Tool["parameters"], input: "yes" }, + { schema: { type: "number" } as Tool["parameters"], input: "abc" }, + { schema: { type: "null" } as Tool["parameters"], input: "nil" }, + { schema: { type: "integer" } as Tool["parameters"], input: "4x" }, ]; for (const testCase of failingCases) { @@ -207,6 +208,25 @@ describe("validateToolArguments", () => { expect(() => validateToolArguments(tool, toolCall)).toThrow("Validation failed"); } }); + it("lists pi-schema's errors, then the arguments it received", () => { + const { tool, toolCall } = createToolCallWithPlainSchema({ type: "number" } as Tool["parameters"], "many"); + expect(() => validateToolArguments(tool, toolCall)).toThrow( + 'Validation failed for tool "echo":\n - value: expected number, got string\n\nReceived arguments:\n{\n "value": "many"\n}', + ); + }); + + it("reports a missing required property at the property", () => { + const { tool, toolCall } = createToolCallWithPlainSchema({ type: "string" } as Tool["parameters"], "x"); + expect(() => validateToolArguments(tool, { ...toolCall, arguments: {} })).toThrow(" - value: missing"); + }); + + it("returns a copy of the arguments, also when nothing is coerced", () => { + const { tool, toolCall } = createToolCallWithPlainSchema({ type: "string" } as Tool["parameters"], "x"); + const args = validateToolArguments(tool, toolCall); + expect(args).toEqual({ value: "x" }); + expect(args).not.toBe(toolCall.arguments); + }); + // #10897 it("reports malformed arguments JSON instead of validating empty arguments", () => { const tool: Tool = { diff --git a/packages/ai/vitest.config.ts b/packages/ai/vitest.config.ts index 84c74692f..397107fa1 100644 --- a/packages/ai/vitest.config.ts +++ b/packages/ai/vitest.config.ts @@ -13,5 +13,8 @@ export default defineConfig({ }, resolve: { alias: [{ find: /^@earendil-works\/pi-telemetry$/, replacement: telemetrySrcIndex }], + // Workspace packages without a build (pi-schema) resolve to their source. + conditions: ["source"], }, + ssr: { resolve: { conditions: ["source"] } }, }); \ No newline at end of file diff --git a/packages/codemode/CHANGELOG.md b/packages/codemode/CHANGELOG.md index decd59f9e..a4c2fd624 100644 --- a/packages/codemode/CHANGELOG.md +++ b/packages/codemode/CHANGELOG.md @@ -2,6 +2,10 @@ ## [Unreleased] +### Changed + +- README: the codemode tool example builds its parameters with `@earendil-works/pi-schema` instead of TypeBox. + ## [1.4.0] - 2026-10-21 ## [1.3.0] - 2026-10-15 diff --git a/packages/codemode/README.md b/packages/codemode/README.md index e4fa87786..d97bd6bf5 100644 --- a/packages/codemode/README.md +++ b/packages/codemode/README.md @@ -275,7 +275,7 @@ import { type CodemodeTool, renderDeclarations, } from "@earendil-works/pi-codemode"; -import { Type } from "typebox"; +import { obj, str } from "@earendil-works/pi-schema"; const sandboxTools: CodemodeTool[] = agentTools.map((tool) => ({ name: tool.name, @@ -293,7 +293,7 @@ const codemodeTool: AgentTool = { name: "codemode", label: "Codemode", description: `Run JavaScript that calls tools as \`await tools.(args)\`. Output with text() or return.\n\n${renderDeclarations({ tools: sandboxTools })}`, - parameters: Type.Object({ code: Type.String() }), + parameters: obj({ code: str() }), execute: async (_toolCallId, { code }, signal) => { const sandbox = new CodemodeSandbox({ tools: sandboxTools }); try { diff --git a/packages/coding-agent/CHANGELOG.md b/packages/coding-agent/CHANGELOG.md index 359250c80..ae5a6f324 100644 --- a/packages/coding-agent/CHANGELOG.md +++ b/packages/coding-agent/CHANGELOG.md @@ -12,6 +12,8 @@ ### Changed +- Tool argument validation runs on `@earendil-works/pi-schema`. TypeBox features outside JSON Schema (`Type.Refine()`, `Type.BigInt()`, `Type.Undefined()`, codecs, `Format.Set()` formats) no longer apply to tool arguments; see [Tool parameter schemas and TypeBox](docs/extensions.md#tool-parameter-schemas-and-typebox). Validation failures the model sees list pi-schema's messages (`- path: missing`, `- limit: expected number, got string`) instead of TypeBox's (`must have required properties path`, `must be number`); which arguments are accepted and how they are coerced is unchanged for extension and built-in tools. +- Extensions keep importing `Type`, `Static` and `TSchema` from `@earendil-works/pi-ai`: Pi maps that import to pi-ai's compat entry plus the TypeBox it bundles, now that pi-ai itself no longer re-exports TypeBox, and its `StringEnum` there stays a TypeBox `TUnsafe`. The built-in tool factories (`createReadTool()` and the others) return `TypeBoxAgentTool`, which keeps `params` typed for TypeBox. - Tightened system prompt - `pi update` for installs from the pi.dev installer no longer passes `--min-release-age=0` to `npm ci`. The release lockfile pins every package, so the flag had no effect, and security scanners that block it stopped the update ([#11262](https://github.com/earendil-works/pi/issues/11262)) - `/model` and `/scoped-models` now show a model's name next to its ID when the name adds words the ID lacks, such as a renamed `models.json` override ([#5916](https://github.com/earendil-works/pi/issues/5916), [#11347](https://github.com/earendil-works/pi/pull/11347)) diff --git a/packages/coding-agent/docs/extensions.md b/packages/coding-agent/docs/extensions.md index 6fbec2b65..5879058c7 100644 --- a/packages/coding-agent/docs/extensions.md +++ b/packages/coding-agent/docs/extensions.md @@ -159,7 +159,7 @@ Settings also accept built-in extensions as `builtin:` entries; see [Setti | `@earendil-works/pi-coding-agent` | Extension types (`ExtensionAPI`, `ExtensionContext`, events) and helpers | | `@earendil-works/pi-agent-core` | Agent message and tool types | | `typebox`, `typebox/compile`, `typebox/value` | Schema definitions for tool parameters | -| `@earendil-works/pi-ai` | AI utilities (`StringEnum` for string enums, transcript helpers, model types) | +| `@earendil-works/pi-ai` | AI utilities (`StringEnum` for string enums, transcript helpers, model types). Inside Pi it also exports TypeBox's `Type`, `Static` and `TSchema`, which the published package no longer re-exports; import them from `typebox` in new code | | `@earendil-works/pi-ai/compat`, `/models`, `/oauth`, `/providers/all` | Provider construction, built-in APIs, and OAuth helpers | | `@earendil-works/pi-tui` | TUI components for custom rendering | @@ -169,6 +169,18 @@ npm dependencies work too. Add a `package.json` next to your extension (or in a For distributed Pi packages installed with `pi install` (npm or git), runtime deps must be in `dependencies`; `devDependencies` are not installed. Do not list the host-provided packages above in `dependencies`. See [Declare dependencies](packages.md#declare-dependencies). +### Tool parameter schemas and TypeBox + +Pi validates tool arguments as JSON Schema with `@earendil-works/pi-schema`, whatever built the schema; TypeBox schemas are JSON Schema and work unchanged. Before validation, loosely typed arguments are coerced: numeric strings to numbers, `"true"` to `true`, a single value to a one-item array, `null` in an optional property dropped. TypeBox features outside JSON Schema do not apply to tool arguments: + +- `Type.Refine()` refinements are not run. Check such constraints in `execute()`. +- `Type.BigInt()`, `Type.Undefined()` and `Type.Void()` reject every argument, since JSON has no such values. Codecs (`Type.Codec()`) do not decode: `execute()` receives the JSON value. +- Formats registered with TypeBox's `Format.Set()` are not checked. The standard formats are, plus `url` and `json-pointer-uri-fragment`. +- Property names are matched as names, not as regular expressions, and `additionalProperties` applies only to properties the object does not declare. +- A `$ref` that names nothing rejects only arguments at its place. + +Validation errors the model sees use pi-schema's messages, such as `- path: missing` or `- limit: expected number, got string`. + Node.js built-ins (`node:fs`, `node:path`, etc.) are also available. ## Writing an Extension diff --git a/packages/coding-agent/examples/tsconfig.json b/packages/coding-agent/examples/tsconfig.json new file mode 100644 index 000000000..03e1a36e1 --- /dev/null +++ b/packages/coding-agent/examples/tsconfig.json @@ -0,0 +1,180 @@ +// Type-checks the example extensions against the modules extensions see at run time (core/extensions/virtual-modules.ts): +// "@earendil-works/pi-ai" is the extension view, which adds TypeBox's Type, Static and TSchema to the compat entry (the +// view imports "@earendil-works/pi-ai/compat", so that path stays the package's own). +// The other paths repeat the root tsconfig.json; keep them in sync. +// Also the test that imports the subagent example: it type-checks only against the extension view. +{ + "extends": "../../../tsconfig.json", + "compilerOptions": { + "paths": { + "@earendil-works/chord": [ + "../../../packages/chord/src/index.ts" + ], + "@earendil-works/chord/context": [ + "../../../packages/chord/src/context/index.ts" + ], + "@earendil-works/chord/delta": [ + "../../../packages/chord/src/delta/index.ts" + ], + "@earendil-works/chord/next": [ + "../../../packages/chord/src/next/index.ts" + ], + "@earendil-works/chord/next/plugins": [ + "../../../packages/chord/src/next-node/plugins/index.ts" + ], + "@earendil-works/chord/next/plugins/runtime": [ + "../../../packages/chord/src/next-node/plugins/runtime.ts" + ], + "@test/contracts/*": [ + "../../../packages/chord/test/next-node/fixtures/plugins/contracts/*.ts" + ], + "@example/contracts/*": [ + "../../../packages/chord/test/next/examples/35-plugins/plugins/contracts/*.ts" + ], + "@example/weather/*": [ + "../../../packages/chord/test/next/examples/36-plugin-failures/plugins/weather/*.ts" + ], + "@example/prices/*": [ + "../../../packages/chord/test/next/examples/36-plugin-failures/plugins/prices/*.ts" + ], + "@example/shapes/*": [ + "../../../packages/chord/test/next/examples/37-plugin-imports/plugins/shapes/*.ts" + ], + "@example/shared/*": [ + "../../../packages/chord/test/next/examples/38-watch-and-rollout/plugins/contracts/*.ts" + ], + "@earendil-works/chord/next/http": [ + "../../../packages/chord/src/next-node/http/index.ts" + ], + "@earendil-works/chord/next/workers": [ + "../../../packages/chord/src/next-node/workers/index.ts" + ], + "@earendil-works/chord/node": [ + "../../../packages/chord/src/node.ts" + ], + "@earendil-works/chord/output": [ + "../../../packages/chord/src/output.ts" + ], + "@earendil-works/chord/telemetry-sqlite": [ + "../../../packages/chord/src/telemetry-sqlite/index.ts" + ], + "@earendil-works/chord/sqlite": [ + "../../../packages/chord/src/sqlite/index.ts" + ], + "@earendil-works/chord/sqlite/node": [ + "../../../packages/chord/src/sqlite/node.ts" + ], + "@earendil-works/chord/sqlite/cloudflare": [ + "../../../packages/chord/src/sqlite/cloudflare.ts" + ], + "@earendil-works/pi-telemetry": [ + "../../../packages/telemetry/src/index.ts" + ], + "@earendil-works/pi-telemetry/testing": [ + "../../../packages/telemetry/src/testing/index.ts" + ], + "@earendil-works/pi-telemetry/render": [ + "../../../packages/telemetry/src/render.ts" + ], + "@earendil-works/pi-codemode": [ + "../../../packages/codemode/src/index.ts" + ], + "@earendil-works/pi-codemode/cloudflare": [ + "../../../packages/codemode/src/cloudflare.ts" + ], + "@earendil-works/pi-codemode/declarations": [ + "../../../packages/codemode/src/declarations.ts" + ], + "@earendil-works/pi-codemode/portable": [ + "../../../packages/codemode/src/portable.ts" + ], + "@earendil-works/pi-codemode/source": [ + "../../../packages/codemode/src/source.ts" + ], + "@earendil-works/pi-codemode/worker": [ + "../../../packages/codemode/src/runtime/worker.ts" + ], + "@earendil-works/pi-schema": [ + "../../../packages/schema/src/index.ts" + ], + "@earendil-works/pi-schema/formats": [ + "../../../packages/schema/src/formats/index.ts" + ], + "@earendil-works/pi-mcp": [ + "../../../packages/mcp/src/index.ts" + ], + "@earendil-works/pi-mcp/oauth": [ + "../../../packages/mcp/src/oauth/index.ts" + ], + "@earendil-works/pi-mcp/testing": [ + "../../../packages/mcp/src/testing/index.ts" + ], + "@earendil-works/pi-ai": [ + "../../../packages/coding-agent/src/core/extensions/pi-ai-extension-view.ts" + ], + "@earendil-works/pi-ai/oauth": [ + "../../../packages/ai/src/oauth.ts" + ], + "@earendil-works/pi-ai/*": [ + "../../../packages/ai/src/*.ts", + "../../../packages/ai/src/providers/*.ts" + ], + "@earendil-works/pi-ai/dist/*": [ + "../../../packages/ai/src/*" + ], + "@earendil-works/pi-durable": [ + "../../../packages/durable/src/index.ts" + ], + "@earendil-works/pi-durable/testing": [ + "../../../packages/durable/src/testing/index.ts" + ], + "@earendil-works/pi-durable/*": [ + "../../../packages/durable/src/*.ts", + "../../../packages/durable/src/*/index.ts", + "../../../packages/durable/src/*" + ], + "@earendil-works/pi-env": [ + "../../../packages/env/src/index.ts" + ], + "@earendil-works/pi-agent-core": [ + "../../../packages/agent/src/index.ts" + ], + "@earendil-works/pi-coding-agent": [ + "../../../packages/coding-agent/src/index.ts" + ], + "@earendil-works/pi-coding-agent/hooks": [ + "../../../packages/coding-agent/src/core/hooks/index.ts" + ], + "@earendil-works/pi-coding-agent/*": [ + "../../../packages/coding-agent/src/*" + ], + "typebox": [ + "../../../node_modules/typebox" + ], + "@earendil-works/pi-tui": [ + "../../../packages/tui/src/index.ts" + ], + "@earendil-works/pi-tui/*": [ + "../../../packages/tui/src/*" + ], + "@earendil-works/pi-agent-old": [ + "../../../packages/agent-old/src/index.ts" + ], + "@earendil-works/pi-agent-old/*": [ + "../../../packages/agent-old/src/*" + ], + "@mariozechner/pi-ai": [ + "../../../packages/coding-agent/src/core/extensions/pi-ai-extension-view.ts" + ] + } + }, + "include": [ + "./extensions/**/*", + "../../*/src/**/*.d.ts", + "../test/suite/regressions/8261-subagent-project-trust.test.ts" + ], + "exclude": [ + "**/node_modules/**", + "./extensions/gondolin/**" + ] +} diff --git a/packages/coding-agent/install-lock/package-lock.json b/packages/coding-agent/install-lock/package-lock.json index ed922e44f..521c82fef 100644 --- a/packages/coding-agent/install-lock/package-lock.json +++ b/packages/coding-agent/install-lock/package-lock.json @@ -391,7 +391,7 @@ "license": "MIT", "dependencies": { "@earendil-works/pi-ai": "1.4.0", - "typebox": "1.3.27" + "@earendil-works/pi-schema": "1.4.0" }, "engines": { "node": ">=22.19.0" @@ -404,14 +404,14 @@ "dependencies": { "@anthropic-ai/sdk": "0.129.0", "@aws-sdk/client-bedrock-runtime": "3.1127.0", + "@earendil-works/pi-schema": "1.4.0", "@earendil-works/pi-telemetry": "1.4.0", "@google/genai": "2.21.0", "@smithy/node-http-handler": "4.12.1", "http-proxy-agent": "9.1.0", "https-proxy-agent": "9.1.0", "openai": "7.19.0", - "partial-json": "0.1.7", - "typebox": "1.3.27" + "partial-json": "0.1.7" }, "bin": { "pi-ai": "dist/cli.js" @@ -537,6 +537,14 @@ "node": ">=22.19.0" } }, + "node_modules/@earendil-works/pi-schema": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/@earendil-works/pi-schema/-/pi-schema-1.4.0.tgz", + "license": "MIT", + "engines": { + "node": ">=22.19.0" + } + }, "node_modules/@earendil-works/pi-telemetry": { "version": "1.4.0", "resolved": "https://registry.npmjs.org/@earendil-works/pi-telemetry/-/pi-telemetry-1.4.0.tgz", diff --git a/packages/coding-agent/schemas/models.schema.json b/packages/coding-agent/schemas/models.schema.json index 39ec3d63e..c51a82c35 100644 --- a/packages/coding-agent/schemas/models.schema.json +++ b/packages/coding-agent/schemas/models.schema.json @@ -255,13 +255,6 @@ "type": "array", "items": { "type": "object", - "required": [ - "inputTokensAbove", - "input", - "output", - "cacheRead", - "cacheWrite" - ], "properties": { "inputTokensAbove": { "type": "number", @@ -283,7 +276,14 @@ "type": "number", "description": "Cache-write cost in USD per million tokens." } - } + }, + "required": [ + "inputTokensAbove", + "input", + "output", + "cacheRead", + "cacheWrite" + ] }, "description": "Request-wide pricing tiers. The highest matching input threshold applies to the full request." } @@ -383,12 +383,6 @@ "$defs": { "ModelCost": { "type": "object", - "required": [ - "input", - "output", - "cacheRead", - "cacheWrite" - ], "properties": { "input": { "type": "number", @@ -410,13 +404,6 @@ "type": "array", "items": { "type": "object", - "required": [ - "inputTokensAbove", - "input", - "output", - "cacheRead", - "cacheWrite" - ], "properties": { "inputTokensAbove": { "type": "number", @@ -438,11 +425,24 @@ "type": "number", "description": "Cache-write cost in USD per million tokens." } - } + }, + "required": [ + "inputTokensAbove", + "input", + "output", + "cacheRead", + "cacheWrite" + ] }, "description": "Request-wide pricing tiers. The highest matching input threshold applies to the full request." } - } + }, + "required": [ + "input", + "output", + "cacheRead", + "cacheWrite" + ] }, "ModelInputLimits": { "type": "object", @@ -628,9 +628,6 @@ }, { "type": "object", - "required": [ - "$var" - ], "properties": { "$var": { "anyOf": [ @@ -651,7 +648,10 @@ "omitWhenOff": { "type": "boolean" } - } + }, + "required": [ + "$var" + ] } ] } @@ -677,9 +677,6 @@ }, { "type": "object", - "required": [ - "$var" - ], "properties": { "$var": { "anyOf": [ @@ -700,7 +697,10 @@ "omitWhenOff": { "type": "boolean" } - } + }, + "required": [ + "$var" + ] } ] } @@ -1085,11 +1085,6 @@ "type": "array", "items": { "type": "object", - "required": [ - "provider", - "model", - "cost" - ], "properties": { "provider": { "type": "string", @@ -1103,6 +1098,11 @@ "$ref": "#/$defs/ModelCost" } }, + "required": [ + "provider", + "model", + "cost" + ], "description": "An Anthropic server-side refusal fallback model with local pricing metadata." }, "maxItems": 3, diff --git a/packages/coding-agent/src/core/extensions/loader.ts b/packages/coding-agent/src/core/extensions/loader.ts index 235a8632e..ff9f127c4 100644 --- a/packages/coding-agent/src/core/extensions/loader.ts +++ b/packages/coding-agent/src/core/extensions/loader.ts @@ -88,10 +88,11 @@ function getAliases(): Record { const piCodingAgentEntry = packageIndex; const piAgentCoreEntry = resolveWorkspaceOrImport("agent/dist/index.js", "@earendil-works/pi-agent-core"); const piTuiEntry = resolveWorkspaceOrImport("tui/dist/index.js", "@earendil-works/pi-tui"); - // Extensions resolve the pi-ai root to the compat entrypoint (a strict - // superset of the core entrypoint): existing extensions using the old - // global API keep working at runtime until compat is removed. - const piAiCompatEntry = resolveWorkspaceOrImport("ai/dist/compat.js", "@earendil-works/pi-ai/compat"); + // Extensions resolve the pi-ai root to the extension view: the compat + // entrypoint (a strict superset of the core entrypoint) plus TypeBox's Type, + // Static and TSchema, so extensions using the old global API and pi-ai's + // former TypeBox re-exports keep working. + const piAiCompatEntry = path.resolve(__dirname, "pi-ai-extension-view.js"); const piAiModelsEntry = resolveWorkspaceOrImport("ai/dist/models.js", "@earendil-works/pi-ai/models"); const piAiOauthEntry = resolveWorkspaceOrImport("ai/dist/oauth.js", "@earendil-works/pi-ai/oauth"); const piAiProvidersEntry = resolveWorkspaceOrImport( diff --git a/packages/coding-agent/src/core/extensions/pi-ai-extension-view.ts b/packages/coding-agent/src/core/extensions/pi-ai-extension-view.ts new file mode 100644 index 000000000..e9c6808f7 --- /dev/null +++ b/packages/coding-agent/src/core/extensions/pi-ai-extension-view.ts @@ -0,0 +1,28 @@ +/** + * The `@earendil-works/pi-ai` module extensions import: pi-ai's compat entry point plus TypeBox's `Type`, `Static` and + * `TSchema`, which pi-ai's main entry no longer re-exports, and a `StringEnum` built with TypeBox (`TUnsafe`), so + * extensions that use it inside `Type.Object` type-check and infer as before. The TypeBox is the coding agent's own, + * the same one extensions get for `typebox`. + */ + +import { type TUnsafe, Type } from "typebox"; + +export * from "@earendil-works/pi-ai/compat"; +export type { Static, TSchema } from "typebox"; +export { Type }; + +/** + * Creates a string enum schema compatible with Google's API and other providers that don't support anyOf/const + * patterns: `{ type: "string", enum: [...] }`, typed for TypeBox's `Static`. + */ +export function StringEnum( + values: T, + options?: { description?: string; default?: T[number] }, +): TUnsafe { + return Type.Unsafe({ + type: "string", + enum: values as unknown as T[number][], + ...(options?.description && { description: options.description }), + ...(options?.default && { default: options.default }), + }); +} diff --git a/packages/coding-agent/src/core/extensions/virtual-modules.ts b/packages/coding-agent/src/core/extensions/virtual-modules.ts index 4b8449338..3c30317d0 100644 --- a/packages/coding-agent/src/core/extensions/virtual-modules.ts +++ b/packages/coding-agent/src/core/extensions/virtual-modules.ts @@ -1,5 +1,4 @@ import * as bundledPiAgentCore from "@earendil-works/pi-agent-core"; -import * as bundledPiAiCompat from "@earendil-works/pi-ai/compat"; import * as bundledPiAiModels from "@earendil-works/pi-ai/models"; import * as bundledPiAiOauth from "@earendil-works/pi-ai/oauth"; import * as bundledPiAiProviders from "@earendil-works/pi-ai/providers/all"; @@ -10,6 +9,7 @@ import * as bundledTypeboxValue from "typebox/value"; // This import is safe because loader.ts exports are not re-exported from index.ts. // Extensions can therefore import from @earendil-works/pi-coding-agent. import * as bundledPiCodingAgent from "../../index.ts"; +import * as bundledPiAiExtensionView from "./pi-ai-extension-view.ts"; /** Modules available to extensions in source and compiled binary runtimes. */ export const VIRTUAL_MODULES: Record = { @@ -22,18 +22,19 @@ export const VIRTUAL_MODULES: Record = { "@earendil-works/pi-agent-core": bundledPiAgentCore, "@earendil-works/pi-tui": bundledPiTui, // Extensions resolve the pi-ai root to the compat entrypoint (a strict - // superset of the core entrypoint): existing extensions using the old - // global API keep working at runtime until compat is removed. - "@earendil-works/pi-ai": bundledPiAiCompat, - "@earendil-works/pi-ai/compat": bundledPiAiCompat, + // superset of the core entrypoint) plus TypeBox's Type, Static and TSchema: + // existing extensions using the old global API and pi-ai's former TypeBox + // re-exports keep working. + "@earendil-works/pi-ai": bundledPiAiExtensionView, + "@earendil-works/pi-ai/compat": bundledPiAiExtensionView, "@earendil-works/pi-ai/models": bundledPiAiModels, "@earendil-works/pi-ai/oauth": bundledPiAiOauth, "@earendil-works/pi-ai/providers/all": bundledPiAiProviders, "@earendil-works/pi-coding-agent": bundledPiCodingAgent, "@mariozechner/pi-agent-core": bundledPiAgentCore, "@mariozechner/pi-tui": bundledPiTui, - "@mariozechner/pi-ai": bundledPiAiCompat, - "@mariozechner/pi-ai/compat": bundledPiAiCompat, + "@mariozechner/pi-ai": bundledPiAiExtensionView, + "@mariozechner/pi-ai/compat": bundledPiAiExtensionView, "@mariozechner/pi-ai/models": bundledPiAiModels, "@mariozechner/pi-ai/oauth": bundledPiAiOauth, "@mariozechner/pi-ai/providers/all": bundledPiAiProviders, diff --git a/packages/coding-agent/src/core/model-config.ts b/packages/coding-agent/src/core/model-config.ts index 553ff35ba..200cdeec0 100644 --- a/packages/coding-agent/src/core/model-config.ts +++ b/packages/coding-agent/src/core/model-config.ts @@ -1,20 +1,21 @@ /** Immutable, credential-blind models.json snapshot. */ import { readFile } from "node:fs/promises"; -import { ProviderCompatSchema } from "@earendil-works/pi-ai/providers/compat-schema"; -import { - ModelCostSchema, - ModelInputLimitsSchema, - ModelInputModalitySchema, - ModelPromptCacheSchema, - ThinkingLevelMapSchema, -} from "@earendil-works/pi-ai/providers/model-schema"; import { type Static, Type } from "typebox"; import { Compile } from "typebox/compile"; import type { TLocalizedValidationError } from "typebox/error"; import { stripJsonComments } from "../utils/json.ts"; import { normalizePath } from "../utils/paths.ts"; import { stripBom } from "../utils/text.ts"; +import { + ModelCostSchema, + ModelInputLimitsSchema, + ModelInputModalitySchema, + ModelPromptCacheSchema, + PartialModelCostSchema, + ProviderCompatSchema, + ThinkingLevelMapSchema, +} from "./pi-ai-schemas.ts"; const SamplingParamsSchema = Type.Record(Type.String(), Type.Unknown()); const SamplingParamsByThinkingLevelSchema = Type.Object( @@ -74,7 +75,7 @@ const ModelOverrideSchema = Type.Object({ thinkingLevelMap: Type.Optional(ThinkingLevelMapSchema), input: Type.Optional(Type.Array(ModelInputModalitySchema)), inputLimits: Type.Optional(ModelInputLimitsSchema), - cost: Type.Optional(Type.Partial(ModelCostSchema)), + cost: Type.Optional(PartialModelCostSchema), promptCache: Type.Optional(ModelPromptCacheSchema), contextWindow: Type.Optional( Type.Number({ exclusiveMinimum: 0, description: ModelFieldDescriptions.contextWindow }), diff --git a/packages/coding-agent/src/core/pi-ai-schemas.ts b/packages/coding-agent/src/core/pi-ai-schemas.ts new file mode 100644 index 000000000..694aa1fe6 --- /dev/null +++ b/packages/coding-agent/src/core/pi-ai-schemas.ts @@ -0,0 +1,40 @@ +/** + * pi-ai's configuration schemas (built with pi-schema) as TypeBox schemas, for the coding agent's own TypeBox-built + * configuration schemas (models.json, settings.json): the same JSON, typed for TypeBox's `Static`. + */ + +import { + type ProviderCompat, + ProviderCompatSchema as ProviderCompatJsonSchema, +} from "@earendil-works/pi-ai/providers/compat-schema"; +import { + type ModelCost, + ModelCostSchema as ModelCostJsonSchema, + type ModelInputLimits, + ModelInputLimitsSchema as ModelInputLimitsJsonSchema, + type ModelInputModality, + ModelInputModalitySchema as ModelInputModalityJsonSchema, + type ModelPromptCache, + ModelPromptCacheSchema as ModelPromptCacheJsonSchema, + type ModelThinkingLevel, + ModelThinkingLevelSchema as ModelThinkingLevelJsonSchema, + type ThinkingLevelMap, + ThinkingLevelMapSchema as ThinkingLevelMapJsonSchema, +} from "@earendil-works/pi-ai/providers/model-schema"; +import { Type } from "typebox"; + +export const ProviderCompatSchema = Type.Unsafe(ProviderCompatJsonSchema); +export const ModelCostSchema = Type.Unsafe(ModelCostJsonSchema); +export const ModelInputLimitsSchema = Type.Unsafe(ModelInputLimitsJsonSchema); +export const ModelInputModalitySchema = Type.Unsafe(ModelInputModalityJsonSchema); +export const ModelPromptCacheSchema = Type.Unsafe(ModelPromptCacheJsonSchema); +export const ModelThinkingLevelSchema = Type.Unsafe(ModelThinkingLevelJsonSchema); +export const ThinkingLevelMapSchema = Type.Unsafe(ThinkingLevelMapJsonSchema); + +/** `ModelCostSchema` with every field optional, for model overrides that change some of the rates. */ +export const PartialModelCostSchema = Type.Unsafe>(withoutRequired(ModelCostJsonSchema)); + +function withoutRequired(schema: object): object { + const { required: _required, ...rest } = schema as { required?: unknown }; + return rest; +} diff --git a/packages/coding-agent/src/core/settings-schema.ts b/packages/coding-agent/src/core/settings-schema.ts index c3ada2872..1ef797338 100644 --- a/packages/coding-agent/src/core/settings-schema.ts +++ b/packages/coding-agent/src/core/settings-schema.ts @@ -1,5 +1,5 @@ -import { ModelThinkingLevelSchema } from "@earendil-works/pi-ai/providers/model-schema"; import { type Static, type TSchemaOptions, Type } from "typebox"; +import { ModelThinkingLevelSchema } from "./pi-ai-schemas.ts"; import { SETTINGS_DEFAULTS } from "./settings-defaults.ts"; function nonNegativeSafeInteger(options: { default?: number } = {}) { diff --git a/packages/coding-agent/src/core/tools/bash.ts b/packages/coding-agent/src/core/tools/bash.ts index 4620e2464..2084d6f7c 100644 --- a/packages/coding-agent/src/core/tools/bash.ts +++ b/packages/coding-agent/src/core/tools/bash.ts @@ -1,7 +1,6 @@ import { constants } from "node:fs"; import { access as fsAccess } from "node:fs/promises"; import { constants as osConstants } from "node:os"; -import type { AgentTool } from "@earendil-works/pi-agent-core"; import { spawn } from "child_process"; import { type Static, Type } from "typebox"; import { waitForChildProcess } from "../../utils/child-process.ts"; @@ -16,7 +15,7 @@ import { import type { ExtensionContext, ToolDefinition } from "../extensions/types.ts"; import { OutputAccumulator } from "./output-accumulator.ts"; import { BASH_UPDATE_THROTTLE_MS, createShellRenderers } from "./renderers/bash.ts"; -import { wrapToolDefinition } from "./tool-definition-wrapper.ts"; +import { type TypeBoxAgentTool, wrapToolDefinition } from "./tool-definition-wrapper.ts"; import { DEFAULT_MAX_BYTES, DEFAULT_MAX_LINES, formatSize, type TruncationResult } from "./truncate.ts"; const MAX_TIMEOUT_MS = 2_147_483_647; @@ -431,7 +430,7 @@ export function createBashToolDefinition( return createShellToolDefinition(cwd, bashToolConfig, options); } -export function createBashTool(cwd: string, options?: BashToolOptions): AgentTool { +export function createBashTool(cwd: string, options?: BashToolOptions): TypeBoxAgentTool { const definition = createBashToolDefinition(cwd, options); const tool = wrapToolDefinition(definition); Object.assign(tool, { diff --git a/packages/coding-agent/src/core/tools/edit.ts b/packages/coding-agent/src/core/tools/edit.ts index ad1cbd2f3..bb6aeff59 100644 --- a/packages/coding-agent/src/core/tools/edit.ts +++ b/packages/coding-agent/src/core/tools/edit.ts @@ -1,4 +1,3 @@ -import type { AgentTool } from "@earendil-works/pi-agent-core"; import { constants } from "fs"; import { access as fsAccess, readFile as fsReadFile, writeFile as fsWriteFile } from "fs/promises"; import { type Static, Type } from "typebox"; @@ -16,7 +15,7 @@ import { import { withFileMutationQueue } from "./file-mutation-queue.ts"; import { resolveToCwd } from "./path-utils.ts"; import { type EditRenderState, editRenderers } from "./renderers/edit.ts"; -import { wrapToolDefinition } from "./tool-definition-wrapper.ts"; +import { type TypeBoxAgentTool, wrapToolDefinition } from "./tool-definition-wrapper.ts"; const replaceEditSchema = Type.Object( { @@ -213,6 +212,6 @@ export function createEditToolDefinition( }; } -export function createEditTool(cwd: string, options?: EditToolOptions): AgentTool { +export function createEditTool(cwd: string, options?: EditToolOptions): TypeBoxAgentTool { return wrapToolDefinition(createEditToolDefinition(cwd, options)); } diff --git a/packages/coding-agent/src/core/tools/find.ts b/packages/coding-agent/src/core/tools/find.ts index 8d44c491e..77ba1b356 100644 --- a/packages/coding-agent/src/core/tools/find.ts +++ b/packages/coding-agent/src/core/tools/find.ts @@ -1,5 +1,4 @@ import { createInterface } from "node:readline"; -import type { AgentTool } from "@earendil-works/pi-agent-core"; import { spawn } from "child_process"; import path from "path"; import { type Static, Type } from "typebox"; @@ -7,7 +6,7 @@ import { ensureTool } from "../../utils/tools-manager.ts"; import type { ExtensionContext, ToolDefinition } from "../extensions/types.ts"; import { pathExists, resolveToCwd } from "./path-utils.ts"; import { findRenderers } from "./renderers/find.ts"; -import { wrapToolDefinition } from "./tool-definition-wrapper.ts"; +import { type TypeBoxAgentTool, wrapToolDefinition } from "./tool-definition-wrapper.ts"; import { DEFAULT_MAX_BYTES, formatSize, type TruncationResult, truncateHead } from "./truncate.ts"; /** Relativize a find result against the search root and normalize it to posix separators. */ @@ -313,6 +312,6 @@ export function createFindToolDefinition( }; } -export function createFindTool(cwd: string, options?: FindToolOptions): AgentTool { +export function createFindTool(cwd: string, options?: FindToolOptions): TypeBoxAgentTool { return wrapToolDefinition(createFindToolDefinition(cwd, options)); } diff --git a/packages/coding-agent/src/core/tools/grep.ts b/packages/coding-agent/src/core/tools/grep.ts index 77d830cd1..e449e0898 100644 --- a/packages/coding-agent/src/core/tools/grep.ts +++ b/packages/coding-agent/src/core/tools/grep.ts @@ -1,6 +1,5 @@ import { readFile as fsReadFile, stat as fsStat } from "node:fs/promises"; import { createInterface } from "node:readline"; -import type { AgentTool } from "@earendil-works/pi-agent-core"; import { spawn } from "child_process"; import path from "path"; import { type Static, Type } from "typebox"; @@ -8,7 +7,7 @@ import { ensureTool } from "../../utils/tools-manager.ts"; import type { ExtensionContext, ToolDefinition } from "../extensions/types.ts"; import { resolveToCwd } from "./path-utils.ts"; import { grepRenderers } from "./renderers/grep.ts"; -import { wrapToolDefinition } from "./tool-definition-wrapper.ts"; +import { type TypeBoxAgentTool, wrapToolDefinition } from "./tool-definition-wrapper.ts"; import { DEFAULT_MAX_BYTES, formatSize, @@ -320,6 +319,6 @@ export function createGrepToolDefinition( }; } -export function createGrepTool(cwd: string, options?: GrepToolOptions): AgentTool { +export function createGrepTool(cwd: string, options?: GrepToolOptions): TypeBoxAgentTool { return wrapToolDefinition(createGrepToolDefinition(cwd, options)); } diff --git a/packages/coding-agent/src/core/tools/ls.ts b/packages/coding-agent/src/core/tools/ls.ts index 558ff9f9f..4f5efc8a7 100644 --- a/packages/coding-agent/src/core/tools/ls.ts +++ b/packages/coding-agent/src/core/tools/ls.ts @@ -1,11 +1,10 @@ import { readdir as fsReaddir, stat as fsStat } from "node:fs/promises"; -import type { AgentTool } from "@earendil-works/pi-agent-core"; import nodePath from "path"; import { type Static, Type } from "typebox"; import type { ExtensionContext, ToolDefinition } from "../extensions/types.ts"; import { pathExists, resolveToCwd } from "./path-utils.ts"; import { lsRenderers } from "./renderers/ls.ts"; -import { wrapToolDefinition } from "./tool-definition-wrapper.ts"; +import { type TypeBoxAgentTool, wrapToolDefinition } from "./tool-definition-wrapper.ts"; import { DEFAULT_MAX_BYTES, formatSize, type TruncationResult, truncateHead } from "./truncate.ts"; const lsSchema = Type.Object({ @@ -170,6 +169,6 @@ export function createLsToolDefinition( }; } -export function createLsTool(cwd: string, options?: LsToolOptions): AgentTool { +export function createLsTool(cwd: string, options?: LsToolOptions): TypeBoxAgentTool { return wrapToolDefinition(createLsToolDefinition(cwd, options)); } diff --git a/packages/coding-agent/src/core/tools/read.ts b/packages/coding-agent/src/core/tools/read.ts index ebe2c2517..8fb93a679 100644 --- a/packages/coding-agent/src/core/tools/read.ts +++ b/packages/coding-agent/src/core/tools/read.ts @@ -1,4 +1,3 @@ -import type { AgentTool } from "@earendil-works/pi-agent-core"; import type { Api, ImageContent, Model, ModelImageResizeOptions, TextContent } from "@earendil-works/pi-ai"; import { constants } from "fs"; import { access as fsAccess, readFile as fsReadFile } from "fs/promises"; @@ -8,7 +7,7 @@ import { detectSupportedImageMimeTypeFromFile } from "../../utils/mime.ts"; import type { ExtensionContext, ToolDefinition } from "../extensions/types.ts"; import { resolveReadPathAsync } from "./path-utils.ts"; import { readRenderers } from "./renderers/read.ts"; -import { wrapToolDefinition } from "./tool-definition-wrapper.ts"; +import { type TypeBoxAgentTool, wrapToolDefinition } from "./tool-definition-wrapper.ts"; import { DEFAULT_MAX_BYTES, DEFAULT_MAX_LINES, formatSize, type TruncationResult, truncateHead } from "./truncate.ts"; const readSchema = Type.Object({ @@ -219,6 +218,6 @@ export function createReadToolDefinition( }; } -export function createReadTool(cwd: string, options?: ReadToolOptions): AgentTool { +export function createReadTool(cwd: string, options?: ReadToolOptions): TypeBoxAgentTool { return wrapToolDefinition(createReadToolDefinition(cwd, options)); } diff --git a/packages/coding-agent/src/core/tools/tool-definition-wrapper.ts b/packages/coding-agent/src/core/tools/tool-definition-wrapper.ts index 9a9465dec..320030fab 100644 --- a/packages/coding-agent/src/core/tools/tool-definition-wrapper.ts +++ b/packages/coding-agent/src/core/tools/tool-definition-wrapper.ts @@ -1,6 +1,25 @@ -import type { AgentTool } from "@earendil-works/pi-agent-core"; +import type { AgentTool, AgentToolResult, AgentToolUpdateCallback } from "@earendil-works/pi-agent-core"; +import type { Static, TSchema } from "typebox"; import type { ExtensionToolContext, ToolDefinition } from "../extensions/types.ts"; +/** + * An `AgentTool` with a TypeBox parameter schema whose `execute` and `prepareArguments` are typed by TypeBox's + * `Static`, as the built-in tools are (pi-agent-core types `params` through pi-schema, which gives `unknown` for + * TypeBox schemas). Spreads into a `ToolDefinition` of the same schema. + */ +export type TypeBoxAgentTool = Omit< + AgentTool, + "prepareArguments" | "execute" +> & { + prepareArguments?: (args: unknown) => Static; + execute: ( + toolCallId: string, + params: Static, + signal?: AbortSignal, + onUpdate?: AgentToolUpdateCallback, + ) => Promise>; +}; + /** Creates the context for one tool call. */ export type ToolContextFactory = (toolCallId: string, signal: AbortSignal | undefined) => ExtensionToolContext; diff --git a/packages/coding-agent/src/core/tools/write.ts b/packages/coding-agent/src/core/tools/write.ts index 374089017..9e6b296dc 100644 --- a/packages/coding-agent/src/core/tools/write.ts +++ b/packages/coding-agent/src/core/tools/write.ts @@ -1,4 +1,3 @@ -import type { AgentTool } from "@earendil-works/pi-agent-core"; import { mkdir as fsMkdir, writeFile as fsWriteFile } from "fs/promises"; import { dirname } from "path"; import { type Static, Type } from "typebox"; @@ -6,7 +5,7 @@ import type { ExtensionContext, ToolDefinition } from "../extensions/types.ts"; import { withFileMutationQueue } from "./file-mutation-queue.ts"; import { resolveToCwd } from "./path-utils.ts"; import { writeRenderers } from "./renderers/write.ts"; -import { wrapToolDefinition } from "./tool-definition-wrapper.ts"; +import { type TypeBoxAgentTool, wrapToolDefinition } from "./tool-definition-wrapper.ts"; const writeSchema = Type.Object({ path: Type.String({ description: "Path to the file to write (relative or absolute)" }), @@ -92,6 +91,6 @@ export function createWriteToolDefinition( }; } -export function createWriteTool(cwd: string, options?: WriteToolOptions): AgentTool { +export function createWriteTool(cwd: string, options?: WriteToolOptions): TypeBoxAgentTool { return wrapToolDefinition(createWriteToolDefinition(cwd, options)); } diff --git a/packages/coding-agent/src/extensions/codemode/tool.ts b/packages/coding-agent/src/extensions/codemode/tool.ts index e28ea999f..648ba6109 100644 --- a/packages/coding-agent/src/extensions/codemode/tool.ts +++ b/packages/coding-agent/src/extensions/codemode/tool.ts @@ -32,7 +32,7 @@ import type { ToolDefinition, ToolLoadout, ToolLoadoutChanges, ToolNamespace } f import type { ModelRegistry } from "../../core/model-registry.ts"; import { SETTINGS_DEFAULTS } from "../../core/settings-defaults.ts"; import type { CodemodeMode } from "../../core/settings-manager.ts"; -import { wrapToolDefinition } from "../../core/tools/tool-definition-wrapper.ts"; +import { type TypeBoxAgentTool, wrapToolDefinition } from "../../core/tools/tool-definition-wrapper.ts"; import { loadCodemodeExecutor } from "./execute.lazy.ts"; import { codemodeRenderers } from "./renderer.ts"; @@ -422,7 +422,7 @@ export function createCodemodeToolDefinition( export function createCodemodeTool( tools: readonly AgentTool[] = [], options: CodemodeToolOptions = {}, -): AgentTool { +): TypeBoxAgentTool { const definition = createCodemodeToolDefinition(options); const tool = wrapToolDefinition(definition); Object.assign(tool, { diff --git a/packages/coding-agent/test/sdk-codex-cache-probe-tool-loop.ts b/packages/coding-agent/test/sdk-codex-cache-probe-tool-loop.ts index 7c569f6e8..2813adc0f 100644 --- a/packages/coding-agent/test/sdk-codex-cache-probe-tool-loop.ts +++ b/packages/coding-agent/test/sdk-codex-cache-probe-tool-loop.ts @@ -21,8 +21,8 @@ import { getModel, type Model, type SimpleStreamOptions, - Type, } from "@earendil-works/pi-ai/compat"; +import { Type } from "typebox"; import { getOpenAICodexWebSocketDebugStats, streamSimple as streamSimpleOpenAICodexResponses, diff --git a/packages/coding-agent/test/suite/agent-session-codemode.test.ts b/packages/coding-agent/test/suite/agent-session-codemode.test.ts index ff39f1cd6..3e0620635 100644 --- a/packages/coding-agent/test/suite/agent-session-codemode.test.ts +++ b/packages/coding-agent/test/suite/agent-session-codemode.test.ts @@ -18,7 +18,10 @@ import { Type } from "typebox"; import { afterEach, describe, expect, it } from "vitest"; import type { ExtensionAPI } from "../../src/core/extensions/types.ts"; import type { CustomEntry } from "../../src/core/session-manager.ts"; -import { createToolDefinitionFromAgentTool } from "../../src/core/tools/tool-definition-wrapper.ts"; +import { + createToolDefinitionFromAgentTool, + type TypeBoxAgentTool, +} from "../../src/core/tools/tool-definition-wrapper.ts"; import { readCodemodeStore } from "../../src/extensions/codemode/execute.ts"; import { createCodemodeExtension } from "../../src/extensions/codemode/index.ts"; import { @@ -33,7 +36,7 @@ const TINY_PNG_BASE64 = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8DwHwAFBQIAX8jx0gAAAABJRU5ErkJggg=="; const echoSchema = Type.Object({ text: Type.String({ description: "Text to echo" }) }); -const echoTool: AgentTool = { +const echoTool: TypeBoxAgentTool = { name: "echo", label: "Echo", description: "Echo text back.\n\nSecond paragraph.", diff --git a/packages/durable/CHANGELOG.md b/packages/durable/CHANGELOG.md index 009e0367f..e852e8fa8 100644 --- a/packages/durable/CHANGELOG.md +++ b/packages/durable/CHANGELOG.md @@ -4,6 +4,7 @@ ### Breaking Changes +- Tool schemas are JSON Schema typed through `@earendil-works/pi-schema`: `defineTool()` and `ToolRegistration` take any JSON Schema object as `parameters` and `structuredOutputSchema`, and type `args` with pi-schema's `Static` (`ToolArgs`; other schemas give `unknown`). The built-in tools (`read`, `write`, `edit`, `bash`, `powershell`, `codemode`, `tool_search`, the model tools) are built with pi-schema, with the same JSON Schema up to key order. `structuredOutput` is validated with pi-schema, so `invalid_structured_output` messages use its wording. `typebox` is no longer a dependency. Migration: replace `Type.Object({ n: Type.Number() })` from `@earendil-works/pi-ai` with `obj({ n: num() })` from `@earendil-works/pi-schema` (closed by default; `{ open: true }` allows other properties). - `SqliteExecutor` (re-exported from Chord) gains `runCounted(sql, ...params)`, resolving with the rows the statement changed; custom `SqliteDatabase` implementations add it. - `Storage.commit()` stores a group of commits in one call: `commit(batches, receive, context)` resolves with a `CommitResult` (`{ stored, refused? }`) and replaces `commit(writes, adopt, context)`. Each batch is one atomic commit with its own seq, checked against the state the batches before it left. The first batch Durable's own checks refuse (global IDs, document actions, copy sources) is not stored, nor anything after it; the batches before it are, and the call resolves with their number in `stored` and the refusal in `refused`. Any other failure (a driver or I/O error, a failed `COMMIT`) rejects the whole call as an unknown outcome and stores nothing the backend could roll back. `receive(index, seq)` is called once per stored batch, in order, synchronously, once they are durable and before the next call starts; receipts made before a rejection stand. Every batch has at least one write. The optional `publishing(publish, context)` runs each commit's publication inside the storage's own scope (`StoreStorage`: the attached host's batch). Memory, SQLite, JSONL, IndexedDB and `StoreStorage` store a call in one transaction or append, and the storage conformance suite checks the contract. Migration for a custom Storage: diff --git a/packages/durable/README.md b/packages/durable/README.md index 95e9c3085..b9e54a068 100644 --- a/packages/durable/README.md +++ b/packages/durable/README.md @@ -257,15 +257,15 @@ registry.install(createCodingTools({ images: await createNodePhotonImages() })); A model without image input sees a placeholder instead of the image, and `read` says so in a diagnostic. `/images/node` prepares images in a worker thread, so a large photo (a second or two of work) never blocks the event loop; the worker exits after 30 s without images, freeing Photon's WebAssembly memory (`createNodePhotonImages({ worker: false, idleMs })`). Elsewhere, `createPhotonImages()` and `/images/cloudflare` work on the calling thread, and decoding takes the image's full size in WebAssembly memory, which does not shrink again; leave the processor out on memory-constrained hosts such as Cloudflare Workers unless images stay small. -Define your own tool with a TypeBox schema. `defineTool()` types `args` from `parameters`, which the Harness validates before `execute()`. `api.output()` streams running output, which becomes the result when `execute()` returns no `output`: +Define your own tool with a JSON Schema, built here with `@earendil-works/pi-schema`. `defineTool()` types `args` from `parameters` (through pi-schema's `Static`; other JSON Schema gives `unknown`), which the Harness coerces and validates before `execute()`. `api.output()` streams running output, which becomes the result when `execute()` returns no `output`: ```typescript -import { Type } from "@earendil-works/pi-ai"; +import { num, obj, str } from "@earendil-works/pi-schema"; const count = defineTool({ name: "count", description: "Count from 1 to n", - parameters: Type.Object({ n: Type.Number() }), + parameters: obj({ n: num() }), execute: async (args, api) => { for (let i = 1; i <= args.n; i++) api.output(`${i}\n`); return {}; @@ -282,8 +282,8 @@ A result has three channels: `output` for the model, `structuredOutput` for prog const lines = defineTool({ name: "lines", description: "Count the lines of a file", - parameters: Type.Object({ path: Type.String() }), - structuredOutputSchema: Type.Object({ lines: Type.Number() }), + parameters: obj({ path: str() }), + structuredOutputSchema: obj({ lines: num() }), execute: async (args, api, context) => { const text = await readText(api, args.path, context); const count = text.split("\n").length; @@ -312,7 +312,7 @@ A tool calls another tool with `api.executeTool()`, as a code mode script or an const testsPass = defineTool({ name: "tests_pass", description: "Run the test suite and say whether it passed", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api, context) => { const run = await api.executeTool("bash", { command: "npm test" }, context, { progress: false }); // Absent when bash did not finish, for example on a timeout. @@ -722,7 +722,7 @@ const Subagent: Extension = defineExtension({ defineTool({ name: "subagent", description: "Delegate a self-contained task to a subagent and get its answer back.", - parameters: Type.Object({ task: Type.String() }), + parameters: obj({ task: str() }), replay: "safe", // a rerun after a crash finds the same child and submission execute: async (args, api, context) => { const child = await api.commit(async (tx) => { @@ -905,7 +905,7 @@ registerStorageConformance({ describe, expect, it }, "My Storage", async (use) = }); ``` -The package root loads TypeBox, because the tool task validates arguments with pi-ai's `validateToolArguments()`. That costs about 23 MB of peak RSS unbundled, about 4 MB in a tree-shaken bundle. +The package root loads pi-schema, because the tool task validates arguments with pi-ai's `validateToolArguments()`. ## Errors diff --git a/packages/durable/package.json b/packages/durable/package.json index 57ced6bbb..495197c59 100644 --- a/packages/durable/package.json +++ b/packages/durable/package.json @@ -161,10 +161,10 @@ "@earendil-works/chord": "1.4.0", "@earendil-works/pi-ai": "1.4.0", "@earendil-works/pi-codemode": "1.4.0", + "@earendil-works/pi-schema": "1.4.0", "@earendil-works/pi-telemetry": "1.4.0", "@silvia-odwyer/photon": "0.3.3", - "diff": "8.0.4", - "typebox": "1.3.27" + "diff": "8.0.4" }, "devDependencies": { "@cloudflare/vitest-plugin": "1.4.10", diff --git a/packages/durable/src/codemode/codemode.ts b/packages/durable/src/codemode/codemode.ts index 8041004d2..a2e765779 100644 --- a/packages/durable/src/codemode/codemode.ts +++ b/packages/durable/src/codemode/codemode.ts @@ -46,7 +46,7 @@ import type { CodemodeSandboxBase, } from "@earendil-works/pi-codemode/portable"; import { CODEMODE_SOURCE_GRAMMAR, parseCodemodeSource } from "@earendil-works/pi-codemode/source"; -import { type Static, Type } from "typebox"; +import { obj, type Static, str } from "@earendil-works/pi-schema"; import { defineDoc } from "../documents.ts"; import type { ExecutionEnv } from "../env/index.ts"; import { errorMessage } from "../errors.ts"; @@ -210,7 +210,7 @@ export type CodemodeGlobal = { execute(args: unknown, api: ToolExecutionApi, context: Context): Promise; }; -const codemodeSchema = Type.Object({ code: Type.String({ description: "Raw JavaScript source." }) }); +const codemodeSchema = obj({ code: str({ description: "Raw JavaScript source." }) }, { open: true }); export type CodemodeToolInput = Static; diff --git a/packages/durable/src/codemode/models.ts b/packages/durable/src/codemode/models.ts index ca22e96ea..fcdf63b68 100644 --- a/packages/durable/src/codemode/models.ts +++ b/packages/durable/src/codemode/models.ts @@ -17,7 +17,7 @@ import type { ModelTypeMap, Usage, } from "@earendil-works/pi-ai"; -import { Type } from "typebox"; +import { obj, unknown } from "@earendil-works/pi-schema"; import { defineTool } from "../harness/define.ts"; import type { ToolRegistration } from "../harness/types.ts"; @@ -227,7 +227,7 @@ function resolveModel( ); } -const ModelCallParameters = Type.Object({ model: Type.Unknown(), context: Type.Unknown() }); +const ModelCallParameters = obj({ model: unknown(), context: unknown() }, { open: true }); /** The result of a model call as the tool returns it: the whole result for the script, usage counted by the Harness. */ function modelResult( @@ -255,7 +255,7 @@ export function createModelTools(): ToolRegistration[] { name: CLASSIFY_TOOL, description: "models.classify() of code mode scripts.", parameters: ModelCallParameters, - structuredOutputSchema: Type.Unknown(), + structuredOutputSchema: unknown(), callers: ["tools"], execute: async (args, api, context) => { const model = resolveModel("models.classify", "classifier", args.model, api.models); @@ -271,7 +271,7 @@ export function createModelTools(): ToolRegistration[] { name: GENERATE_IMAGES_TOOL, description: "models.generateImages() of code mode scripts.", parameters: ModelCallParameters, - structuredOutputSchema: Type.Unknown(), + structuredOutputSchema: unknown(), callers: ["tools"], execute: async (args, api, context) => { const model = resolveModel("models.generateImages", "image", args.model, api.models); diff --git a/packages/durable/src/harness/define.ts b/packages/durable/src/harness/define.ts index dd4bd2e8d..f5e0df8be 100644 --- a/packages/durable/src/harness/define.ts +++ b/packages/durable/src/harness/define.ts @@ -1,5 +1,4 @@ import type { JsonValue } from "@earendil-works/chord"; -import type { TSchema } from "@earendil-works/pi-ai"; import type { Task } from "../types.ts"; import type { AnyTask, @@ -20,7 +19,7 @@ export function defineExtension( +export function defineTool( tool: ToolRegistration, ): ToolRegistration { return tool; diff --git a/packages/durable/src/harness/tool.ts b/packages/durable/src/harness/tool.ts index 533795a72..dc9e72ca2 100644 --- a/packages/durable/src/harness/tool.ts +++ b/packages/durable/src/harness/tool.ts @@ -11,7 +11,8 @@ import { } from "@earendil-works/chord/output"; import type { ImageContent, TextContent, ToolCall, ToolResultMessage } from "@earendil-works/pi-ai"; import { validateToolArguments } from "@earendil-works/pi-ai/utils/validation"; -import { Compile } from "typebox/compile"; +import { compile, type Validator } from "@earendil-works/pi-schema"; +import { formats } from "@earendil-works/pi-schema/formats"; import { defineDocFamily } from "../documents.ts"; import { AssistantEntry, ToolResultEntry } from "../entries.ts"; import type { ExecutionEnv } from "../env/index.ts"; @@ -770,7 +771,7 @@ async function afterHarnessResult( return { ...rest, ...value, output: bounded.content, diagnostics: [...(rest.diagnostics ?? []), ...harness] }; } -const structuredValidators = new WeakMap>(); +const structuredValidators = new WeakMap(); /** Why a result's `structuredOutput` breaks the tool's contract, or `undefined` when it keeps it. */ function structuredOutputError(tool: ToolRegistration, result: ToolExecutionResult): string | undefined { @@ -786,12 +787,17 @@ function structuredOutputError(tool: ToolRegistration, result: ToolExecutionResu } let validator = structuredValidators.get(schema); if (validator === undefined) { - validator = Compile(schema); + validator = compile(schema, { formats, unresolvedRefs: "reject" }); structuredValidators.set(schema, validator); } - if (validator.Check(value)) return undefined; - const [first] = validator.Errors(value); - const path = first?.instancePath.replace(/^\//, "").replace(/\//g, ".") || "root"; + if (validator.check(value)) return undefined; + const [first] = validator.errors(value); + const path = + first?.path + .slice(1) + .split("/") + .map((segment) => segment.replace(/~1/g, "/").replace(/~0/g, "~")) + .join(".") || "root"; return `Tool ${tool.name} returned structuredOutput that does not match its schema: ${path}: ${first?.message ?? "invalid"}`; } diff --git a/packages/durable/src/harness/types.ts b/packages/durable/src/harness/types.ts index d114bf6cc..c0f03e65d 100644 --- a/packages/durable/src/harness/types.ts +++ b/packages/durable/src/harness/types.ts @@ -8,15 +8,14 @@ import type { Model, Models, ModelThinkingLevel, - Static, Tool, ToolCall, ToolResultMessage, Transport, - TSchema, Usage, UserMessage, } from "@earendil-works/pi-ai"; +import type { Static } from "@earendil-works/pi-schema"; import type { TelemetryContext } from "@earendil-works/pi-telemetry"; import type { ExecutionEnv, ShellOutputSkip, ShellOutputWindow } from "../env/index.ts"; import type { @@ -333,12 +332,22 @@ export interface ToolExecutionApi extend ): Promise; } +/** + * The arguments type of a tool with parameter schema `TParameters`: `Static` for schemas built with + * `@earendil-works/pi-schema`'s builders, `unknown` for other JSON Schema, and `any` for `ToolRegistration`. + */ +export type ToolArgs = IsAny extends true ? any : Static; + +// `0 extends 1 & T` is resolved too early for a `T` constrained to `object`; this form waits for the argument. +type IsAny = boolean extends (T extends never ? true : false) ? true : false; + /** * Executable tool registered in a registry. Only pi-ai `Tool` fields enter the transcript. `args` are typed by - * `parameters`, which the Harness validates them against before `execute()`; `defineTool()` infers both generics. + * `parameters` (any JSON Schema object; see {@link ToolArgs}), which the Harness validates them against before + * `execute()`; `defineTool()` infers both generics. */ export type ToolRegistration< - TParameters extends TSchema = TSchema, + TParameters extends object = object, TDetails extends JsonValue = JsonValue, > = Tool & { /** Whether an interrupted execution may rerun on recovery. Default `unsafe`. */ @@ -350,12 +359,12 @@ export type ToolRegistration< * pure and must not mutate `args`: it runs again when a call is retried before its intent is recorded. Its result is * still validated against `parameters`. */ - prepareArguments?(args: unknown): Static; + prepareArguments?(args: unknown): ToolArgs; /** * Schema of `structuredOutput`, which programs that call the tool receive. Omitted: they receive the result's bounded * output, success or error, as `NestedToolExecutionResult.structuredOutput` describes. */ - readonly structuredOutputSchema?: TSchema; + readonly structuredOutputSchema?: object; /** * Who may call the tool: the model, other tools through `executeTool()`, or both (default). A code mode tool is * `["model"]`, so scripts cannot start scripts; tools behind code mode, such as MCP tools, can be `["tools"]`, so the @@ -373,7 +382,7 @@ export type ToolRegistration< readonly retain?: "head" | "tail"; }; execute( - args: Static, + args: ToolArgs, api: ToolExecutionApi, context: Context, ): Promise>; diff --git a/packages/durable/src/tool-search/tool-search.ts b/packages/durable/src/tool-search/tool-search.ts index 1a4607c17..5d79b007e 100644 --- a/packages/durable/src/tool-search/tool-search.ts +++ b/packages/durable/src/tool-search/tool-search.ts @@ -3,7 +3,7 @@ * and the matches are loaded with `control.addTools`, so the next request offers them. Loading writes the * conversation's `modelTools` (its `add`), so loaded tools stay offered on that branch, also after a compaction. */ -import { Type } from "@earendil-works/pi-ai"; +import { num, obj, opt, str } from "@earendil-works/pi-schema"; import { defineExtension, defineTool } from "../harness/define.ts"; import type { ToolRegistration } from "../harness/types.ts"; import { rankDocuments } from "../search.ts"; @@ -14,12 +14,13 @@ export const DEFAULT_TOOL_SEARCH_LIMIT = 8; /** Fixed, so it does not change while deferred tools are registered, for example as MCP servers connect. */ const TOOL_SEARCH_DESCRIPTION = `# Tool discovery\n\nSearches over deferred tool metadata with BM25 and exposes matching tools for the next model call.\n\nSome of the tools, such as tools of MCP servers, may not have been provided to you upfront, and you should use this tool (\`${TOOL_SEARCH_TOOL_NAME}\`) to search for the required tools. For MCP tool discovery, always use \`${TOOL_SEARCH_TOOL_NAME}\`.`; -const toolSearchSchema = Type.Object({ - query: Type.String({ description: "Search query for deferred tools." }), - limit: Type.Optional( - Type.Number({ description: `Maximum number of tools to return. Defaults to ${DEFAULT_TOOL_SEARCH_LIMIT}.` }), - ), -}); +const toolSearchSchema = obj( + { + query: str({ description: "Search query for deferred tools." }), + limit: opt(num({ description: `Maximum number of tools to return. Defaults to ${DEFAULT_TOOL_SEARCH_LIMIT}.` })), + }, + { open: true }, +); /** `tool_search` result details: the tools it loaded. */ export type ToolSearchToolDetails = { loaded: string[] }; diff --git a/packages/durable/src/tools/bash.ts b/packages/durable/src/tools/bash.ts index 21d1e1d03..211338f16 100644 --- a/packages/durable/src/tools/bash.ts +++ b/packages/durable/src/tools/bash.ts @@ -1,5 +1,5 @@ import type { Context } from "@earendil-works/chord"; -import { type Static, Type } from "typebox"; +import { bool, num, obj, opt, type Static, str } from "@earendil-works/pi-schema"; import { defineTool } from "../harness/define.ts"; import type { ToolExecutionApi, ToolExecutionResult, ToolRegistration } from "../harness/types.ts"; import { DEFAULT_MAX_BYTES, DEFAULT_MAX_LINES } from "../truncate.ts"; @@ -7,23 +7,33 @@ import { requireEnv } from "./env.ts"; const MAX_TIMEOUT_SECONDS = 2_147_483_647 / 1000; -const bashSchema = Type.Object({ - command: Type.String({ description: "Bash command to execute" }), - timeout: Type.Optional(Type.Number({ description: "Timeout in seconds (optional, no default timeout)" })), -}); +// Open objects, as the tools accepted extra arguments when their schemas were TypeBox's. +const bashSchema = obj( + { + command: str({ description: "Bash command to execute" }), + timeout: opt(num({ description: "Timeout in seconds (optional, no default timeout)" })), + }, + { open: true }, +); -const powershellSchema = Type.Object({ - command: Type.String({ description: "PowerShell command to execute" }), - timeout: Type.Optional(Type.Number({ description: "Timeout in seconds (optional, no default timeout)" })), -}); +const powershellSchema = obj( + { + command: str({ description: "PowerShell command to execute" }), + timeout: opt(num({ description: "Timeout in seconds (optional, no default timeout)" })), + }, + { open: true }, +); /** What a program that calls `bash` or `powershell` receives. */ -const shellStructuredOutputSchema = Type.Object({ - output: Type.String({ description: "Combined stdout and stderr: the retained tail, as the model sees it" }), - truncated: Type.Boolean({ description: "Whether earlier output was dropped" }), - fullOutputPath: Type.Optional(Type.String({ description: "File with the complete output, when it was spilled" })), - exitCode: Type.Number(), -}); +const shellStructuredOutputSchema = obj( + { + output: str({ description: "Combined stdout and stderr: the retained tail, as the model sees it" }), + truncated: bool({ description: "Whether earlier output was dropped" }), + fullOutputPath: opt(str({ description: "File with the complete output, when it was spilled" })), + exitCode: num(), + }, + { open: true }, +); export type BashToolInput = Static; export type PowerShellToolInput = Static; diff --git a/packages/durable/src/tools/edit.ts b/packages/durable/src/tools/edit.ts index 802150152..e5b9be73d 100644 --- a/packages/durable/src/tools/edit.ts +++ b/packages/durable/src/tools/edit.ts @@ -1,4 +1,4 @@ -import { type Static, Type } from "typebox"; +import { arr, obj, type Static, str } from "@earendil-works/pi-schema"; import type { FileError } from "../env/index.ts"; import { defineTool } from "../harness/define.ts"; import type { ToolRegistration } from "../harness/types.ts"; @@ -16,21 +16,27 @@ import { requireEnv } from "./env.ts"; import { withFileMutationQueue } from "./file-mutation-queue.ts"; import { resolveToolPath } from "./path-utils.ts"; -const replaceEditSchema = Type.Object({ - oldText: Type.String({ - description: - "Exact text for one targeted replacement. It must be unique in the original file and must not overlap with any other edits[].oldText in the same call.", - }), - newText: Type.String({ description: "Replacement text for this targeted edit." }), -}); +const replaceEditSchema = obj( + { + oldText: str({ + description: + "Exact text for one targeted replacement. It must be unique in the original file and must not overlap with any other edits[].oldText in the same call.", + }), + newText: str({ description: "Replacement text for this targeted edit." }), + }, + { open: true }, +); -const editSchema = Type.Object({ - path: Type.String({ description: "Path to the file to edit (relative or absolute)" }), - edits: Type.Array(replaceEditSchema, { - description: - "One or more targeted replacements. Each edit is matched against the original file, not incrementally. Do not include overlapping or nested edits. If two changes touch the same block or nearby lines, merge them into one edit instead.", - }), -}); +const editSchema = obj( + { + path: str({ description: "Path to the file to edit (relative or absolute)" }), + edits: arr(replaceEditSchema, { + description: + "One or more targeted replacements. Each edit is matched against the original file, not incrementally. Do not include overlapping or nested edits. If two changes touch the same block or nearby lines, merge them into one edit instead.", + }), + }, + { open: true }, +); export type EditToolInput = Static; type LegacyEditToolInput = EditToolInput & { oldText?: unknown; newText?: unknown }; diff --git a/packages/durable/src/tools/read.ts b/packages/durable/src/tools/read.ts index b0312f3b3..de84a2b2f 100644 --- a/packages/durable/src/tools/read.ts +++ b/packages/durable/src/tools/read.ts @@ -1,6 +1,6 @@ import type { Context } from "@earendil-works/chord"; import { characterEnd } from "@earendil-works/chord/output"; -import { type Static, Type } from "typebox"; +import { lit, num, obj, opt, type Static, str, union } from "@earendil-works/pi-schema"; import { type BinaryReader, type FileInfo, @@ -31,11 +31,14 @@ import { } from "./image-processor.ts"; import { resolveReadToolPath } from "./path-utils.ts"; -const readSchema = Type.Object({ - path: Type.String({ description: "Path to the file to read (relative or absolute)" }), - offset: Type.Optional(Type.Number({ description: "Line number to start reading from (1-indexed)" })), - limit: Type.Optional(Type.Number({ description: "Maximum number of lines to read" })), -}); +const readSchema = obj( + { + path: str({ description: "Path to the file to read (relative or absolute)" }), + offset: opt(num({ description: "Line number to start reading from (1-indexed)" })), + limit: opt(num({ description: "Maximum number of lines to read" })), + }, + { open: true }, +); export type ReadToolInput = Static; @@ -43,9 +46,9 @@ export type ReadToolInput = Static; * What programs such as code mode scripts get: the text of a text file, or the image block of an image, which code * mode's `image()` shows. */ -const readStructuredOutputSchema = Type.Union([ - Type.String(), - Type.Object({ type: Type.Literal("image"), data: Type.String(), mimeType: Type.String() }), +const readStructuredOutputSchema = union([ + str(), + obj({ type: lit("image"), data: str(), mimeType: str() }, { open: true }), ]); export type ReadToolDetails = { diff --git a/packages/durable/src/tools/write.ts b/packages/durable/src/tools/write.ts index ee9f724cb..d8c804558 100644 --- a/packages/durable/src/tools/write.ts +++ b/packages/durable/src/tools/write.ts @@ -1,4 +1,4 @@ -import { type Static, Type } from "typebox"; +import { obj, type Static, str } from "@earendil-works/pi-schema"; import { getOrThrow } from "../env/index.ts"; import { defineTool } from "../harness/define.ts"; import type { ToolRegistration } from "../harness/types.ts"; @@ -6,10 +6,13 @@ import { requireEnv } from "./env.ts"; import { withFileMutationQueue } from "./file-mutation-queue.ts"; import { resolveToolPath } from "./path-utils.ts"; -const writeSchema = Type.Object({ - path: Type.String({ description: "Path to the file to write (relative or absolute)" }), - content: Type.String({ description: "Content to write to the file" }), -}); +const writeSchema = obj( + { + path: str({ description: "Path to the file to write (relative or absolute)" }), + content: str({ description: "Content to write to the file" }), + }, + { open: true }, +); export type WriteToolInput = Static; diff --git a/packages/durable/test/codemode-bench.ts b/packages/durable/test/codemode-bench.ts index c9c3c2b17..d79df3da6 100644 --- a/packages/durable/test/codemode-bench.ts +++ b/packages/durable/test/codemode-bench.ts @@ -5,6 +5,7 @@ * * node --conditions=source --experimental-strip-types test/codemode-bench.ts [N,...] */ + import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -12,7 +13,7 @@ import { BACKGROUND_CONTEXT as context } from "@earendil-works/chord/context"; import { createModels } from "@earendil-works/pi-ai/models"; import { fauxAssistantMessage, fauxProvider, fauxText, fauxToolCall } from "@earendil-works/pi-ai/providers/faux"; import { CodemodeSandbox } from "@earendil-works/pi-codemode"; -import { Type } from "typebox"; +import { num, obj, str } from "@earendil-works/pi-schema"; import { createCodemodeTool } from "../src/codemode/codemode.ts"; import { createRegistry, defineExtension, defineTool, Harness } from "../src/index.ts"; import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts"; @@ -22,14 +23,14 @@ type Shape = "sequential" | "parallel"; const echo = defineTool({ name: "echo", description: "echo", - parameters: Type.Object({ i: Type.Number() }), + parameters: obj({ i: num() }), execute: async (args) => ({ output: [{ type: "text", text: String(args.i) }] }), }); const plain = defineTool({ name: "plain", description: "plain", - parameters: Type.Object({ n: Type.Number(), shape: Type.String() }), + parameters: obj({ n: num(), shape: str() }), replay: "safe", execute: async (args, api, callContext) => { if (args.shape === "sequential") { diff --git a/packages/durable/test/codemode-describe.test.ts b/packages/durable/test/codemode-describe.test.ts index 99ced0465..85dfdbe99 100644 --- a/packages/durable/test/codemode-describe.test.ts +++ b/packages/durable/test/codemode-describe.test.ts @@ -1,7 +1,8 @@ -import { type AssistantMessage, fauxAssistantMessage, fauxText, Type } from "@earendil-works/pi-ai"; +import { type AssistantMessage, fauxAssistantMessage, fauxText } from "@earendil-works/pi-ai"; import { CodemodeSandbox } from "@earendil-works/pi-codemode"; import { CODEMODE_SOURCE_GRAMMAR } from "@earendil-works/pi-codemode/source"; import { defineTool, MemoryStorage, type ToolRegistration } from "@earendil-works/pi-durable"; +import { obj, str } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; import { createCodemodeExtension } from "../src/codemode/codemode.ts"; import { type CodemodeNamespace, codemodeDescription, codemodeSection } from "../src/codemode/describe.ts"; @@ -12,7 +13,7 @@ function tool(name: string, description: string, extra: Partial ({ output: [] }), ...extra, }); diff --git a/packages/durable/test/codemode.test.ts b/packages/durable/test/codemode.test.ts index e96b9b773..c12eba6d2 100644 --- a/packages/durable/test/codemode.test.ts +++ b/packages/durable/test/codemode.test.ts @@ -2,7 +2,7 @@ import { mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; import type { Context } from "@earendil-works/chord"; -import { type AssistantMessage, fauxAssistantMessage, fauxText, fauxToolCall, Type } from "@earendil-works/pi-ai"; +import { type AssistantMessage, fauxAssistantMessage, fauxText, fauxToolCall } from "@earendil-works/pi-ai"; import { type CodemodeRun, CodemodeSandbox } from "@earendil-works/pi-codemode"; import { defineTool, @@ -13,6 +13,7 @@ import { ToolTask, type ToolTaskInput, } from "@earendil-works/pi-durable"; +import { bool, num, obj, opt, str } from "@earendil-works/pi-schema"; import { afterEach, describe, expect, it } from "vitest"; import { type CodemodeGlobal, @@ -30,7 +31,7 @@ import { addHooks, addTool } from "./harness-support.ts"; import { ControlledStorage, context } from "./session-support.ts"; import { aborted, type Deferred, deferred, eventually } from "./task-support.ts"; -const Parameters = Type.Object({ text: Type.Optional(Type.String()), id: Type.Optional(Type.Number()) }); +const Parameters = obj({ text: opt(str()), id: opt(num()) }); function tool( name: string, @@ -715,13 +716,13 @@ describe("durable code mode", () => { it("resolves a schema tool's error result that carries its value, and rejects one without with its text", async () => { const chat = codemodeChat(); - const Exit = Type.Object({ output: Type.String(), exitCode: Type.Number() }); + const Exit = obj({ output: str(), exitCode: num() }); addTool( chat.setup.registry, defineTool({ name: "run", description: "run", - parameters: Type.Object({}), + parameters: obj({}), structuredOutputSchema: Exit, // Like bash on a nonzero exit: an error result with its value. execute: async () => ({ @@ -736,7 +737,7 @@ describe("durable code mode", () => { defineTool({ name: "denied", description: "denied", - parameters: Type.Object({}), + parameters: obj({}), structuredOutputSchema: Exit, execute: async () => ({ output: [{ type: "text", text: "permission denied" }], isError: true }), }), @@ -767,7 +768,7 @@ describe("durable code mode", () => { defineTool({ name: "picture", description: "picture", - parameters: Type.Object({ caption: Type.Boolean() }), + parameters: obj({ caption: bool() }), execute: async (args) => ({ output: [ ...(args.caption ? [{ type: "text" as const, text: "a cat" }] : []), @@ -1078,7 +1079,7 @@ describe("durable code mode output and tool search", () => { defineTool({ name: "later", description: "A deferred tool", - parameters: Type.Object({}), + parameters: obj({}), deferred: true, execute: async () => ({ output: [{ type: "text", text: "later ran" }] }), }), diff --git a/packages/durable/test/examples/05-harness.ts b/packages/durable/test/examples/05-harness.ts index ad33c0d0c..e0fd947fb 100644 --- a/packages/durable/test/examples/05-harness.ts +++ b/packages/durable/test/examples/05-harness.ts @@ -1,9 +1,10 @@ // Open a Harness with a registry of extensions. // Run from packages/durable: // node --conditions=source --experimental-strip-types test/examples/05-harness.ts + import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context"; -import { Type } from "@earendil-works/pi-ai"; import { createModels } from "@earendil-works/pi-ai/models"; +import { obj, str } from "@earendil-works/pi-schema"; import { AgentDoc, createRegistry, @@ -23,7 +24,7 @@ const context = BACKGROUND_CONTEXT; const read = defineTool({ name: "read", description: "Read a file", - parameters: Type.Object({ path: Type.String() }), + parameters: obj({ path: str() }), execute: async (args) => ({ output: [{ type: "text", text: `contents of ${args.path}` }] }), }); const Files = defineExtension({ name: "files", tools: [read] }); diff --git a/packages/durable/test/examples/06-configuration.ts b/packages/durable/test/examples/06-configuration.ts index 03184b6d8..fb50401db 100644 --- a/packages/durable/test/examples/06-configuration.ts +++ b/packages/durable/test/examples/06-configuration.ts @@ -1,9 +1,10 @@ // Per-conversation agent choices and Harness-wide settings. // Run from packages/durable: // node --conditions=source --experimental-strip-types test/examples/06-configuration.ts + import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context"; -import { Type } from "@earendil-works/pi-ai"; import { createModels } from "@earendil-works/pi-ai/models"; +import { obj, str } from "@earendil-works/pi-schema"; import { AgentDoc, createRegistry, @@ -25,7 +26,7 @@ function exampleTool(name: string, description: string): AppTool { const tool = defineTool({ name, description, - parameters: Type.Object({ path: Type.String() }), + parameters: obj({ path: str() }), execute: async (args) => ({ output: [{ type: "text", text: `${name} ${args.path}` }] }), }); return { ...tool, snippet: `Use ${name} for files.` }; diff --git a/packages/durable/test/examples/09-registry-reload.ts b/packages/durable/test/examples/09-registry-reload.ts index 4135650ee..e1ee8319e 100644 --- a/packages/durable/test/examples/09-registry-reload.ts +++ b/packages/durable/test/examples/09-registry-reload.ts @@ -1,9 +1,10 @@ // Reload extension code through the registry. // Run from packages/durable: // node --conditions=source --experimental-strip-types test/examples/09-registry-reload.ts + import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context"; -import { Type } from "@earendil-works/pi-ai"; import { createModels } from "@earendil-works/pi-ai/models"; +import { obj, str } from "@earendil-works/pi-schema"; import { createRegistry, defineExtension, defineTool, Harness, MemoryStorage, wrapTool } from "../../src/index.ts"; const context = BACKGROUND_CONTEXT; @@ -12,7 +13,7 @@ function exampleTool(name: string, description: string) { return defineTool({ name, description, - parameters: Type.Object({ path: Type.String() }), + parameters: obj({ path: str() }), execute: async (args) => ({ output: [{ type: "text", text: `${name} ${args.path}` }] }), }); } diff --git a/packages/durable/test/examples/10-extension-state.ts b/packages/durable/test/examples/10-extension-state.ts index f2d518214..bfd6613e7 100644 --- a/packages/durable/test/examples/10-extension-state.ts +++ b/packages/durable/test/examples/10-extension-state.ts @@ -1,10 +1,11 @@ // An extension that keeps its own per-conversation document. // Run from packages/durable: // node --conditions=source --experimental-strip-types test/examples/10-extension-state.ts + import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context"; -import { Type } from "@earendil-works/pi-ai"; import { createModels } from "@earendil-works/pi-ai/models"; import { fauxAssistantMessage, fauxProvider, fauxToolCall } from "@earendil-works/pi-ai/providers/faux"; +import { obj, str } from "@earendil-works/pi-schema"; import { createRegistry, defineDoc, @@ -38,7 +39,7 @@ const Todo = defineExtension({ defineTool({ name: "todo", description: "Add an item to your todo list", - parameters: Type.Object({ item: Type.String() }), + parameters: obj({ item: str() }), execute: async (args, api, callContext) => { await api.commit(async (tx) => { (await tx.doc(Todos, api.conversationId)).items.push(args.item); diff --git a/packages/durable/test/examples/20-late-join.ts b/packages/durable/test/examples/20-late-join.ts index 73b53076b..59eb2d0d1 100644 --- a/packages/durable/test/examples/20-late-join.ts +++ b/packages/durable/test/examples/20-late-join.ts @@ -2,10 +2,11 @@ // view, and then only what changes after that. // Run from packages/durable: // node --conditions=source --experimental-strip-types test/examples/20-late-join.ts + import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context"; -import { Type } from "@earendil-works/pi-ai"; import { createModels } from "@earendil-works/pi-ai/models"; import { fauxAssistantMessage, fauxProvider, fauxText, fauxToolCall } from "@earendil-works/pi-ai/providers/faux"; +import { obj } from "@earendil-works/pi-schema"; import { createRegistry, defineExtension, @@ -27,7 +28,7 @@ registry.install( defineTool({ name: "count", description: "Counts to ten", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api) => { for (let n = 1; n <= 10; n++) { api.output(`${n}\n`); diff --git a/packages/durable/test/examples/21-subagent-foreground.ts b/packages/durable/test/examples/21-subagent-foreground.ts index af4a31823..7f624a018 100644 --- a/packages/durable/test/examples/21-subagent-foreground.ts +++ b/packages/durable/test/examples/21-subagent-foreground.ts @@ -4,12 +4,14 @@ // Uses OpenAI when OPENAI_API_KEY is set, and a scripted faux model otherwise. // Run from packages/durable: // node --conditions=source --experimental-strip-types test/examples/21-subagent-foreground.ts + import type { Context } from "@earendil-works/chord"; import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context"; -import { type AssistantMessage, type FauxResponseStep, Type } from "@earendil-works/pi-ai"; +import type { AssistantMessage, FauxResponseStep } from "@earendil-works/pi-ai"; import { createModels } from "@earendil-works/pi-ai/models"; import { fauxAssistantMessage, fauxProvider, fauxText, fauxToolCall } from "@earendil-works/pi-ai/providers/faux"; import { openaiProvider } from "@earendil-works/pi-ai/providers/openai"; +import { obj, str } from "@earendil-works/pi-schema"; import { AssistantEntry, type ConversationId, @@ -42,7 +44,7 @@ const Subagent: Extension = defineExtension({ defineTool({ name: "subagent", description: "Delegate a self-contained task to a subagent and get its answer back.", - parameters: Type.Object({ task: Type.String({ description: "What the subagent should do" }) }), + parameters: obj({ task: str({ description: "What the subagent should do" }) }), // Safe to rerun after a crash: a rerun finds the child it already created and the submission it already made. replay: "safe", execute: async (args, api, callContext) => { diff --git a/packages/durable/test/examples/22-subagent-background.ts b/packages/durable/test/examples/22-subagent-background.ts index 7aaf64d8b..f01ee89b4 100644 --- a/packages/durable/test/examples/22-subagent-background.ts +++ b/packages/durable/test/examples/22-subagent-background.ts @@ -5,14 +5,16 @@ // Uses OpenAI when OPENAI_API_KEY is set, and a scripted faux model otherwise. // Run from packages/durable: // node --conditions=source --experimental-strip-types test/examples/22-subagent-background.ts + import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context"; -import { type AssistantMessage, type FauxResponseStep, Type } from "@earendil-works/pi-ai"; +import type { AssistantMessage, FauxResponseStep } from "@earendil-works/pi-ai"; import { createModels } from "@earendil-works/pi-ai/models"; import { fauxAssistantMessage, fauxProvider, fauxText, fauxToolCall } from "@earendil-works/pi-ai/providers/faux"; import { openaiProvider } from "@earendil-works/pi-ai/providers/openai"; +import { bool, obj, oneOf, opt, str } from "@earendil-works/pi-schema"; import { AssistantEntry, type ConversationId, @@ -134,11 +136,11 @@ const subagentTool = defineTool({ "Manage persistent subagents that work in the background. Actions: spawn (name, message), send (name, message; " + "followUp: true queues it after the current answer instead of steering), stop (name: aborts its current work), " + "status (name, or all subagents without one). Answers are reported back to you when they arrive.", - parameters: Type.Object({ - action: Type.Union([Type.Literal("spawn"), Type.Literal("send"), Type.Literal("stop"), Type.Literal("status")]), - name: Type.Optional(Type.String()), - message: Type.Optional(Type.String()), - followUp: Type.Optional(Type.Boolean()), + parameters: obj({ + action: oneOf(["spawn", "send", "stop", "status"]), + name: opt(str()), + message: opt(str()), + followUp: opt(bool()), }), // A call interrupted by a crash is not rerun: repeating `stop` could stop newer work. The model sees that the call // was interrupted and can check with `status`. diff --git a/packages/durable/test/examples/26-plan-mode.ts b/packages/durable/test/examples/26-plan-mode.ts index b0bedd056..0eaf5b939 100644 --- a/packages/durable/test/examples/26-plan-mode.ts +++ b/packages/durable/test/examples/26-plan-mode.ts @@ -2,13 +2,14 @@ // extension's own document, and switching back restores the full tool set. // Run from packages/durable: // node --conditions=source --experimental-strip-types test/examples/26-plan-mode.ts + import { mkdtemp, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context"; -import { Type } from "@earendil-works/pi-ai"; import { createModels } from "@earendil-works/pi-ai/models"; import { fauxAssistantMessage, fauxProvider, fauxToolCall } from "@earendil-works/pi-ai/providers/faux"; +import { arr, obj, str } from "@earendil-works/pi-schema"; import { NodeExecutionEnv } from "../../src/env/node.ts"; import { createRegistry, @@ -39,7 +40,7 @@ const PlanDoc = defineDoc<{ steps: string[] }>({ const submitPlan = defineTool({ name: "submit_plan", description: "Submit the plan as a list of steps.", - parameters: Type.Object({ steps: Type.Array(Type.String()) }), + parameters: obj({ steps: arr(str()) }), execute: async (args, api, callContext) => { await api.commit(async (tx) => { (await tx.doc(PlanDoc, api.conversationId)).steps = args.steps; diff --git a/packages/durable/test/examples/30-reload-and-restart.ts b/packages/durable/test/examples/30-reload-and-restart.ts index 6813a7659..4918cea2f 100644 --- a/packages/durable/test/examples/30-reload-and-restart.ts +++ b/packages/durable/test/examples/30-reload-and-restart.ts @@ -2,13 +2,15 @@ // with; stored choices are names, so they survive a restart and bind to whatever code the new process installs. // Run from packages/durable: // node --conditions=source --experimental-strip-types test/examples/30-reload-and-restart.ts + import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context"; -import { type ToolResultMessage, Type } from "@earendil-works/pi-ai"; +import type { ToolResultMessage } from "@earendil-works/pi-ai"; import { createModels } from "@earendil-works/pi-ai/models"; import { fauxAssistantMessage, fauxProvider, fauxToolCall } from "@earendil-works/pi-ai/providers/faux"; +import { obj } from "@earendil-works/pi-schema"; import { createRegistry, defineExtension, defineTool, Harness, ToolResultEntry } from "../../src/index.ts"; import { openNodeSqliteStorage } from "../../src/storage/sqlite/node.ts"; @@ -24,7 +26,7 @@ function loadVersioned(version: string) { defineTool({ name: "version", description: "Report the tool's code version", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => { started(); await gate; diff --git a/packages/durable/test/examples/32-tests-after-edits.ts b/packages/durable/test/examples/32-tests-after-edits.ts index 32292c521..240bfec31 100644 --- a/packages/durable/test/examples/32-tests-after-edits.ts +++ b/packages/durable/test/examples/32-tests-after-edits.ts @@ -2,10 +2,11 @@ // tests, a failure starts a fix run, and input sent meanwhile waits for them too. // Run from packages/durable: // node --conditions=source --experimental-strip-types test/examples/32-tests-after-edits.ts + import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context"; -import { Type } from "@earendil-works/pi-ai"; import { createModels } from "@earendil-works/pi-ai/models"; import { fauxAssistantMessage, fauxProvider, fauxToolCall } from "@earendil-works/pi-ai/providers/faux"; +import { obj, str } from "@earendil-works/pi-schema"; import { createRegistry, defineExtension, @@ -34,7 +35,7 @@ const runTests = () => { const edit = defineTool({ name: "edit", description: "Replace a file's content", - parameters: Type.Object({ path: Type.String(), content: Type.String() }), + parameters: obj({ path: str(), content: str() }), execute: async (args) => { files.set(args.path, args.content); return { output: [{ type: "text", text: `wrote ${args.path}` }] }; diff --git a/packages/durable/test/examples/33-turn-limit-and-continue.ts b/packages/durable/test/examples/33-turn-limit-and-continue.ts index cac9d315c..817bd1fcb 100644 --- a/packages/durable/test/examples/33-turn-limit-and-continue.ts +++ b/packages/durable/test/examples/33-turn-limit-and-continue.ts @@ -2,10 +2,11 @@ // message, with a fresh budget. A host learns that a run stopped from `pi.live.runFailure`. // Run from packages/durable: // node --conditions=source --experimental-strip-types test/examples/33-turn-limit-and-continue.ts + import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context"; -import { Type } from "@earendil-works/pi-ai"; import { createModels } from "@earendil-works/pi-ai/models"; import { fauxAssistantMessage, fauxProvider, fauxToolCall } from "@earendil-works/pi-ai/providers/faux"; +import { num, obj } from "@earendil-works/pi-schema"; import { ConversationBusy, createRegistry, @@ -21,7 +22,7 @@ const context = BACKGROUND_CONTEXT; const step = defineTool({ name: "step", description: "Do one step of the work", - parameters: Type.Object({ n: Type.Number() }), + parameters: obj({ n: num() }), execute: async (args) => ({ output: [{ type: "text", text: `step ${args.n} done` }] }), }); const call = (n: number) => fauxAssistantMessage(fauxToolCall("step", { n }), { stopReason: "toolUse" }); diff --git a/packages/durable/test/examples/34-submit-in-commit.ts b/packages/durable/test/examples/34-submit-in-commit.ts index 291305cc6..5cdfa918f 100644 --- a/packages/durable/test/examples/34-submit-in-commit.ts +++ b/packages/durable/test/examples/34-submit-in-commit.ts @@ -3,10 +3,11 @@ // state; `whenIdle: "reject"` refuses it when no run is working. // Run from packages/durable: // node --conditions=source --experimental-strip-types test/examples/34-submit-in-commit.ts + import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context"; -import { Type } from "@earendil-works/pi-ai"; import { createModels } from "@earendil-works/pi-ai/models"; import { fauxAssistantMessage, fauxProvider, fauxToolCall } from "@earendil-works/pi-ai/providers/faux"; +import { obj } from "@earendil-works/pi-schema"; import { ConversationIdle, createRegistry, @@ -38,7 +39,7 @@ const working = new Promise((resolve) => { const build = defineTool({ name: "build", description: "Build the project", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => { await working; return { output: [{ type: "text", text: "built" }] }; diff --git a/packages/durable/test/harness-after-run.test.ts b/packages/durable/test/harness-after-run.test.ts index 373922122..7201fe6af 100644 --- a/packages/durable/test/harness-after-run.test.ts +++ b/packages/durable/test/harness-after-run.test.ts @@ -2,7 +2,7 @@ import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import type { Draft, JsonValue } from "@earendil-works/chord"; -import { fauxAssistantMessage, fauxToolCall, Type } from "@earendil-works/pi-ai"; +import { fauxAssistantMessage, fauxToolCall } from "@earendil-works/pi-ai"; import { CompactionTask, ConversationBusy, @@ -25,6 +25,7 @@ import { type TaskId, type Tx, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { afterEach, describe, expect, it } from "vitest"; import { endRun } from "../src/harness/live.ts"; import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts"; @@ -132,7 +133,7 @@ function install(setup: ChatSetup, state: Probe, afterRun?: GenerationHooks["aft return Tests; } -const Parameters = Type.Object({}); +const Parameters = obj({}); const tools = defineExtension({ name: "tools", tools: [ diff --git a/packages/durable/test/harness-callers.test.ts b/packages/durable/test/harness-callers.test.ts index ff9a8311e..bbf4e7db9 100644 --- a/packages/durable/test/harness-callers.test.ts +++ b/packages/durable/test/harness-callers.test.ts @@ -1,4 +1,4 @@ -import { type AssistantMessage, fauxAssistantMessage, fauxText, fauxToolCall, Type } from "@earendil-works/pi-ai"; +import { type AssistantMessage, fauxAssistantMessage, fauxText, fauxToolCall } from "@earendil-works/pi-ai"; import { AgentDoc, type Conversation, @@ -7,6 +7,7 @@ import { type NestedToolExecutionResult, type ToolRegistration, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; import { type ChatSetup, chatSetup, openChat } from "./chat-support.ts"; import { addTool } from "./harness-support.ts"; @@ -16,7 +17,7 @@ function tool(name: string, extra: Partial = {}): ToolRegistra return defineTool({ name, description: name, - parameters: Type.Object({}), + parameters: obj({}), execute: async () => ({ output: [{ type: "text", text: `${name} ran` }] }), ...extra, }); @@ -34,7 +35,7 @@ function prober(names: readonly string[]) { const registration = defineTool({ name: "probe", description: "probe", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api, callContext) => { for (const name of names) received.set(name, await api.executeTool(name, {}, callContext)); return {}; diff --git a/packages/durable/test/harness-compaction.test.ts b/packages/durable/test/harness-compaction.test.ts index b60eb2dae..1be32ae35 100644 --- a/packages/durable/test/harness-compaction.test.ts +++ b/packages/durable/test/harness-compaction.test.ts @@ -9,7 +9,6 @@ import { type Message, type SimpleStreamOptions, type TranscriptContext, - Type, type UserMessage, } from "@earendil-works/pi-ai"; import { @@ -33,6 +32,7 @@ import { type TaskId, UsageDoc, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { afterEach, describe, expect, it } from "vitest"; import { selectCut, serializeConversation } from "../src/harness/compaction.ts"; import { orderToolResults } from "../src/harness/context.ts"; @@ -471,7 +471,7 @@ describe("manual compaction", () => { defineTool({ name: "wait", description: "wait", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => ({ output: [{ type: "text", text: "waited" }] }), }), ); @@ -886,7 +886,7 @@ describe("compaction outcomes", () => { defineTool({ name: "read", description: "read", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => ({ output: [] }), }), ); @@ -1343,7 +1343,7 @@ describe("compaction estimates and interactions", () => { defineTool({ name: "slow", description: "slow", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => { toolReached.resolve(); await toolGate.promise; diff --git a/packages/durable/test/harness-continue.test.ts b/packages/durable/test/harness-continue.test.ts index 8b609b8eb..627e6218f 100644 --- a/packages/durable/test/harness-continue.test.ts +++ b/packages/durable/test/harness-continue.test.ts @@ -1,4 +1,5 @@ // `submit({ type: "continue" })`: a run without a new user message (#11043). + import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -10,7 +11,6 @@ import { fauxText, fauxToolCall, type Message, - Type, } from "@earendil-works/pi-ai"; import { AssistantEntry, @@ -26,6 +26,7 @@ import { type Storage, UserEntry, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { InMemoryTelemetryContext } from "@earendil-works/pi-telemetry"; import { afterEach, describe, expect, it } from "vitest"; import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts"; @@ -125,7 +126,7 @@ describe("continue", () => { defineTool({ name: "wait", description: "Waits until aborted", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, _api, callContext) => { started.resolve(); return aborted(callContext.abortSignal!); @@ -312,7 +313,7 @@ describe("continue", () => { defineTool({ name: "delegate", description: "Delegates", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api, callContext) => { const child = await api.commit(async (tx) => { const created = await tx.createConversation({ ownership: { kind: "task", taskId: api.taskId } }); diff --git a/packages/durable/test/harness-deferred-progress.test.ts b/packages/durable/test/harness-deferred-progress.test.ts index 9506ab931..07116cc08 100644 --- a/packages/durable/test/harness-deferred-progress.test.ts +++ b/packages/durable/test/harness-deferred-progress.test.ts @@ -2,8 +2,9 @@ // the next commit that reads or writes `pi.live` or within `settings.progress.persistWithinMs`. Crashes are simulated // by failing the Harness: from that moment nothing more reaches Storage, as when the process dies; the reopened Harness // resumes from what was stored. + import { applyImmutable } from "@earendil-works/chord/delta"; -import { fauxAssistantMessage, fauxText, fauxToolCall, Type } from "@earendil-works/pi-ai"; +import { fauxAssistantMessage, fauxText, fauxToolCall } from "@earendil-works/pi-ai"; import { AssistantEntry, type ConversationId, @@ -15,6 +16,7 @@ import { type LiveState, ToolResultEntry, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; import type { PartialMessage } from "../src/harness/partial-changes.ts"; import type { SessionImpl } from "../src/session/session.ts"; @@ -257,7 +259,7 @@ describe("tool output", () => { const registration = defineTool({ name: "work", description: "work", - parameters: Type.Object({}), + parameters: obj({}), replay, execute: async (_args, api, callContext) => { const run = ++state.runs; @@ -306,7 +308,7 @@ describe("tool output", () => { defineTool({ name: "work", description: "work", - parameters: Type.Object({}), + parameters: obj({}), replay: "safe", execute: async (_args, api, callContext) => { runs++; diff --git a/packages/durable/test/harness-host-inputs.test.ts b/packages/durable/test/harness-host-inputs.test.ts index 4719cea94..4922cd0d4 100644 --- a/packages/durable/test/harness-host-inputs.test.ts +++ b/packages/durable/test/harness-host-inputs.test.ts @@ -6,7 +6,6 @@ import { fauxToolCall, type Models, type SimpleStreamOptions, - Type, } from "@earendil-works/pi-ai"; import { type Conversation, @@ -19,6 +18,7 @@ import { MemoryStorage, type Registry, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; import { allEntries, textOf } from "./chat-support.ts"; import { context } from "./session-support.ts"; @@ -86,7 +86,7 @@ describe("HarnessOptions.models and settings as functions", () => { defineTool({ name: "probe", description: "Records api.models before and after the host swaps it", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api) => { seen.push(api.models); current = second; diff --git a/packages/durable/test/harness-inbox.test.ts b/packages/durable/test/harness-inbox.test.ts index 0062b197a..a4cb91837 100644 --- a/packages/durable/test/harness-inbox.test.ts +++ b/packages/durable/test/harness-inbox.test.ts @@ -9,7 +9,6 @@ import { fauxText, fauxToolCall, type ToolResultMessage, - Type, type Usage, } from "@earendil-works/pi-ai"; import { @@ -29,6 +28,7 @@ import { ToolTask, UsageDoc, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { afterEach, describe, expect, it } from "vitest"; import { recordUsage } from "../src/harness/usage.ts"; import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts"; @@ -67,7 +67,7 @@ function holdTool(setup: ChatSetup, gate: Deferred, result: ToolExecutionResult defineTool({ name: "hold", description: "Waits for the test", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => { await gate.promise; return result; @@ -474,7 +474,7 @@ describe("inbox", () => { defineTool({ name: "hold", description: "Hands off", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => ({ output: [], control: { handoff: `h${++handoffs}` } }), }), ); @@ -796,7 +796,7 @@ describe("inbox", () => { defineTool({ name: "later", description: "Finishes first", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => { await secondGate.promise; return { output: [], control: { handoff: "two" } }; diff --git a/packages/durable/test/harness-input-entries.test.ts b/packages/durable/test/harness-input-entries.test.ts index f0c44b9fb..ce6baa468 100644 --- a/packages/durable/test/harness-input-entries.test.ts +++ b/packages/durable/test/harness-input-entries.test.ts @@ -1,4 +1,5 @@ // Inputs placed as application entries, and data-only writes appended at once (#10407). + import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -9,7 +10,6 @@ import { fauxText, fauxToolCall, type Context as ModelContext, - Type, } from "@earendil-works/pi-ai"; import { type Conversation, @@ -23,6 +23,7 @@ import { MemoryStorage, type SubmissionDraft, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts"; import { allEntries, type ChatSetup, chatSetup, openChat, textOf, waitFor } from "./chat-support.ts"; @@ -54,7 +55,7 @@ function holdTool(setup: ChatSetup, gate: Deferred): void { defineTool({ name: "hold", description: "Waits for the test", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => { await gate.promise; return { output: [{ type: "text", text: "held" }] }; diff --git a/packages/durable/test/harness-lifecycle.test.ts b/packages/durable/test/harness-lifecycle.test.ts index e41cc259d..bf044120b 100644 --- a/packages/durable/test/harness-lifecycle.test.ts +++ b/packages/durable/test/harness-lifecycle.test.ts @@ -3,7 +3,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import type { Context, JsonValue } from "@earendil-works/chord"; import { withAbortSignal } from "@earendil-works/chord/context"; -import { createModels, fauxAssistantMessage, fauxText, fauxToolCall, Type } from "@earendil-works/pi-ai"; +import { createModels, fauxAssistantMessage, fauxText, fauxToolCall } from "@earendil-works/pi-ai"; import { type Conversation, type ConversationId, @@ -17,6 +17,7 @@ import { type Storage, type TaskId, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; import type { SessionImpl } from "../src/session/session.ts"; import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts"; @@ -168,7 +169,7 @@ describe("Harness close", () => { defineTool({ name: "wait", description: "wait", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => { if (where === "tool") await stubborn(); return { output: [] }; diff --git a/packages/durable/test/harness-live-deltas.test.ts b/packages/durable/test/harness-live-deltas.test.ts index 14d3dd67b..93ac8187c 100644 --- a/packages/durable/test/harness-live-deltas.test.ts +++ b/packages/durable/test/harness-live-deltas.test.ts @@ -1,5 +1,5 @@ import type { Op } from "@earendil-works/chord/delta"; -import { fauxAssistantMessage, fauxText, fauxToolCall, Type } from "@earendil-works/pi-ai"; +import { fauxAssistantMessage, fauxText, fauxToolCall } from "@earendil-works/pi-ai"; import { type CommitPublication, defineTool, @@ -10,6 +10,7 @@ import { type ToolExecutionApi, type ToolRegistration, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; import { chatSetup, openChat, waitFor } from "./chat-support.ts"; import { addTool } from "./harness-support.ts"; @@ -30,7 +31,7 @@ async function drive(outputLimits: ToolRegistration["outputLimits"] = {}) { defineTool({ name: "drive", description: "Driven by the test", - parameters: Type.Object({}), + parameters: obj({}), outputLimits, execute: async (_args, api, callContext) => { for (;;) { @@ -104,7 +105,7 @@ describe("pi.live deltas", () => { defineTool({ name: "noop", description: "noop", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => ({ output: [] }), }), ); @@ -317,7 +318,7 @@ describe("pi.live deltas", () => { defineTool({ name, description: name, - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api) => { api.output(`${name} output\n`); await new Promise((resolve) => setTimeout(resolve, 150)); @@ -368,7 +369,7 @@ describe("pi.live deltas", () => { defineTool({ name: "bad", description: "bad", - parameters: Type.Object({}), + parameters: obj({}), // Not strict JSON: the result commit throws and the scheduler faults the task. execute: async () => ({ output: [], details: { fn: (() => 1) as never } }), }), @@ -412,7 +413,7 @@ describe("pi.live deltas", () => { defineTool({ name: "noop", description: "noop", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => ({ output: [] }), }), ); diff --git a/packages/durable/test/harness-nested-tools-restart.test.ts b/packages/durable/test/harness-nested-tools-restart.test.ts index 0eb29ff76..8d80aa903 100644 --- a/packages/durable/test/harness-nested-tools-restart.test.ts +++ b/packages/durable/test/harness-nested-tools-restart.test.ts @@ -8,7 +8,6 @@ import { fauxText, fauxToolCall, type ToolResultMessage, - Type, } from "@earendil-works/pi-ai"; import { defineExtension, @@ -23,6 +22,7 @@ import { ToolTask, type ToolTaskInput, } from "@earendil-works/pi-durable"; +import { obj, opt, str } from "@earendil-works/pi-schema"; import { afterEach, describe, expect, it } from "vitest"; import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts"; import { allEntries, type ChatSetup, chatSetup, openChat } from "./chat-support.ts"; @@ -31,7 +31,7 @@ import { nestedResultOf, nestedResults, viewHistory } from "./live-support.ts"; import { context } from "./session-support.ts"; import { aborted, deferred, eventually } from "./task-support.ts"; -const Parameters = Type.Object({ text: Type.Optional(Type.String()) }); +const Parameters = obj({ text: opt(str()) }); function tool( name: string, diff --git a/packages/durable/test/harness-nested-tools.test.ts b/packages/durable/test/harness-nested-tools.test.ts index 31325b395..66de32766 100644 --- a/packages/durable/test/harness-nested-tools.test.ts +++ b/packages/durable/test/harness-nested-tools.test.ts @@ -10,7 +10,6 @@ import { fauxText, fauxToolCall, type ToolResultMessage, - Type, type Usage, } from "@earendil-works/pi-ai"; import { @@ -33,6 +32,7 @@ import { type ToolTaskResult, UsageDoc, } from "@earendil-works/pi-durable"; +import { obj, opt, str } from "@earendil-works/pi-schema"; import { afterEach, describe, expect, it } from "vitest"; import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts"; import { commitWrites } from "../src/testing/commit-writes.ts"; @@ -42,7 +42,7 @@ import { endedAt, nestedResultOf, nestedResults, slotsOf, startOrder, viewHistor import { context, onValue } from "./session-support.ts"; import { aborted, deferred, eventually } from "./task-support.ts"; -const Parameters = Type.Object({ text: Type.Optional(Type.String()) }); +const Parameters = obj({ text: opt(str()) }); type Tool = ToolRegistration; function tool(name: string, execute: Tool["execute"], extra: Partial = {}): Tool { @@ -329,7 +329,7 @@ describe("nested tool calls", () => { defineTool({ name: "pair", description: "pair", - parameters: Type.Object({ a: Type.String(), b: Type.String() }), + parameters: obj({ a: str(), b: str() }), execute: async (args) => { runs++; return { output: [{ type: "text", text: `${args.a}${args.b}` }] }; diff --git a/packages/durable/test/harness-ownership.test.ts b/packages/durable/test/harness-ownership.test.ts index 6ba3ad8d9..d0251e24c 100644 --- a/packages/durable/test/harness-ownership.test.ts +++ b/packages/durable/test/harness-ownership.test.ts @@ -1,7 +1,7 @@ import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { type AssistantMessage, fauxAssistantMessage, fauxText, fauxToolCall, Type } from "@earendil-works/pi-ai"; +import { type AssistantMessage, fauxAssistantMessage, fauxText, fauxToolCall } from "@earendil-works/pi-ai"; import { type Conversation, type ConversationHandle, @@ -20,6 +20,7 @@ import { type Submission, type TaskId, } from "@earendil-works/pi-durable"; +import { obj, str } from "@earendil-works/pi-schema"; import { afterEach, describe, expect, it } from "vitest"; import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts"; import { chatSetup, openChat } from "./chat-support.ts"; @@ -1019,7 +1020,7 @@ describe("owned conversations from tools and supervisors", () => { defineTool({ name: "delegate", description: "Delegates", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api, callContext) => { const child = await api.commit(async (tx) => { const created = await tx.createConversation({ ownership: { kind: "task", taskId: api.taskId } }); @@ -1074,7 +1075,7 @@ describe("owned conversations from tools and supervisors", () => { defineTool({ name: "delegate", description: "Delegates late", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api, callContext) => { const child = await api.commit(async (tx) => { const created = await tx.createConversation({ ownership: { kind: "task", taskId: api.taskId } }); @@ -1159,7 +1160,7 @@ describe("owned conversations from tools and supervisors", () => { defineTool({ name: "delegate", description: "Delegates", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api, callContext) => { child = await api.commit(async (tx) => { const created = await tx.createConversation({ ownership: { kind: "task", taskId: api.taskId } }); @@ -1208,7 +1209,7 @@ describe("owned conversations from tools and supervisors", () => { defineTool({ name: "delegate", description: "Delegates and cancels", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api, callContext) => { const child = await api.commit(async (tx) => { const created = await tx.createConversation({ ownership: { kind: "task", taskId: api.taskId } }); @@ -1247,7 +1248,7 @@ describe("owned conversations from tools and supervisors", () => { defineTool({ name: "spawn", description: "Starts owned work, then throws or hangs", - parameters: Type.Object({ mode: Type.String() }), + parameters: obj({ mode: str() }), execute: async (args, api, callContext) => { const child = await api.commit(async (tx) => { const created = await tx.createConversation({ ownership: { kind: "task", taskId: api.taskId } }); @@ -1305,7 +1306,7 @@ describe("owned conversations from tools and supervisors", () => { defineTool({ name: "subagent", description: "Delegates", - parameters: Type.Object({}), + parameters: obj({}), replay: "safe", execute: async (_args, api, callContext) => { const child = await api.commit(async (tx) => { diff --git a/packages/durable/test/harness-prompt.test.ts b/packages/durable/test/harness-prompt.test.ts index 40ef2e09e..39affbc58 100644 --- a/packages/durable/test/harness-prompt.test.ts +++ b/packages/durable/test/harness-prompt.test.ts @@ -1,4 +1,4 @@ -import { type SystemMessage, type Tool, Type } from "@earendil-works/pi-ai"; +import type { SystemMessage, Tool } from "@earendil-works/pi-ai"; import { getCurrentTools, toToolDeclaration } from "@earendil-works/pi-ai/utils/transcript"; import { type Conversation, @@ -12,6 +12,7 @@ import { type ToolRegistration, wrapSection, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; import { resolveAgent, resolveSettings } from "../src/harness/agent.ts"; import { planSystemEntries, renderSections, replaySections } from "../src/harness/prompt.ts"; @@ -207,7 +208,7 @@ describe("tool loadout preparation", () => { const declaration = (name: string, description = name): Tool => ({ name, description, - parameters: Type.Object({}), + parameters: obj({}), }); /** Plan tools only, append the plan, check that replay offers `tools` in order, and return each message's changes. */ diff --git a/packages/durable/test/harness-registry.test.ts b/packages/durable/test/harness-registry.test.ts index af852b0c1..b51002576 100644 --- a/packages/durable/test/harness-registry.test.ts +++ b/packages/durable/test/harness-registry.test.ts @@ -1,4 +1,3 @@ -import { Type } from "@earendil-works/pi-ai"; import { type AgentState, CompactionTask, @@ -15,6 +14,7 @@ import { wrapSection, wrapTool, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; import { agentHooks, resolveAgent, resolveSettings } from "../src/harness/agent.ts"; import { context } from "./session-support.ts"; @@ -25,7 +25,7 @@ function tool(name: string, extra: Partial = {}): AppTool { return { name, description: `${name} tool`, - parameters: Type.Object({}), + parameters: obj({}), execute: async () => ({ output: [] }), ...extra, }; diff --git a/packages/durable/test/harness-reports.test.ts b/packages/durable/test/harness-reports.test.ts index 623fafcdb..97a7acf51 100644 --- a/packages/durable/test/harness-reports.test.ts +++ b/packages/durable/test/harness-reports.test.ts @@ -1,4 +1,4 @@ -import { createModels, fauxAssistantMessage, Type } from "@earendil-works/pi-ai"; +import { createModels, fauxAssistantMessage } from "@earendil-works/pi-ai"; import { type AnyTask, createRegistry, @@ -14,6 +14,7 @@ import { section, wrapTool, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; import { chatSetup } from "./chat-support.ts"; import { ControlledStorage, context } from "./session-support.ts"; @@ -189,7 +190,7 @@ describe("reports", () => { const echo = defineTool({ name: "echo", description: "echo", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => ({}), }); registry.install(defineExtension({ name: "tools", tools: [echo] })); diff --git a/packages/durable/test/harness-restart-failure.test.ts b/packages/durable/test/harness-restart-failure.test.ts index 3ae60d042..c8ece30c7 100644 --- a/packages/durable/test/harness-restart-failure.test.ts +++ b/packages/durable/test/harness-restart-failure.test.ts @@ -1,6 +1,7 @@ // What a crash (a host that dies, is evicted, or is reset) does to work that was running: a generation's interrupted // request is a failed attempt under the retry policy, and every other task ends visibly once it was cut off // `settings.restartLimit` times in a row (#11139). + import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -13,7 +14,6 @@ import { type Message, type Models, type ToolResultMessage, - Type, } from "@earendil-works/pi-ai"; import { defineTool, @@ -27,6 +27,7 @@ import { type SubmissionId, type TaskId, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { afterEach, describe, expect, it } from "vitest"; import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts"; import { commitWrites } from "../src/testing/commit-writes.ts"; @@ -282,7 +283,7 @@ describe("a generation interrupted while the model answers (#11139)", () => { defineTool({ name: "hold", description: "Waits", - parameters: Type.Object({}), + parameters: obj({}), replay: "safe", execute: async () => { started.resolve(); @@ -375,7 +376,7 @@ describe("other tasks cut off restartLimit times in a row (#11139)", () => { defineTool({ name: "heavy", description: "Never finishes", - parameters: Type.Object({}), + parameters: obj({}), replay: "safe", execute: async (_args, api, ctx) => { runs++; @@ -751,7 +752,7 @@ describe("the restart limit on the normal path (#11139)", () => { defineTool({ name: "t", description: "t", - parameters: Type.Object({}), + parameters: obj({}), replay: "safe", execute: async () => ({}), }), diff --git a/packages/durable/test/harness-run-failure.test.ts b/packages/durable/test/harness-run-failure.test.ts index ded8c8ebc..638a4715b 100644 --- a/packages/durable/test/harness-run-failure.test.ts +++ b/packages/durable/test/harness-run-failure.test.ts @@ -1,5 +1,6 @@ // `pi.live.runFailure`: why the newest run ended without an answer, as state a renderer shows (#11327). -import { type FauxResponseStep, fauxAssistantMessage, fauxText, fauxToolCall, Type } from "@earendil-works/pi-ai"; + +import { type FauxResponseStep, fauxAssistantMessage, fauxText, fauxToolCall } from "@earendil-works/pi-ai"; import { type AfterFailedRun, type Conversation, @@ -10,6 +11,7 @@ import { MemoryStorage, type RunFailure, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; import { chatSetup, openChat, unanswered } from "./chat-support.ts"; import { addTool } from "./harness-support.ts"; @@ -104,7 +106,7 @@ describe("pi.live.runFailure", () => { defineTool({ name: "hold", description: "Waits", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => { started.resolve(); await gate.promise; diff --git a/packages/durable/test/harness-structured-output.test.ts b/packages/durable/test/harness-structured-output.test.ts index 829932323..5ef6f75ba 100644 --- a/packages/durable/test/harness-structured-output.test.ts +++ b/packages/durable/test/harness-structured-output.test.ts @@ -8,7 +8,6 @@ import { fauxText, fauxToolCall, type ToolResultMessage, - Type, } from "@earendil-works/pi-ai"; import { defineTool, @@ -21,6 +20,7 @@ import { ToolResultEntry, ToolTask, } from "@earendil-works/pi-durable"; +import { nul, num, obj } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; import { NodeExecutionEnv } from "../src/env/node.ts"; import { createBashTool } from "../src/tools/index.ts"; @@ -28,14 +28,14 @@ import { allEntries, type ChatSetup, chatSetup, openChat } from "./chat-support. import { addHooks, addTool } from "./harness-support.ts"; import { context } from "./session-support.ts"; -const Count = Type.Object({ count: Type.Number() }); +const Count = obj({ count: num() }); /** A tool that returns `result`, with `extra` such as a schema. */ function returning(name: string, result: ToolExecutionResult, extra: Partial = {}): ToolRegistration { return defineTool({ name, description: name, - parameters: Type.Object({}), + parameters: obj({}), execute: async () => result, ...extra, }); @@ -55,7 +55,7 @@ async function nested(setup: ChatSetup, name: string): Promise { received = await api.executeTool(name, {}, callContext); return {}; @@ -132,17 +132,14 @@ describe("structured output", () => { it("stores exactly what callers and events get, without the model's output, and keeps a stored null", async () => { const setup = chatSetup(); addTool(setup.registry, returning("plain", { output: [{ type: "text", text: "a" }] })); - addTool( - setup.registry, - returning("nothing", { structuredOutput: null }, { structuredOutputSchema: Type.Null() }), - ); + addTool(setup.registry, returning("nothing", { structuredOutput: null }, { structuredOutputSchema: nul() })); const received: NestedToolExecutionResult[] = []; addTool( setup.registry, defineTool({ name: "probe", description: "probe", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api, callContext) => { received.push(await api.executeTool("plain", {}, callContext)); received.push(await api.executeTool("nothing", {}, callContext)); @@ -272,7 +269,7 @@ describe("structured output", () => { defineTool({ name: "throws", description: "throws", - parameters: Type.Object({}), + parameters: obj({}), structuredOutputSchema: Count, execute: async () => { throw new Error("permission denied"); @@ -344,7 +341,7 @@ describe("structured output", () => { defineTool({ name: "probe", description: "probe", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api, callContext) => { plain = await api.executeTool("plain", {}, callContext); counter = await api.executeTool("counter", {}, callContext); @@ -375,7 +372,7 @@ describe("structured output", () => { defineTool({ name: "probe", description: "probe", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api, callContext) => { received.push(await api.executeTool("bash", { command: "printf ok" }, callContext)); received.push(await api.executeTool("bash", { command: "printf bad; exit 4" }, callContext)); diff --git a/packages/durable/test/harness-structured.test.ts b/packages/durable/test/harness-structured.test.ts index 78d4e9e76..26c7d7fb6 100644 --- a/packages/durable/test/harness-structured.test.ts +++ b/packages/durable/test/harness-structured.test.ts @@ -8,7 +8,6 @@ import { fauxToolCall, type Models, type ToolResultMessage, - Type, } from "@earendil-works/pi-ai"; import { type Conversation, @@ -35,6 +34,7 @@ import { ToolResultEntry, type Tx, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { afterEach, describe, expect, it } from "vitest"; import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts"; import { allEntries, chatSetup, openChat, waitFor } from "./chat-support.ts"; @@ -1192,7 +1192,7 @@ function blockingTool(name: string): { started: Promise; registration: Too registration: defineTool({ name, description: `The ${name} tool`, - parameters: Type.Object({}), + parameters: obj({}), executionMode: "sequential", execute: async (_args, api, callContext) => { api.output("partial"); @@ -1218,7 +1218,7 @@ describe("tool rounds", () => { defineTool({ name: "noop", description: "Does nothing", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => ({ output: [] }), }), ); @@ -1298,7 +1298,7 @@ describe("tool rounds", () => { defineTool({ name: "noop", description: "Does nothing", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => ({ output: [] }), }), ); @@ -1826,7 +1826,7 @@ describe("tool rounds as the view shows them", () => { defineTool({ name: "noop", description: "Does nothing", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => ({ output: [] }), }), ); @@ -1866,7 +1866,7 @@ describe("tool rounds as the view shows them", () => { defineTool({ name: "delegate", description: "Starts work in a conversation it owns", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api, callContext) => { await api.commit(async (tx) => { const child = await tx.createConversation({ ownership: { kind: "task", taskId: api.taskId } }); @@ -1906,7 +1906,7 @@ describe("tool rounds as the view shows them", () => { defineTool({ name: "broken", description: "Starts owned work, then returns a result that is not strict JSON", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api, callContext) => { await api.commit((tx) => tx.createTask(Node, { name: "held" }, owned(api.taskId)), callContext); return { output: [], details: { fn: (() => 1) as unknown as JsonValue } }; diff --git a/packages/durable/test/harness-submit-in-commit.test.ts b/packages/durable/test/harness-submit-in-commit.test.ts index 3d8ad2986..9f4e08d9e 100644 --- a/packages/durable/test/harness-submit-in-commit.test.ts +++ b/packages/durable/test/harness-submit-in-commit.test.ts @@ -1,5 +1,6 @@ // Admission for hosts that must not start runs: `whenIdle: "reject"`, and `submit(tx, draft)` inside the caller's // commit (#11149). + import { withTelemetry } from "@earendil-works/chord/context"; import { type AssistantMessage, @@ -7,7 +8,6 @@ import { fauxAssistantMessage, fauxText, fauxToolCall, - Type, } from "@earendil-works/pi-ai"; import { type Conversation, @@ -26,6 +26,7 @@ import { type Tx, UserEntry, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { InMemoryTelemetryContext } from "@earendil-works/pi-telemetry"; import { describe, expect, it } from "vitest"; import { type ChatSetup, chatSetup, openChat, unanswered, waitFor } from "./chat-support.ts"; @@ -80,7 +81,7 @@ function addHold(setup: ChatSetup) { defineTool({ name: "hold", description: "Holds until released", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => { reached.resolve(); await release.promise; @@ -180,7 +181,7 @@ describe("whenIdle reject", () => { defineTool({ name: "nudge", description: "Steers another conversation", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api, callContext) => { const handle = (await api.conversation(other!.id, callContext))!; const draft = { type: "input", content: "nudge", whenBusy: "steer", whenIdle: "reject" } as const; @@ -525,7 +526,7 @@ describe("submit inside the caller's commit", () => { defineTool({ name: "delegate", description: "Delegates to another conversation", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api, callContext) => { toolTask = api.taskId; const handle = (await api.conversation(other!.id, callContext))!; @@ -611,7 +612,7 @@ describe("submit inside the caller's commit", () => { defineTool({ name: "hold", description: "Holds until released, also when aborted", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api, callContext) => { handle = (await api.conversation(other!.id, callContext))!; callContext.abortSignal!.addEventListener("abort", () => abortSeen.resolve(), { once: true }); diff --git a/packages/durable/test/harness-support.ts b/packages/durable/test/harness-support.ts index 499f81846..d2c443c45 100644 --- a/packages/durable/test/harness-support.ts +++ b/packages/durable/test/harness-support.ts @@ -7,7 +7,7 @@ import type { ToolResultMessage, UserMessage, } from "@earendil-works/pi-ai"; -import { createModels, Type } from "@earendil-works/pi-ai"; +import { createModels } from "@earendil-works/pi-ai"; import { type AnyTask, createRegistry, @@ -23,10 +23,11 @@ import { section, type ToolRegistration, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { context } from "./session-support.ts"; export function tool(name: string, description = `${name} tool`): ToolRegistration { - return defineTool({ name, description, parameters: Type.Object({}), execute: async () => ({ output: [] }) }); + return defineTool({ name, description, parameters: obj({}), execute: async () => ({ output: [] }) }); } /** Open a Harness with a fresh registry holding the named tools. */ diff --git a/packages/durable/test/harness-tasks-recovery.test.ts b/packages/durable/test/harness-tasks-recovery.test.ts index 714fdc202..07fcd7f27 100644 --- a/packages/durable/test/harness-tasks-recovery.test.ts +++ b/packages/durable/test/harness-tasks-recovery.test.ts @@ -2,7 +2,7 @@ import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import type { Context, JsonValue } from "@earendil-works/chord"; -import { createModels, Type } from "@earendil-works/pi-ai"; +import { createModels } from "@earendil-works/pi-ai"; import { createRegistry, defineDoc, @@ -15,6 +15,7 @@ import { type TaskId, type TaskRuntime, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { afterEach, describe, expect, it } from "vitest"; import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts"; import { addTask, addTool } from "./harness-support.ts"; @@ -482,7 +483,7 @@ describe("blocked tasks", () => { defineTool({ name: "unrelated", description: "unrelated", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => ({}), }), ); diff --git a/packages/durable/test/harness-tasks.test.ts b/packages/durable/test/harness-tasks.test.ts index cc40953dc..17bfe604f 100644 --- a/packages/durable/test/harness-tasks.test.ts +++ b/packages/durable/test/harness-tasks.test.ts @@ -1,6 +1,6 @@ import type { Context } from "@earendil-works/chord"; import { withAbortSignal, withCancel } from "@earendil-works/chord/context"; -import { createModels, Type } from "@earendil-works/pi-ai"; +import { createModels } from "@earendil-works/pi-ai"; import { type CommitPublication, type Conversation, @@ -18,6 +18,7 @@ import { type TaskId, type TaskRuntime, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { InMemoryTelemetryContext } from "@earendil-works/pi-telemetry"; import { describe, expect, it } from "vitest"; import type { SessionImpl } from "../src/session/session.ts"; @@ -424,7 +425,7 @@ describe("task phases", () => { defineTool({ name: "late", description: "late", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => ({}), }), ); diff --git a/packages/durable/test/harness-telemetry.test.ts b/packages/durable/test/harness-telemetry.test.ts index 59537179e..58bcbc9e8 100644 --- a/packages/durable/test/harness-telemetry.test.ts +++ b/packages/durable/test/harness-telemetry.test.ts @@ -1,5 +1,5 @@ import { withTelemetry } from "@earendil-works/chord/context"; -import { fauxAssistantMessage, fauxToolCall, Type } from "@earendil-works/pi-ai"; +import { fauxAssistantMessage, fauxToolCall } from "@earendil-works/pi-ai"; import { CodemodeSandbox } from "@earendil-works/pi-codemode"; import { CompactionTask, @@ -11,6 +11,7 @@ import { type Storage, type TaskId, } from "@earendil-works/pi-durable"; +import { obj, str } from "@earendil-works/pi-schema"; import { InMemoryTelemetryContext, NOOP_TELEMETRY_CONTEXT, @@ -53,7 +54,7 @@ describe("Durable telemetry", () => { defineTool({ name: "inner", description: "inner", - parameters: Type.Object({ secret: Type.String() }), + parameters: obj({ secret: str() }), execute: async () => ({ output: [{ type: "text", text: "private-result" }] }), }), ); @@ -62,7 +63,7 @@ describe("Durable telemetry", () => { defineTool({ name: "outer", description: "outer", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api, ctx) => { await api.executeTool("inner", { secret: "private-argument" }, ctx); return { output: [{ type: "text", text: "private-output" }] }; @@ -347,7 +348,7 @@ describe("Durable telemetry", () => { defineTool({ name: "hold", description: "hold", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => { reached.resolve(); await gate.promise; @@ -415,7 +416,7 @@ describe("Durable telemetry", () => { defineTool({ name: "delegate", description: "delegate", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api, ctx) => { const child = await api.commit( (tx) => tx.createConversation({ ownership: { kind: "task", taskId: api.taskId } }), @@ -952,7 +953,7 @@ describe("Durable telemetry", () => { defineTool({ name: "once", description: "once", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => { runs++; return { output: [] }; @@ -965,7 +966,7 @@ describe("Durable telemetry", () => { name: "hold", description: "hold", replay: "safe", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, _api, ctx) => { if (!recover) { reached.resolve(); diff --git a/packages/durable/test/harness-tool-hooks.test.ts b/packages/durable/test/harness-tool-hooks.test.ts index 97010c5b3..beca471e1 100644 --- a/packages/durable/test/harness-tool-hooks.test.ts +++ b/packages/durable/test/harness-tool-hooks.test.ts @@ -1,5 +1,6 @@ // `afterTool` runs on every result a tool task writes, also those the Harness decides without executing the tool // (#11043). + import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -10,7 +11,6 @@ import { fauxText, fauxToolCall, type ToolResultMessage, - Type, } from "@earendil-works/pi-ai"; import { defineTool, @@ -29,6 +29,7 @@ import { ToolResultEntry, ToolTask, } from "@earendil-works/pi-durable"; +import { bool, obj, opt, str } from "@earendil-works/pi-schema"; import { afterEach, describe, expect, it } from "vitest"; import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts"; import { commitWrites } from "../src/testing/commit-writes.ts"; @@ -79,7 +80,7 @@ function stored(): Promise { return new Promise((resolve) => setTimeout(resolve, 50)); } -const Parameters = Type.Object({ text: Type.Optional(Type.String()) }); +const Parameters = obj({ text: opt(str()) }); function tool( name: string, @@ -318,7 +319,7 @@ describe("afterTool on results the call phase decides", () => { addTool( setup.registry, tool("typed", async () => ({ output: [], structuredOutput: { ok: true } }), { - structuredOutputSchema: Type.Object({ ok: Type.Boolean() }), + structuredOutputSchema: obj({ ok: bool() }), }), ); const nested: NestedToolExecutionResult[] = []; diff --git a/packages/durable/test/harness-tools-recovery.test.ts b/packages/durable/test/harness-tools-recovery.test.ts index ad4ffbe98..cf050f8df 100644 --- a/packages/durable/test/harness-tools-recovery.test.ts +++ b/packages/durable/test/harness-tools-recovery.test.ts @@ -8,7 +8,6 @@ import { fauxText, fauxToolCall, type ToolResultMessage, - Type, } from "@earendil-works/pi-ai"; import { defineTool, @@ -21,6 +20,7 @@ import { ToolResultEntry, ToolTask, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { afterEach, describe, expect, it } from "vitest"; import type { ExecutionEnv } from "../src/env/index.ts"; import { NodeExecutionEnv } from "../src/env/node.ts"; @@ -55,7 +55,7 @@ function tool( execute: ToolRegistration["execute"], extra: Partial = {}, ): ToolRegistration { - return defineTool({ name, description: name, parameters: Type.Object({}), execute, ...extra }); + return defineTool({ name, description: name, parameters: obj({}), execute, ...extra }); } function call(name: string, id = "c1"): AssistantMessage { diff --git a/packages/durable/test/harness-tools.test.ts b/packages/durable/test/harness-tools.test.ts index d2937fd77..92aec1cf6 100644 --- a/packages/durable/test/harness-tools.test.ts +++ b/packages/durable/test/harness-tools.test.ts @@ -11,7 +11,6 @@ import { type Models, type SystemMessage, type ToolResultMessage, - Type, } from "@earendil-works/pi-ai"; import { AgentDoc, @@ -32,6 +31,7 @@ import { ToolResultEntry, ToolTask, } from "@earendil-works/pi-durable"; +import { obj, opt, str } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; import { NodeExecutionEnv } from "../src/env/node.ts"; import { createBashTool, createEditTool, createReadTool } from "../src/tools/index.ts"; @@ -40,7 +40,7 @@ import { addHooks, addTask, addTool, type Installed } from "./harness-support.ts import { context } from "./session-support.ts"; import { aborted, deferred } from "./task-support.ts"; -const EchoParameters = Type.Object({ text: Type.Optional(Type.String()) }); +const EchoParameters = obj({ text: opt(str()) }); type Echo = ToolRegistration; type Execute = Echo["execute"]; diff --git a/packages/durable/test/harness-turn-limit.test.ts b/packages/durable/test/harness-turn-limit.test.ts index e5d819468..7f8ad61e1 100644 --- a/packages/durable/test/harness-turn-limit.test.ts +++ b/packages/durable/test/harness-turn-limit.test.ts @@ -1,8 +1,9 @@ // Per-conversation turn limits (`AgentState.maxTurns`) and continuing a run that reached one (#11043). + import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { type AssistantMessage, fauxAssistantMessage, fauxText, fauxToolCall, Type } from "@earendil-works/pi-ai"; +import { type AssistantMessage, fauxAssistantMessage, fauxText, fauxToolCall } from "@earendil-works/pi-ai"; import { type AfterFailedRun, type Conversation, @@ -16,6 +17,7 @@ import { ToolResultEntry, ToolTask, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { afterEach, describe, expect, it } from "vitest"; import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts"; import { allEntries, type ChatSetup, chatSetup, openChat, textOf } from "./chat-support.ts"; @@ -33,7 +35,7 @@ afterEach(async () => { directories.clear(); }); -const Parameters = Type.Object({}); +const Parameters = obj({}); function tool( name: string, diff --git a/packages/durable/test/harness-view-run.test.ts b/packages/durable/test/harness-view-run.test.ts index 61431cd13..79f446098 100644 --- a/packages/durable/test/harness-view-run.test.ts +++ b/packages/durable/test/harness-view-run.test.ts @@ -1,5 +1,6 @@ // What a renderer sees of a run through the conversation view: the transcript window and `pi.live`, every change a // commit makes, rebuilt from operations alone. This replaced the event stream (`watchEvents`, #11327). + import { GCProfiler, getHeapStatistics, setFlagsFromString } from "node:v8"; import { runInNewContext } from "node:vm"; import type { Draft } from "@earendil-works/chord"; @@ -11,7 +12,6 @@ import { fauxText, fauxThinking, fauxToolCall, - Type, } from "@earendil-works/pi-ai"; import { type Conversation, @@ -22,6 +22,7 @@ import { type LiveState, MemoryStorage, } from "@earendil-works/pi-durable"; +import { num, obj } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; import { type ChatSetup, chatSetup, openChat, textOf, waitFor } from "./chat-support.ts"; import { addTool } from "./harness-support.ts"; @@ -126,7 +127,7 @@ describe("a run as the conversation view shows it", () => { defineTool({ name: "print", description: "Prints", - parameters: Type.Object({ n: Type.Number() }), + parameters: obj({ n: num() }), execute: async (_args, api) => { api.output("one\n"); await new Promise((resolve) => setTimeout(resolve, 150)); @@ -259,7 +260,7 @@ describe("a run as the conversation view shows it", () => { defineTool({ name: "tail", description: "Prints lines", - parameters: Type.Object({}), + parameters: obj({}), outputLimits: { maxLines: 3, retain: "tail" }, execute: async (_args, api) => { for (let line = 0; line < 6; line++) { @@ -339,7 +340,7 @@ describe("a run as the conversation view shows it", () => { defineTool({ name: "wait", description: "Waits until aborted", - parameters: Type.Object({}), + parameters: obj({}), execute: (_args, _api, callContext) => aborted(callContext.abortSignal!), }), ); @@ -380,7 +381,7 @@ describe("a run as the conversation view shows it", () => { defineTool({ name: "hold", description: "Waits", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => { await gate.promise; return {}; diff --git a/packages/durable/test/nested-images-bench.ts b/packages/durable/test/nested-images-bench.ts index d0539a7bf..858f21672 100644 --- a/packages/durable/test/nested-images-bench.ts +++ b/packages/durable/test/nested-images-bench.ts @@ -10,6 +10,7 @@ * * node --conditions=source --experimental-strip-types --expose-gc test/nested-images-bench.ts [count] */ + import { randomBytes } from "node:crypto"; import { mkdtemp, readdir, rm, stat } from "node:fs/promises"; import { tmpdir } from "node:os"; @@ -18,7 +19,7 @@ import { BACKGROUND_CONTEXT as context } from "@earendil-works/chord/context"; import type { ImageContent } from "@earendil-works/pi-ai"; import { createModels } from "@earendil-works/pi-ai/models"; import { fauxAssistantMessage, fauxProvider, fauxText, fauxToolCall } from "@earendil-works/pi-ai/providers/faux"; -import { Type } from "typebox"; +import { arr, lit, num, obj, opt, str } from "@earendil-works/pi-schema"; import { createRegistry, defineExtension, @@ -95,11 +96,11 @@ function noWrites(): Written { }; } -const ImageArgument = Type.Object({ - type: Type.Literal("image"), - mimeType: Type.String(), - data: Type.Optional(Type.String()), - source: Type.Optional(Type.Object({ taskId: Type.Number() })), +const ImageArgument = obj({ + type: lit("image"), + mimeType: str(), + data: opt(str()), + source: opt(obj({ taskId: num() })), }); function registryFor(scenario: Scenario, probe: Probe) { @@ -107,7 +108,7 @@ function registryFor(scenario: Scenario, probe: Probe) { const readImage = defineTool({ name: "read_image", description: "Read an image", - parameters: Type.Object({ n: Type.Number() }), + parameters: obj({ n: num() }), // One image, so programs get it as an `ImageContent`. Unique per call: base64 of random bytes. execute: async () => ({ output: [ @@ -118,7 +119,7 @@ function registryFor(scenario: Scenario, probe: Probe) { const classify = defineTool({ name: "classify", description: "Classify images", - parameters: Type.Object({ images: Type.Array(ImageArgument) }), + parameters: obj({ images: arr(ImageArgument) }), execute: async (args, api, callContext) => { for (const image of args.images) { let data = image.data; @@ -141,7 +142,7 @@ function registryFor(scenario: Scenario, probe: Probe) { const script = defineTool({ name: "script", description: "Read and classify images", - parameters: Type.Object({ count: Type.Number() }), + parameters: obj({ count: num() }), callers: ["model"], execute: async (args, api, callContext) => { for (let n = 0; n < args.count; n++) { diff --git a/packages/durable/test/nested-tools-bench.ts b/packages/durable/test/nested-tools-bench.ts index 0c4ed5224..002383fec 100644 --- a/packages/durable/test/nested-tools-bench.ts +++ b/packages/durable/test/nested-tools-bench.ts @@ -9,13 +9,14 @@ * * node --conditions=source --experimental-strip-types --expose-gc test/nested-tools-bench.ts [streaming|nesting] */ + import { mkdtemp, readdir, rm, stat } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { BACKGROUND_CONTEXT as context } from "@earendil-works/chord/context"; import { createModels } from "@earendil-works/pi-ai/models"; import { fauxAssistantMessage, fauxProvider, fauxText, fauxToolCall } from "@earendil-works/pi-ai/providers/faux"; -import { Type } from "typebox"; +import { num, obj } from "@earendil-works/pi-schema"; import { type Conversation, createRegistry, @@ -103,7 +104,7 @@ async function runScenario(scenario: Scenario): Promise<{ perRun: Run[]; closedB const blob = defineTool({ name: "blob", description: "Stream a blob", - parameters: Type.Object({ n: Type.Number() }), + parameters: obj({ n: num() }), execute: async (_args, api) => { for (let n = 0; n < CHUNKS; n++) { api.output(chunk); @@ -118,7 +119,7 @@ async function runScenario(scenario: Scenario): Promise<{ perRun: Run[]; closedB defineTool({ name, description: `Call ${target} in parallel`, - parameters: Type.Object({ count: Type.Number() }), + parameters: obj({ count: num() }), execute: async (args, api, callContext) => { const calls = Array.from({ length: args.count }, (_, n) => api.executeTool(target, { n, count: FAN_OUT }, callContext, options), diff --git a/packages/durable/test/scheduler-fanout-bench.ts b/packages/durable/test/scheduler-fanout-bench.ts index ab00d9881..2d2fc381f 100644 --- a/packages/durable/test/scheduler-fanout-bench.ts +++ b/packages/durable/test/scheduler-fanout-bench.ts @@ -5,10 +5,11 @@ * * node --conditions=source --experimental-strip-types test/scheduler-fanout-bench.ts [calls...] */ + import { BACKGROUND_CONTEXT as context } from "@earendil-works/chord/context"; import { createModels } from "@earendil-works/pi-ai/models"; import { fauxAssistantMessage, fauxProvider, fauxText, fauxToolCall } from "@earendil-works/pi-ai/providers/faux"; -import { Type } from "typebox"; +import { num, obj } from "@earendil-works/pi-schema"; import { createRegistry, defineExtension, defineTool, Harness, ToolResultEntry } from "../src/index.ts"; import { MemoryStorage } from "../src/storage/memory.ts"; @@ -20,13 +21,13 @@ async function run(calls: number): Promise<{ ms: number; commits: number }> { const leaf = defineTool({ name: "leaf", description: "Return at once", - parameters: Type.Object({ n: Type.Number() }), + parameters: obj({ n: num() }), execute: async () => ({ output: [{ type: "text", text: "ok" }] }), }); const fan = defineTool({ name: "fan", description: "Call leaf in parallel", - parameters: Type.Object({ count: Type.Number() }), + parameters: obj({ count: num() }), execute: async (args, api, callContext) => { const results = await Promise.all( Array.from({ length: args.count }, (_, n) => diff --git a/packages/durable/test/session-document-retention.test.ts b/packages/durable/test/session-document-retention.test.ts index 2712a1817..8d1b5e927 100644 --- a/packages/durable/test/session-document-retention.test.ts +++ b/packages/durable/test/session-document-retention.test.ts @@ -1,6 +1,7 @@ // The Session's document cache is bounded by use: unattached documents leave it after the retention period and load // from Storage again; documents with a watch or a document state never leave it. -import { fauxAssistantMessage, fauxText, fauxToolCall, Type } from "@earendil-works/pi-ai"; + +import { fauxAssistantMessage, fauxText, fauxToolCall } from "@earendil-works/pi-ai"; import { type ConversationId, defineDoc, @@ -11,6 +12,7 @@ import { MemoryStorage, type StorageWrite, } from "@earendil-works/pi-durable"; +import { obj } from "@earendil-works/pi-schema"; import { describe, expect, it, onTestFinished } from "vitest"; import { SessionImpl, setDefaultDocumentRetention } from "../src/session/session.ts"; import { chatSetup, openChat } from "./chat-support.ts"; @@ -519,7 +521,7 @@ describe("document cache retention", () => { defineTool({ name: "steps", description: "Streams three steps", - parameters: Type.Object({}), + parameters: obj({}), execute: async (_args, api) => { for (const step of ["a\n", "b\n", "c\n"]) { api.output(step); diff --git a/packages/durable/test/spec-usage.test.ts b/packages/durable/test/spec-usage.test.ts index c50cc255e..b1896bf66 100644 --- a/packages/durable/test/spec-usage.test.ts +++ b/packages/durable/test/spec-usage.test.ts @@ -3,8 +3,9 @@ * the names the spec leaves to the application declared below. Keep the two in sync; `test/examples/` runs the same * patterns end to end. */ + import type { Context, Draft } from "@earendil-works/chord"; -import { type AssistantMessage, type Models, type ToolCall, Type } from "@earendil-works/pi-ai"; +import type { AssistantMessage, Models, ToolCall } from "@earendil-works/pi-ai"; import { type Conversation, type ConversationId, @@ -34,6 +35,7 @@ import { type UserInput, wrapTool, } from "@earendil-works/pi-durable"; +import { obj, str } from "@earendil-works/pi-schema"; import { expect, expectTypeOf, it } from "vitest"; import type { ExecutionEnv } from "../src/env/index.ts"; import { CodingTools, createBashTool, createEditTool, createReadTool } from "../src/tools/index.ts"; @@ -169,7 +171,7 @@ function examples() { defineTool({ name: "subagent", description: "Delegate a self-contained task to a subagent and get its answer back.", - parameters: Type.Object({ task: Type.String() }), + parameters: obj({ task: str() }), replay: "safe", execute: async (args, api, context) => { const { task } = args; diff --git a/packages/durable/test/system-order-cache-e2e.test.ts b/packages/durable/test/system-order-cache-e2e.test.ts index 0d67629d5..f6581ac4a 100644 --- a/packages/durable/test/system-order-cache-e2e.test.ts +++ b/packages/durable/test/system-order-cache-e2e.test.ts @@ -4,7 +4,6 @@ import { type Model, type Models, normalizeContext, - Type, uuidv7, } from "@earendil-works/pi-ai"; import { builtinModels } from "@earendil-works/pi-ai/providers/all"; @@ -18,6 +17,7 @@ import { hook, MemoryStorage, } from "@earendil-works/pi-durable"; +import { obj, str } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; import { streamSimple as streamSimpleAnthropic } from "../../ai/src/api/anthropic-messages.ts"; import { resolveApiKey } from "../../ai/test/oauth.ts"; @@ -41,7 +41,7 @@ function instructions(nonce: string): string { return lines.join("\n"); } -const Parameters = Type.Object({ text: Type.String() }); +const Parameters = obj({ text: str() }); const first = defineTool({ name: "probe_first", description: "Unused probe tool.", diff --git a/packages/durable/test/tool-output-bench.ts b/packages/durable/test/tool-output-bench.ts index 19140c1ff..ac7a54d29 100644 --- a/packages/durable/test/tool-output-bench.ts +++ b/packages/durable/test/tool-output-bench.ts @@ -7,6 +7,7 @@ * then close the Harness mid-round (persistent backends) to measure replay of the round's delta chain on reopen, and * finish the run to measure the stored size after the round's base and reclamation. Throughput scenarios push 1 GiB. */ + import { execFile } from "node:child_process"; import { mkdtemp, open, readdir, rm, stat } from "node:fs/promises"; import { tmpdir } from "node:os"; @@ -16,7 +17,7 @@ import { promisify } from "node:util"; import { BACKGROUND_CONTEXT as context } from "@earendil-works/chord/context"; import { createModels } from "@earendil-works/pi-ai/models"; import { fauxAssistantMessage, fauxProvider, fauxText, fauxToolCall } from "@earendil-works/pi-ai/providers/faux"; -import { Type } from "typebox"; +import { obj } from "@earendil-works/pi-schema"; import { NodeExecutionEnv } from "../src/env/node.ts"; import { createRegistry, defineTool, Harness, LiveDoc, MemoryStorage, type Storage } from "../src/index.ts"; import { openNodeJsonlStorage } from "../src/storage/jsonl/node.ts"; @@ -145,7 +146,7 @@ async function runScenario(scenario: Scenario): Promise { defineTool({ name: "emit", description: "", - parameters: Type.Object({}), + parameters: obj({}), outputLimits: { retain: scenario.retain }, execute: async (_args, api, callContext) => { let counter = 0; diff --git a/packages/durable/test/tool-search.test.ts b/packages/durable/test/tool-search.test.ts index 895240bbb..988c85d5a 100644 --- a/packages/durable/test/tool-search.test.ts +++ b/packages/durable/test/tool-search.test.ts @@ -1,4 +1,4 @@ -import { type AssistantMessage, fauxAssistantMessage, fauxText, fauxToolCall, Type } from "@earendil-works/pi-ai"; +import { type AssistantMessage, fauxAssistantMessage, fauxText, fauxToolCall } from "@earendil-works/pi-ai"; import { AgentDoc, type Conversation, @@ -9,6 +9,7 @@ import { MemoryStorage, type ToolRegistration, } from "@earendil-works/pi-durable"; +import { num, obj, opt } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; import { createToolSearchExtension } from "../src/tool-search/index.ts"; import { allEntries, type ChatSetup, chatSetup, openChat } from "./chat-support.ts"; @@ -19,7 +20,7 @@ function tool(name: string, description: string, extra: Partial ({ output: [{ type: "text", text: `${name} ran` }] }), ...extra, }); @@ -120,7 +121,7 @@ describe("deferred tools and tool_search", () => { const grow = defineTool({ name: "grow", description: "grow", - parameters: Type.Object({}), + parameters: obj({}), execute: async () => ({ output: [], control: { addTools: ["github_get_issue", "alpha", "ghost"] } }), }); addTool(chat.registry, grow); diff --git a/packages/mcp/CHANGELOG.md b/packages/mcp/CHANGELOG.md index 74908a649..92e01bf9d 100644 --- a/packages/mcp/CHANGELOG.md +++ b/packages/mcp/CHANGELOG.md @@ -2,6 +2,10 @@ ## [Unreleased] +### Changed + +- README: the `AgentTool` wrapping example passes the MCP input schema as plain JSON Schema instead of through TypeBox's `Type.Unsafe`. + ### Fixed - OAuth metadata parsing and `startAuthorization()` now accept only `http:` and `https:` URLs. Before, a server could supply a `file:` or custom protocol authorization endpoint (for example `search-ms:`), which the platform URL handler would launch on sign-in. diff --git a/packages/mcp/README.md b/packages/mcp/README.md index b09e043fb..ebb1b2973 100644 --- a/packages/mcp/README.md +++ b/packages/mcp/README.md @@ -36,7 +36,6 @@ Wrapping an MCP tool as a `pi-agent-core` `AgentTool`: ```typescript import type { AgentTool } from "@earendil-works/pi-agent-core"; import { toLlmContent } from "@earendil-works/pi-mcp"; -import { Type } from "typebox"; const tools: AgentTool[] = (await client.listTools()).map((tool) => ({ // Providers allow at most 64 characters of [A-Za-z0-9_-]. @@ -44,7 +43,7 @@ const tools: AgentTool[] = (await client.listTools()).map((tool) => ({ label: tool.title ?? tool.name, description: tool.description ?? tool.name, // Providers require an object schema, and some reject one without `properties`. - parameters: Type.Unsafe({ ...tool.inputSchema, type: "object", properties: tool.inputSchema.properties ?? {} }), + parameters: { ...tool.inputSchema, type: "object", properties: tool.inputSchema.properties ?? {} }, execute: async (_toolCallId, params, signal) => { const result = await client.callTool(tool.name, params as Record, { signal }); // MCP reports tool failures in the result instead of as a protocol error. diff --git a/packages/schema/CHANGELOG.md b/packages/schema/CHANGELOG.md index d23ca137c..69650940a 100644 --- a/packages/schema/CHANGELOG.md +++ b/packages/schema/CHANGELOG.md @@ -6,4 +6,7 @@ - Initial release: `compile(schema, options)` validates JSON Schema drafts 2020-12 and 7 without `eval` or `new Function`, compiling a schema once into plain closures. `Validator.check` is the boolean check (a type guard for schemas built with the builders), `Validator.errors` reports each failure with a JSON Pointer path. Format checks are in the optional `@earendil-works/pi-schema/formats` entry point. - Schema builders (`str`, `num`, `int`, `bool`, `nul`, `unknown`, `lit`, `oneOf`, `obj`, `opt`, `arr`, `tuple`, `map`, `union`, `nullable`, `bytes`) that return plain JSON Schema typed with the values it accepts (`Static`). +- `compile(schema, { coerce: true })` adds `coerce(value)` and `parse(value)` for loosely typed input such as a model's tool call arguments: optional `null`s dropped, TypeBox 1.3.27's `Value.Convert` rules keyed on JSON Schema keywords (numeric strings, `"true"`, single values to arrays, `const`/`enum` members, union branches), then Ajv-style primitive coercion by `type`. `coerce` copies only the paths it changes and returns the input itself when nothing changes; `parse` coerces, then checks. Validators compiled without the option have neither; the README lists every rule. +- `unresolvedRefs: "reject"` compiles a `$ref` that names nothing as `false` (the error is `unresolved reference `) instead of throwing, for schemas whose references may dangle. +- `formats` also checks `url` and `json-pointer-uri-fragment`, as TypeBox and ajv-formats do. - `npm run bench` compares conformance, speed, setup cost, memory and bundle size with TypeBox, @cfworker/json-schema and omptype. diff --git a/packages/schema/README.md b/packages/schema/README.md index 023ebdc02..100186dd9 100644 --- a/packages/schema/README.md +++ b/packages/schema/README.md @@ -86,11 +86,19 @@ const validator = compile(schema, { draft: "7", // "2020-12" (default) or "7" formats, // check `format`; without it `format` is an annotation, as 2020-12 specifies schemas: { "https://example.com/point.json": point }, // documents for $refs to other documents, by URI + unresolvedRefs: "reject", // a $ref that names nothing rejects values at its place; default "throw" + coerce: true, // adds coerce() and parse(), see Coercion }); ``` -`formats` has every format 2020-12 defines (`date-time`, `email`, `uri`, `uuid`, `ipv6`, ...). Pass your own map to -check fewer formats, or custom ones. A `$ref` that names nothing throws when the schema compiles. +`formats` has every format 2020-12 defines (`date-time`, `email`, `uri`, `uuid`, `ipv6`, ...), plus `url` and +`json-pointer-uri-fragment`, which TypeBox and ajv-formats also check. Pass your own map to check fewer formats, or +custom ones. + +A `$ref` that names nothing (a missing definition, a document not in `schemas`) throws when the schema compiles. With +`unresolvedRefs: "reject"` it compiles as `false` instead: only values at that place fail, with the error +`unresolved reference `, and the rest of the schema works. Use it for schemas from other systems, such as MCP +servers, whose references may dangle. ### TypeBox schemas @@ -105,6 +113,56 @@ const user = compile(Type.Object({ name: Type.String(), tags: Type.Array(Type.St TypeBox's JavaScript-only types (`BigInt`, `Undefined`, `Function`, ...), refinements and codecs are not JSON Schema and are not supported. +### Coercion + +`compile(schema, { coerce: true })` adds `coerce()` and `parse()`, for loosely typed input such as a model's tool call +arguments. Without the option the validator has neither, and its `check` pays nothing for them: the coercion rules are +built on the first `coerce()` or `parse()`. + +```typescript +const read = compile(Read, { coerce: true }); + +read.coerce({ path: "a.txt", offset: "10", mode: "text" }); // { path: "a.txt", offset: 10, mode: "text" } + +const result = read.parse({ path: "a.txt", offset: "ten", mode: "text" }); +if (result.ok) result.value.offset; // number +else result.errors; // [{ path: "/offset", keyword: "type", message: "expected integer, got string" }] +``` + +`coerce(value)` returns the coerced value. It copies only the paths it changes, returns `value` itself when nothing +changes, and never modifies `value`. A value no rule applies to stays as it is. `parse(value)` coerces, then checks: +`{ ok: true, value }` with the coerced value, or `{ ok: false, errors }` with the errors of the coerced value. + +Three passes run in order: + +1. **Optional nulls.** A `null` in an optional property whose schema rejects `null` is dropped, as if omitted. +2. **Conversions**, ported from TypeBox 1.3.27's `Value.Convert` and keyed on JSON Schema keywords (the first that + applies to a schema): `$ref` follows the reference; `const` and `enum` take the member the value converts to; + `anyOf`/`oneOf` keep a value one branch accepts, else take the first branch's conversion the whole schema + accepts; `allOf` converts with each member in turn; then by `type`: + - `string`: numbers and booleans to their text, `null` to `"null"`. + - `number`: numeric strings (`""` is 0), `"true"`/`"false"`, booleans to 1/0, `null` to 0. + - `integer`: as `number`, then truncated (`"3.5"` and `3.5` become 3). + - `boolean`: `"true"`/`"false"` (any case), `"1"`/`"0"`, 1/0, `null` to `false`. + - `null`: `"null"`, `"undefined"`, `""`, `"0"`, 0 and `false`. + - `array`: a single value becomes a one-item array, then each item converts; a tuple (`prefixItems` or an `items` + array) converts its items and never wraps. + - `object`: each property converts by `properties`, `patternProperties` or `additionalProperties`. +3. **Primitive coercion by type**, as Ajv's `coerceTypes`, for what pass 2 leaves (type arrays such as + `["integer", "null"]`): numbers from numeric strings and booleans, booleans from `"true"`/`"false"` and 1/0, + strings from numbers and booleans, `null` from `""`, 0 and `false`, and `null` to `""`, 0 or `false`. A value that + already has one of a type array's types stays. + +Where passes 2 and 3 disagree, pass 2 wins: `null` for a required string becomes `"null"`, not `""`. A union's +branches convert in order and only until one is accepted; a union node that also declares properties converts them into +each branch's candidate before checking it. A recursive schema (a tree through `$ref`) that meets a value it cannot +convert leaves the value as it is, so `check` reports it, and results through a `$ref` are reused within one call, so a +recursive union does not convert (or explain) the same child once per branch. + +Pass 1 drops optional nulls inside the values of `properties`, `patternProperties` and `additionalProperties` alike; a +pattern or additional key itself is never dropped. Items past a draft-7 `items` array (`additionalItems`) are not +coerced by any pass. + ### Values - A property whose value is `undefined` counts as absent, as in TypeBox. diff --git a/packages/schema/src/checks.ts b/packages/schema/src/checks.ts index 701707a28..222d3dbde 100644 --- a/packages/schema/src/checks.ts +++ b/packages/schema/src/checks.ts @@ -127,7 +127,11 @@ export function codePoints(s: string): number { let n = 0; for (let i = 0; i < s.length; i++, n++) { const c = s.charCodeAt(i); - if (c >= 0xd800 && c <= 0xdbff) i++; + // A high surrogate starts a pair only when a low surrogate follows; an unpaired one is a code point of its own. + if (c >= 0xd800 && c <= 0xdbff && i + 1 < s.length) { + const next = s.charCodeAt(i + 1); + if (next >= 0xdc00 && next <= 0xdfff) i++; + } } return n; } diff --git a/packages/schema/src/coerce.ts b/packages/schema/src/coerce.ts new file mode 100644 index 000000000..5dee2487f --- /dev/null +++ b/packages/schema/src/coerce.ts @@ -0,0 +1,556 @@ +// Coercion: turns a loosely typed value (a model's tool call arguments, form input, query strings) into the value a +// schema most likely meant, before it is checked. Only validators compiled with `{ coerce: true }` have it, and they +// build it on their first coerce() or parse(); the check path never runs it. +// +// Three passes, in this order: +// 1. Optional nulls: a `null` in an optional property whose schema rejects null is dropped, as if omitted. +// 2. Conversion rules ported from TypeBox 1.3.27's Value.Convert (MIT), keyed on JSON Schema structure (`$ref`, `const`, +// `enum`, `anyOf`/`oneOf`, `allOf`, `type`, `items`/`prefixItems`, `properties`/`patternProperties`) instead of +// TypeBox's kind marks: `null` becomes "null" for a string, `"3.5"` becomes 3 for an integer, a single value becomes +// a one-item array, and a union takes the first branch whose conversion passes. +// 3. Primitive coercion by `type` (also type arrays), as in Ajv's `coerceTypes`: what pass 2 left, for example a type +// array `["integer", "null"]`, or `""` for `null`. +// A value that cannot be coerced is left as it is; the check then reports it. +// +// Every pass is copy-on-write: an object or array is copied only when one of its members changes, so an unchanged +// value comes back as the same reference and the input is never modified. All passes walk the same keywords for +// members: `properties`, then `patternProperties`, then `additionalProperties` for keys neither names; tuple items by +// `prefixItems` (or a draft-7 `items` array), the rest by `items`. Draft 7's `additionalItems` is not coerced. + +import type { Compiler } from "./compiler.ts"; +import { isSchemaObject } from "./resolver.ts"; +import { sharedRegex } from "./shared.ts"; +import type { SchemaObject } from "./types.ts"; + +/** Coerces values for one compiled schema. */ +export class Coercer { + private readonly c: Compiler; + private readonly schema: unknown; + /** + * The (schema node, value) pairs pass 2 is converting through a re-entry point, as a stack. Conversion can enter + * a node again with the same value only through a `$ref` or by wrapping a single value into an array (also for + * schemas that cycle through shared JS objects): a recursive schema that meets a value no branch accepts would + * convert it without end. A pair already on the stack (primitives by value, objects by reference) leaves the value + * unchanged. Only those two points push, so schemas without references or wraps pay nothing. + */ + private readonly entered: unknown[] = []; + /** How often a re-entry was cut short in this coerce() call; results computed across one are not memoized. */ + private cuts = 0; + /** + * Results of `$ref` targets by (node, value) within one coerce() call: a recursive union whose branches all + * convert the same children (and all fail) would convert them once per branch, exponentially in the depth. + */ + private memo: Map> | undefined = undefined; + + constructor(compiler: Compiler, schema: unknown) { + this.c = compiler; + this.schema = schema; + } + + /** `value` coerced: a copy along the changed paths only, or `value` itself when nothing changes. */ + coerce(value: unknown): unknown { + const base = this.c.r.rootBase; + const withoutNulls = this.dropOptionalNulls(value, this.schema, base); + return this.coerceTypes(this.convertOnly(withoutNulls), this.schema, base); + } + + /** Pass 2 alone, as TypeBox's `Value.Convert` (the TypeBox conformance tests run it). */ + convertOnly(value: unknown): unknown { + try { + return this.convert(this.schema, this.c.r.rootBase, value); + } finally { + this.memo = undefined; + this.cuts = 0; + } + } + + /** `node` converted for `value` at a re-entry point: unchanged when that pair is already being converted. */ + private reenter(node: object, base: string, value: unknown, memoize: boolean): unknown { + const entered = this.entered; + for (let i = 0; i < entered.length; i += 2) { + if (entered[i] === node && Object.is(entered[i + 1], value)) { + this.cuts++; + return value; + } + } + const memo = memoize ? this.memo?.get(node) : undefined; + if (memo?.has(value)) return memo.get(value); + const cuts = this.cuts; + entered.push(node, value); + let result: unknown; + try { + result = this.convert(node, base, value); + } finally { + entered.length -= 2; + } + if (memoize && this.cuts === cuts) { + this.memo ??= new Map(); + let values = this.memo.get(node); + if (values === undefined) { + values = new Map(); + this.memo.set(node, values); + } + values.set(value, result); + } + return result; + } + + /** Whether `value` passes the schema `node`; undefined when `node` cannot be compiled. */ + private passes(node: unknown, base: string, value: unknown): boolean | undefined { + try { + return this.c.check(node, this.baseOf(node, base))(value); + } catch { + return undefined; + } + } + + private baseOf(node: unknown, base: string): string { + return isSchemaObject(node) ? (this.c.r.baseOf.get(node) ?? base) : base; + } + + // --------------------------------------------------------------------------------------------------------------- + // Pass 1: optional nulls + // --------------------------------------------------------------------------------------------------------------- + + private dropOptionalNulls(value: unknown, schema: unknown, base: string): unknown { + if (!isSchemaObject(schema)) return value; + const b = this.baseOf(schema, base); + if (Array.isArray(value)) { + return mapItems(value, (item, i) => { + const itemSchema = itemSchemaAt(schema, i); + return itemSchema === undefined ? item : this.dropOptionalNulls(item, itemSchema, b); + }); + } + if (!isPlainObject(value)) return value; + const properties = isSchemaObject(schema.properties) ? schema.properties : {}; + const patterns = isSchemaObject(schema.patternProperties) + ? Object.entries(schema.patternProperties).map(([pattern, item]) => [sharedRegex(pattern), item] as const) + : []; + const additional = isSchemaObject(schema.additionalProperties) ? schema.additionalProperties : undefined; + if (patterns.length === 0 && additional === undefined && !isSchemaObject(schema.properties)) return value; + const required = new Set(Array.isArray(schema.required) ? schema.required : []); + let out: Record | undefined; + const set = (key: string, next: unknown) => { + if (next === (out ?? value)[key]) return; + out ??= { ...value }; + out[key] = next; + }; + for (const [key, property] of Object.entries(properties)) { + if (!Object.hasOwn(value, key)) continue; + const isRef = isSchemaObject(property) && typeof property.$ref === "string"; + if (value[key] === null && !required.has(key) && !isRef && this.passes(property, b, null) === false) { + out ??= { ...value }; + delete out[key]; + continue; + } + set(key, this.dropOptionalNulls(value[key], property, b)); + } + // Values of pattern and additional properties are walked for their own optional nulls; the keys themselves + // are not optional properties, so they stay. + if (patterns.length > 0 || additional !== undefined) { + for (const key of Object.keys(value)) { + let matched = Object.hasOwn(properties, key); + for (const [pattern, item] of patterns) { + if (!pattern.test(key)) continue; + matched = true; + set(key, this.dropOptionalNulls((out ?? value)[key], item, b)); + } + if (!matched && additional !== undefined) set(key, this.dropOptionalNulls(value[key], additional, b)); + } + } + return out ?? value; + } + + // --------------------------------------------------------------------------------------------------------------- + // Pass 2: TypeBox's conversion rules + // --------------------------------------------------------------------------------------------------------------- + + private convert(node: unknown, base: string, value: unknown): unknown { + if (!isSchemaObject(node)) return value; + const b = this.baseOf(node, base); + if (typeof node.$ref === "string") { + let target: { node: unknown; base: string }; + try { + target = this.c.r.resolve(node.$ref, b); + } catch { + return value; + } + return isSchemaObject(target.node) ? this.reenter(target.node, target.base, value, true) : value; + } + return this.convertNode(node, b, value); + } + + /** One schema node's conversion: its first keyword among `const`, `enum`, the unions, `allOf` and `type`. */ + private convertNode(node: SchemaObject, b: string, value: unknown): unknown { + if (Object.hasOwn(node, "const")) return convertLiteral(node.const, value); + if (Array.isArray(node.enum)) return this.convertEnum(node, node.enum, b, value); + // A union or intersection node may also declare members (plain JSON Schema; TypeBox's never do): they convert + // after the branches. + if (Array.isArray(node.anyOf)) return this.convertUnion(node, node.anyOf, b, value); + if (Array.isArray(node.oneOf)) return this.convertUnion(node, node.oneOf, b, value); + if (Array.isArray(node.allOf)) { + let next = value; + for (const member of node.allOf) next = this.convert(member, b, next); + return this.convertMembers(node, b, next); + } + switch (node.type) { + case "string": + return orValue(tryString(value), value); + case "number": + return orValue(tryNumber(value), value); + case "integer": { + const result = tryNumber(value); + return result === undefined ? value : Math.trunc(result.value); + } + case "boolean": + return orValue(tryBoolean(value), value); + case "null": + return orValue(tryNull(value), value); + case "array": + return this.convertArray(node, b, value); + case "object": + return this.convertObject(node, b, value); + case undefined: + if (node.properties !== undefined || node.patternProperties !== undefined) { + return this.convertObject(node, b, value); + } + if (node.items !== undefined || node.prefixItems !== undefined) return this.convertArray(node, b, value); + return value; + default: + return value; + } + } + + /** An enum is a union of its values: a member stays, else the first value the input converts to. */ + private convertEnum(node: SchemaObject, values: readonly unknown[], base: string, value: unknown): unknown { + if (values.some((member) => member === value)) return value; + for (const member of values) { + const candidate = convertLiteral(member, value); + if (candidate !== value && this.passes(node, base, candidate) === true) return candidate; + } + return value; + } + + /** + * A value one branch accepts stays; else the first branch's conversion that the whole union accepts. Branches are + * converted in order and only until one is accepted, so a later recursive branch is not entered needlessly. A node + * that also declares members (plain JSON Schema; TypeBox's never do) converts them into each candidate before it is + * checked, so a branch is not discarded for a sibling property that is converted after it. + */ + private convertUnion(node: SchemaObject, branches: readonly unknown[], base: string, value: unknown): unknown { + if (branches.some((branch) => this.passes(branch, base, value) === true)) + return this.convertMembers(node, base, value); + for (const branch of branches) { + const candidate = this.convertMembers(node, base, this.convert(branch, base, value)); + if (this.passes(node, base, candidate) === true) return candidate; + } + return this.convertMembers(node, base, value); + } + + /** The object members of a union or intersection node, when it declares any. */ + private convertMembers(node: SchemaObject, base: string, value: unknown): unknown { + const declares = + node.properties !== undefined || + node.patternProperties !== undefined || + node.additionalProperties !== undefined; + return declares ? this.convertObject(node, base, value) : value; + } + + private convertArray(node: SchemaObject, base: string, value: unknown): unknown { + const prefix = Array.isArray(node.items) + ? node.items + : Array.isArray(node.prefixItems) + ? node.prefixItems + : undefined; + if (prefix !== undefined) { + // A tuple converts its items and never wraps a single value. + if (!Array.isArray(value)) return value; + const rest = Array.isArray(node.items) ? undefined : node.items; + return mapItems(value, (item, i) => { + const schema = i < prefix.length ? prefix[i] : rest; + return schema === undefined ? item : this.convert(schema, base, item); + }); + } + if (Array.isArray(value)) { + return node.items === undefined ? value : mapItems(value, (item) => this.convert(node.items, base, item)); + } + // The wrapped single value is converted by the item schema with the same value: a re-entry point. + const item = node.items; + return [isSchemaObject(item) ? this.reenter(item, base, value, false) : value]; + } + + private convertObject(node: SchemaObject, base: string, value: unknown): unknown { + if (!isPlainObject(value)) return value; + const properties = isSchemaObject(node.properties) ? node.properties : {}; + const patterns = isSchemaObject(node.patternProperties) + ? Object.entries(node.patternProperties).map(([pattern, schema]) => [sharedRegex(pattern), schema] as const) + : []; + const additional = isSchemaObject(node.additionalProperties) ? node.additionalProperties : undefined; + let out: Record | undefined; + for (const key of Object.keys(value)) { + // An undefined property counts as absent. + if (value[key] === undefined) continue; + let next: unknown = value[key]; + let matched = Object.hasOwn(properties, key); + if (matched) next = this.convert(properties[key], base, next); + for (const [pattern, schema] of patterns) { + if (!pattern.test(key)) continue; + matched = true; + next = this.convert(schema, base, next); + } + if (!matched && additional !== undefined) next = this.convert(additional, base, next); + if (next !== value[key]) { + out ??= { ...value }; + out[key] = next; + } + } + return out ?? value; + } + + // --------------------------------------------------------------------------------------------------------------- + // Pass 3: primitive coercion by type + // --------------------------------------------------------------------------------------------------------------- + + private coerceTypes(value: unknown, schema: unknown, base: string): unknown { + if (!isSchemaObject(schema)) return value; + const b = this.baseOf(schema, base); + let next = value; + if (Array.isArray(schema.allOf)) for (const member of schema.allOf) next = this.coerceTypes(next, member, b); + if (Array.isArray(schema.anyOf)) next = this.coerceUnion(next, schema.anyOf, b); + if (Array.isArray(schema.oneOf)) next = this.coerceUnion(next, schema.oneOf, b); + const types = typesOf(schema); + const matchesUnionMember = types.length > 1 && types.some((type) => matchesType(next, type)); + if (types.length > 0 && !matchesUnionMember) { + for (const type of types) { + const candidate = coercePrimitive(next, type); + if (candidate !== next) { + next = candidate; + break; + } + } + } + if (types.includes("object") && isPlainObject(next)) next = this.coerceProperties(next, schema, b); + if (types.includes("array") && Array.isArray(next)) next = this.coerceItems(next, schema, b); + return next; + } + + private coerceUnion(value: unknown, branches: readonly unknown[], base: string): unknown { + for (const branch of branches) if (this.passes(branch, base, value) === true) return value; + for (const branch of branches) { + const coerced = this.coerceTypes(value, branch, base); + if (this.passes(branch, base, coerced) === true) return coerced; + } + return value; + } + + private coerceProperties( + value: Record, + schema: SchemaObject, + base: string, + ): Record { + const properties = isSchemaObject(schema.properties) ? schema.properties : {}; + const patterns = isSchemaObject(schema.patternProperties) + ? Object.entries(schema.patternProperties).map(([pattern, item]) => [sharedRegex(pattern), item] as const) + : []; + const additional = isSchemaObject(schema.additionalProperties) ? schema.additionalProperties : undefined; + let out: Record | undefined; + const set = (key: string, next: unknown) => { + if (next === (out ?? value)[key]) return; + out ??= { ...value }; + out[key] = next; + }; + for (const [key, property] of Object.entries(properties)) { + if (key in value) set(key, this.coerceTypes(value[key], property, base)); + } + for (const key of Object.keys(value)) { + let matched = Object.hasOwn(properties, key); + for (const [pattern, item] of patterns) { + if (!pattern.test(key)) continue; + matched = true; + set(key, this.coerceTypes((out ?? value)[key], item, base)); + } + if (!matched && additional !== undefined) set(key, this.coerceTypes(value[key], additional, base)); + } + return out ?? value; + } + + private coerceItems(value: unknown[], schema: SchemaObject, base: string): unknown[] { + return mapItems(value, (item, i) => { + const itemSchema = itemSchemaAt(schema, i); + return isSchemaObject(itemSchema) ? this.coerceTypes(item, itemSchema, base) : item; + }); + } +} + +/** + * The schema of an array's item at `index`: a tuple's (`prefixItems`, or a draft-7 `items` array) for its prefix, + * `items` after `prefixItems`, or `items` for every item. Undefined when nothing applies. + */ +function itemSchemaAt(schema: SchemaObject, index: number): unknown { + if (Array.isArray(schema.items)) return schema.items[index]; + if (Array.isArray(schema.prefixItems) && index < schema.prefixItems.length) return schema.prefixItems[index]; + return schema.items; +} + +/** `items` mapped by `map`; the same array when every item maps to itself. */ +function mapItems(items: unknown[], map: (item: unknown, index: number) => unknown): unknown[] { + let out: unknown[] | undefined; + for (let i = 0; i < items.length; i++) { + const next = map(items[i], i); + if (next === items[i]) continue; + out ??= items.slice(); + out[i] = next; + } + return out ?? items; +} + +// ------------------------------------------------------------------------------------------------------------------- +// TypeBox's conversions of one value (value/convert/try in TypeBox 1.3.27): a result, or undefined when there is none +// ------------------------------------------------------------------------------------------------------------------- + +type Converted = { readonly value: T } | undefined; + +function orValue(result: Converted, value: unknown): unknown { + return result === undefined ? value : result.value; +} + +/** A literal (`const`, an enum value) taken from a value of another type that converts to exactly it. */ +function convertLiteral(literal: unknown, value: unknown): unknown { + if (literal === value) return value; + const result = + typeof literal === "string" + ? tryString(value) + : typeof literal === "number" + ? tryNumber(value) + : typeof literal === "boolean" + ? tryBoolean(value) + : undefined; + return result !== undefined && result.value === literal ? result.value : value; +} + +function tryString(value: unknown): Converted { + if (typeof value === "bigint" || typeof value === "boolean" || isFiniteNumber(value)) + return { value: String(value) }; + if (value === null) return { value: "null" }; + if (typeof value === "string") return { value }; + if (value === undefined) return { value: "" }; + return undefined; +} + +const MAX_BIGINT = BigInt(Number.MAX_SAFE_INTEGER); +const MIN_BIGINT = BigInt(Number.MIN_SAFE_INTEGER); + +function tryNumber(value: unknown): Converted { + if (typeof value === "bigint") + return value <= MAX_BIGINT && value >= MIN_BIGINT ? { value: Number(value) } : undefined; + if (typeof value === "boolean") return { value: value ? 1 : 0 }; + if (isFiniteNumber(value)) return { value }; + if (value === null || value === undefined) return { value: 0 }; + if (typeof value !== "string") return undefined; + const coerced = +value; + if (Number.isFinite(coerced)) return { value: coerced }; + const lowercase = value.toLowerCase(); + if (lowercase === "false") return { value: 0 }; + if (lowercase === "true") return { value: 1 }; + const big = bigIntOfString(value); + return big !== undefined && big <= MAX_BIGINT && big >= MIN_BIGINT ? { value: Number(big) } : undefined; +} + +/** TypeBox's TryBigInt of a string: `12n`, `12.5` (truncated) or `12`; true and false are not reached from tryNumber. */ +function bigIntOfString(value: string): bigint | undefined { + if (/^-?(0|[1-9]\d*)n$/.test(value)) return BigInt(value.slice(0, -1)); + if (/^-?(0|[1-9]\d*)\.\d+$/.test(value)) return BigInt(value.split(".")[0]); + if (/^-?(0|[1-9]\d*)$/.test(value)) return BigInt(value); + return undefined; +} + +function tryBoolean(value: unknown): Converted { + if (typeof value === "bigint") return value === 0n ? { value: false } : value === 1n ? { value: true } : undefined; + if (typeof value === "boolean") return { value }; + if (isFiniteNumber(value)) return value === 0 ? { value: false } : value === 1 ? { value: true } : undefined; + if (value === null || value === undefined) return { value: false }; + if (typeof value !== "string") return undefined; + const lowercase = value.toLowerCase(); + if (lowercase === "false" || value === "0") return { value: false }; + if (lowercase === "true" || value === "1") return { value: true }; + return undefined; +} + +function tryNull(value: unknown): Converted { + if (typeof value === "bigint") return value === 0n ? { value: null } : undefined; + if (typeof value === "boolean") return value === false ? { value: null } : undefined; + if (isFiniteNumber(value)) return value === 0 ? { value: null } : undefined; + if (value === null || value === undefined) return { value: null }; + if (typeof value !== "string") return undefined; + const lowercase = value.toLowerCase(); + return lowercase === "undefined" || lowercase === "null" || value === "" || value === "0" + ? { value: null } + : undefined; +} + +function isFiniteNumber(value: unknown): value is number { + return typeof value === "number" && Number.isFinite(value); +} + +// ------------------------------------------------------------------------------------------------------------------- +// Primitive coercion by type (pass 3) +// ------------------------------------------------------------------------------------------------------------------- + +function typesOf(schema: SchemaObject): string[] { + if (typeof schema.type === "string") return [schema.type]; + if (Array.isArray(schema.type)) return schema.type.filter((type): type is string => typeof type === "string"); + return []; +} + +function matchesType(value: unknown, type: string): boolean { + switch (type) { + case "number": + return typeof value === "number"; + case "integer": + return typeof value === "number" && Number.isInteger(value); + case "boolean": + return typeof value === "boolean"; + case "string": + return typeof value === "string"; + case "null": + return value === null; + case "array": + return Array.isArray(value); + case "object": + return isPlainObject(value); + default: + return false; + } +} + +function coercePrimitive(value: unknown, type: string): unknown { + switch (type) { + case "number": + case "integer": { + if (value === null) return 0; + if (typeof value === "string" && value.trim() !== "") { + const parsed = Number(value); + if (type === "number" ? Number.isFinite(parsed) : Number.isInteger(parsed)) return parsed; + } + if (typeof value === "boolean") return value ? 1 : 0; + return value; + } + case "boolean": + if (value === null) return false; + if (value === "true" || value === 1) return true; + if (value === "false" || value === 0) return false; + return value; + case "string": + if (value === null) return ""; + if (typeof value === "number" || typeof value === "boolean") return String(value); + return value; + case "null": + return value === "" || value === 0 || value === false ? null : value; + default: + return value; + } +} + +function isPlainObject(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} diff --git a/packages/schema/src/explain.ts b/packages/schema/src/explain.ts index da55edb43..3ab363986 100644 --- a/packages/schema/src/explain.ts +++ b/packages/schema/src/explain.ts @@ -22,6 +22,11 @@ import type { SchemaObject, ValidationError } from "./types.ts"; export class Explainer { private readonly c: Compiler; private readonly schema: unknown; + /** + * The errors of `$ref` targets for object values, within one errors() call: a recursive union explains each branch + * to pick the one to blame, and every branch would explain the same children again, exponentially in the depth. + */ + private memo: Map> | undefined = undefined; constructor(compiler: Compiler, schema: unknown) { this.c = compiler; @@ -32,7 +37,11 @@ export class Explainer { /** The errors of `value`, at least one (`value` failed the check). */ errors(value: unknown): ValidationError[] { const out: ValidationError[] = []; - this.walk(this.schema, this.c.r.rootBase, value, "", out); + try { + this.walk(this.schema, this.c.r.rootBase, value, "", out); + } finally { + this.memo = undefined; + } if (out.length === 0) out.push({ path: "", keyword: "schema", message: "invalid" }); return out; } @@ -56,8 +65,7 @@ export class Explainer { out.push({ path: at, keyword, message }); }; if (this.c.draft === "7" && typeof node.$ref === "string") { - const target = this.c.r.resolve(node.$ref, b); - this.walk(target.node, target.base, v, path, out); + this.walkRef(node.$ref, b, v, path, out); return; } if (node.type !== undefined) { @@ -69,7 +77,7 @@ export class Explainer { } } if (Array.isArray(node.enum) && !enumCheck(node.enum)(v)) - push("enum", `expected one of ${JSON.stringify(node.enum)}`); + push("enum", `expected one of ${listValues(node.enum)}`); if (Object.hasOwn(node, "const") && !constCheck(node.const)(v)) push("const", `expected ${JSON.stringify(node.const)}`); if (typeof v === "string" && node.type !== "bytes") this.walkString(node, v, push); @@ -85,10 +93,7 @@ export class Explainer { const branch = this.ok(node.if, b, v) ? node.then : node.else; if (branch !== undefined) this.walk(branch, b, v, path, out); } - if (typeof node.$ref === "string") { - const target = this.c.r.resolve(node.$ref, b); - this.walk(target.node, target.base, v, path, out); - } + if (typeof node.$ref === "string") this.walkRef(node.$ref, b, v, path, out); if (out.length === start && (node.unevaluatedProperties !== undefined || node.unevaluatedItems !== undefined)) { this.walkUnevaluated(node, b, v, path, out); } @@ -97,6 +102,33 @@ export class Explainer { } } + private walkRef(ref: string, base: string, v: unknown, path: string, out: ValidationError[]): void { + const target = this.c.r.resolve(ref, base); + if (target.node === false && this.c.r.isUnresolved(ref, base)) { + out.push({ path, keyword: "$ref", message: `unresolved reference ${ref}` }); + return; + } + if (!isSchemaObject(target.node) || typeof v !== "object" || v === null) { + this.walk(target.node, target.base, v, path, out); + return; + } + this.memo ??= new Map(); + let values = this.memo.get(target.node); + const known = values?.get(v); + if (known !== undefined && known.path === path) { + out.push(...known.errors); + return; + } + const errors: ValidationError[] = []; + this.walk(target.node, target.base, v, path, errors); + if (values === undefined) { + values = new Map(); + this.memo.set(target.node, values); + } + values.set(v, { path, errors }); + out.push(...errors); + } + private walkString(node: SchemaObject, v: string, push: Push): void { const n = codePoints(v); if (typeof node.minLength === "number" && n < node.minLength) { @@ -230,7 +262,7 @@ export class Explainer { const min = draft7 || typeof node.minContains !== "number" ? 1 : node.minContains; const max = draft7 || typeof node.maxContains !== "number" ? Number.POSITIVE_INFINITY : node.maxContains; if (!containsCheck(this.c.check(node.contains, b), min, max)(v)) { - push("contains", `expected between ${min} and ${max} matching items`); + push("contains", containsMessage(min, max)); } } } @@ -258,18 +290,33 @@ export class Explainer { out.push({ path: pointer(path, tag.key), keyword, - message: `expected one of ${JSON.stringify(tag.values)}`, + message: `expected one of ${listValues(tag.values)}`, }); else this.walk(branches[i], b, v, path, out); return; } - let best: ValidationError[] | undefined; + // An object is blamed on the branch that declares most of its properties (the one it was most likely meant for), + // then on the branch whose first error is deepest. + let best: { errors: ValidationError[]; declared: number } | undefined; for (const s of branches) { const errors: ValidationError[] = []; this.walk(s, b, v, path, errors); - if (errors.length > 0 && (best === undefined || errors[0].path.length > best[0].path.length)) best = errors; + if (errors.length === 0) continue; + const declared = isObject(v) ? this.declaredKeys(s, b, v) : 0; + const better = + best === undefined || + declared > best.declared || + (declared === best.declared && errors[0].path.length > best.errors[0].path.length); + if (better) best = { errors, declared }; } - out.push(...(best ?? [{ path, keyword, message: "matches no alternative" }])); + out.push(...(best?.errors ?? [{ path, keyword, message: "matches no alternative" }])); + } + + /** How many of the keys of `v` the branch `s` (through a `$ref`) declares in `properties`. */ + private declaredKeys(s: unknown, b: string, v: Record): number { + const node = isSchemaObject(s) && typeof s.$ref === "string" ? this.c.r.resolve(s.$ref, b).node : s; + const properties = isSchemaObject(node) && isSchemaObject(node.properties) ? node.properties : {}; + return Object.keys(v).filter((k) => Object.hasOwn(properties, k)).length; } private tagOf(branches: readonly unknown[], b: string): { key: string; values: unknown[] } | undefined { @@ -314,6 +361,22 @@ export class Explainer { type Push = (keyword: string, message: string, at?: string) => void; +// Enum errors name the allowed values; a long list (an enum of thousands) is cut, so the message stays readable. +const LISTED_VALUES = 20; + +/** `values` as JSON, at most LISTED_VALUES of them, then how many more there are. */ +function listValues(values: readonly unknown[]): string { + if (values.length <= LISTED_VALUES) return JSON.stringify(values); + const listed = JSON.stringify(values.slice(0, LISTED_VALUES)); + return `${listed.slice(0, -1)}, ... and ${values.length - LISTED_VALUES} more]`; +} + +function containsMessage(min: number, max: number): string { + if (max === Number.POSITIVE_INFINITY) return `expected at least ${min} matching ${min === 1 ? "item" : "items"}`; + if (min === 0) return `expected at most ${max} matching ${max === 1 ? "item" : "items"}`; + return `expected between ${min} and ${max} matching items`; +} + function pointer(path: string, key: string | number): string { return `${path}/${String(key).replace(/~/g, "~0").replace(/\//g, "~1")}`; } diff --git a/packages/schema/src/formats/index.ts b/packages/schema/src/formats/index.ts index 6919b9256..05d8824ed 100644 --- a/packages/schema/src/formats/index.ts +++ b/packages/schema/src/formats/index.ts @@ -14,15 +14,20 @@ import { IsIPv6 } from "./typebox/ipv6.ts"; import { IsIri } from "./typebox/iri.ts"; import { IsIriReference } from "./typebox/iri_reference.ts"; import { IsJsonPointer } from "./typebox/json_pointer.ts"; +import { IsJsonPointerUriFragment } from "./typebox/json_pointer_uri_fragment.ts"; import { IsRegex } from "./typebox/regex.ts"; import { IsRelativeJsonPointer } from "./typebox/relative_json_pointer.ts"; import { IsTime } from "./typebox/time.ts"; import { IsUri } from "./typebox/uri.ts"; import { IsUriReference } from "./typebox/uri_reference.ts"; import { IsUriTemplate } from "./typebox/uri_template.ts"; +import { IsUrl } from "./typebox/url.ts"; import { IsUuid } from "./typebox/uuid.ts"; -/** Every format JSON Schema 2020-12 defines, by name. The checks are adapted from TypeBox 1.3.27 (MIT). */ +/** + * Every format JSON Schema 2020-12 defines, by name, plus `url` and `json-pointer-uri-fragment`, which TypeBox and + * ajv-formats also check. The checks are adapted from TypeBox 1.3.27 (MIT). + */ export const formats: Readonly> = Object.freeze({ date: IsDate, "date-time": IsDateTime, @@ -36,11 +41,13 @@ export const formats: Readonly> = Object.freeze({ iri: IsIri, "iri-reference": IsIriReference, "json-pointer": IsJsonPointer, + "json-pointer-uri-fragment": IsJsonPointerUriFragment, regex: IsRegex, "relative-json-pointer": IsRelativeJsonPointer, time: IsTime, uri: IsUri, "uri-reference": IsUriReference, "uri-template": IsUriTemplate, + url: IsUrl, uuid: IsUuid, }); diff --git a/packages/schema/src/formats/typebox/json_pointer_uri_fragment.ts b/packages/schema/src/formats/typebox/json_pointer_uri_fragment.ts new file mode 100644 index 000000000..cac809cef --- /dev/null +++ b/packages/schema/src/formats/typebox/json_pointer_uri_fragment.ts @@ -0,0 +1,38 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +const JsonPointerUriFragment = /^#(?:\/(?:[a-z0-9_\-.!$&'()*+,;:=@]|%[0-9a-f]{2}|~0|~1)*)*$/i; + +/** + * Returns true if the value is a json pointer uri fragment + * @specification https://datatracker.ietf.org/doc/html/rfc6901 + * @source https://github.com/ajv-validator/ajv-formats + */ +export function IsJsonPointerUriFragment(value: string): boolean { + return JsonPointerUriFragment.test(value); +} diff --git a/packages/schema/src/formats/typebox/url.ts b/packages/schema/src/formats/typebox/url.ts new file mode 100644 index 000000000..dfe00c4b8 --- /dev/null +++ b/packages/schema/src/formats/typebox/url.ts @@ -0,0 +1,36 @@ +/*-------------------------------------------------------------------------- + +TypeBox + +The MIT License (MIT) + +Copyright (c) 2017-2026 Haydn Paterson + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + +---------------------------------------------------------------------------*/ + +/** + * Returns true if the value is a valid Uniform Resource Locator + * @specification https://datatracker.ietf.org/doc/html/rfc3986 + * @specification https://datatracker.ietf.org/doc/html/rfc3987 + */ +export function IsUrl(value: string): boolean { + return URL.canParse(value); +} diff --git a/packages/schema/src/index.ts b/packages/schema/src/index.ts index c421d6fdb..c57ede2e9 100644 --- a/packages/schema/src/index.ts +++ b/packages/schema/src/index.ts @@ -27,5 +27,5 @@ export { union, unknown, } from "./builders.ts"; -export type { Check, CompileOptions, FormatCheck, JsonSchema, ValidationError } from "./types.ts"; -export { compile, Validator } from "./validator.ts"; +export type { Check, CompileOptions, FormatCheck, JsonSchema, ParseResult, ValidationError } from "./types.ts"; +export { CoercingValidator, compile, Validator } from "./validator.ts"; diff --git a/packages/schema/src/resolver.ts b/packages/schema/src/resolver.ts index 9541d2c33..5c8b97543 100644 --- a/packages/schema/src/resolver.ts +++ b/packages/schema/src/resolver.ts @@ -26,6 +26,9 @@ export class Resolver { private readonly root: JsonSchema; private readonly options: CompileOptions; private readonly registry = new Map(); + private readonly rejectUnresolved: boolean; + /** The references that named nothing and resolved to `false` (with `unresolvedRefs: "reject"`). */ + private readonly unresolved = new Set(); private readonly resources = new Map(); private readonly anchors = new Map(); private indexed = false; @@ -34,6 +37,7 @@ export class Resolver { this.draft = options.draft === "7" ? "7" : "2020-12"; this.root = root; this.options = options; + this.rejectUnresolved = options.unresolvedRefs === "reject"; this.rootBase = isSchemaObject(root) ? this.baseFor(root, DEFAULT_BASE) : DEFAULT_BASE; } @@ -61,9 +65,27 @@ export class Resolver { this.index(this.root, DEFAULT_BASE); } - /** The schema a reference names, with the base URI it compiles under. Throws if it names nothing. */ + /** + * The schema a reference names, with the base URI it compiles under. Throws if it names nothing, or, with + * `unresolvedRefs: "reject"`, resolves to `false`, the schema no value passes. + */ resolve(ref: string, base: string): Resolved { this.ensureIndexed(); + if (!this.rejectUnresolved) return this.find(ref, base); + try { + return this.find(ref, base); + } catch { + this.unresolved.add(resolveUri(ref, base)); + return { node: false, base }; + } + } + + /** Whether `ref`, from `base`, named nothing and resolved to `false`. */ + isUnresolved(ref: string, base: string): boolean { + return this.unresolved.has(resolveUri(ref, base)); + } + + private find(ref: string, base: string): Resolved { const uri = resolveUri(ref, base); const doc = documentOf(uri); const fragment = uri.length > doc.length ? decodeURIComponent(uri.slice(doc.length + 1)) : ""; diff --git a/packages/schema/src/types.ts b/packages/schema/src/types.ts index a9e2135d1..d3f635adb 100644 --- a/packages/schema/src/types.ts +++ b/packages/schema/src/types.ts @@ -20,8 +20,21 @@ export interface CompileOptions { readonly formats?: Readonly>; /** Schema documents by URI, for `$ref`s to other documents. */ readonly schemas?: Readonly>; + /** Add `coerce()` and `parse()` to the validator, for loosely typed input (see the README's Coercion section). */ + readonly coerce?: boolean; + /** + * What a `$ref` that names nothing (a missing definition, a document not in `schemas`) does: `"throw"` (default) + * makes compile() throw; `"reject"` compiles it as `false`, so only values at that place fail, with the error + * "unresolved reference". + */ + readonly unresolvedRefs?: "throw" | "reject"; } +/** The result of `parse()`: the coerced value when it is valid, else the errors of the coerced value. */ +export type ParseResult = + | { readonly ok: true; readonly value: T } + | { readonly ok: false; readonly errors: ValidationError[] }; + /** One failure of a value: where it is (a JSON Pointer, "" for the value itself), the failing keyword and a message. */ export interface ValidationError { readonly path: string; diff --git a/packages/schema/src/validator.ts b/packages/schema/src/validator.ts index 3d915b33b..65781e5dd 100644 --- a/packages/schema/src/validator.ts +++ b/packages/schema/src/validator.ts @@ -1,16 +1,29 @@ // compile(): a schema to a Validator, whose `check` is the fast boolean path and whose `errors` explains failures. import type { Static } from "./builders.ts"; +import { Coercer } from "./coerce.ts"; import { Compiler, RESTART } from "./compiler.ts"; import { Explainer } from "./explain.ts"; -import type { Check, CompileOptions, JsonSchema, ValidationError } from "./types.ts"; +import type { Check, CompileOptions, JsonSchema, ParseResult, ValidationError } from "./types.ts"; /** * Compiles `schema` into a Validator. Throws if the schema is malformed or a `$ref` names nothing (register other - * documents with `options.schemas`). The schema is read, never modified. + * documents with `options.schemas`). The schema is read, never modified. With `coerce: true` the validator also + * coerces and parses loosely typed input. */ -export function compile(schema: S, options: CompileOptions = NO_OPTIONS): Validator> { - return new Validator(schema, options, compileCheck(schema, options)); +export function compile( + schema: S, + options: CompileOptions & { readonly coerce: true }, +): CoercingValidator>; +export function compile(schema: S, options?: CompileOptions): Validator>; +export function compile( + schema: S, + options: CompileOptions = NO_OPTIONS, +): Validator> | CoercingValidator> { + const check = compileCheck(schema, options); + return options.coerce === true + ? new CoercingValidator(schema, options, check) + : new Validator(schema, options, check); } const NO_OPTIONS: CompileOptions = Object.freeze({}); @@ -19,8 +32,8 @@ const NO_OPTIONS: CompileOptions = Object.freeze({}); export class Validator { /** Whether `value` is valid. A plain function: it may be passed around without its validator. */ readonly check: (value: unknown) => value is T; - private readonly schema: JsonSchema; - private readonly options: CompileOptions; + protected readonly schema: JsonSchema; + protected readonly options: CompileOptions; private explainer: Explainer | undefined = undefined; constructor(schema: JsonSchema, options: CompileOptions, check: Check) { @@ -41,6 +54,28 @@ export class Validator { } } +/** A validator compiled with `coerce: true`: it also coerces loosely typed input towards the schema. */ +export class CoercingValidator extends Validator { + private coercer: Coercer | undefined = undefined; + + /** + * `value` coerced towards the schema: numeric strings to numbers, `"true"` to `true`, a single value to a one-item + * array, an optional `null` dropped, the first union branch the value converts to (the README lists every rule). + * Only the changed paths are copied; `value` comes back itself when nothing changes, and is never modified. Values + * that cannot be coerced stay as they are. The rules are built on the first call. + */ + coerce(value: unknown): unknown { + this.coercer ??= new Coercer(compiler(this.schema, this.options), this.schema); + return this.coercer.coerce(value); + } + + /** Coerces `value`, then checks it: the coerced value when it is valid, else its errors. */ + parse(value: unknown): ParseResult { + const coerced = this.coerce(value); + return this.check(coerced) ? { ok: true, value: coerced } : { ok: false, errors: this.errors(coerced) }; + } +} + function compileCheck(schema: JsonSchema, options: CompileOptions): Check { return compiler(schema, options).root(); } diff --git a/packages/schema/test/coerce-typebox.test.ts b/packages/schema/test/coerce-typebox.test.ts new file mode 100644 index 000000000..f9a83140d --- /dev/null +++ b/packages/schema/test/coerce-typebox.test.ts @@ -0,0 +1,1344 @@ +// biome-ignore-all lint/suspicious/noTemplateCurlyInString: TypeBox template literal syntax, as in the original tests. +// biome-ignore-all lint/correctness/noPrecisionLoss: the original tests' literal, kept as written. +// TypeBox 1.3.27's Value.Convert tests (test/typebox/runtime/value/convert, MIT, (c) Haydn Paterson), ported to the +// conversion pass of pi-schema's coercion. Schemas are still built with TypeBox: its schemas are JSON Schema, and the +// port reads only their JSON structure, never TypeBox's kind marks. Run with vitest. + +import { Type } from "typebox"; +import { describe, expect, it } from "vitest"; +import { Coercer } from "../src/coerce.ts"; +import { Compiler } from "../src/compiler.ts"; +import { formats } from "../src/formats/index.ts"; +import { compile } from "../src/index.ts"; + +/** Pass 2 alone, as TypeBox's Value.Convert(schema, value). */ +function convert(schema: unknown, value: unknown): unknown { + return new Coercer(new Compiler(schema as object, { formats }, false), schema).convertOnly(value); +} + +const Value = { + Convert: convert, + Check: (schema: unknown, value: unknown) => compile(schema as object, { formats }).check(value), +}; +const Convert = convert; +const Assert = { + IsEqual: (actual: unknown, expected: unknown) => expect(actual).toEqual(expected), + IsTrue: (value: boolean) => expect(value).toBe(true), + IsFalse: (value: boolean) => expect(value).toBe(false), +}; + +describe("Value.Convert.Any", () => { + it("Should Convert 1", () => { + const T = Type.Any(); + const R = Value.Convert(T, null); + Assert.IsEqual(R, null); + }); + it("Should Convert 2", () => { + const T = Type.Any(); + const R = Value.Convert(T, undefined); + Assert.IsEqual(R, undefined); + }); + it("Should Convert 3", () => { + const T = Type.Any(); + const R = Value.Convert(T, "hello"); + Assert.IsEqual(R, "hello"); + }); + it("Should Convert 4", () => { + const T = Type.Any(); + const R = Value.Convert(T, 42); + Assert.IsEqual(R, 42); + }); + it("Should Convert 5", () => { + const T = Type.Any(); + const R = Value.Convert(T, true); + Assert.IsEqual(R, true); + }); + it("Should Convert 6", () => { + const T = Type.Any(); + const R = Value.Convert(T, { x: 1 }); + Assert.IsEqual(R, { x: 1 }); + }); + it("Should Convert 7", () => { + const T = Type.Any(); + const R = Value.Convert(T, [1, 2, 3]); + Assert.IsEqual(R, [1, 2, 3]); + }); + // Skipped: the schema or the value has no JSON form. + it.skip("Should Convert 8", () => { + const S = Symbol("x-symbol"); + const T = Type.Symbol(); + const R = Value.Convert(T, S); + Assert.IsEqual(R, S); + }); +}); + +describe("Value.Convert.Array", () => { + it("Should Convert 1", () => { + const T = Type.Array(Type.Number()); + const R = Value.Convert(T, [1, 3.14, "1", "3.14", true, false, "true", "false", "hello"]); + Assert.IsEqual(R, [1, 3.14, 1, 3.14, 1, 0, 1, 0, "hello"]); + }); + it("Should Convert 2", () => { + const T = Type.Array(Type.Boolean()); + const R = Value.Convert(T, [1, 3.14, "1", "3.14", true, false, "true", "false", "hello"]); + Assert.IsEqual(R, [true, 3.14, true, "3.14", true, false, true, false, "hello"]); + }); + it("Should Convert 3", () => { + const T = Type.Array(Type.String()); + const R = Value.Convert(T, [1, 3.14, "1", "3.14", true, false, "true", "false", "hello"]); + Assert.IsEqual(R, ["1", "3.14", "1", "3.14", "true", "false", "true", "false", "hello"]); + }); + // ------------------------------------------------------------------ + // Added support for Array coercion + // + // https://github.com/sinclairzx81/typebox/issues/1469 + // ------------------------------------------------------------------ + it("Should Convert 4", () => { + const T = Type.Array(Type.Number()); + const R = Value.Convert(T, "1"); + Assert.IsEqual(R, [1]); + }); + it("Should Convert 5", () => { + const T = Type.Array(Type.Number()); + const R = Value.Convert(T, []); + Assert.IsEqual(R, []); + }); + it("Should Convert 6", () => { + const T = Type.Array(Type.Number()); + const R = Value.Convert(T, 42); + Assert.IsEqual(R, [42]); + }); + it("Should Convert 7", () => { + const T = Type.Array(Type.Boolean()); + const R = Value.Convert(T, true); + Assert.IsEqual(R, [true]); + }); + it("Should Convert 8", () => { + const T = Type.Array(Type.Null()); + const R = Value.Convert(T, null); + Assert.IsEqual(R, [null]); + }); + it("Should Convert 9", () => { + const T = Type.Array(Type.Unknown()); + const R = Value.Convert(T, undefined); + Assert.IsEqual(R, [undefined]); + }); + // Skipped: the schema or the value has no JSON form. + it.skip("Should Convert 10", () => { + const T = Type.Array(Type.BigInt()); + const R = Value.Convert(T, 1n); + Assert.IsEqual(R, [1n]); + }); + it("Should Convert 11", () => { + const T = Type.Array(Type.Object({})); + const R = Value.Convert(T, {}); + Assert.IsEqual(R, [{}]); + }); + it("Should Convert 12", () => { + const T = Type.Array(Type.Object({ x: Type.Number() })); + const R = Value.Convert(T, { x: 1 }); + Assert.IsEqual(R, [{ x: 1 }]); + }); + it("Should Convert 13", () => { + const T = Type.Array(Type.Number()); + const R = Value.Convert(T, "42"); + Assert.IsEqual(R, [42]); + }); + it("Should Convert 14", () => { + const T = Type.Array(Type.Boolean()); + const R = Value.Convert(T, "true"); + Assert.IsEqual(R, [true]); + }); + it("Should Convert 15", () => { + const T = Type.Array(Type.Boolean()); + const R = Value.Convert(T, 0); + Assert.IsEqual(R, [false]); + }); + it("Should Convert 16", () => { + const T = Type.Array(Type.Number()); + const R = Value.Convert(T, "hello"); + Assert.IsEqual(R, ["hello"]); + }); + it("Should Convert 17", () => { + const T = Type.Array(Type.Number()); + const R = Value.Convert(T, [1]); + Assert.IsEqual(R, [1]); + }); + it("Should Convert 18", () => { + const T = Type.Array(Type.Array(Type.Number())); + const R = Value.Convert(T, [1, 2, 3]); + Assert.IsTrue(Value.Check(T, R)); + }); + it("Should Convert 19", () => { + const T = Type.Array(Type.Number()); + const R = Value.Convert(T, { 0: 1, length: 1 }); + Assert.IsEqual(R, [{ 0: 1, length: 1 }]); + }); + it("Should Convert 20", () => { + const T = Type.Array(Type.Array(Type.Number())); + const R = Value.Convert(T, 42); + Assert.IsTrue(Value.Check(T, R)); + }); + // ------------------------------------------------------------------ + // Union + // ------------------------------------------------------------------ + it("Should Convert 21", () => { + const T = Type.Array(Type.Union([Type.Number(), Type.String()])); + const R = Value.Convert(T, 1); + Assert.IsTrue(Value.Check(T, R)); + }); + it("Should Convert 22", () => { + const T = Type.Array(Type.Union([Type.Number(), Type.String()])); + const R = Value.Convert(T, "hello"); + Assert.IsTrue(Value.Check(T, R)); + }); + it("Should Convert 23", () => { + const T = Type.Array(Type.Union([Type.Number(), Type.String()])); + const R = Value.Convert(T, [1, "hello", true, "3.14"]); + Assert.IsTrue(Value.Check(T, R)); + }); + it("Should Convert 24", () => { + const T = Type.Union([Type.Array(Type.Number()), Type.String()]); + const R = Value.Convert(T, "1"); + Assert.IsTrue(Value.Check(T, R)); + }); + it("Should Convert 25", () => { + const T = Type.Array(Type.Union([Type.Array(Type.Number()), Type.String()])); + const R = Value.Convert(T, 42); + Assert.IsTrue(Value.Check(T, R)); + }); + // ------------------------------------------------------------------ + // Embedded Intersect + // ------------------------------------------------------------------ + it("Should Convert 26", () => { + const T = Type.Array(Type.Intersect([Type.Object({ x: Type.Number() }), Type.Object({ y: Type.Number() })])); + const R = Value.Convert(T, { x: 1, y: 2 }); + Assert.IsTrue(Value.Check(T, R)); + }); + it("Should Convert 27", () => { + const T = Type.Array(Type.Intersect([Type.Object({ x: Type.Number() }), Type.Object({ y: Type.Number() })])); + const R = Value.Convert(T, [ + { x: "1", y: "2" }, + { x: "3", y: "4" }, + ]); + Assert.IsTrue(Value.Check(T, R)); + }); + it("Should Convert 28", () => { + const T = Type.Intersect([Type.Array(Type.Number()), Type.Array(Type.Number())]); + const R = Value.Convert(T, 42); + Assert.IsTrue(Value.Check(T, R)); + }); + it("Should Convert 29", () => { + const T = Type.Intersect([Type.Array(Type.Number()), Type.Object({ x: Type.Number() })]); + const R = Value.Convert(T, 42); + Assert.IsFalse(Value.Check(T, R)); + }); +}); + +describe("Value.Convert.Boolean", () => { + it("Should Convert 1", () => { + const T = Type.Boolean(); + const R = Value.Convert(T, "1"); + Assert.IsEqual(R, true); + }); + it("Should Convert 2", () => { + const T = Type.Boolean(); + const R = Value.Convert(T, "3.14"); + Assert.IsEqual(R, "3.14"); + }); + it("Should Convert 3", () => { + const T = Type.Boolean(); + const R = Value.Convert(T, "true"); + Assert.IsEqual(R, true); + }); + it("Should Convert 4", () => { + const T = Type.Boolean(); + const R = Value.Convert(T, "false"); + Assert.IsEqual(R, false); + }); + it("Should Convert 5", () => { + const T = Type.Boolean(); + const R = Value.Convert(T, 1); + Assert.IsEqual(R, true); + }); + it("Should Convert 6", () => { + const T = Type.Boolean(); + const R = Value.Convert(T, 3.14); + Assert.IsEqual(R, 3.14); + }); + it("Should Convert 7", () => { + const T = Type.Boolean(); + const R = Value.Convert(T, 1.1); + Assert.IsEqual(R, 1.1); + }); + it("Should Convert 8", () => { + const T = Type.Boolean(); + const R = Value.Convert(T, true); + Assert.IsEqual(R, true); + }); + it("Should Convert 9", () => { + const T = Type.Boolean(); + const R = Value.Convert(T, false); + Assert.IsEqual(R, false); + }); + // ---------------------------------------------------------- + // Casts + // ---------------------------------------------------------- + it("Should Convert 10", () => { + const value = "hello"; + const result = Value.Convert(Type.Boolean(), value); + Assert.IsEqual(result, "hello"); + }); + it("Should Convert 11", () => { + const value = "true"; + const result = Value.Convert(Type.Boolean(), value); + Assert.IsEqual(result, true); + }); + it("Should Convert 12", () => { + const value = "TRUE"; + const result = Value.Convert(Type.Boolean(), value); + Assert.IsEqual(result, true); + }); + it("Should Convert 13", () => { + const value = "false"; + const result = Value.Convert(Type.Boolean(), value); + Assert.IsEqual(result, false); + }); + it("Should Convert 14", () => { + const value = "0"; + const result = Value.Convert(Type.Boolean(), value); + Assert.IsEqual(result, false); + }); + it("Should Convert 15", () => { + const value = "1"; + const result = Value.Convert(Type.Boolean(), value); + Assert.IsEqual(result, true); + }); + it("Should Convert 16", () => { + const value = "0"; + const result = Value.Convert(Type.Boolean({ default: true }), value); + Assert.IsEqual(result, false); + }); + it("Should Convert 17", () => { + const value = "1"; + const result = Value.Convert(Type.Boolean({ default: false }), value); + Assert.IsEqual(result, true); + }); + it("Should Convert 18", () => { + const value = "2"; + const result = Value.Convert(Type.Boolean({ default: true }), value); + Assert.IsEqual(result, "2"); + }); + it("Should Convert 19", () => { + const value = 0; + const result = Value.Convert(Type.Boolean(), value); + Assert.IsEqual(result, false); + }); + it("Should Convert 20", () => { + const value = 1n; + const result = Value.Convert(Type.Boolean(), value); + Assert.IsEqual(result, true); + }); + it("Should Convert 21", () => { + const value = 1; + const result = Value.Convert(Type.Boolean(), value); + Assert.IsEqual(result, true); + }); + it("Should Convert 22", () => { + const value = 2; + const result = Value.Convert(Type.Boolean(), value); + Assert.IsEqual(result, 2); + }); + it("Should Convert 23", () => { + const value = 0; + const result = Value.Convert(Type.Boolean({ default: true }), value); + Assert.IsEqual(result, false); + }); + it("Should Convert 24", () => { + const value = 1; + const result = Value.Convert(Type.Boolean({ default: false }), value); + Assert.IsEqual(result, true); + }); + it("Should Convert 25", () => { + const value = 2; + const result = Value.Convert(Type.Boolean({ default: true }), value); + Assert.IsEqual(result, 2); + }); + it("Should Convert 26", () => { + const value = true; + const result = Value.Convert(Type.Boolean(), value); + Assert.IsEqual(result, true); + }); + it("Should Convert 27", () => { + const value = false; + const result = Value.Convert(Type.Boolean(), value); + Assert.IsEqual(result, false); + }); + it("Should Convert 28", () => { + const value = {}; + const result = Value.Convert(Type.Boolean(), value); + Assert.IsEqual(result, {}); + }); + it("Should Convert 29", () => { + const value = [] as any[]; + const result = Value.Convert(Type.Boolean(), value); + Assert.IsEqual(result, []); + }); + // ------------------------------------------------------------------ + // Coverage + // ------------------------------------------------------------------ + it("Should Convert 17", () => { + const T = Type.Boolean(); + const R = Value.Convert(T, BigInt(0)); + Assert.IsEqual(R, false); + }); + it("Should Convert 18", () => { + const T = Type.Boolean(); + const R = Value.Convert(T, BigInt(1)); + Assert.IsEqual(R, true); + }); + it("Should Convert 19", () => { + const T = Type.Boolean(); + const R = Value.Convert(T, BigInt(2)); + Assert.IsEqual(R, BigInt(2)); + }); + it("Should Convert 20", () => { + const T = Type.Boolean(); + const R = Value.Convert(T, true); + Assert.IsEqual(R, true); + }); + it("Should Convert 21", () => { + const T = Type.Boolean(); + const R = Value.Convert(T, false); + Assert.IsEqual(R, false); + }); + it("Should Convert 22", () => { + const T = Type.Boolean(); + const R = Value.Convert(T, null); + Assert.IsEqual(R, false); + }); + it("Should Convert 23", () => { + const T = Type.Boolean(); + const R = Value.Convert(T, undefined); + Assert.IsEqual(R, false); + }); +}); + +describe("Value.Convert.Enum", () => { + it("Should Convert 1", () => { + const T = Type.Object({ + x: Type.Enum([1, 2, "hello"]), + }); + const R0 = Value.Convert(T, { x: "1" }); + const R1 = Value.Convert(T, { x: "2" }); + const R2 = Value.Convert(T, { x: "hello" }); + const R3 = Value.Convert(T, { x: "world" }); + Assert.IsEqual(R0, { x: 1 }); + Assert.IsEqual(R1, { x: 2 }); + Assert.IsEqual(R2, { x: "hello" }); + Assert.IsEqual(R3, { x: "world" }); + }); +}); + +describe("Value.Convert.Integer", () => { + it("Should Convert 1", () => { + const T = Type.Integer(); + const R = Value.Convert(T, "3.14"); + Assert.IsEqual(R, 3); + }); + it("Should Convert 2", () => { + const T = Type.Integer(); + const R = Value.Convert(T, "42"); + Assert.IsEqual(R, 42); + }); + it("Should Convert 3", () => { + const T = Type.Integer(); + const R = Value.Convert(T, true); + Assert.IsEqual(R, 1); + }); + it("Should Convert 4", () => { + const T = Type.Integer(); + const R = Value.Convert(T, false); + Assert.IsEqual(R, 0); + }); + it("Should Convert 5", () => { + const T = Type.Integer(); + const R = Value.Convert(T, 3.14); + Assert.IsEqual(R, 3); + }); + // ---------------------------------------------------------- + // Casts + // ---------------------------------------------------------- + it("Should Convert 6", () => { + const value = "hello"; + const result = Value.Convert(Type.Integer(), value); + Assert.IsEqual(result, "hello"); + }); + it("Should Convert 7", () => { + const value = "3.14"; + const result = Value.Convert(Type.Integer(), value); + Assert.IsEqual(result, 3); + }); + it("Should Convert 8", () => { + const value = "-0"; + const result = Value.Convert(Type.Integer(), value); + Assert.IsEqual(result, -0); + }); + it("Should Convert 9", () => { + const value = "-100"; + const result = Value.Convert(Type.Integer(), value); + Assert.IsEqual(result, -100); + }); + it("Should Convert 10", () => { + const value = 42; + const result = Value.Convert(Type.Integer(), value); + Assert.IsEqual(result, 42); + }); + it("Should Convert 11", () => { + const value = true; + const result = Value.Convert(Type.Integer(), value); + Assert.IsEqual(result, 1); + }); + it("Should Convert 12", () => { + const value = false; + const result = Value.Convert(Type.Integer(), value); + Assert.IsEqual(result, 0); + }); + it("Should Convert 13", () => { + const value = {}; + const result = Value.Convert(Type.Integer(), value); + Assert.IsEqual(result, {}); + }); + it("Should Convert 14", () => { + const value = [] as any[]; + const result = Value.Convert(Type.Integer(), value); + Assert.IsEqual(result, []); + }); + // ---------------------------------------------------------- + // https://github.com/sinclairzx81/typebox/issues/1147 + // ---------------------------------------------------------- + it("Should Convert 15", () => { + const N = 1738213389080; + const R = Value.Convert(Type.Integer(), N); + Assert.IsEqual(R, N); + }); + it("Should Convert 16", () => { + const N = 1738213389080.5555; + const R = Value.Convert(Type.Integer(), N); + Assert.IsEqual(R, 1738213389080); + }); + it("Should Convert 17", () => { + const N = "1738213389080"; + const R = Value.Convert(Type.Integer(), N); + Assert.IsEqual(R, 1738213389080); + }); + it("Should Convert 18", () => { + const N = "1738213389080.555"; + const R = Value.Convert(Type.Integer(), N); + Assert.IsEqual(R, 1738213389080); + }); +}); + +describe("Value.Convert.Intersect", () => { + it("Should Convert intersected objects", () => { + const T = Type.Intersect([Type.Object({ x: Type.Number() }), Type.Object({ y: Type.Number() })]); + const R = Value.Convert(T, { x: "1", y: "2" }); + Assert.IsEqual(R, { x: 1, y: 2 }); + }); + // ---------------------------------------------------------------- + // Intersection Complex + // ---------------------------------------------------------------- + it("Should complex intersect 1", () => { + const T = Type.Intersect([Type.Number(), Type.Object({ x: Type.Number() }), Type.Object({ y: Type.Number() })]); + const R = Value.Convert(T, { x: "1", y: "2" }); + Assert.IsEqual(R, { x: 1, y: 2 }); + }); + it("Should complex intersect 2", () => { + const T = Type.Intersect([Type.Object({ x: Type.Number() }), Type.Object({ y: Type.Number() }), Type.Number()]); + const R = Value.Convert(T, { x: "3", y: "4" }); + Assert.IsEqual(R, { x: 3, y: 4 }); + }); + // Differs from TypeBox: allOf converts with each member in turn; TypeBox first merges an intersection into one object and drops the number. + it("Should complex intersect 3", () => { + const T = Type.Intersect([ + Type.Number(), // lost in Evaluate + Type.Object({ x: Type.Number() }), + Type.Object({ y: Type.Number() }), + ]); + const R = Value.Convert(T, "123"); + Assert.IsEqual(R, 123); + }); +}); + +describe("Value.Convert.Literal", () => { + it("LiteralString: Should convert 1", () => { + const T = Type.Literal("1"); + const R = Value.Convert(T, 1); + Assert.IsEqual(R, "1"); + }); + it("LiteralString: Should convert 2", () => { + const T = Type.Literal("1"); + const R = Value.Convert(T, 2); + Assert.IsEqual(R, 2); + }); + it("LiteralString: Should convert 3", () => { + const T = Type.Literal("true"); + const R = Value.Convert(T, true); + Assert.IsEqual(R, "true"); + }); + // ------------------------------------------------------------------ + // LiteralNumber + // ------------------------------------------------------------------ + it("LiteralNumber: Should convert 1", () => { + const T = Type.Literal(3.14); + const R = Value.Convert(T, "3.14"); + Assert.IsEqual(R, 3.14); + }); + it("LiteralNumber: Should convert 2", () => { + const T = Type.Literal(3.14); + const R = Value.Convert(T, "3.15"); + Assert.IsEqual(R, "3.15"); + }); + it("LiteralNumber: Should convert 3", () => { + const T = Type.Literal(1); + const R = Value.Convert(T, true); + Assert.IsEqual(R, 1); + }); + it("LiteralNumber: Should convert 4", () => { + const T = Type.Literal(0); + const R = Value.Convert(T, false); + Assert.IsEqual(R, 0); + }); + it("LiteralNumber: Should convert 5", () => { + const T = Type.Literal(2); + const R = Value.Convert(T, true); + Assert.IsEqual(R, true); + }); + // ------------------------------------------------------------------ + // LiteralBoolean + // ------------------------------------------------------------------ + it("LiteralBoolean: Should convert 1", () => { + const T = Type.Literal(true); + const R = Value.Convert(T, 3.14); + Assert.IsEqual(R, 3.14); + }); + it("LiteralBoolean: Should convert 2", () => { + const T = Type.Literal(true); + const R = Value.Convert(T, 1); + Assert.IsEqual(R, true); + }); + it("LiteralBoolean: Should convert 3", () => { + const T = Type.Literal(true); + const R = Value.Convert(T, "true"); + Assert.IsEqual(R, true); + }); + it("LiteralBoolean: Should convert 4", () => { + const T = Type.Literal(false); + const R = Value.Convert(T, "false"); + Assert.IsEqual(R, false); + }); + it("LiteralBoolean: Should convert 5", () => { + const T = Type.Literal(true); + const R = Value.Convert(T, "1"); + Assert.IsEqual(R, true); + }); + it("LiteralBoolean: Should convert 6", () => { + const T = Type.Literal(false); + const R = Value.Convert(T, "0"); + Assert.IsEqual(R, false); + }); + it("LiteralBoolean: Should convert 7", () => { + const T = Type.Literal(false); + const R = Value.Convert(T, "2"); + Assert.IsEqual(R, "2"); + }); + // ------------------------------------------------------------------ + // LiteralBigInt + // ------------------------------------------------------------------ + // Skipped: the schema or the value has no JSON form. + it.skip("LiteralBigInt: Should convert 1", () => { + const T = Type.Literal(100n); + const R = Value.Convert(T, 100n); + Assert.IsEqual(R, 100n); + }); + // Skipped: the schema or the value has no JSON form. + it.skip("LiteralBigInt: Should convert 2", () => { + const T = Type.Literal(100n); + const R = Value.Convert(T, 100); + Assert.IsEqual(R, 100n); + }); + // Skipped: the schema or the value has no JSON form. + it.skip("LiteralBigInt: Should convert 3", () => { + const T = Type.Literal(1n); + const R = Value.Convert(T, "true"); + Assert.IsEqual(R, 1n); + }); + // Skipped: the schema or the value has no JSON form. + it.skip("LiteralBigInt: Should convert 4", () => { + const T = Type.Literal(0n); + const R = Value.Convert(T, "false"); + Assert.IsEqual(R, 0n); + }); + // Skipped: the schema or the value has no JSON form. + it.skip("LiteralBigInt: Should convert 5", () => { + const T = Type.Literal(1n); + const R = Value.Convert(T, "1"); + Assert.IsEqual(R, 1n); + }); + // Skipped: the schema or the value has no JSON form. + it.skip("LiteralBigInt: Should convert 6", () => { + const T = Type.Literal(0n); + const R = Value.Convert(T, "0"); + Assert.IsEqual(R, 0n); + }); + // Skipped: the schema or the value has no JSON form. + it.skip("LiteralBigInt: Should convert 7", () => { + const T = Type.Literal(100n); + const R = Value.Convert(T, "2"); + Assert.IsEqual(R, "2"); + }); + // ------------------------------------------------------------------ + // LiteralBigInt + // ------------------------------------------------------------------ + // Skipped: the schema or the value has no JSON form. + it.skip("LiteralBigInt: Should convert 1", () => { + const T = Type.Literal(100n); + const R = Value.Convert(T, 100n); + Assert.IsEqual(R, 100n); + }); + // Skipped: the schema or the value has no JSON form. + it.skip("LiteralBigInt: Should convert 2", () => { + const T = Type.Literal(100n); + const R = Value.Convert(T, 100); + Assert.IsEqual(R, 100n); + }); + // Skipped: the schema or the value has no JSON form. + it.skip("LiteralBigInt: Should convert 3", () => { + const T = Type.Literal(1n); + const R = Value.Convert(T, "true"); + Assert.IsEqual(R, 1n); + }); + // Skipped: the schema or the value has no JSON form. + it.skip("LiteralBigInt: Should convert 4", () => { + const T = Type.Literal(0n); + const R = Value.Convert(T, "false"); + Assert.IsEqual(R, 0n); + }); + // Skipped: the schema or the value has no JSON form. + it.skip("LiteralBigInt: Should convert 5", () => { + const T = Type.Literal(1n); + const R = Value.Convert(T, "1"); + Assert.IsEqual(R, 1n); + }); + // Skipped: the schema or the value has no JSON form. + it.skip("LiteralBigInt: Should convert 6", () => { + const T = Type.Literal(0n); + const R = Value.Convert(T, "0"); + Assert.IsEqual(R, 0n); + }); + // Skipped: the schema or the value has no JSON form. + it.skip("LiteralBigInt: Should convert 7", () => { + const T = Type.Literal(100n); + const R = Value.Convert(T, "2"); + Assert.IsEqual(R, "2"); + }); +}); + +describe("Value.Convert.Never", () => { + it("Should not 1", () => { + const T = Type.Never(); + const R = Value.Convert(T, true); + Assert.IsEqual(R, true); + }); + it("Should not 2", () => { + const T = Type.Never(); + const R = Value.Convert(T, 42); + Assert.IsEqual(R, 42); + }); + it("Should not 3", () => { + const T = Type.Never(); + const R = Value.Convert(T, "true"); + Assert.IsEqual(R, "true"); + }); +}); + +describe("Value.Convert.Null", () => { + it("Should Convert 1", () => { + const T = Type.Null(); + const R = Value.Convert(T, "null"); + Assert.IsEqual(R, null); + }); + it("Should Convert 2", () => { + const T = Type.Null(); + const R = Value.Convert(T, "NULL"); + Assert.IsEqual(R, null); + }); + it("Should Convert 3", () => { + const T = Type.Null(); + const R = Value.Convert(T, "nil"); + Assert.IsEqual(R, "nil"); + }); + it("Should Convert 4", () => { + const T = Type.Null(); + const R = Value.Convert(T, "undefined"); + Assert.IsEqual(R, null); + }); + it("Should Convert 5", () => { + const T = Type.Null(); + const R = Value.Convert(T, "hello"); + Assert.IsEqual(R, "hello"); + }); + it("Should Convert 6", () => { + const T = Type.Null(); + const R = Value.Convert(T, null); + Assert.IsEqual(R, null); + }); + it("Should Convert 7", () => { + const T = Type.Null(); + const R = Value.Convert(T, 0); + Assert.IsEqual(R, null); + }); + it("Should Convert 8", () => { + const T = Type.Null(); + const R = Value.Convert(T, false); + Assert.IsEqual(R, null); + }); + it("Should Convert 9", () => { + const T = Type.Null(); + const R = Value.Convert(T, ""); + Assert.IsEqual(R, null); + }); + // ------------------------------------------------------------------ + // Coverage + // ------------------------------------------------------------------ + it("Should Convert 10", () => { + const T = Type.Null(); + const R = Value.Convert(T, BigInt(0)); + Assert.IsEqual(R, null); + }); + it("Should Convert 11", () => { + const T = Type.Null(); + const R = Value.Convert(T, BigInt(1)); + Assert.IsEqual(R, BigInt(1)); + }); + it("Should Convert 12", () => { + const T = Type.Null(); + const R = Value.Convert(T, false); + Assert.IsEqual(R, null); + }); + it("Should Convert 13", () => { + const T = Type.Null(); + const R = Value.Convert(T, true); + Assert.IsEqual(R, true); + }); + it("Should Convert 14", () => { + const T = Type.Null(); + const R = Value.Convert(T, 0); + Assert.IsEqual(R, null); + }); + it("Should Convert 15", () => { + const T = Type.Null(); + const R = Value.Convert(T, 1); + Assert.IsEqual(R, 1); + }); + it("Should Convert 16", () => { + const T = Type.Null(); + const R = Value.Convert(T, null); + Assert.IsEqual(R, null); + }); + it("Should Convert 17", () => { + const T = Type.Null(); + const R = Value.Convert(T, undefined); + Assert.IsEqual(R, null); + }); + // Skipped: the schema or the value has no JSON form. + it.skip("Should Convert 18", () => { + const T = Type.Null(); + const S = Symbol("hello"); + const R = Value.Convert(T, S); + Assert.IsEqual(R, S); + }); +}); + +describe("Value.Convert.Number", () => { + it("Should Convert 1", () => { + const T = Type.Number(); + const R = Value.Convert(T, "3.14"); + Assert.IsEqual(R, 3.14); + }); + it("Should Convert 2", () => { + const T = Type.Number(); + const R = Value.Convert(T, "42"); + Assert.IsEqual(R, 42); + }); + it("Should Convert 3", () => { + const T = Type.Number(); + const R = Value.Convert(T, true); + Assert.IsEqual(R, 1); + }); + it("Should Convert 4", () => { + const T = Type.Number(); + const R = Value.Convert(T, false); + Assert.IsEqual(R, 0); + }); + it("Should Convert 5", () => { + const T = Type.Number(); + const R = Value.Convert(T, 3.14); + Assert.IsEqual(R, 3.14); + }); + // ---------------------------------------------------------- + // Casts + // ---------------------------------------------------------- + it("Should Convert 6", () => { + const T = Type.Number(); + const R = Value.Convert(T, "hello"); + Assert.IsEqual(R, "hello"); + }); + it("Should Convert 7", () => { + const T = Type.Number(); + const R = Value.Convert(T, "3.14"); + Assert.IsEqual(R, 3.14); + }); + it("Should Convert 8", () => { + const T = Type.Number(); + const R = Value.Convert(T, "-0"); + Assert.IsEqual(R, -0); + }); + it("Should Convert 9", () => { + const T = Type.Number(); + const R = Value.Convert(T, "-100"); + Assert.IsEqual(R, -100); + }); + it("Should Convert 10", () => { + const T = Type.Number(); + const R = Value.Convert(T, 42); + Assert.IsEqual(R, 42); + }); + it("Should Convert 11", () => { + const T = Type.Number(); + const R = Value.Convert(T, true); + Assert.IsEqual(R, 1); + }); + it("Should Convert 12", () => { + const T = Type.Number(); + const R = Value.Convert(T, false); + Assert.IsEqual(R, 0); + }); + it("Should Convert 13", () => { + const T = Type.Number(); + const R = Value.Convert(T, {}); + Assert.IsEqual(R, {}); + }); + it("Should Convert 14", () => { + const T = Type.Number(); + const R = Value.Convert(T, []); + Assert.IsEqual(R, []); + }); + // ------------------------------------------------------------------ + // Coverage + // ------------------------------------------------------------------ + it("Should Convert 10", () => { + const T = Type.Number(); + const R = Value.Convert(T, BigInt(0)); + Assert.IsEqual(R, 0); + }); + it("Should Convert 11", () => { + const T = Type.Number(); + const R = Value.Convert(T, BigInt(10000000000000000000000000000000000000000000000000000)); + Assert.IsEqual(R, BigInt(10000000000000000000000000000000000000000000000000000)); + }); + it("Should Convert 12", () => { + const T = Type.Number(); + const R = Value.Convert(T, null); + Assert.IsEqual(R, 0); + }); + it("Should Convert 13", () => { + const T = Type.Number(); + const R = Value.Convert(T, "false"); + Assert.IsEqual(R, 0); + }); + it("Should Convert 14", () => { + const T = Type.Number(); + const R = Value.Convert(T, "true"); + Assert.IsEqual(R, 1); + }); + it("Should Convert 15", () => { + const T = Type.Number(); + const R = Value.Convert(T, "123"); + Assert.IsEqual(R, 123); + }); + it("Should Convert 16", () => { + const T = Type.Number(); + const R = Value.Convert(T, "123.456"); + Assert.IsEqual(R, 123.456); + }); + it("Should Convert 17", () => { + const T = Type.Number(); + const R = Value.Convert(T, "123.456"); + Assert.IsEqual(R, 123.456); + }); + it("Should Convert 18", () => { + const T = Type.Number(); + const R = Value.Convert(T, "123n"); + Assert.IsEqual(R, 123); + }); + it("Should Convert 19", () => { + const T = Type.Number(); + const R = Value.Convert(T, undefined); + Assert.IsEqual(R, 0); + }); + it("Should Convert 20", () => { + const T = Type.Number(); + const R = Value.Convert(T, `90071992547409910n`); // trailing 0 too large + Assert.IsEqual(R, "90071992547409910n"); + }); +}); + +describe("Value.Convert.Object", () => { + it("Should Convert 1", () => { + const T = Type.Object({ + x: Type.Number(), + y: Type.Boolean(), + z: Type.Boolean(), + }); + const R = Value.Convert(T, { x: "42", y: "true", z: "hello" }); + Assert.IsEqual(R, { x: 42, y: true, z: "hello" }); + }); + it("Should Convert 2", () => { + const T = Type.Object({ + x: Type.Number(), + y: Type.Boolean(), + }); + const R = Value.Convert(T, { x: "42", y: "true", z: "hello" }); + Assert.IsEqual(R, { x: 42, y: true, z: "hello" }); + }); + it("Should Convert 3", () => { + const T = Type.Object({ x: Type.Number() }); + const R = Value.Convert(T, {}); + Assert.IsEqual(R, {}); + }); + it("Should Convert 4", () => { + const T = Type.Object( + { x: Type.Number() }, + { + additionalProperties: Type.Null(), + }, + ); + const R = Value.Convert(T, { x: "1", y: 0 }); + Assert.IsEqual(R, { x: 1, y: null }); + }); + // ------------------------------------------------------------------ + // IsOptionalUndefined + // + // https://github.com/sinclairzx81/typebox/issues/1336#issuecomment-3312808962 + // ------------------------------------------------------------------ + it("Should Convert 5", () => { + const T = Type.Object({ x: Type.Optional(Type.Number()) }); + const R = Value.Convert(T, { x: 1 }); + Assert.IsEqual(R, { x: 1 }); + }); + it("Should Convert 6", () => { + const T = Type.Object({ x: Type.Optional(Type.Number()) }); + const R = Value.Convert(T, { x: "1" }); + Assert.IsEqual(R, { x: 1 }); + }); + it("Should Convert 7", () => { + const T = Type.Object({ x: Type.Optional(Type.Number()) }); + const R = Value.Convert(T, { x: undefined }); + Assert.IsEqual(R, { x: undefined }); + }); +}); + +describe("Value.Convert.Record", () => { + it("Should Convert 1", () => { + const T = Type.Record(Type.String(), Type.Number()); + const R = Value.Convert(T, { x: "42", y: "24", z: "hello" }); + Assert.IsEqual(R, { x: 42, y: 24, z: "hello" }); + }); + // ---------------------------------------------------------------- + // https://github.com/sinclairzx81/typebox/issues/930 + // ---------------------------------------------------------------- + it("Should Convert 2", () => { + const T = Type.Union([Type.Null(), Type.Record(Type.Number(), Type.Any())]); + const R = Value.Convert(T, {}); + Assert.IsEqual(R, {}); + }); + it("Should Convert 3", () => { + const T = Type.Union([Type.Record(Type.Number(), Type.Any()), Type.Null()]); + const R = Value.Convert(T, {}); + Assert.IsEqual(R, {}); + }); + it("Should Convert 4", () => { + const T = Type.Union([Type.Null(), Type.Record(Type.Number(), Type.Any())]); + const R = Value.Convert(T, null); + Assert.IsEqual(R, null); + }); + it("Should Convert 5", () => { + const T = Type.Union([Type.Record(Type.Number(), Type.Any()), Type.Null()]); + const R = Value.Convert(T, null); + Assert.IsEqual(R, null); + }); + it("Should Convert 6", () => { + const T = Type.Union([Type.Null(), Type.Record(Type.Number(), Type.Any())]); + const R = Value.Convert(T, "NULL"); + Assert.IsEqual(R, null); + }); + it("Should Convert 7", () => { + const T = Type.Union([Type.Record(Type.Number(), Type.Any()), Type.Null()]); + const R = Value.Convert(T, "NULL"); + Assert.IsEqual(R, null); + }); + it("Should Convert 8", () => { + const T = Type.Record(Type.Number(), Type.Number(), { + additionalProperties: Type.Null(), + }); + const R = Value.Convert(T, { "0": "1", x: 0 }); + Assert.IsEqual(R, { "0": 1, x: null }); + }); +}); + +describe("Value.Convert.String", () => { + it("Should Convert 1", () => { + const T = Type.String(); + const R = Value.Convert(T, 3.14); + Assert.IsEqual(R, "3.14"); + }); + it("Should Convert 2", () => { + const T = Type.String(); + const R = Value.Convert(T, 3); + Assert.IsEqual(R, "3"); + }); + it("Should Convert 3", () => { + const T = Type.String(); + const R = Value.Convert(T, true); + Assert.IsEqual(R, "true"); + }); + it("Should Convert 4", () => { + const T = Type.String(); + const R = Value.Convert(T, false); + Assert.IsEqual(R, "false"); + }); + it("Should Convert 5", () => { + const T = Type.String(); + const R = Value.Convert(T, BigInt(12345)); + Assert.IsEqual(R, "12345"); + }); + // ---------------------------------------------------------- + // Casts + // ---------------------------------------------------------- + it("Should Convert 6", () => { + const T = Type.String(); + const R = Value.Convert(T, "hello"); + Assert.IsEqual(R, "hello"); + }); + it("Should Convert 7", () => { + const T = Type.String(); + const R = Value.Convert(T, 42); + Assert.IsEqual(R, "42"); + }); + it("Should Convert 8", () => { + const T = Type.String(); + const R = Value.Convert(T, 42n); + Assert.IsEqual(R, "42"); + }); + it("Should Convert 9", () => { + const T = Type.String(); + const R = Value.Convert(T, true); + Assert.IsEqual(R, "true"); + }); + it("Should Convert 10", () => { + const T = Type.String(); + const R = Value.Convert(T, false); + Assert.IsEqual(R, "false"); + }); + it("Should Convert 11", () => { + const T = Type.String(); + const R = Value.Convert(T, {}); + Assert.IsEqual(R, {}); + }); + it("Should Convert 12", () => { + const T = Type.String(); + const R = Value.Convert(T, []); + Assert.IsEqual(R, []); + }); + // ------------------------------------------------------------------ + // Coverage + // ------------------------------------------------------------------ + it("Should Convert 13", () => { + const T = Type.String(); + const R = Value.Convert(T, null); + Assert.IsEqual(R, "null"); + }); + it("Should Convert 14", () => { + const T = Type.String(); + const R = Value.Convert(T, undefined); + Assert.IsEqual(R, ""); + }); +}); + +describe("Value.Convert.TemplateLiteral", () => { + // Differs from TypeBox: a template literal is a string with a pattern in JSON Schema; TypeBox converts a finite one as a union of literals. + it("Should Convert 1", () => { + const T = Type.TemplateLiteral("${1 | 2 | 3}"); + const R0 = Value.Convert(T, 1); + const R1 = Value.Convert(T, 2); + const R2 = Value.Convert(T, 3); + const R3 = Value.Convert(T, 4); + Assert.IsEqual(R0, "1"); + Assert.IsEqual(R1, "2"); + Assert.IsEqual(R2, "3"); + Assert.IsEqual(R3, "4"); + }); + // ------------------------------------------------------------------ + // https://github.com/sinclairzx81/typebox/issues/1510 + // ------------------------------------------------------------------ + it("Should Convert 2", () => { + const T = Type.Object({ + test: Type.TemplateLiteral('${number}${"a"|"b"|"c"}'), + }); + const R0 = Value.Convert(T, { test: "123b" }); + Assert.IsEqual(R0, { test: "123b" }); + }); +}); + +describe("Value.Convert.Tuple", () => { + it("Should Convert 1", () => { + const T = Type.Tuple([Type.Number(), Type.Number()]); + const R = Value.Convert(T, ["1", "true"]); + Assert.IsEqual(R, [1, 1]); + }); + it("Should Convert 2", () => { + const T = Type.Tuple([Type.Number(), Type.Number()]); + const R = Value.Convert(T, ["1"]); + Assert.IsEqual(R, [1]); + }); + it("Should Convert 3", () => { + const T = Type.Tuple([Type.Number(), Type.Number()]); + const R = Value.Convert(T, ["1", "2", "3"]); + Assert.IsEqual(R, [1, 2, "3"]); + }); + it("Should Convert 4", () => { + const T = Type.Tuple([Type.Number(), Type.Number()]); + const R = Value.Convert(T, "not an array"); + Assert.IsEqual(R, "not an array"); + }); + it("Should Convert 5", () => { + const T = Type.Tuple([Type.Number(), Type.Number()]); + const R = Value.Convert(T, 42); + Assert.IsEqual(R, 42); + }); + it("Should Convert 6", () => { + const T = Type.Tuple([Type.Number(), Type.Number()]); + const R = Value.Convert(T, null); + Assert.IsEqual(R, null); + }); + it("Should Convert 7", () => { + const T = Type.Tuple([Type.Number(), Type.Number()]); + const R = Value.Convert(T, { 0: "1", 1: "2" }); + Assert.IsEqual(R, { 0: "1", 1: "2" }); + }); +}); + +describe("Value.Convert.Union", () => { + it("Should Convert 1", () => { + const T = Type.Object({ + x: Type.Union([Type.Number(), Type.Null()]), + }); + const R0 = Value.Convert(T, { x: "42" }); + const R1 = Value.Convert(T, { x: "null" }); + const R2 = Value.Convert(T, { x: "hello" }); + Assert.IsEqual(R0, { x: 42 }); + Assert.IsEqual(R1, { x: null }); + Assert.IsEqual(R2, { x: "hello" }); + }); + it("Should Convert 2", () => { + const T = Type.Object({ + x: Type.Union([Type.Boolean(), Type.Number()]), + }); + const R = Value.Convert(T, { x: "1" }); + Assert.IsEqual(R, { x: true }); + }); + + // ---------------------------------------------------------------- + // https://github.com/sinclairzx81/typebox/issues/787 + // ---------------------------------------------------------------- + it("Should Convert 3", () => { + const T = Type.Intersect([ + Type.Union([Type.Object({ a: Type.Number() }), Type.Object({ b: Type.Number() })]), + Type.Object({ c: Type.Number() }), + ]); + const R0 = Convert(T, { a: "1", c: "2" }); + const R1 = Convert(T, { b: "1", c: "2" }); + const R2 = Convert(T, { a: "1", b: "2", c: "3" }); + Assert.IsEqual(R0, { a: 1, c: 2 }); + Assert.IsEqual(R1, { b: 1, c: 2 }); + Assert.IsEqual(R2, { a: 1, b: "2", c: 3 }); // note: matching on first + }); + // ---------------------------------------------------------------- + // https://github.com/sinclairzx81/typebox/issues/1295 + // ---------------------------------------------------------------- + it("Should guard against Array conversion in Object", () => { + const T = Type.Union([ + Type.Object({ + type: Type.Literal("A"), + values: Type.Union([Type.String(), Type.Number()]), + }), + Type.Object({ + type: Type.Literal("B"), + values: Type.String(), + }), + ]); + const converted = Value.Convert(T, [{ type: "A", values: 1 }]); + Assert.IsEqual(converted, [{ type: "A", values: 1 }]); + }); + it("Should guard against Array conversion in Record", () => { + const T = Type.Union([ + Type.Record(Type.String({ pattern: "^values$" }), Type.Union([Type.String(), Type.Number()])), + Type.Record(Type.String({ pattern: "^type$" }), Type.Union([Type.String(), Type.Number()])), + ]); + const converted = Value.Convert(T, [{ type: "A", values: 1 }]); + Assert.IsEqual(converted, [{ type: "A", values: 1 }]); + }); +}); + +describe("Value.Convert.Unknown", () => { + it("Should Convert 1", () => { + const V = null; + const T = Type.Unknown(); + const R = Value.Convert(T, V); + Assert.IsEqual(R, V); + }); + it("Should Convert 2", () => { + const V = undefined; + const T = Type.Unknown(); + const R = Value.Convert(T, V); + Assert.IsEqual(R, V); + }); + it("Should Convert 3", () => { + const V = "hello"; + const T = Type.Unknown(); + const R = Value.Convert(T, V); + Assert.IsEqual(R, V); + }); + it("Should Convert 4", () => { + const V = 42; + const T = Type.Unknown(); + const R = Value.Convert(T, V); + Assert.IsEqual(R, V); + }); + it("Should Convert 5", () => { + const T = Type.Unknown(); + const R = Value.Convert(T, true); + Assert.IsEqual(R, true); + }); + it("Should Convert 6", () => { + const T = Type.Unknown(); + const R = Value.Convert(T, { x: 1 }); + Assert.IsEqual(R, { x: 1 }); + }); + it("Should Convert 7", () => { + const T = Type.Unknown(); + const R = Value.Convert(T, [1, 2, 3]); + Assert.IsEqual(R, [1, 2, 3]); + }); +}); diff --git a/packages/schema/test/coerce.test.ts b/packages/schema/test/coerce.test.ts new file mode 100644 index 000000000..4444f0cc9 --- /dev/null +++ b/packages/schema/test/coerce.test.ts @@ -0,0 +1,458 @@ +// compile(schema, { coerce: true }): coerce() and parse(), the three coercion passes (optional nulls, TypeBox's +// conversion rules, primitive coercion by type), their order where they conflict, and the guarantees around them. +// Ports pi-ai's tool argument coercion tests. + +import Type from "typebox"; +import { describe, expect, expectTypeOf, it } from "vitest"; +import { formats } from "../src/formats/index.ts"; +import { + arr, + compile, + int, + map, + nullable, + num, + obj, + opt, + str, + tuple, + type ValidationError, + type Validator, +} from "../src/index.ts"; + +function coerce(schema: object, value: unknown): unknown { + return compile(schema, { formats, coerce: true }).coerce(value); +} + +/** The coerced `value` property of `{ value }` under an object schema with one required property `value`. */ +function coerceValue(schema: unknown, value: unknown): unknown { + const root = { type: "object", properties: { value: schema }, required: ["value"] }; + return (coerce(root, { value }) as { value: unknown }).value; +} + +describe("compile options", () => { + it("adds coerce and parse only with coerce: true", () => { + const plain = compile(obj({ n: num() })); + expectTypeOf(plain).toEqualTypeOf>(); + expect("coerce" in plain).toBe(false); + expect("parse" in plain).toBe(false); + const coercing = compile(obj({ n: num() }), { coerce: true }); + expectTypeOf(coercing.parse).returns.toEqualTypeOf< + | { readonly ok: true; readonly value: { n: number } } + | { readonly ok: false; readonly errors: ValidationError[] } + >(); + expect(coercing.check({ n: 1 })).toBe(true); + }); +}); + +describe("coerce", () => { + it("never modifies the value", () => { + const value = { count: "42", tags: "a" }; + const schema = obj({ count: num(), tags: arr(str()) }); + expect(coerce(schema, value)).toEqual({ count: 42, tags: ["a"] }); + expect(value).toEqual({ count: "42", tags: "a" }); + }); + + it("returns the value itself when nothing changes", () => { + const value = { a: { b: [1, 2] }, c: "x" }; + const schema = obj({ a: obj({ b: arr(num()) }), c: str() }); + expect(coerce(schema, value)).toBe(value); + }); + + it("copies only the changed paths", () => { + const unchanged = { b: [1, 2] }; + const value = { a: unchanged, c: { d: "1" } }; + const schema = obj({ a: obj({ b: arr(num()) }), c: obj({ d: num() }) }); + const coerced = coerce(schema, value) as typeof value; + expect(coerced).toEqual({ a: { b: [1, 2] }, c: { d: 1 } }); + expect(coerced).not.toBe(value); + expect(coerced.a).toBe(unchanged); + expect(value.c).toEqual({ d: "1" }); + }); + + it("works on frozen schemas", () => { + const schema = Object.freeze({ + type: "object", + properties: Object.freeze({ n: Object.freeze({ type: "integer" }) }), + }); + expect(coerce(schema, { n: "7" })).toEqual({ n: 7 }); + }); + + it("still coerces when the Function constructor is unavailable", () => { + const original = globalThis.Function; + globalThis.Function = (() => { + throw new EvalError("Code generation from strings disallowed for this context"); + }) as unknown as FunctionConstructor; + try { + expect(coerce(obj({ count: num() }), { count: "42" })).toEqual({ count: 42 }); + } finally { + globalThis.Function = original; + } + }); + + it("leaves values it cannot coerce for the check to report", () => { + const validator = compile(obj({ count: num() }), { coerce: true }); + const coerced = validator.coerce({ count: "many" }); + expect(coerced).toEqual({ count: "many" }); + expect(validator.check(coerced)).toBe(false); + }); +}); + +describe("parse", () => { + it("returns the coerced value when it is valid", () => { + const validator = compile(obj({ count: num(), tags: arr(str()) }), { coerce: true }); + const result = validator.parse({ count: "42", tags: "a" }); + expect(result).toEqual({ ok: true, value: { count: 42, tags: ["a"] } }); + if (result.ok) expectTypeOf(result.value).toEqualTypeOf<{ count: number; tags: string[] }>(); + }); + + it("returns the errors of the coerced value when it is invalid", () => { + const validator = compile(obj({ count: num(), name: str() }), { coerce: true }); + expect(validator.parse({ count: "many", name: 7 })).toEqual({ + ok: false, + errors: [{ path: "/count", keyword: "type", message: "expected number, got string" }], + }); + }); +}); + +describe("pass 1: optional nulls", () => { + it("drops null in optional non-nullable properties, also nested", () => { + const schema = obj( + { + path: str(), + offset: opt(num()), + nullable: opt(nullable(str())), + metadata: obj({ enabled: opt({ type: "boolean" }) }), + }, + { open: true }, + ); + expect(coerce(schema, { path: "file.txt", offset: null, nullable: null, metadata: { enabled: null } })).toEqual({ + path: "file.txt", + nullable: null, + metadata: {}, + }); + }); + + it("keeps a null whose property is a $ref (the reference may allow it)", () => { + const schema = { + type: "object", + properties: { value: { $ref: "#/$defs/value" } }, + $defs: { value: { anyOf: [{ type: "number" }, { type: "null" }] } }, + }; + expect(coerce(schema, { value: null })).toEqual({ value: null }); + }); + + it("walks array items", () => { + const schema = { type: "array", items: obj({ a: str(), b: opt(int()) }) }; + expect(coerce(schema, [{ a: "x", b: null }])).toEqual([{ a: "x" }]); + }); +}); + +describe("pass 2: TypeBox's conversion rules on JSON Schema", () => { + it("converts primitives by type", () => { + expect(coerceValue({ type: "number" }, "42")).toBe(42); + expect(coerceValue({ type: "number" }, "")).toBe(0); + expect(coerceValue({ type: "number" }, "true")).toBe(1); + expect(coerceValue({ type: "integer" }, "3.5")).toBe(3); + expect(coerceValue({ type: "integer" }, 1.5)).toBe(1); + expect(coerceValue({ type: "boolean" }, "1")).toBe(true); + expect(coerceValue({ type: "boolean" }, "FALSE")).toBe(false); + expect(coerceValue({ type: "null" }, "null")).toBe(null); + expect(coerceValue({ type: "string" }, 42)).toBe("42"); + }); + + it("wraps a single value into an array and converts the items", () => { + expect(coerceValue({ type: "array", items: { type: "number" } }, "1")).toEqual([1]); + expect(coerceValue({ type: "array", items: { type: "string" } }, null)).toEqual(["null"]); + }); + + it("converts tuple items in place and never wraps for a tuple", () => { + const tuple = { type: "array", prefixItems: [{ type: "number" }, { type: "boolean" }], items: false }; + expect(coerceValue(tuple, ["1", "true"])).toEqual([1, true]); + expect(coerceValue(tuple, "1")).toBe("1"); + }); + + it("converts const and enum values only when the result is a member", () => { + expect(coerceValue({ const: 1 }, "1")).toBe(1); + expect(coerceValue({ const: 1 }, "2")).toBe("2"); + expect(coerceValue({ enum: [1, 2, "hello"] }, "2")).toBe(2); + expect(coerceValue({ enum: [1, 2, "hello"] }, "world")).toBe("world"); + }); + + it("keeps a value one union branch accepts, else takes the first branch whose conversion the union accepts", () => { + const union = { anyOf: [{ type: "number" }, { type: "null" }] }; + expect(coerceValue(union, null)).toBe(null); + expect(coerceValue(union, "42")).toBe(42); + expect(coerceValue(union, "null")).toBe(null); + expect(coerceValue({ oneOf: [{ type: "boolean" }, { type: "number" }] }, "1")).toBe(true); + }); + + it("converts allOf with each member in turn", () => { + const schema = { allOf: [obj({ a: num() }, { open: true }), obj({ b: { type: "boolean" } }, { open: true })] }; + expect(coerce(schema, { a: "1", b: "true" })).toEqual({ a: 1, b: true }); + }); + + it("follows $ref", () => { + const schema = { + type: "object", + properties: { item: { $ref: "#/$defs/item" } }, + $defs: { item: { type: "object", properties: { id: { type: "integer" } } } }, + }; + expect(coerce(schema, { item: { id: "7" } })).toEqual({ item: { id: 7 } }); + }); + + it("converts patternProperties and additionalProperties", () => { + const schema = { + type: "object", + properties: { a: { type: "number" } }, + patternProperties: { "^n_": { type: "number" } }, + additionalProperties: { type: "boolean" }, + }; + expect(coerce(schema, { a: "1", n_x: "2", other: "true" })).toEqual({ a: 1, n_x: 2, other: true }); + }); +}); + +describe("pass 3: primitive coercion by type", () => { + it("coerces like Ajv's coerceTypes", () => { + const cases: Array<[unknown, unknown, unknown]> = [ + [{ type: "number" }, true, 1], + [{ type: "number" }, null, 0], + [{ type: "integer" }, "42", 42], + [{ type: "boolean" }, "true", true], + [{ type: "boolean" }, 0, false], + [{ type: "string" }, true, "true"], + [{ type: "null" }, "", null], + [{ type: "null" }, 0, null], + [{ type: "null" }, false, null], + ]; + for (const [schema, input, expected] of cases) expect(coerceValue(schema, input)).toEqual(expected); + }); + + it("leaves a value that matches one of a type array's types, else takes the first coercible type", () => { + expect(coerceValue({ type: ["number", "string"] }, "1")).toBe("1"); + expect(coerceValue({ type: ["boolean", "number"] }, "1")).toBe(1); + expect(coerceValue({ type: ["integer", "null"] }, "5")).toBe(5); + }); + + it("accepts null for nullable array schemas with items", () => { + expect(coerceValue({ type: ["array", "null"], items: { type: "string" } }, null)).toBe(null); + }); + + it("coerces nullable unions when the value matches no branch", () => { + expect(coerceValue({ anyOf: [{ type: "number" }, { type: "null" }] }, "42")).toBe(42); + expect(coerceValue({ oneOf: [{ type: "number" }, { type: "null" }] }, null)).toBe(null); + }); + + it("leaves values no rule coerces", () => { + const validator = compile( + { type: "object", properties: { value: { type: "integer" } }, required: ["value"] }, + { coerce: true }, + ); + for (const input of ["abc", "4x", {}, ["1", "2"]]) { + const coerced = validator.coerce({ value: input }); + expect(validator.check(coerced)).toBe(false); + } + }); +}); + +describe("where the passes conflict, the conversion rules win", () => { + it("turns null into 'null' for a required string (pass 3 alone would give '')", () => { + expect(coerceValue({ type: "string" }, null)).toBe("null"); + }); + + it("turns '' into 0 for a number (pass 3 leaves an empty string)", () => { + expect(coerceValue({ type: "number" }, "")).toBe(0); + }); + + it("turns '1' and '0' into booleans (pass 3 takes only 'true', 'false', 1 and 0)", () => { + expect(coerceValue({ type: "boolean" }, "0")).toBe(false); + }); + + it("truncates '42.1' for an integer (pass 3 rejects it)", () => { + expect(coerceValue({ type: "integer" }, "42.1")).toBe(42); + }); + + it("drops an optional null before converting it (pass 2 alone would give 'null')", () => { + expect(coerce(obj({ name: opt(str()) }), { name: null })).toEqual({}); + }); +}); + +describe("recursive schemas", () => { + const tree = { + type: "object", + properties: { t: { $ref: "#/$defs/Tree" } }, + $defs: { Tree: { anyOf: [{ type: "string" }, { type: "array", items: { $ref: "#/$defs/Tree" } }] } }, + }; + const json = { + type: "object", + properties: { data: { $ref: "#/$defs/Json" } }, + required: ["data"], + $defs: { + Json: { + anyOf: [ + { type: "string" }, + { type: "number" }, + { type: "boolean" }, + { type: "array", items: { $ref: "#/$defs/Json" } }, + { type: "object", additionalProperties: { $ref: "#/$defs/Json" } }, + ], + }, + }, + }; + + it("convert through a recursive union by its first accepting branch", () => { + const validator = compile(tree, { coerce: true }); + expect(validator.parse({ t: ["a", ["b"]] })).toEqual({ ok: true, value: { t: ["a", ["b"]] } }); + expect(validator.parse({ t: 5 })).toEqual({ ok: true, value: { t: "5" } }); + expect(validator.parse({ t: [5] })).toEqual({ ok: true, value: { t: ["5"] } }); + expect(compile(json, { coerce: true }).parse({ data: { a: [1, null] } })).toEqual({ + ok: true, + value: { data: { a: [1, "null"] } }, + }); + }); + + it("leave a value a recursive schema cannot convert as it is, for the check to report", () => { + const validator = compile(tree, { coerce: true }); + const result = validator.parse({ t: {} }); + expect(result.ok).toBe(false); + expect(validator.coerce({ t: { a: 1 } })).toEqual({ t: { a: 1 } }); + }); + + it("convert a scalar-or-array schema that refers to itself", () => { + const schema = { + type: "object", + properties: { + value: { anyOf: [{ type: "number" }, { type: "array", items: { $ref: "#/properties/value" } }] }, + }, + required: ["value"], + }; + const validator = compile(schema, { coerce: true }); + expect(validator.parse({ value: "42" })).toEqual({ ok: true, value: { value: 42 } }); + expect(validator.parse({ value: ["1", ["2"]] })).toEqual({ ok: true, value: { value: [1, [2]] } }); + expect(validator.parse({ value: {} }).ok).toBe(false); + }); + + it("stop at a schema that cycles through shared JS objects instead of $ref", () => { + const node: { anyOf: unknown[] } = { anyOf: [{ type: "number" }] }; + node.anyOf.push({ type: "array", items: node }); + const validator = compile(node, { coerce: true }); + expect(validator.parse("42")).toEqual({ ok: true, value: 42 }); + expect(validator.parse(["1", ["2"]])).toEqual({ ok: true, value: [1, [2]] }); + expect(validator.parse({}).ok).toBe(false); + }); + + it("reset the re-entry state between calls and leave the input unchanged", () => { + const validator = compile( + { anyOf: [{ type: "number" }, { type: "array", items: { $ref: "#" } }] }, + { coerce: true }, + ); + for (let i = 0; i < 20; i++) { + expect(validator.parse({}).ok).toBe(false); + expect(validator.parse("42")).toEqual({ ok: true, value: 42 }); + const input = [1, "2", ["3"]]; + expect(validator.parse(input)).toEqual({ ok: true, value: [1, 2, [3]] }); + expect(input).toEqual([1, "2", ["3"]]); + } + }); + + it("convert and explain a deep value that every branch of a recursive union rejects, in linear time", () => { + // Each object branch converts and explains every child: without memoizing per call this is 2^depth work. + const S = { + anyOf: [ + { type: "object", additionalProperties: { $ref: "#/$defs/S" }, required: ["a"] }, + { type: "object", additionalProperties: { $ref: "#/$defs/S" }, required: ["b"] }, + ], + }; + const validator = compile( + { type: "object", properties: { s: { $ref: "#/$defs/S" } }, $defs: { S } }, + { coerce: true }, + ); + let value: unknown = { x: 1 }; + for (let i = 0; i < 40; i++) value = { c: value }; + const result = validator.parse({ s: value }); + expect(result.ok).toBe(false); + const missing = Array.from({ length: 41 }, (_, i) => ({ + path: `/s${"/c".repeat(i)}/a`, + keyword: "required", + message: "missing", + })); + const leaf = { path: `/s${"/c".repeat(40)}/x`, keyword: "type", message: "expected object, got integer" }; + expect(result).toEqual({ ok: false, errors: [...missing, leaf] }); + }); + + it("convert a TypeBox Cyclic union", () => { + const Tree = Type.Cyclic({ Tree: Type.Union([Type.String(), Type.Array(Type.Ref("Tree"))]) }, "Tree"); + const validator = compile(Type.Object({ t: Tree }), { coerce: true }); + expect(validator.parse({ t: [5] })).toEqual({ ok: true, value: { t: ["5"] } }); + expect(validator.parse({ t: { a: 1 } }).ok).toBe(false); + }); +}); + +describe("member keywords across the passes", () => { + it("keep a pattern-matched property from additionalProperties in every pass", () => { + const record = Type.Record(Type.String(), Type.Number(), { additionalProperties: Type.String() }); + expect(compile(record, { coerce: true }).parse({ n1: "1" })).toEqual({ ok: true, value: { n1: 1 } }); + const plain = { + type: "object", + patternProperties: { "^n_": { type: ["integer", "null"] } }, + additionalProperties: { type: "string" }, + }; + expect(coerce(plain, { n_a: "5", other: 7 })).toEqual({ n_a: 5, other: "7" }); + }); + + it("drop optional nulls inside declared, pattern and additional property values alike", () => { + const item = obj({ n: opt(int({ minimum: 1 })) }); + const schemas = [ + obj({ entry: item }), + map(item), + { type: "object", patternProperties: { "^entry$": item }, additionalProperties: false }, + ]; + for (const schema of schemas) { + const input = { entry: { n: null } }; + expect(compile(schema, { coerce: true }).parse(input)).toEqual({ ok: true, value: { entry: {} } }); + expect(input).toEqual({ entry: { n: null } }); + } + }); + + it("keep a null under pattern and additional properties: their keys are not optional properties", () => { + const schema = { type: "object", additionalProperties: { type: ["integer", "null"] } }; + expect(compile(schema, { coerce: true }).parse({ n: null })).toEqual({ ok: true, value: { n: null } }); + }); + + it("drop optional nulls in tuple items as in array items", () => { + const item = obj({ n: opt(int()) }); + expect(compile(arr(item), { coerce: true }).coerce([{ n: null }])).toEqual([{}]); + expect(compile(tuple([item]), { coerce: true }).coerce([{ n: null }])).toEqual([{}]); + }); + + it("coerce tuple items by type in pass 3", () => { + const schema = { type: "array", prefixItems: [{ type: ["integer", "null"] }, { type: ["boolean", "null"] }] }; + expect(coerce(schema, ["5", "true"])).toEqual([5, true]); + }); + + it("convert the properties a union node declares besides its branches", () => { + const schema = { + type: "object", + properties: { a: { type: "integer" }, b: { type: "integer" } }, + anyOf: [{ required: ["a"] }, { required: ["b"] }], + }; + expect(compile(schema, { coerce: true }).parse({ a: "3.5" })).toEqual({ ok: true, value: { a: 3 } }); + }); + + for (const keyword of ["anyOf", "oneOf"]) { + it(`check ${keyword} candidates after the node's own properties convert`, () => { + const schema = { + type: "object", + [keyword]: [ + { type: "object", properties: { a: { type: "array", items: { type: "integer" } } }, required: ["a"] }, + { type: "object", properties: { b: { type: "boolean" } }, required: ["b"] }, + ], + properties: { c: { type: "integer" } }, + required: ["c"], + }; + const input = { a: "3", c: "2" }; + expect(compile(schema, { coerce: true }).parse(input)).toEqual({ ok: true, value: { a: [3], c: 2 } }); + expect(input).toEqual({ a: "3", c: "2" }); + }); + } +}); diff --git a/packages/schema/test/validator.test.ts b/packages/schema/test/validator.test.ts index a62b38e31..c5ebd6edf 100644 --- a/packages/schema/test/validator.test.ts +++ b/packages/schema/test/validator.test.ts @@ -188,3 +188,136 @@ function deepFreeze(value: T): T { } return value; } + +describe("unresolved references", () => { + const dangling = { type: "object", properties: { t: { $ref: "#/definitions/Missing" } } }; + const external = { type: "object", properties: { t: { $ref: "https://example.com/schemas/t.json" } } }; + + it("throw when the schema compiles, by default", () => { + expect(() => compile(dangling)).toThrow("Unresolved $ref #/definitions/Missing"); + expect(() => compile(external)).toThrow("Unresolved $ref https://example.com/schemas/t.json"); + }); + + it("reject only values at their place with unresolvedRefs: reject", () => { + for (const schema of [dangling, external]) { + const validator = compile(schema, { unresolvedRefs: "reject" }); + expect(validator.check({})).toBe(true); + expect(validator.check({ t: 1 })).toBe(false); + } + expect(compile(dangling, { unresolvedRefs: "reject" }).errors({ t: 1 })).toEqual([ + { path: "/t", keyword: "$ref", message: "unresolved reference #/definitions/Missing" }, + ]); + }); + + it("leave values at their place uncoerced", () => { + const validator = compile(dangling, { unresolvedRefs: "reject", coerce: true }); + expect(validator.parse({ t: "1" })).toEqual({ + ok: false, + errors: [{ path: "/t", keyword: "$ref", message: "unresolved reference #/definitions/Missing" }], + }); + }); + + it("reject a TypeBox Ref without its definition", () => { + const validator = compile(Type.Object({ a: Type.Ref("A") }), { unresolvedRefs: "reject" }); + expect(validator.errors({ a: { x: "1" } })).toEqual([ + { path: "/a", keyword: "$ref", message: "unresolved reference A" }, + ]); + }); +}); + +describe("string lengths", () => { + it("count an unpaired high surrogate as one code point and a pair as one", () => { + expect(compile({ type: "string", minLength: 2 }).check("\ud800A")).toBe(true); + expect(compile({ type: "string", maxLength: 1 }).check("\ud800A")).toBe(false); + expect(compile({ type: "string", maxLength: 1 }).check("\ud800")).toBe(true); + expect(compile({ type: "string", maxLength: 1 }).check("😀")).toBe(true); + expect(compile({ type: "string", minLength: 2 }).check("😀")).toBe(false); + expect(compile({ type: "string", minLength: 2 }).check("\udc00\ud800")).toBe(true); + }); +}); + +describe("error messages", () => { + it("list at most 20 enum values", () => { + const values = Array.from({ length: 5000 }, (_, i) => `v${i}`); + const validator = compile({ type: "object", properties: { e: { type: "string", enum: values } } }); + expect(validator.check({ e: "v4999" })).toBe(true); + const [error] = validator.errors({ e: "v5000" }); + expect(error.message).toBe( + `expected one of ${JSON.stringify(values.slice(0, 20)).slice(0, -1)}, ... and 4980 more]`, + ); + expect(compile({ enum: ["a", "b"] }).errors("c")).toEqual([ + { path: "", keyword: "enum", message: 'expected one of ["a","b"]' }, + ]); + }); + + it("accept a 10,000-value enum", () => { + const values = Array.from({ length: 10_000 }, (_, i) => `member${i}`); + expect(compile({ enum: values }).check("member9999")).toBe(true); + }); + + it("describe contains bounds in words", () => { + const at = (schema: object) => compile({ type: "array", ...schema }).errors(["a"])[0]?.message; + expect(at({ contains: { type: "number" } })).toBe("expected at least 1 matching item"); + expect(at({ contains: { type: "number" }, minContains: 2 })).toBe("expected at least 2 matching items"); + expect(at({ contains: { type: "string" }, maxContains: 0, minContains: 0 })).toBe( + "expected at most 0 matching items", + ); + expect(at({ contains: { type: "number" }, minContains: 1, maxContains: 3 })).toBe( + "expected between 1 and 3 matching items", + ); + }); + + it("blame a oneOf failure on the branch that declares most of the object's properties", () => { + const filter = { + type: "object", + properties: { where: { $ref: "#/$defs/F" } }, + $defs: { + F: { + oneOf: [ + { + type: "object", + properties: { field: { type: "string" }, eq: { type: "string" } }, + required: ["field", "eq"], + additionalProperties: false, + }, + { + type: "object", + properties: { and: { type: "array", items: { $ref: "#/$defs/F" } } }, + required: ["and"], + additionalProperties: false, + }, + ], + }, + }, + }; + expect(compile(filter).errors({ where: { and: [{ field: "a", eq: 1, x: 2 }] } })).toEqual([ + { path: "/where/and/0/eq", keyword: "type", message: "expected string, got integer" }, + { path: "/where/and/0/x", keyword: "additionalProperties", message: "unknown field" }, + ]); + }); + + it("explain one object reached through the same reference at two paths at each path", () => { + const schema = { + type: "object", + properties: { a: { $ref: "#/$defs/P" }, b: { $ref: "#/$defs/P" } }, + $defs: { P: { type: "object", properties: { n: { type: "integer" } } } }, + }; + const shared = { n: "x" }; + expect(compile(schema).errors({ a: shared, b: shared })).toEqual([ + { path: "/a/n", keyword: "type", message: "expected integer, got string" }, + { path: "/b/n", keyword: "type", message: "expected integer, got string" }, + ]); + }); +}); + +describe("TypeBox's extra formats", () => { + it("check url and json-pointer-uri-fragment", () => { + const url = compile({ type: "string", format: "url" }, { formats }); + expect(url.check("https://example.com/a?b=1")).toBe(true); + expect(url.check("not a url")).toBe(false); + const fragment = compile({ type: "string", format: "json-pointer-uri-fragment" }, { formats }); + expect(fragment.check("#/a/b~1c")).toBe(true); + expect(fragment.check("#/a b")).toBe(false); + expect(fragment.check("/a")).toBe(false); + }); +}); diff --git a/scripts/browser-smoke-entry.ts b/scripts/browser-smoke-entry.ts index 4c6d07c38..9cf69450d 100644 --- a/scripts/browser-smoke-entry.ts +++ b/scripts/browser-smoke-entry.ts @@ -1,12 +1,17 @@ import { createHost, memoryLinkPair, replicatedState } from "@earendil-works/chord"; -import { createAssistantMessageEventStream, Type } from "@earendil-works/pi-ai"; +import { createAssistantMessageEventStream, validateToolArguments } from "@earendil-works/pi-ai"; import { complete, getModel, getProviders, streamSimple } from "@earendil-works/pi-ai/compat"; import { Agent, streamProxy } from "@earendil-works/pi-agent-core"; +import { obj, str } from "@earendil-works/pi-schema"; // Keep this entry browser-safe. It is bundled by scripts/check-browser-smoke.mjs // to catch accidental Node-only runtime imports in browser-facing package exports. const model = getModel("google", "gemini-2.5-flash"); -const schema = Type.Object({ prompt: Type.String() }); +const schema = obj({ prompt: str() }); +const args = validateToolArguments( + { name: "prompt", description: "", parameters: schema }, + { type: "toolCall", id: "1", name: "prompt", arguments: { prompt: 1 } }, +); const stream = createAssistantMessageEventStream(); const agent = new Agent({ initialState: { model }, streamFn: streamSimple }); @@ -17,6 +22,7 @@ console.log( getProviders().length, typeof complete, schema.type, + args.prompt, typeof stream.push, agent.hasQueuedMessages(), typeof streamProxy, diff --git a/scripts/check-entry-graphs.mjs b/scripts/check-entry-graphs.mjs index 54dd30c5d..4aea32757 100644 --- a/scripts/check-entry-graphs.mjs +++ b/scripts/check-entry-graphs.mjs @@ -44,10 +44,18 @@ const BUDGETS = { // Request telemetry reaches pi-ai's telemetry.ts and pi-telemetry's root (index, noop, metrics, memory). // Request identity reaches utils/identity.ts and its version constant. maxFiles: 20, - forbid: ["providers/", "models.generated.ts", "index.ts", "utils/validation.ts", "utils/typebox-helpers.ts"], + forbid: ["providers/", "models.generated.ts", "index.ts", "utils/validation.ts", "utils/string-enum.ts"], allow: ["packages/telemetry/src/index.ts"], }, - "./utils/*": { maxFiles: 3, forbid: ["providers/", "api/", "index.ts"] }, + "./utils/*": { + maxFiles: 3, + forbid: ["providers/", "api/", "index.ts"], + overrides: { + // Tool argument validation runs on pi-schema: its validator, coercion and format checks (the checks TypeBox's + // validator ran by default). TypeBox, an external package, was not counted before. + "./utils/validation": { maxFiles: 42, forbid: ["providers/", "api/", "packages/ai/src/index.ts"] }, + }, + }, }, "packages/codemode": { // Node and Bun users never load the Cloudflare entry, which imports the `cloudflare:workers` built-in. @@ -65,7 +73,8 @@ const BUDGETS = { }, "packages/durable": { ".": { - // Tool argument validation reaches TypeBox; provider-session creation reaches pi-ai's lean UUID utility; the + // Tool argument validation reaches pi-schema (validator, coercion and format checks; 39 files that replace + // TypeBox, which this walk did not count); provider-session creation reaches pi-ai's lean UUID utility; the // memory and SQLite storages share scan order and cursor handling. Values from Chord's root (copyJson, // replicatedState) reach its host runtime through the root barrel. Chord's context reaches pi-telemetry's root // (index, noop, metrics, memory) for its NOOP telemetry default; harness/telemetry.ts is Durable's schema. @@ -77,16 +86,17 @@ const BUDGETS = { // each conversation's newest entry and head marker (session/conversation-tails.ts). Group commit sends batches // through `commit()`, or one `commit()` each (storage/commit-batch.ts), from a writer off the Session line // (session/writer.ts) that gates reads by the keys of unpublished writes (session/keys.ts). - maxFiles: 83, + maxFiles: 123, forbid: [ "packages/ai/src/index.ts", - "packages/ai/src/utils/typebox-helpers.ts", + "packages/ai/src/utils/string-enum.ts", // Photon (WebAssembly) loads only through `./images`. "packages/durable/src/images/", ], }, - // The file tools reach their environment helpers and the Harness definitions they need, not the Harness itself. - "./tools": { maxFiles: 17, forbid: ["packages/durable/src/images/"] }, + // The file tools reach their environment helpers and the Harness definitions they need, not the Harness itself, + // and pi-schema's root for their schema builders (10 files; its format checks stay out). + "./tools": { maxFiles: 27, forbid: ["packages/durable/src/images/", "packages/schema/src/formats/"] }, }, }; @@ -156,6 +166,8 @@ for (const [pkgDir, budgets] of Object.entries(BUDGETS)) { } const target = typeof declared === "string" ? declared : declared.import; for (const [name, distPath] of expand(pkgDir, entry, target)) { + // A wildcard entry's budget applies to every file it exports, unless `overrides` names that file. + const limits = budget.overrides?.[name] ?? budget; const source = sourceFor(pkgDir, distPath); if (!source) { console.error(`${pkgDir} export "${name}" points at ${distPath}, which has no source file`); @@ -163,15 +175,15 @@ for (const [pkgDir, budgets] of Object.entries(BUDGETS)) { continue; } const graph = [...walk(source)].map((file) => relative(ROOT, file).split(sep).join("/")); - if (graph.length > budget.maxFiles) { + if (graph.length > limits.maxFiles) { console.error( - `${pkgDir} export "${name}" reaches ${graph.length} files, budget ${budget.maxFiles}\n` + + `${pkgDir} export "${name}" reaches ${graph.length} files, budget ${limits.maxFiles}\n` + graph.map((file) => ` ${file}`).join("\n"), ); failures += 1; } - for (const pattern of budget.forbid ?? []) { - const hit = graph.filter((file) => file.includes(pattern) && !(budget.allow ?? []).some((allowed) => file.endsWith(allowed))); + for (const pattern of limits.forbid ?? []) { + const hit = graph.filter((file) => file.includes(pattern) && !(limits.allow ?? []).some((allowed) => file.endsWith(allowed))); if (hit.length > 0) { console.error(`${pkgDir} export "${name}" must not reach ${pattern}:\n${hit.map((f) => ` ${f}`).join("\n")}`); failures += 1; diff --git a/tsconfig.json b/tsconfig.json index 96c54539c..04557184d 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -63,5 +63,5 @@ } }, "include": ["scripts/model-catalog-protocol.ts", "scripts/model-catalog-protocol.test.ts", "packages/*/src/**/*", "packages/*/test/**/*", "packages/ai/catalog/**/*", "packages/durable/test-workers/**/*", "packages/coding-agent/examples/**/*", "packages/agent/examples/**/*"], - "exclude": ["**/dist/**", "packages/coding-agent/examples/extensions/gondolin/**"] + "exclude": ["**/dist/**", "packages/coding-agent/examples/extensions/**", "packages/coding-agent/test/suite/regressions/8261-subagent-project-trust.test.ts"] } From be039fb316a22c70cdf22eddf4b992110d84d6fb Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sun, 1 Nov 2026 10:14:38 +0000 Subject: [PATCH 115/175] feat(chord): Chord v1 and v2 on pi-schema, drop TypeBox Schema helpers are a thin layer over pi-schema's builders with the same JSON, so fingerprints and stored data are unchanged (proven by schema-switch tests and a store written by the TypeBox code). Checks through pi-schema; error messages are pi-schema's; format is an annotation. v1 contracts take any JSON Schema. map(bytes()) works. --- package-lock.json | 4 +- packages/chord/CHANGELOG.md | 2 + packages/chord/README.md | 12 +- packages/chord/docs/guide.md | 30 +- packages/chord/docs/spec.md | 16 +- packages/chord/docs/tutorial.md | 51 +- packages/chord/package.json | 4 +- packages/chord/src/cell.ts | 14 +- packages/chord/src/contract.ts | 93 +- packages/chord/src/next/contracts/README.md | 7 +- packages/chord/src/next/contracts/contract.ts | 5 +- packages/chord/src/next/contracts/declare.ts | 6 +- packages/chord/src/next/contracts/evolve.ts | 62 +- packages/chord/src/next/contracts/history.ts | 6 +- packages/chord/src/next/contracts/index.ts | 8 +- packages/chord/src/next/contracts/params.ts | 27 +- packages/chord/src/next/contracts/schema.ts | 213 +-- packages/chord/src/next/contracts/snapshot.ts | 28 +- packages/chord/src/next/data/README.md | 5 +- packages/chord/src/next/data/args.ts | 2 +- packages/chord/src/next/data/validator.ts | 49 +- .../chord/src/next/documents/definition.ts | 20 +- packages/chord/src/next/jobs/definition.ts | 12 +- packages/chord/src/next/link/README.md | 2 +- packages/chord/src/next/link/message.ts | 16 +- packages/chord/src/next/store/archive.ts | 30 +- packages/chord/src/next/store/columns.ts | 32 +- packages/chord/src/next/store/expression.ts | 5 +- packages/chord/src/next/store/json-column.ts | 21 +- packages/chord/src/next/store/table-evolve.ts | 10 +- packages/chord/src/object.ts | 21 +- packages/chord/src/remote.ts | 4 +- packages/chord/src/schema.ts | 183 ++- packages/chord/src/wire.ts | 178 +-- packages/chord/test/awaiting.test.ts | 8 +- packages/chord/test/bindings.test.ts | 42 +- packages/chord/test/boundary.test.ts | 7 +- packages/chord/test/cluster.test.ts | 30 +- packages/chord/test/contract.test.ts | 53 +- packages/chord/test/fixtures/echo.ts | 6 +- packages/chord/test/fixtures/loader.ts | 22 +- packages/chord/test/fixtures/schemas.ts | 14 + packages/chord/test/flow-units.test.ts | 16 +- packages/chord/test/flow.test.ts | 10 +- packages/chord/test/generations.test.ts | 16 +- .../chord/test/host-objects-reload.test.ts | 35 +- packages/chord/test/host.test.ts | 106 +- packages/chord/test/hydration.test.ts | 30 +- packages/chord/test/link.test.ts | 32 +- packages/chord/test/links-review.test.ts | 40 +- .../chord/test/next/contracts/declare.test.ts | 18 +- .../chord/test/next/contracts/evolve.test.ts | 9 +- .../test/next/contracts/fingerprint.test.ts | 28 +- .../chord/test/next/contracts/params.test.ts | 3 +- .../chord/test/next/contracts/schema.test.ts | 41 +- .../test/next/contracts/snapshot.test.ts | 110 +- packages/chord/test/next/data/args.test.ts | 12 +- packages/chord/test/next/data/params.test.ts | 4 +- .../chord/test/next/data/validator.test.ts | 48 +- .../37-plugin-imports/plugins/shapes/units.ts | 4 +- .../chord/test/next/schema/declarations.ts | 180 +++ packages/chord/test/next/schema/owner.ts | 85 ++ packages/chord/test/next/schema/proofs.ts | 113 ++ packages/chord/test/next/schema/recorded.json | 1143 +++++++++++++++++ .../test/next/schema/schema-switch.test.ts | 22 + .../test/next/schema/stored/archive.jsonl | 36 + .../chord/test/next/schema/stored/store.db | Bin 0 -> 57344 bytes .../chord/test/next/schema/write-stored.ts | 40 + packages/chord/test/placement.test.ts | 22 +- packages/chord/test/principals.test.ts | 10 +- packages/chord/test/provider-cost.test.ts | 8 +- packages/chord/test/readiness.test.ts | 10 +- packages/chord/test/reload.test.ts | 52 +- packages/chord/test/requests.test.ts | 58 +- packages/chord/test/review.test.ts | 103 +- packages/chord/test/state-init.test.ts | 18 +- packages/chord/test/state.test.ts | 40 +- packages/chord/test/tutorial.test.ts | 40 +- packages/chord/test/types.test.ts | 14 +- packages/chord/test/variants.test.ts | 14 +- packages/chord/test/websocket.test.ts | 12 +- 81 files changed, 2917 insertions(+), 1025 deletions(-) create mode 100644 packages/chord/test/fixtures/schemas.ts create mode 100644 packages/chord/test/next/schema/declarations.ts create mode 100644 packages/chord/test/next/schema/owner.ts create mode 100644 packages/chord/test/next/schema/proofs.ts create mode 100644 packages/chord/test/next/schema/recorded.json create mode 100644 packages/chord/test/next/schema/schema-switch.test.ts create mode 100644 packages/chord/test/next/schema/stored/archive.jsonl create mode 100644 packages/chord/test/next/schema/stored/store.db create mode 100644 packages/chord/test/next/schema/write-stored.ts diff --git a/package-lock.json b/package-lock.json index d37cfbbba..4bc15ba5c 100644 --- a/package-lock.json +++ b/package-lock.json @@ -5553,8 +5553,8 @@ "version": "1.4.0", "license": "MIT", "dependencies": { - "@earendil-works/pi-telemetry": "1.4.0", - "typebox": "1.3.27" + "@earendil-works/pi-schema": "1.4.0", + "@earendil-works/pi-telemetry": "1.4.0" }, "devDependencies": { "@types/ws": "8.18.2", diff --git a/packages/chord/CHANGELOG.md b/packages/chord/CHANGELOG.md index 326d4b8de..4f50cc098 100644 --- a/packages/chord/CHANGELOG.md +++ b/packages/chord/CHANGELOG.md @@ -4,6 +4,7 @@ ### Breaking Changes +- Schemas are JSON Schema checked by `@earendil-works/pi-schema` instead of TypeBox, which is no longer a dependency of Chord. Contracts, `self.state(...)` schemas and method signatures take pi-schema builders or any JSON Schema object (TypeBox-built schemas still validate, being JSON Schema, but type as `unknown`); handler and reference types come from pi-schema's `Static`. Validation, input defaults and the stripping of unknown fields work on any JSON Schema by its shape (before, defaults and stripping applied only to TypeBox-built schemas). `format` is an annotation, no longer checked: `str({ format: "email" })` accepts `"nope"` (TypeBox checked the formats it knew); check a format in the handler where it matters. An `invalid` error's message is now ` ` with pi-schema's paths and messages (a missing field at its own path, `missing`; `expected string, got number` instead of `must be string`), and its `details` are pi-schema's `ValidationError`s (`{ path, keyword, message }`) instead of TypeBox's errors. Link frames, descriptions, fingerprints and identities are unchanged for schemas with the same JSON. Migration, keeping a contract's fingerprint (it is computed from the schema's JSON): `Type.String/Number/Integer/Boolean/Null/Unknown/Array/Union/Optional(...)` become `str/num/int/bool/nul/unknown/arr/union/opt(...)` with the same options; `Type.Object(p)` becomes `obj(p, { open: true })` (pi-schema's `obj` is closed by default) and `Type.Object(p, { additionalProperties: false })` `obj(p)`; `Type.Enum(values)` becomes `oneOf(values)`; `Type.Literal(v)` becomes `{ type: typeof v, const: v }` (pi-schema's `lit(v)` has no `type`); `Type.Record(Type.String(), v)` becomes `{ type: "object", patternProperties: { "^.*$": v } }` (pi-schema's `map(v)` uses `additionalProperties`); `Type.Intersect(list)` becomes `{ allOf: list }`; `Type.Function(params, result)` and `Type.Void()` become `{ type: "function", parameters, returnType }` and `{ type: "void" }`; `Type.Unsafe(schema)` becomes `schema as Schema`; `Type.Immutable(schema)` becomes the schema (cast its `Schema` type); `opt(...)` marks only an `obj` property; `Static` and `TSchema` come from pi-schema (`Static`, `Schema`). pi-schema's own `lit`, `map` and closed `obj` give other JSON and so other fingerprints, which matters only while peers of both versions talk. - `traceIn(options, fn, context)` takes the context last (was `traceIn(context, options, fn)`), like the rest of the API. - Watch listeners take one change object before the context: `WatchHandle.start((change, context) => …)` with `change: WatchChange` = `{ value, ops, retired }`, and `NativeWatchHandle.start(({ value }, context) => …)` with `NativeWatchChange`. A change is plain data, those fields only; the frame's context is the second argument. `Watch.advance(change, context)` takes the same object, and a frame delivered alone, with nothing folded into it, reaches the listener as the object the source passed. Migration: `start(async (value, ops, ctx) => …)` becomes `start(async ({ value, ops }, ctx) => …)`, `start(async (value, ctx) => …)` on a native watch `start(async ({ value }, ctx) => …)`, and `advance(value, ops, ctx)` becomes `advance({ value, ops, retired: false }, ctx)`. - `SqliteExecutor.runCounted(sql, ...params)` runs one statement like `run` and resolves with the rows it inserted, updated or deleted, as SQLite's `changes()` counts them (index writes and rows changed by triggers do not count). The Node driver takes it from `node:sqlite`; the Durable Object driver reads `SELECT changes()` after the statement (the cursor's `rowsWritten` also counts index writes), so `run` stays as cheap as before. Migration: custom `SqliteDatabase` and `SqliteExecutor` implementations add `runCounted`. @@ -30,6 +31,7 @@ ### Changed +- Chord v2 (`@earendil-works/chord/next`) checks its schemas with `@earendil-works/pi-schema` instead of TypeBox: its helpers (`str`, `num`, `int`, `bool`, `nul`, `lit`, `oneOf`, `union`, `obj`, `arr`, `map`, `opt`, `bytes`, `unknown`) build on pi-schema's builders and keep the JSON contracts were fingerprinted with, so every fingerprint, normal form and stored record stays the same (a test records them and reads a store written before the switch). `Static` and `Schema` (was `TSchema`) come from pi-schema; `opt(...)` marks an object field or a table column and is no schema elsewhere (a top-level `opt` had no effect before); a schema with a non-enumerable key (a TypeBox schema's metadata) is refused at declaration; `map(bytes())` is accepted now (it was refused while `bytes()` carried a TypeBox refinement). `format` is an annotation, no longer checked (`{ type: "string", format: "email" }` accepts `"nope"`; TypeBox checked the formats it knew). A failed check's message comes from pi-schema (`args/title: expected string, got integer`), its `details.path` escapes `~` and `/` (JSON Pointer); reasons and codes are unchanged. Declaring and compiling are cheaper (a 20-method contract 1.07 ms to 0.86 ms, a validator's first use 172 µs to 48 µs), a check costs the same (0.25 µs), the message envelope check up to 100 ns more per message. - The delta tracker keeps its one open change in a plain field and needs a `WeakRef` only for a second change open at the same time (one per change before). A `WeakRef` target survives the scavenges of its job, so a run of changes in one job promoted every change's overlay to old space: 200,000 small changes in one job now promote 2 MB instead of 47.5 MB and take 3.1 µs each instead of 3.8 µs (3.0 instead of 3.5 µs when they span jobs). A draft dropped without `prepare()` or `abort()` now stays referenced by its tracker until the tracker's next `beginChange()` or `adopt()`. - `applyImmutable()`, `applyImmutableBatches()` and tracker revisions copy each container on a changed path with spread (or `Object.assign()` onto a null-prototype object) instead of one `Object.defineProperty()` per key, about 25 times faster per container ([#9265](https://github.com/earendil-works/pi/issues/9265)) diff --git a/packages/chord/README.md b/packages/chord/README.md index 9d9d95587..081e3fe21 100644 --- a/packages/chord/README.md +++ b/packages/chord/README.md @@ -8,7 +8,7 @@ Chord builds applications out of **objects** that live in several processes (a s - **State** is observable JSON. Remote viewers get a snapshot followed by deltas. - **Hot reload** swaps code while it runs. Work in flight finishes on the code it started with, state is kept, and incompatible changes migrate at a safe point. -Contracts are runtime values described with [TypeBox](https://github.com/sinclairzx81/typebox) schemas. Hosts exchange JSON frames over links: in memory, a Node IPC channel, or a WebSocket. +Contracts are runtime values described with JSON Schema, written with the builders of `@earendil-works/pi-schema` (any JSON Schema object works; `format` is an annotation, never checked). Hosts exchange JSON frames over links: in memory, a Node IPC channel, or a WebSocket. ## Entry points @@ -27,18 +27,18 @@ Contracts are runtime values described with [TypeBox](https://github.com/sinclai ## Quick start ```ts -import { Type } from "typebox"; +import { num, obj, str } from "@earendil-works/pi-schema"; import { createHost, hostNamed, method, remote } from "@earendil-works/chord"; // contract.ts: shared by provider and consumers export const Counter = remote("demo.counter", { version: 1, - input: Type.Object({ name: Type.String() }), + input: obj({ name: str() }), host: hostNamed("server"), // where its objects live; callers never name a host variants: { default: { - state: { count: Type.Number() }, - methods: { increment: method(Type.Object({ by: Type.Number() })) }, + state: { count: num() }, + methods: { increment: method(obj({ by: num() })) }, }, }, }); @@ -46,7 +46,7 @@ export const Counter = remote("demo.counter", { const host = createHost({ name: "server", self: { kind: "server" } }); host.serve(Counter, ({ name }, _principal, self) => { - const count = self.state("count", Type.Number(), 0); + const count = self.state("count", num(), 0); return { variant: "default", count, diff --git a/packages/chord/docs/guide.md b/packages/chord/docs/guide.md index 59c1414b7..c3b7b3211 100644 --- a/packages/chord/docs/guide.md +++ b/packages/chord/docs/guide.md @@ -36,17 +36,17 @@ Chord builds applications out of **objects** that live in several processes (a s ## Quick start ```ts -import { Type } from "typebox"; +import { num, obj, str } from "@earendil-works/pi-schema"; import { createHost, method, remote } from "@earendil-works/chord"; // contract.ts: shared by provider and consumers export const Counter = remote("demo.counter", { version: 1, - input: Type.Object({ name: Type.String() }), + input: obj({ name: str() }), variants: { default: { - state: { count: Type.Number() }, - methods: { increment: method(Type.Object({ by: Type.Number() })) }, + state: { count: num() }, + methods: { increment: method(obj({ by: num() })) }, }, }, }); @@ -54,7 +54,7 @@ export const Counter = remote("demo.counter", { const host = createHost({ name: "server", self: { kind: "server" } }); host.serve(Counter, ({ name }, _principal, self) => { - const count = self.state("count", Type.Number(), 0); + const count = self.state("count", num(), 0); return { variant: "default", count, @@ -82,7 +82,7 @@ Remote types can be reached from other hosts. Everything that crosses the wire h ```ts export const Chat = remote("pi.chat", { version: 1, - input: Type.Object({ session: Type.String(), conversation: Type.Number() }), + input: obj({ session: str(), conversation: num() }), host: hostFamily("session:", "session"), // lives on `session:S1` for `{ session: "S1", … }` variants: { viewer: { @@ -91,7 +91,7 @@ export const Chat = remote("pi.chat", { participant: { state: { view: ConversationViewSchema }, methods: { - send: method(SendInputSchema, Type.Object({ submissionId: Type.Number() })), + send: method(SendInputSchema, obj({ submissionId: num() })), stop: method(), // no arguments, no result: `chat.stop(ctx)` }, }, @@ -119,10 +119,10 @@ Local types are reachable only inside their host. Their members may be anything: ```ts export const Index = local("pi.index", { version: 1, - input: Type.Object({ session: Type.String() }), + input: obj({ session: str() }), members: { - search: method(Type.Object({ query: Type.String() }), Type.Array(HitSchema)), - onChange: method(Type.Function([ChangeSchema], Type.Void()), UnsubscribeSchema), + search: method(obj({ query: str() }), arr(HitSchema)), + onChange: method({ type: "function", parameters: [ChangeSchema], returnType: { type: "void" } }, UnsubscribeSchema), harness: opaque("pi.durable-harness@1"), }, }); @@ -181,7 +181,7 @@ A reload runs the factory again over the same object. Declarations that keep som ```ts host.serve(StatusStore, ({ session }, _host, self) => { const status = self.state("status", StatusSchema, { running: false, tasks: 0 }); - const errors = self.state("errors", Type.Array(Type.String()), []); + const errors = self.state("errors", arr(str()), []); const catalog = self.get(Catalog, {}); const db = self.resource("db", { version: 1, @@ -230,7 +230,7 @@ host.serve(Visits, (_, _p, self) => { // right host.serve(Visits, (_, _p, self) => { - const visits = self.state("visits", Type.Number(), 0); + const visits = self.state("visits", num(), 0); return { hit: async (ctx) => { visits.change(ctx, (n) => n + 1); @@ -240,7 +240,7 @@ host.serve(Visits, (_, _p, self) => { }); ``` -**Methods** are declared with `method(args?, result?)` in the contract. Without `args` a method takes none: callers write `ref.hit(ctx)` (or `ref.hit()`), and its implementation is `(ctx) => …`. Without `result` it returns nothing: the implementation returns nothing, and callers get `undefined`. `method(undefined, Type.Number())` takes no arguments and returns a number. +**Methods** are declared with `method(args?, result?)` in the contract. Without `args` a method takes none: callers write `ref.hit(ctx)` (or `ref.hit()`), and its implementation is `(ctx) => …`. Without `result` it returns nothing: the implementation returns nothing, and callers get `undefined`. `method(undefined, num())` takes no arguments and returns a number. ## Activation @@ -428,7 +428,7 @@ chat.release(); ## State ```ts -const todos = self.state("todos", Type.Array(TodoSchema), []); +const todos = self.state("todos", arr(TodoSchema), []); todos.change(ctx, (draft) => { draft.push({ id: "t1", text: "Write docs", done: false }); @@ -457,7 +457,7 @@ view.replace(ctx, rebuilt, { diff: true }); // always diff ```ts const db = self.resource("db", { version: 1, open: () => openSqlite(path), close: (db) => db.close() }); -const notes = self.state("notes", Type.Array(NoteSchema), { init: () => db.value.all("SELECT * FROM notes") }); +const notes = self.state("notes", arr(NoteSchema), { init: () => db.value.all("SELECT * FROM notes") }); ``` `init(ctx)` runs once the resources are open and the references live, before the object goes live, so no method, diff --git a/packages/chord/docs/spec.md b/packages/chord/docs/spec.md index ca9c93b38..ca910d758 100644 --- a/packages/chord/docs/spec.md +++ b/packages/chord/docs/spec.md @@ -3,7 +3,7 @@ This document specifies the runtime in `src`. Where it and the code disagree, the disagreement is a bug. [Known limitations and decisions](#12-known-limitations-and-decisions) lists what the runtime does not do yet and the choices the sections below leave open. Hard constraints: -- Chord's runtime depends only on its own `delta` module (tracker, applyImmutable, encoder/decoder, diff) and Node/browser built-ins plus `typebox` for schemas. No other Pi packages. +- Chord's runtime depends only on its own `delta` module (tracker, applyImmutable, encoder/decoder, diff) and Node/browser built-ins, plus `@earendil-works/pi-telemetry` for telemetry and `@earendil-works/pi-schema` for schemas. No other Pi packages. - Built-in plugins are trusted, loaded from source in-process. The plugin loader uses dynamic `import()` with a cache-busting query (owner-approved exception, `TODO(loader)`: replace with an isolating, evicting loader; old module copies leak; TUI plugins need isolation). - Hot reload is required. Host restart only for Chord/runtime code changes or host-level crash recovery. @@ -29,23 +29,23 @@ Contracts are runtime values for every type (shipped bundles have no TypeScript; ```ts export const Chat = remote("pi.chat", { version: 1, // contract version: a plain number, bumped only on breaking changes - input: Type.Object({ session: Type.String(), conversation: Type.Number() }), + input: obj({ session: str(), conversation: num() }), host: hostFamily("session:", "session"), // where its objects live: `session:` + input.session variants: { viewer: { state: { view: ViewSchema } }, participant: { state: { view: ViewSchema }, - methods: { send: method(SendSchema, Type.Object({ submissionId: Type.Number() })) }, + methods: { send: method(SendSchema, obj({ submissionId: num() })) }, }, }, }); export const Index = local("pi.index", { version: 1, - input: Type.Object({ session: Type.String() }), + input: obj({ session: str() }), members: { - search: method(Type.Object({ query: Type.String() }), Type.Array(Hit)), - onChange: method(Type.Function([Change], Type.Void()), Unsubscribe), // non-JSON allowed for locals + search: method(obj({ query: str() }), arr(Hit)), + onChange: method({ type: "function", parameters: [Change], returnType: { type: "void" } }, Unsubscribe), // non-JSON allowed for locals harness: opaque("pi.durable-harness@1"), // native object, by name+version }, }); @@ -81,7 +81,7 @@ Factories are synchronous; returning a promise throws. A declaration has a name host.serve(StatusStore, ({ session }, _host, self) => { const status = self.state("status", StatusSchema, { running: false, tasks: 0 }, // schema required, initial value { version: 2, migrate: (old, from) => upgrade(old, from) }); // per cell; default version 1 - const errors = self.state("errors", Type.Array(Type.String()), []); + const errors = self.state("errors", arr(str()), []); const catalog = self.get(Catalog, {}); // reference; identity = canonical address (+ at) const watch = self.resource("watch", { version: 1, deps: [catalog], exclusive: false, @@ -278,7 +278,7 @@ The tests cover at least: lifetime/references/linger/forced disposal/gone; activ - **Joining during a reload.** Participants are the child hosts in step when a reload starts. A child whose hello arrives while it runs is decided when it ends: in step, or reconciled to the committed code version; a child already behind is fenced at once, so the calls its roots make meanwhile are admitted in the parent's code version. A reconciliation that is running when a reload starts syncs its child only after that reload ended, against the code version it committed. A reconciliation announces the parent's retired code versions before its `prepare`. - **Two-version rule.** An object of a changed type created after a commit has only new code; a previous-code-version root calling it gets `superseded`. - **Validation.** Remote calls are validated at reference dispatch, in-host too: a direct binding against the provider's schema, a binding through an adapter against the consumer's. A provider validates each snapshot and reset of a remote state member once per (re)subscription, against the consumer's contract through an `accepts` adapter and the provider's otherwise; ops are validated only in dev mode. At reload, the new provider's description must cover the old provider's when the version is unchanged, in addition to the check against live consumers and facets' `uses`. -- **Contracts.** Local contracts list their members, so a loading reference knows whether a member is a method, state or opaque. Canonical identity omits input fields equal to their default; with TypeBox, use `Type.Optional(Type.X({ default }))` to let callers omit them. Provider variants must cover consumer variants; members the consumer's contract does not list are hidden. Schema change detection compares canonical JSON, so reordering `anyOf` branches counts as a change. Fingerprints use a non-cryptographic 64-bit hash: synchronous and browser-safe; contracts are trusted. +- **Contracts.** Local contracts list their members, so a loading reference knows whether a member is a method, state or opaque. Canonical identity omits input fields equal to their default; use `opt(str({ default }))` (pi-schema) to let callers omit them. Provider variants must cover consumer variants; members the consumer's contract does not list are hidden. Schema change detection compares canonical JSON, so reordering `anyOf` branches counts as a change. Fingerprints use a non-cryptographic 64-bit hash: synchronous and browser-safe; contracts are trusted. - **Principals.** A root ctx at ingress acts as its principal (`host.root(fn, { principal })`); contexts inside objects act as `host`. The acceptor decides a link's kind and name (`accept(link, { principal, kind, name })`); only configured host links are addressable, carry roots and may vote, and control frames are accepted only from the parent. Clients never carry roots or code versions. Relaying needs a matching `route`; `authorize(principal, hostName)` runs before `resolve` and again after, and revoking or re-stamping a principal closes its relays at once. `invalidatePrincipal(match)` re-stamps or closes matching links and propagates pattern matches to child hosts. - **Cells and readables.** A value returned from `change` replaces the value and is always validated; `replace` transfers ownership without copying; by default (`{ diff }` overrides) it publishes a diff when the new value reuses a container of the old one at the same key, else `r`. `subscribe(fn)` delivers values, starting with the current one; `events(fn)` delivers `snap`, then `ops`, or `reset`. Reference state members are stable readables that throw `not_ready` while loading and `not_in_variant` outside the variant, and reset when their source moves; a listener added before they are readable gets its snapshot then. Remote state members stay hydrated while the reference is held. - **References and calls.** A reference's identity is its address; a second `self.get` of the same address in one run returns the same reference. A reference call without ctx, or with a task ctx, starts a fresh root in the current code version; a ctx whose call or root already settled is refused (`aborted/expired`), one from another host `invalid/foreign_ctx`. Calling a reference inside a factory throws `not_ready`. diff --git a/packages/chord/docs/tutorial.md b/packages/chord/docs/tutorial.md index fe16dce4c..65a6334ed 100644 --- a/packages/chord/docs/tutorial.md +++ b/packages/chord/docs/tutorial.md @@ -45,11 +45,12 @@ shuts down its objects and connections. Before callers can use an object, they need to agree on its name and members. That agreement is a **contract**: a runtime value that also gives TypeScript its -types. A **schema** describes a value's shape. We use TypeBox's `Type` to write -schemas. `method(args, result)` declares a method by its argument and result -schemas: `method()` takes no arguments and returns nothing, and -`method(undefined, Type.String())`, used below, takes no arguments and returns a -string. +types. A **schema** describes a value's shape: plain JSON Schema, which we write +with the builders of `@earendil-works/pi-schema` (`obj`, `str`, `num`, ...; +`{ open: true }` lets an object carry fields it does not name). +`method(args, result)` declares a method by its argument and result schemas: +`method()` takes no arguments and returns nothing, and +`method(undefined, str())`, used below, takes no arguments and returns a string. Start with a label for a named list. `local` means only code inside this host can reach it. The input `{ list: "family" }` identifies which list we want; @@ -59,8 +60,8 @@ it is not an argument to `label`. ```ts const TodoLabel = local("tutorial.todos", { version: 1, - input: Type.Object({ list: Type.String() }), - members: { label: method(undefined, Type.String()) }, + input: obj({ list: str() }, { open: true }), + members: { label: method(undefined, str()) }, }); ``` @@ -116,17 +117,17 @@ name, because it holds nothing a reload must keep. ```ts -const Todo = Type.Object({ text: Type.String(), done: Type.Boolean() }); -const Rows = Type.Array(Todo); +const Todo = obj({ text: str(), done: bool() }, { open: true }); +const Rows = arr(Todo); const TodoStore = local("tutorial.todos", { version: 1, - input: Type.Object({ list: Type.String() }), + input: obj({ list: str() }, { open: true }), members: { - label: method(undefined, Type.String()), + label: method(undefined, str()), rows: Rows, - openCount: Type.Number(), - add: method(Type.Object({ text: Type.String() })), - finish: method(Type.Object({ index: Type.Integer() })), + openCount: num(), + add: method(obj({ text: str() }, { open: true })), + finish: method(obj({ index: int() }, { open: true })), reset: method(Rows), }, }); @@ -230,13 +231,13 @@ to a declared resource, without reading the resource during the factory. ```ts const Database = local("tutorial.database", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), members: { db: opaque<{ saved: string[] }>("tutorial.database@1") }, }); const SavedTodos = local("tutorial.saved-todos", { version: 1, - input: Type.Object({ list: Type.String() }), - members: { add: method(Type.Object({ text: Type.String() })) }, + input: obj({ list: str() }, { open: true }), + members: { add: method(obj({ text: str() }, { open: true })) }, }); const host = createHost({ self: "server" }); let opens = 0; @@ -338,13 +339,13 @@ puts every `Todos` object on the host named `server`. Section 6 uses that. ```ts const Todos = remote("tutorial.todo-view", { version: 1, - input: Type.Object({ list: Type.String() }), + input: obj({ list: str() }, { open: true }), host: hostNamed("server"), variants: { viewer: { state: { rows: Rows } }, editor: { state: { rows: Rows }, - methods: { add: method(Type.Object({ text: Type.String() })) }, + methods: { add: method(obj({ text: str() }, { open: true })) }, }, }, }); @@ -499,8 +500,8 @@ test; they are not stored application state. ```ts const Activity = local("tutorial.activity", { version: 1, - input: Type.Object({}), - members: { seen: Type.Number() }, + input: obj({}, { open: true }), + members: { seen: num() }, }); const host = createHost({ self: "server" }); host.serve(TodoStore, todoFactory); @@ -509,7 +510,7 @@ const seeded = Promise.withResolvers(); const stopped = Promise.withResolvers(); host.serve(Activity, (_input, _principal, self) => { const todos = self.get(TodoStore, { list: "family" }); - const seen = self.state("seen", Type.Number(), 0); + const seen = self.state("seen", num(), 0); self.on("rows", todos.rows, (rows, ctx) => { seen.replace(ctx, rows?.length ?? 0); if (rows?.length === 1) observed.resolve(); @@ -570,7 +571,7 @@ default lifetime, `"referenced"`, are still created only when referenced. ```ts const Actions = defineRegistry<(ctx: Ctx) => Promise>("tutorial.actions"); -const Menu = local("tutorial.menu", { version: 1, input: Type.Object({}) }); +const Menu = local("tutorial.menu", { version: 1, input: obj({}, { open: true }) }); let click: (() => Promise) | undefined; const storeFacet: Facet = { name: "todo-store", @@ -745,7 +746,7 @@ const { label: _label, ...members } = TodoStore.def.members; const RenamedStore = local("tutorial.todos", { version: 2, input: TodoStore.def.input, - members: { ...members, heading: method(undefined, Type.String()) }, + members: { ...members, heading: method(undefined, str()) }, }); await host.reload([ { @@ -890,7 +891,7 @@ tutorial's source-matching check, not by the application. ```ts import assert from "node:assert/strict"; import { readFileSync } from "node:fs"; -import { type Static, Type } from "typebox"; +import { arr, bool, int, num, obj, type Static, str } from "@earendil-works/pi-schema"; import { it, onTestFinished } from "vitest"; import { type Ctx, diff --git a/packages/chord/package.json b/packages/chord/package.json index 908005452..edced8cd5 100644 --- a/packages/chord/package.json +++ b/packages/chord/package.json @@ -90,8 +90,8 @@ "node": ">=22.19.0" }, "dependencies": { - "@earendil-works/pi-telemetry": "1.4.0", - "typebox": "1.3.27" + "@earendil-works/pi-schema": "1.4.0", + "@earendil-works/pi-telemetry": "1.4.0" }, "peerDependencies": { "esbuild": "0.28.2", diff --git a/packages/chord/src/cell.ts b/packages/chord/src/cell.ts index 65d391128..d520258e3 100644 --- a/packages/chord/src/cell.ts +++ b/packages/chord/src/cell.ts @@ -1,4 +1,4 @@ -import type { TSchema } from "typebox"; +import type { Schema } from "@earendil-works/pi-schema"; import { type Draft, diffRevisions, type JsonValue, type Op, type Tracker, track } from "./delta/index.ts"; import { ChordError } from "./errors.ts"; import { enqueue, holdPublications } from "./publish.ts"; @@ -87,7 +87,7 @@ export abstract class Source implements Readable { readonly seq = ++sequence; readonly dependents = new Set>(); /** Schemas of remote state members currently exposing this source (checked per write in dev mode). */ - schemas: readonly TSchema[] = []; + schemas: readonly Schema[] = []; protected readonly options: SourceOptions; protected readonly fanout: Fanout; @@ -114,7 +114,7 @@ export abstract class Source implements Readable { for (const schema of this.schemas) assertValid("state", `state ${this.key}`, schema, value); } - protected ownSchemas(): readonly TSchema[] { + protected ownSchemas(): readonly Schema[] { return []; } } @@ -149,7 +149,7 @@ function unbox(ops: readonly Op[], value: unknown): Op[] { /** A state cell holding any JSON value, validated against its schema at creation and (dev mode) on writes. */ export class Cell extends Source { /** Replaced at a reload commit that changes the schema (state-version bump). */ - schema: TSchema; + schema: Schema; #tracker: Tracker>; readonly #guard: WriteGuard; /** False until `initialize` for a cell declared with `init`: reads and writes throw `not_ready`. */ @@ -161,7 +161,7 @@ export class Cell extends Source { */ constructor( key: string, - schema: TSchema, + schema: Schema, initial: { readonly value: T } | undefined, guard: WriteGuard, options: SourceOptions, @@ -182,7 +182,7 @@ export class Cell extends Source { this.#initialized = true; } - protected override ownSchemas(): readonly TSchema[] { + protected override ownSchemas(): readonly Schema[] { return [this.schema]; } @@ -238,7 +238,7 @@ export class Cell extends Source { * Installs a migrated value and its schema (reload commit). The caller validated the value and the derivations * over it against every schema that will apply, so nothing is checked here. Subscribers receive a reset. */ - reset(value: T, schema: TSchema): void { + reset(value: T, schema: Schema): void { this.schema = schema; this.#publish(this.#tracker.prepareReplace({ v: value }), "reset", false); } diff --git a/packages/chord/src/contract.ts b/packages/chord/src/contract.ts index c2ebd6b96..1a6d64031 100644 --- a/packages/chord/src/contract.ts +++ b/packages/chord/src/contract.ts @@ -1,15 +1,14 @@ -import { type Static, type TSchema, Type } from "typebox"; -import { Value } from "typebox/value"; +import { int, lit, map, obj, opt, type Schema, type Static, str, union } from "@earendil-works/pi-schema"; import { ChordError, invalid } from "./errors.ts"; -import { canonicalJson, fits, hash64 } from "./schema.ts"; +import { canonicalJson, fits, hash64, isValid } from "./schema.ts"; import type { Ctx, Exposed, Readable, RefBase, StateSource } from "./types.ts"; const MEMBER = Symbol("chord.member"); /** A method's arguments and result; `undefined`: it takes no arguments, or returns nothing (void). */ export interface MethodSpec< - A extends TSchema | undefined = TSchema | undefined, - R extends TSchema | undefined = TSchema | undefined, + A extends Schema | undefined = Schema | undefined, + R extends Schema | undefined = Schema | undefined, > { readonly [MEMBER]: "method"; readonly args: A; @@ -29,9 +28,9 @@ export interface OpaqueSpec { * for a method without arguments that returns something. */ export function method(): MethodSpec; -export function method(args: A): MethodSpec; -export function method(args: A, result: R): MethodSpec; -export function method(args?: TSchema, result?: TSchema): MethodSpec { +export function method(args: A): MethodSpec; +export function method(args: A, result: R): MethodSpec; +export function method(args?: Schema, result?: Schema): MethodSpec { return { [MEMBER]: "method", args, result }; } @@ -42,7 +41,7 @@ export const opaque = (id: string): OpaqueSpec => { }; export interface VariantDef { - readonly state?: Readonly>; + readonly state?: Readonly>; readonly methods?: Readonly>; } @@ -102,63 +101,59 @@ export const placementText = (placement: Placement): string => export interface RemoteDef { readonly version: number; - readonly input: TSchema; + readonly input: Schema; /** Where its objects live: `hostNamed(…)` or `hostFamily(…)`. Part of the contract (and its fingerprint). */ readonly host: Placement; readonly variants: Readonly>; } /** A local member: a method, an opaque native object, or a state member described by its schema. */ -export type LocalMember = MethodSpec | OpaqueSpec | TSchema; +export type LocalMember = MethodSpec | OpaqueSpec | Schema; export interface LocalDef { readonly version: number; - readonly input: TSchema; + readonly input: Schema; readonly members?: Readonly>; } // A description travels as a reference's contract token (wire.ts), so it is declared once, as the schema that validates it. -// `TSchema` is an empty interface: any object is one, which is all a peer's schema is checked for here. -const SchemaObject = Type.Unsafe(Type.Object({})); +// Any object is a schema, which is all a peer's schema is checked for here. +const SchemaObject = obj({}, { open: true }) as Schema; -const MemberDescriptionSchema = Type.Union([ +const MemberDescriptionSchema = union([ // Absent `args`: the method takes none. Absent `result`: it returns nothing. - Type.Object({ - kind: Type.Literal("method"), - args: Type.Optional(SchemaObject), - result: Type.Optional(SchemaObject), - }), - Type.Object({ kind: Type.Literal("state"), schema: SchemaObject }), - Type.Object({ kind: Type.Literal("opaque"), id: Type.String() }), + obj({ kind: lit("method"), args: opt(SchemaObject), result: opt(SchemaObject) }, { open: true }), + obj({ kind: lit("state"), schema: SchemaObject }, { open: true }), + obj({ kind: lit("opaque"), id: str() }, { open: true }), ]); export type MemberDescription = Static; /** Local contracts have one unnamed variant. */ export const LOCAL_VARIANT = ""; -const PlacementSchema = Type.Union([ - Type.Object({ name: Type.String({ minLength: 1 }) }, { additionalProperties: false }), - Type.Object( - { prefix: Type.String({ minLength: 1 }), key: Type.String({ minLength: 1 }) }, - { additionalProperties: false }, - ), +const PlacementSchema = union([ + obj({ name: str({ minLength: 1 }) }), + obj({ prefix: str({ minLength: 1 }), key: str({ minLength: 1 }) }), ]); -export const DescriptionSchema = Type.Object({ - name: Type.String({ minLength: 1 }), - scope: Type.Union([Type.Literal("remote"), Type.Literal("local")]), - version: Type.Integer(), - input: SchemaObject, - /** Remote types only (required there): where their objects live. */ - host: Type.Optional(PlacementSchema), - variants: Type.Record(Type.String(), Type.Record(Type.String(), MemberDescriptionSchema)), -}); +export const DescriptionSchema = obj( + { + name: str({ minLength: 1 }), + scope: union([lit("remote"), lit("local")]), + version: int(), + input: SchemaObject, + /** Remote types only (required there): where their objects live. */ + host: opt(PlacementSchema), + variants: map(map(MemberDescriptionSchema)), + }, + { open: true }, +); export type Description = Static; interface ContractBase { readonly name: string; readonly version: number; - readonly input: TSchema; + readonly input: Schema; readonly description: Description; readonly fingerprint: string; } @@ -204,7 +199,7 @@ function describeMember(name: string, member: unknown, scope: "remote" | "local" if (scope === "remote") throw new TypeError(`remote contracts cannot expose opaque members: ${name}`); return { kind: "opaque", id: (member as OpaqueSpec).id }; } - return { kind: "state", schema: member as TSchema }; + return { kind: "state", schema: member as Schema }; } function finish(contract: Omit): C { @@ -222,7 +217,7 @@ export function remote( ): RemoteContract { checkHeader(name, def.version); const placement = def.host as Placement | undefined; - if (placement === undefined || !Value.Check(PlacementSchema, placement)) + if (placement === undefined || !isValid(PlacementSchema, placement)) throw new TypeError(`remote contract ${name} needs \`host\`: hostNamed(…) or hostFamily(…)`); // A JSON copy: the description travels, and a placement object may carry nothing else. const host: Placement = @@ -353,23 +348,23 @@ export function bind(provider: AnyContract, accepts: Accepts | undefined, consum // ─── Static types ──────────────────────────────────────────────────────────── type Empty = Record; -type StateOf = V extends { readonly state: infer S extends Readonly> } ? S : Empty; -type StaticOf = S extends TSchema ? Static : never; +type StateOf = V extends { readonly state: infer S extends Readonly> } ? S : Empty; +type StaticOf = S extends Schema ? Static : never; type MethodsOf = V extends { readonly methods: infer M extends Readonly> } ? M : Empty; /** What a handler returns: the result's value, or nothing for a void method. */ -type Returned = [R] extends [TSchema] ? Static | Promise> : void | Promise; +type Returned = [R] extends [Schema] ? Static | Promise> : void | Promise; /** What a call resolves to. */ -type Resolved = [R] extends [TSchema] ? Static : undefined; +type Resolved = [R] extends [Schema] ? Static : undefined; /** A method's handler; `P` is the app's principal type (`ctx.principal`). */ export type MethodImplementation = M extends MethodSpec - ? [A] extends [TSchema] + ? [A] extends [Schema] ? (args: Static, ctx: Ctx

    ) => Returned : (ctx: Ctx

    ) => Returned : never; export type Call = M extends MethodSpec - ? [A] extends [TSchema] + ? [A] extends [Schema] ? (args: Static, ctx?: Ctx) => Promise> : (ctx?: Ctx) => Promise> : never; @@ -402,21 +397,21 @@ type LocalImplementationMember = M extends MethodSpec ? MethodImplementation : M extends OpaqueSpec ? T | Exposed - : M extends TSchema + : M extends Schema ? StateSource> : never; type LocalRefMember = M extends MethodSpec ? Call : M extends OpaqueSpec ? T | undefined - : M extends TSchema + : M extends Schema ? Readable> : never; type LocalMembers = D extends { readonly members: infer M extends Readonly> } ? M : Empty; -export type InputOf = C extends { readonly def: { readonly input: infer I extends TSchema } } ? Static : never; +export type InputOf = C extends { readonly def: { readonly input: infer I extends Schema } } ? Static : never; /** What a factory returns for contract C; its methods' `ctx.principal` is `P`. */ export type ImplementationOf = C extends RemoteContract diff --git a/packages/chord/src/next/contracts/README.md b/packages/chord/src/next/contracts/README.md index bd79e9bc0..7cfd5fd9d 100644 --- a/packages/chord/src/next/contracts/README.md +++ b/packages/chord/src/next/contracts/README.md @@ -34,7 +34,12 @@ type NotesProvider = Implementation; // { variant: "editor"; membe key built like an identity key (`session|S1`); two addresses are equal when their keys are. The params are checked where the address is routed or served (`invalid/host_address`), not when it is made. No host-name strings in app code. - Schema helpers: `str`, `num`, `int`, `bool`, `nul`, `lit`, `oneOf`, `union`, `obj`, `arr`, `map`, `opt`, `bytes`, - `unknown`, and `Static`. They return ordinary TypeBox schemas. + `unknown`, `Schema` and `Static`, over `@earendil-works/pi-schema`'s builders: plain JSON Schema typed with the values + it accepts. They keep the shapes contracts were fingerprinted with: `lit` carries its JSON `type`, `oneOf(...)` is a + union of `lit`s, `map` one pattern property (`^.*$`), `obj` is closed. `opt(...)` marks an object field (or a table + column) optional and is no schema elsewhere. Any other JSON Schema object of the vocabulary below works too; a + non-enumerable key (a TypeBox schema's metadata) is refused. `format` is an annotation: it is part of the normal form + and the fingerprint, but no format is checked. - `evolve(previous, (e) => e...)`: the next version from structured edits (`state`, `addState`, `removeState`, `args`, `result`, `method`, `addMethod`, `removeMethod`); it must return the editor it was given. - Types: `Contract`, `Implementation` (what a factory returns: for `local` a `Watchable` per state member and a diff --git a/packages/chord/src/next/contracts/contract.ts b/packages/chord/src/next/contracts/contract.ts index 123757d70..696696b19 100644 --- a/packages/chord/src/next/contracts/contract.ts +++ b/packages/chord/src/next/contracts/contract.ts @@ -1,13 +1,12 @@ // Contract handles: the branded, immutable value an app declares, its type-level shape, and the Chord-owned data // behind it. -import type { TNull } from "typebox"; import type { Context } from "../../context/index.ts"; import type { Watchable, WatchHandle } from "../../watch.ts"; import type { Change, CurrentStep, History, VariantShape } from "./history.ts"; import type { Address } from "./host-kind.ts"; import type { Params, ParamsData } from "./params.ts"; -import type { Method, Static } from "./schema.ts"; +import type { Method, NullSchema, Static } from "./schema.ts"; import type { Snapshot } from "./snapshot.ts"; export type Kind = "remote" | "local"; @@ -88,7 +87,7 @@ export interface KeyedWatchable { } /** A method as a function: without the argument when its argument schema is `null` (`null` stays the wire form). */ -type MethodFunction = M["args"] extends TNull +type MethodFunction = M["args"] extends NullSchema ? (context: X) => Promise> : (args: Static, context: X) => Promise>; diff --git a/packages/chord/src/next/contracts/declare.ts b/packages/chord/src/next/contracts/declare.ts index 8bd3f5533..1ef0b0ee8 100644 --- a/packages/chord/src/next/contracts/declare.ts +++ b/packages/chord/src/next/contracts/declare.ts @@ -16,11 +16,11 @@ import type { MemberNames, MethodAt, VariantShape } from "./history.ts"; import type { Address } from "./host-kind.ts"; import { checkName } from "./names.ts"; import { type Params, type RequiredFields, readParams } from "./params.ts"; -import { KeyedMember, Method, type Static, type TSchema } from "./schema.ts"; +import { KeyedMember, Method, type Schema, type Static } from "./schema.ts"; import { type Snapshot, snapshotArguments, snapshotSchema } from "./snapshot.ts"; type VariantDeclaration = { - readonly state?: { readonly [member: string]: TSchema }; + readonly state?: { readonly [member: string]: Schema }; readonly methods?: { readonly [member: string]: Method }; }; @@ -42,7 +42,7 @@ type LocalVariants = { }; /** A local contract's member: a state schema, `method(...)` or `keyed(schema)`. */ -type LocalMember = TSchema | Method | KeyedMember; +type LocalMember = Schema | Method | KeyedMember; type SchemaOf = M extends KeyedMember ? S : never; diff --git a/packages/chord/src/next/contracts/evolve.ts b/packages/chord/src/next/contracts/evolve.ts index 04462793b..d3cb975f4 100644 --- a/packages/chord/src/next/contracts/evolve.ts +++ b/packages/chord/src/next/contracts/evolve.ts @@ -16,7 +16,7 @@ import { import { checkMembers, memberPath, readMethod } from "./declare.ts"; import { materialize } from "./fingerprint.ts"; import type { Change, CurrentStep, Increment, MemberNames, MethodAt, Variants, VariantsWith } from "./history.ts"; -import { Method, type TSchema } from "./schema.ts"; +import { Method, type Schema } from "./schema.ts"; import { type Snapshot, snapshotArguments, snapshotSchema } from "./snapshot.ts"; /** Method edits, the same for both kinds: each changes the method in every variant that has it. */ @@ -25,7 +25,7 @@ interface MethodEditor { readonly "~variants"?: V; /** Type only: the changes made by the edits so far. */ readonly "~changes"?: C; - args, A extends TSchema>( + args, A extends Schema>( member: M, args: A, ): Editor< @@ -33,7 +33,7 @@ interface MethodEditor { Put, "methods", M, Method["result"]>>, C | [VariantsWith, M, "args"] >; - result, R extends TSchema>( + result, R extends Schema>( member: M, result: R, ): Editor< @@ -41,7 +41,7 @@ interface MethodEditor { Put, "methods", M, Method["args"], R>>, C | [VariantsWith, M, "result"] >; - method, A extends TSchema, R extends TSchema>( + method, A extends Schema, R extends Schema>( member: M, args: A, result: R, @@ -51,12 +51,12 @@ interface MethodEditor { /** Edits of a remote contract. */ interface RemoteEditor extends MethodEditor<"remote", V, C> { - state( + state( variant: X, member: M, schema: S, ): RemoteEditor, C | [X, M, "state"]>; - addState( + addState( variant: X, member: M, schema: S, @@ -65,7 +65,7 @@ interface RemoteEditor extends MethodEdito variant: X, member: M, ): RemoteEditor, C>; - addMethod( + addMethod( variant: X, member: M, args: A, @@ -75,26 +75,26 @@ interface RemoteEditor extends MethodEdito /** Edits of a local contract: like remote ones, without a variant. */ interface LocalEditor extends MethodEditor<"local", V, C> { - state( + state( member: M, schema: S, ): LocalEditor, C | [Local, M, "state"]>; - addState( + addState( member: M, schema: S, ): LocalEditor, C | [Local, M, "state"]>; removeState(member: M): LocalEditor, C>; - addMethod( + addMethod( member: M, args: A, result: R, ): LocalEditor>, C | [Local, M, "args" | "result"]>; /** Changes keyed member `member`'s schema (slice 7b: an older consumer reads each key mapped, or is refused). */ - keyed & string, S extends TSchema>( + keyed & string, S extends Schema>( member: M, schema: S, ): LocalEditor, C | [Local, M, "keyed"]>; - addKeyed( + addKeyed( member: M, schema: S, ): LocalEditor, C | [Local, M, "keyed"]>; @@ -220,14 +220,14 @@ class Draft { this.#kinds = new Map(data.kinds); } - state(variant: string, member: string, schema: TSchema): object { + state(variant: string, member: string, schema: Schema): object { if (!this.#variant(variant).state.has(member)) { throw new Error(`${this.#path(variant, member, "state")}: no such state member`); } return this.#putState(variant, member, schema); } - addState(variant: string, member: string, schema: TSchema): object { + addState(variant: string, member: string, schema: Schema): object { this.#refuseDeclared(variant, member, "state"); return this.#putState(variant, member, schema); } @@ -239,19 +239,19 @@ class Draft { return this.editor; } - args(member: string, args: TSchema): object { + args(member: string, args: Schema): object { return this.#changeMethod(member, args, undefined); } - result(member: string, result: TSchema): object { + result(member: string, result: Schema): object { return this.#changeMethod(member, undefined, result); } - method(member: string, args: TSchema, result: TSchema): object { + method(member: string, args: Schema, result: Schema): object { return this.#changeMethod(member, args, result); } - addMethod(variant: string, member: string, args: TSchema, result: TSchema): object { + addMethod(variant: string, member: string, args: Schema, result: Schema): object { this.#refuseDeclared(variant, member, "method"); this.#variant(variant).methods.set( member, @@ -261,14 +261,14 @@ class Draft { return this.#checked(); } - keyed(variant: string, member: string, schema: TSchema): object { + keyed(variant: string, member: string, schema: Schema): object { if (!this.#variant(variant).keyed.has(member)) { throw new Error(`${this.#path(variant, member, "keyed")}: no such keyed member`); } return this.#putKeyed(variant, member, schema); } - addKeyed(variant: string, member: string, schema: TSchema): object { + addKeyed(variant: string, member: string, schema: Schema): object { this.#refuseDeclared(variant, member, "keyed"); return this.#putKeyed(variant, member, schema); } @@ -292,20 +292,20 @@ class Draft { return { ...this.#data, versions: [...versions, version], kinds: this.#kinds }; } - #putState(variant: string, member: string, schema: TSchema): object { + #putState(variant: string, member: string, schema: Schema): object { this.#variant(variant).state.set(member, snapshotSchema(schema, this.#path(variant, member, "state"))); this.#changes.push([variant, member, "state"]); return this.#checked(); } - #putKeyed(variant: string, member: string, schema: TSchema): object { + #putKeyed(variant: string, member: string, schema: Schema): object { this.#variant(variant).keyed.set(member, snapshotSchema(schema, this.#path(variant, member, "keyed"))); this.#changes.push([variant, member, "keyed"]); return this.#checked(); } /** Changes the given sides of a method in every variant that has it. */ - #changeMethod(member: string, args: TSchema | undefined, result: TSchema | undefined): object { + #changeMethod(member: string, args: Schema | undefined, result: Schema | undefined): object { const variants = this.#variantsWith(member); const path = this.#path(variants[0]!, member, "method"); const newArgs = args === undefined ? undefined : snapshotArguments(args, `${path}.args`); @@ -355,16 +355,16 @@ class Draft { /** The local editor: the draft's edits with the variant filled in. */ function localEditor(draft: Draft): object { draft.editor = { - state: (member: string, schema: TSchema) => draft.state(LOCAL, member, schema), - addState: (member: string, schema: TSchema) => draft.addState(LOCAL, member, schema), + state: (member: string, schema: Schema) => draft.state(LOCAL, member, schema), + addState: (member: string, schema: Schema) => draft.addState(LOCAL, member, schema), removeState: (member: string) => draft.removeState(LOCAL, member), - addMethod: (member: string, args: TSchema, result: TSchema) => draft.addMethod(LOCAL, member, args, result), - args: (member: string, args: TSchema) => draft.args(member, args), - result: (member: string, result: TSchema) => draft.result(member, result), - method: (member: string, args: TSchema, result: TSchema) => draft.method(member, args, result), + addMethod: (member: string, args: Schema, result: Schema) => draft.addMethod(LOCAL, member, args, result), + args: (member: string, args: Schema) => draft.args(member, args), + result: (member: string, result: Schema) => draft.result(member, result), + method: (member: string, args: Schema, result: Schema) => draft.method(member, args, result), removeMethod: (member: string) => draft.removeMethod(member), - keyed: (member: string, schema: TSchema) => draft.keyed(LOCAL, member, schema), - addKeyed: (member: string, schema: TSchema) => draft.addKeyed(LOCAL, member, schema), + keyed: (member: string, schema: Schema) => draft.keyed(LOCAL, member, schema), + addKeyed: (member: string, schema: Schema) => draft.addKeyed(LOCAL, member, schema), removeKeyed: (member: string) => draft.removeKeyed(LOCAL, member), }; return draft.editor; diff --git a/packages/chord/src/next/contracts/history.ts b/packages/chord/src/next/contracts/history.ts index a664f98ff..2a3fb1a80 100644 --- a/packages/chord/src/next/contracts/history.ts +++ b/packages/chord/src/next/contracts/history.ts @@ -1,13 +1,13 @@ // The type-level history of a contract: every version's members and the changes that produced it. -import type { Method, TSchema } from "./schema.ts"; +import type { Method, Schema } from "./schema.ts"; /** One variant's members at one version: state schemas and method signatures. */ export type VariantShape = { - readonly state: { readonly [member: string]: TSchema }; + readonly state: { readonly [member: string]: Schema }; readonly methods: { readonly [member: string]: Method }; /** A local contract's keyed state members (slice 6a §12). */ - readonly keyed?: { readonly [member: string]: TSchema }; + readonly keyed?: { readonly [member: string]: Schema }; }; /** Variants by name. A local contract has the single variant `members`. */ diff --git a/packages/chord/src/next/contracts/index.ts b/packages/chord/src/next/contracts/index.ts index 299bac9d3..da4a7fbe9 100644 --- a/packages/chord/src/next/contracts/index.ts +++ b/packages/chord/src/next/contracts/index.ts @@ -7,24 +7,30 @@ export { evolve } from "./evolve.ts"; export { type Address, type HostKind, hostKind, type KindShape, type ParamsOfKind } from "./host-kind.ts"; export { arr, + type BooleanSchema, type Bounds, bool, bytes, type Defaulted, + type IntegerSchema, int, type KeyedMember, keyed, lit, map, method, + type NullSchema, + type NumberSchema, nul, num, + type Optional, obj, oneOf, opt, + type Schema, type Static, + type StringSchema, str, - type TSchema, union, unknown, } from "./schema.ts"; diff --git a/packages/chord/src/next/contracts/params.ts b/packages/chord/src/next/contracts/params.ts index 565c88500..d827edc3a 100644 --- a/packages/chord/src/next/contracts/params.ts +++ b/packages/chord/src/next/contracts/params.ts @@ -1,26 +1,35 @@ // Params declarations: the policy for params fields and the closed object schema checked on every get. -import type { TBoolean, TInteger, TLiteral, TString, TUnion } from "typebox"; import type { JsonValue } from "../../json.ts"; -import type { TSchema } from "./schema.ts"; +import { + type BooleanSchema, + type IntegerSchema, + isOptional, + type LiteralsSchema, + type Optional, + type Schema, + type StringSchema, +} from "./schema.ts"; import { snapshotSchema } from "./snapshot.ts"; -/** A params field: a string, an integer, a boolean or a union of string literals; constraints allowed. */ -type ParamSchema = TString | TInteger | TBoolean | TUnion[]>; +/** A params field: a string, an integer, a boolean or a union of string literals; constraints allowed, no default. */ +type ParamSchema = (StringSchema | IntegerSchema | BooleanSchema | LiteralsSchema) & { + readonly default?: never; +}; /** Params: a flat record of required fields, in identity order. */ export type Params = { readonly [field: string]: ParamSchema }; /** Refuses optional params fields, which the field types alone admit. */ export type RequiredFields

    = { - [F in keyof P]: P[F] extends { readonly "~optional": true } ? never : P[F]; + [F in keyof P]: P[F] extends Optional ? never : P[F]; }; export type ParamsData = { /** Field names in identity order. */ readonly fields: readonly string[]; /** The closed object schema of all fields. */ - readonly schema: TSchema; + readonly schema: Schema; /** `[field, normal form]` pairs in identity order. */ readonly normal: JsonValue; }; @@ -29,10 +38,10 @@ export type ParamsData = { export function readParams(declared: readonly [string, unknown][], path: string): ParamsData { const fields = declared.map(([field, schema]) => { const fieldPath = `${path}.${field}`; + const refused = new Error(`${fieldPath}: a param is a required str(), int(), bool() or oneOf(...) of strings`); + if (isOptional(schema)) throw refused; const snapshot = snapshotSchema(schema, fieldPath); - if (Object.hasOwn(schema as object, "~optional") || !isParamSchema(snapshot.normal)) { - throw new Error(`${fieldPath}: a param is a required str(), int(), bool() or oneOf(...) of strings`); - } + if (!isParamSchema(snapshot.normal)) throw refused; return [field, snapshot] as const; }); const schema = { diff --git a/packages/chord/src/next/contracts/schema.ts b/packages/chord/src/next/contracts/schema.ts index fc987dbb2..230d3a039 100644 --- a/packages/chord/src/next/contracts/schema.ts +++ b/packages/chord/src/next/contracts/schema.ts @@ -1,24 +1,26 @@ -// Schema helpers: short constructors for TypeBox schemas, the bytes leaf, and method declarations. +// Schema helpers: Chord's short constructors over pi-schema's builders (plain JSON Schema, typed with the values they +// accept), and method and keyed member declarations. The helpers keep the shapes contracts were fingerprinted with: +// literals carry their `type`, `oneOf` is a union of literals, `map` a pattern-property record. -import Type, { - type TArray, - type TBoolean, - type TInteger, - type TLiteral, - type TNull, - type TNumber, - type TObject, - type TOptional, - type TProperties, - type TRecord, - type TSchema, - type TString, - type TUnion, - type TUnknown, - type TUnsafe, -} from "typebox"; +import { + arr as arrSchema, + bool as boolSchema, + bytes as bytesSchema, + int as intSchema, + nul as nulSchema, + num as numSchema, + Optional, + obj as objSchema, + type Properties, + type Schema, + type Static, + type StaticObject, + str as strSchema, + union as unionSchema, + unknown as unknownSchema, +} from "@earendil-works/pi-schema"; -export type { Static, TSchema } from "typebox"; +export { Optional, opt, type Schema, type Static } from "@earendil-works/pi-schema"; /** Bounds of a string length, a number or an array length. */ export type Bounds = { readonly min?: number; readonly max?: number }; @@ -26,118 +28,132 @@ export type Bounds = { readonly min?: number; readonly max?: number }; /** A declared default value: a table column with one may be left out of a `put` (slice 6a §5). */ export type Defaulted = { readonly default: T }; +// Interfaces, not intersections: declarations of apps' exported contracts and tables name them (an intersection with +// `Schema` would print pi-schema's private brand, which declaration files cannot name). + +/** A string schema (`str()`), told apart from the other scalars by its `type`. */ +export interface StringSchema extends Schema { + readonly type: "string"; +} + +/** A number schema (`num()`). */ +export interface NumberSchema extends Schema { + readonly type: "number"; +} + +/** An integer schema (`int()`). */ +export interface IntegerSchema extends Schema { + readonly type: "integer"; +} + +/** A boolean schema (`bool()`). */ +export interface BooleanSchema extends Schema { + readonly type: "boolean"; +} + +/** The `null` schema (`nul()`): also the omitted side of a method. */ +export interface NullSchema extends Schema { + readonly type: "null"; +} + +/** A union of literals (`oneOf(...)`). */ +export interface LiteralsSchema extends Schema { + readonly anyOf: readonly Schema[]; +} + /** A string, with optional length bounds, pattern and default. */ export function str( options: Bounds & { readonly pattern?: string; readonly default: string }, -): TString & Defaulted; -export function str(options?: Bounds & { readonly pattern?: string }): TString; -export function str(options: Bounds & { readonly pattern?: string; readonly default?: string } = {}): TString { +): StringSchema & Defaulted; +export function str(options?: Bounds & { readonly pattern?: string }): StringSchema; +export function str(options: Bounds & { readonly pattern?: string; readonly default?: string } = {}): StringSchema { const pattern = options.pattern === undefined ? {} : { pattern: options.pattern }; - return Type.String({ ...limits(options, "minLength", "maxLength"), ...pattern, ...defaultOf(options) }); + return strSchema({ + ...limits(options, "minLength", "maxLength"), + ...pattern, + ...defaultOf(options), + }) as StringSchema; } /** A finite number within optional bounds, with an optional default. */ -export function num(options: Bounds & { readonly default: number }): TNumber & Defaulted; -export function num(options?: Bounds): TNumber; -export function num(options: Bounds & { readonly default?: number } = {}): TNumber { - return Type.Number({ ...limits(options, "minimum", "maximum"), ...defaultOf(options) }); +export function num(options: Bounds & { readonly default: number }): NumberSchema & Defaulted; +export function num(options?: Bounds): NumberSchema; +export function num(options: Bounds & { readonly default?: number } = {}): NumberSchema { + return numSchema({ ...limits(options, "minimum", "maximum"), ...defaultOf(options) }) as NumberSchema; } /** An integer within optional bounds, with an optional default. */ -export function int(options: Bounds & { readonly default: number }): TInteger & Defaulted; -export function int(options?: Bounds): TInteger; -export function int(options: Bounds & { readonly default?: number } = {}): TInteger { - return Type.Integer({ ...limits(options, "minimum", "maximum"), ...defaultOf(options) }); +export function int(options: Bounds & { readonly default: number }): IntegerSchema & Defaulted; +export function int(options?: Bounds): IntegerSchema; +export function int(options: Bounds & { readonly default?: number } = {}): IntegerSchema { + return intSchema({ ...limits(options, "minimum", "maximum"), ...defaultOf(options) }) as IntegerSchema; } /** A boolean, with an optional default. */ -export function bool(options: { readonly default: boolean }): TBoolean & Defaulted; -export function bool(options?: Record): TBoolean; -export function bool(options: { readonly default?: boolean } = {}): TBoolean { - return Type.Boolean(defaultOf(options)); +export function bool(options: { readonly default: boolean }): BooleanSchema & Defaulted; +export function bool(options?: Record): BooleanSchema; +export function bool(options: { readonly default?: boolean } = {}): BooleanSchema { + return boolSchema(defaultOf(options)) as BooleanSchema; } /** The `default` keyword of `options`, left out when absent (snapshots refuse `undefined`). */ -function defaultOf(options: { readonly default?: string | number | boolean }): { default?: string | number | boolean } { +function defaultOf(options: { readonly default?: T }): { default?: T } { return options.default === undefined ? {} : { default: options.default }; } /** `null`; also the omitted side of a method. */ -export function nul(): TNull { - return Type.Null(); +export function nul(): NullSchema { + // The builder's `type: "null"`, typed as such. + return nulSchema() as NullSchema; } -/** Exactly the given value. */ -export function lit(value: V): TLiteral { - return Type.Literal(value); +/** Exactly the given value, with its JSON type (`{ type: "string", const: "a" }`). */ +export function lit(value: V): Schema { + return { type: typeof value, const: value }; } -/** One of the given literals. */ -export function oneOf(...values: V): TUnion> { - // One literal schema per value, in order. - return Type.Union(values.map((value) => Type.Literal(value))) as TUnion>; +/** A value `lit(...)` and `oneOf(...)` take. */ +export type Literal = string | number | boolean; + +/** One of the given literals: a union of `lit(...)` schemas, in order. */ +export function oneOf(...values: V): LiteralsSchema { + return { anyOf: values.map((value) => lit(value)) }; } -type Literal = string | number | boolean; - -type Literals = V extends [infer First extends Literal, ...infer Rest extends Literal[]] - ? [TLiteral, ...Literals] - : []; - /** Any of the given schemas. */ -export function union(schemas: [...S]): TUnion { - return Type.Union(schemas); +export function union(schemas: S): Schema> { + return unionSchema(schemas); } -/** A closed object: unknown fields are refused. */ -export function obj

    (properties: P): TObject

    { - return Type.Object(properties, { additionalProperties: false }); +/** A closed object: unknown fields are refused; `opt(...)` marks an optional field. */ +export function obj(properties: P): Schema> { + return objSchema(properties); } /** An array, with optional length bounds. */ -export function arr(items: S, options: Bounds = {}): TArray { - return Type.Array(items, limits(options, "minItems", "maxItems")); +export function arr(items: Schema, options: Bounds = {}): Schema { + return arrSchema(items, limits(options, "minItems", "maxItems")); } -/** A string-keyed record. */ -export function map(value: S): TRecord<"^.*$", S> { - return Type.Record(Type.String(), value); -} - -/** An optional object field. */ -export function opt(schema: S): TOptional { - return Type.Optional(schema) as TOptional; +/** A string-keyed record: one pattern property matching every key. */ +export function map(value: Schema): Schema> { + return { type: "object", patternProperties: { "^.*$": value } }; } /** Any value. */ -export function unknown(): TUnknown { - return Type.Unknown(); +export function unknown(): Schema { + return unknownSchema(); } /** - * A `Uint8Array` leaf, checked by `instanceof` and length, never enumerated. Its enumerable form is the data marker - * `{ type: "bytes", minLength?, maxLength? }`. + * A `Uint8Array` leaf, checked by `instanceof` and length, never enumerated: `{ type: "bytes", minLength?, maxLength? }` + * (pi-schema's extension of JSON Schema for values that never pass through JSON). */ -export function bytes(options: Bounds = {}): TUnsafe { - return withBytesCheck({ type: "bytes", ...limits(options, "minLength", "maxLength") }); +export function bytes(options: Bounds = {}): Schema { + return bytesSchema(limits(options, "minLength", "maxLength")); } -/** The data marker of a bytes schema. */ -export type BytesMarker = { readonly type: "bytes"; readonly minLength?: number; readonly maxLength?: number }; - -/** Attaches the TypeBox check of a bytes marker, non-enumerably so it is never copied as data. */ -export function withBytesCheck(marker: BytesMarker): TUnsafe { - const { minLength = 0, maxLength = Number.POSITIVE_INFINITY } = marker; - const refinement = { - check: (value: unknown) => value instanceof Uint8Array && value.length >= minLength && value.length <= maxLength, - error: () => "expected bytes of the declared length", - }; - Object.defineProperty(marker, "~refine", { value: [refinement] }); - // The marker object is the schema; TypeBox reads the attached check. - return marker as unknown as TUnsafe; -} - -/** TypeBox options for the given bounds; an absent bound is left out, since snapshots refuse `undefined`. */ +/** Builder options for the given bounds; an absent bound is left out, since snapshots refuse `undefined`. */ function limits(options: Bounds, min: string, max: string): { [keyword: string]: number } { const result: { [keyword: string]: number } = {}; if (options.min !== undefined) result[min] = options.min; @@ -145,8 +161,13 @@ function limits(options: Bounds, min: string, max: string): { [keyword: string]: return result; } +/** Whether a declared object field is `opt(...)`. */ +export function isOptional(value: unknown): value is Optional { + return value instanceof Optional; +} + /** A method declaration: its argument and result schemas; an omitted side is `null` on the wire. */ -export class Method { +export class Method { readonly args: A; readonly result: R; @@ -157,7 +178,7 @@ export class Method { } /** A keyed state member's declaration (slice 6a §12): documents of `schema` by string key; only in `local` contracts. */ -export class KeyedMember { +export class KeyedMember { readonly schema: S; constructor(schema: S) { @@ -166,15 +187,15 @@ export class KeyedMember { } /** Declares a keyed state member of a `local` contract: documents of `schema` by string key, each watchable. */ -export function keyed(schema: S): KeyedMember { +export function keyed(schema: S): KeyedMember { return new KeyedMember(schema); } /** Declares a method: `method()`, `method(args)`, `method(args, result)` or `method(undefined, result)`. */ -export function method(): Method; -export function method(args: A): Method; -export function method(args: undefined, result: R): Method; -export function method(args: A, result: R): Method; -export function method(args?: TSchema, result?: TSchema): Method { +export function method(): Method; +export function method(args: A): Method; +export function method(args: undefined, result: R): Method; +export function method(args: A, result: R): Method; +export function method(args?: Schema, result?: Schema): Method { return new Method(args ?? nul(), result ?? nul()); } diff --git a/packages/chord/src/next/contracts/snapshot.ts b/packages/chord/src/next/contracts/snapshot.ts index a84ed79c0..c8ec2eafb 100644 --- a/packages/chord/src/next/contracts/snapshot.ts +++ b/packages/chord/src/next/contracts/snapshot.ts @@ -1,14 +1,14 @@ // Schema snapshots: one schema-aware walk checks an app schema against Chord's vocabulary and produces a // Chord-owned copy and its normal form (the fingerprint input). +import type { JsonSchema } from "@earendil-works/pi-schema"; import { isJsonValue, type JsonValue } from "../../json.ts"; -import { type BytesMarker, type TSchema, withBytesCheck } from "./schema.ts"; /** * A Chord-owned copy of an app schema, its normal form, and whether it has a `bytes()` leaf (values crossing a link * then carry raw byte segments); immutable by convention. */ -export type Snapshot = { readonly schema: TSchema; readonly normal: JsonValue; readonly bytes: boolean }; +export type Snapshot = { readonly schema: JsonSchema; readonly normal: JsonValue; readonly bytes: boolean }; /** Snapshot of a state, result or params schema. Throws an Error naming `path` for anything outside the vocabulary. */ export function snapshotSchema(source: unknown, path: string): Snapshot { @@ -51,11 +51,6 @@ const KEYWORDS = new Map( const TYPES = new Set(["null", "boolean", "number", "integer", "string", "array", "object", "bytes"]); -/** TypeBox's executable features, attached non-enumerably. */ -const EXECUTABLE = ["~refine", "~codec", "~unsafe"]; - -const BYTES_EXECUTABLE = ["~refine", "~unsafe"]; - const BYTES_KEYWORDS = new Set(["type", "minLength", "maxLength"]); function walk(source: unknown, path: string, objects: Objects, ancestors: Set): Snapshot { @@ -63,13 +58,6 @@ function walk(source: unknown, path: string, objects: Objects, ancestors: Set): Copy { @@ -167,15 +151,15 @@ function sortedCopy(value: JsonValue): JsonValue { return Object.fromEntries(keys.map((key) => [key, sortedCopy(value[key]!)])); } -/** Enumerable data entries sorted by key; symbol keys and accessors are refused. */ +/** Data entries sorted by key; symbol keys, accessors and non-enumerable keys (not data) are refused. */ function ownEntries(source: object, path: string): [string, unknown][] { const entries: [string, unknown][] = []; for (const key of Reflect.ownKeys(source)) { if (typeof key === "symbol") throw new Error(`${path}: symbol keys are not supported`); const descriptor = Object.getOwnPropertyDescriptor(source, key)!; if (!("value" in descriptor)) throw new Error(`${path}.${key}: accessors are not supported`); - // Non-enumerable keys are TypeBox metadata (`~kind`, `~optional`), never schema data. - if (descriptor.enumerable) entries.push([key, descriptor.value]); + if (!descriptor.enumerable) throw new Error(`${path}.${key}: non-enumerable keys are not supported`); + entries.push([key, descriptor.value]); } return entries.sort(([a], [b]) => compare(a, b)); } diff --git a/packages/chord/src/next/data/README.md b/packages/chord/src/next/data/README.md index c3caba867..74248adb4 100644 --- a/packages/chord/src/next/data/README.md +++ b/packages/chord/src/next/data/README.md @@ -51,8 +51,9 @@ Rules: them. Results and state are never checked. Nothing in L1 is frozen. - A method without arguments (argument schema `null`) omits the argument in references and implementations; `null` is its internal and wire form, which `checkArgs` checks. Unknown fields fail. -- A failed check is `invalid/args`, `invalid/params` or `invalid/host_address` with `details.path`, a JSON Pointer from TypeBox's first error. - TypeBox does not escape `~` and `/`, so keys containing them make the path ambiguous. +- A failed check is `invalid/args`, `invalid/params` or `invalid/host_address` with `details.path`, the JSON Pointer of + the first error pi-schema explains (`~` and `/` in keys escaped as `~0` and `~1`); a missing field is reported at its + own path ("missing"), an unknown one too ("unknown field"), the value itself at `/`. - `checkArgs` refuses an unknown method with `invalid/member` and an unaccepted version or a refused method with `contract_mismatch`. It never invokes a `call` replacement. - Error codes, reasons and their pairings are link protocol: changing them bumps `PROTOCOL_VERSION`. diff --git a/packages/chord/src/next/data/args.ts b/packages/chord/src/next/data/args.ts index 3239d263e..83fe60013 100644 --- a/packages/chord/src/next/data/args.ts +++ b/packages/chord/src/next/data/args.ts @@ -1,6 +1,6 @@ // Arguments: a provider's binding (its dispatch table per accepted version) and the per-call argument check. -import type { Validator } from "typebox/compile"; +import type { Validator } from "@earendil-works/pi-schema"; import type { Context } from "../../context/index.ts"; import { type AcceptedVersion, type AcceptsTable, checkAccepts } from "../contracts/accepts.ts"; import { diff --git a/packages/chord/src/next/data/validator.ts b/packages/chord/src/next/data/validator.ts index 4a16599de..f7995f9ab 100644 --- a/packages/chord/src/next/data/validator.ts +++ b/packages/chord/src/next/data/validator.ts @@ -1,28 +1,19 @@ -// Compiled checks: TypeBox validators of Chord-owned schemas, and the `invalid` error of a failed check. +// Compiled checks: pi-schema validators of Chord-owned schemas, and the `invalid` error of a failed check. -import { Compile, type Validator } from "typebox/compile"; -import type { TLocalizedValidationError } from "typebox/error"; -import { ResetExternal, ResetFunctions } from "typebox/schema"; -import { Errors } from "typebox/value"; +import { compile, type JsonSchema, type Validator } from "@earendil-works/pi-schema"; import type { Context } from "../../context/index.ts"; -import type { TSchema } from "../contracts/schema.ts"; import { type ChordError, invalid } from "./errors.ts"; /** A Chord-owned record holding a schema: a snapshot, or a contract's params. */ -type SchemaOwner = { readonly schema: TSchema }; +type SchemaOwner = { readonly schema: JsonSchema }; /** The validator of an owner's schema, compiled on first use (counted) and cached weakly by the owner. */ export function validatorFor(owner: SchemaOwner, context: Context): Validator { let validator = validators.get(owner); if (validator === undefined) { context.telemetry.metrics?.counter("chord.data.compilations").add(1); - try { - validator = Compile(owner.schema); - } finally { - // TypeBox keeps its last build in module-level accumulators; reset them so no build state is retained. - ResetFunctions(); - ResetExternal(); - } + // `format` is an annotation (decided for the schema switch): no format checks are compiled in. + validator = compile(owner.schema); validators.set(owner, validator); } return validator; @@ -38,35 +29,23 @@ export function check( context: Context, ): void { context.telemetry.metrics?.counter("chord.data.validations").add(1); - if (!validator.Check(value)) throw failedCheck(validator.Type(), value, reason, context); + if (!validator.check(value)) throw failedCheck(validator, value, reason, context); } /** - * The error of a failed check: TypeBox's first error from its diagnostic walk (counted; it builds the whole list). - * TypeBox does not escape `~` and `/` in its instance paths, so keys containing them make the path ambiguous. + * The error of a failed check: the first error of the validator's explanation (counted: it walks the schema). A + * missing field is reported at the field ("missing"), an unknown one at its own path ("unknown field"); the value + * itself at `/`. */ function failedCheck( - schema: TSchema, + validator: Validator, value: unknown, reason: "args" | "params" | "host_address", context: Context, ): ChordError { context.telemetry.metrics?.counter("chord.data.walks").add(1); - const [error] = Errors(schema, value); - const { path, problem } = describe(error); - return invalid(reason, `${reason}${path === "/" ? "" : path}: ${problem}`, { path }); -} - -/** - * The path and problem of a TypeBox error: TypeBox reports a missing field at its parent, and an unknown one as a - * false schema (`additionalProperties: false`) at its own path. Anything else keeps TypeBox's message. - */ -function describe(error: TLocalizedValidationError): { path: string; problem: string } { - if (error.keyword === "required") { - return { path: `${error.instancePath}/${error.params.requiredProperties[0]}`, problem: "missing" }; - } - if (error.keyword === "boolean" && error.schemaPath.endsWith("/additionalProperties")) { - return { path: error.instancePath, problem: "unknown field" }; - } - return { path: error.instancePath || "/", problem: error.message }; + // A value that failed the check has at least one error. + const error = validator.errors(value)[0]!; + const path = error.path || "/"; + return invalid(reason, `${reason}${path === "/" ? "" : path}: ${error.message}`, { path }); } diff --git a/packages/chord/src/next/documents/definition.ts b/packages/chord/src/next/documents/definition.ts index e9a2eecf1..26c2105e6 100644 --- a/packages/chord/src/next/documents/definition.ts +++ b/packages/chord/src/next/documents/definition.ts @@ -4,7 +4,7 @@ import type { Context } from "../../context/index.ts"; import type { Op } from "../../delta/index.ts"; import type { JsonValue } from "../../json.ts"; -import type { Static, TSchema } from "../contracts/index.ts"; +import type { Schema, Static } from "../contracts/index.ts"; import { checkName, fingerprintOf } from "../contracts/internal.ts"; import { snapshotSchema } from "../contracts/snapshot.ts"; import { validatorFor } from "../data/internal.ts"; @@ -31,7 +31,7 @@ export type CheckpointInfo = { export type CheckpointWhen = (value: T, ops: readonly Op[], info: CheckpointInfo) => boolean; /** What `defineState` takes for any document. */ -type StateOptions = { +type StateOptions = { readonly schema: S; readonly version: number; readonly initial: () => DocumentOf; @@ -41,7 +41,7 @@ type StateOptions = { }; /** What a persisted document adds (slice 6a §10): `checkpointWhen` is required. */ -type PersistOptions = { +type PersistOptions = { /** Stored in the host's store; only in `local` types with empty params (P3). */ readonly persist: true; /** Called once per commit per changed document; creations and version changes always store a base. */ @@ -52,15 +52,15 @@ type PersistOptions = { * Declares a document: a singleton per object, or with `keyed: true` many per object, each by a string key; with * `persist: true` stored in the host's store. Its name is a restricted name (lowercase `[a-z0-9@/._~-]`). */ -export function defineState( +export function defineState( name: string, options: StateOptions & { readonly persist?: false; readonly checkpointWhen?: never }, ): StateDefinition, K, false>; -export function defineState( +export function defineState( name: string, options: StateOptions & PersistOptions, ): StateDefinition, K, true>; -export function defineState( +export function defineState( name: string, options: StateOptions & { readonly persist?: boolean; @@ -82,14 +82,14 @@ export function defineState( } /** The value of a document whose schema is `S`: its static type, which must be a JSON object. */ -type DocumentOf = Static extends JsonObject ? Static : never; +type DocumentOf = Static extends JsonObject ? Static : never; const NEVER: CheckpointWhen = () => false; /** A declared document; `T` is its value, `K` whether it is keyed, `P` whether it is persisted. */ export class StateDefinition { readonly name: string; - readonly schema: TSchema; + readonly schema: Schema; readonly version: number; readonly keyed: boolean; readonly persist: boolean; @@ -110,7 +110,7 @@ export class StateDefinition T, migrate: ((old: JsonObject, from: number) => T) | undefined, @@ -159,7 +159,7 @@ export class StateDefinition | undefined, > = { /** The job's version; every change of its args, result or state bumps it. */ @@ -40,7 +40,7 @@ export type JobOptions< export type JobPersistence = { readonly resume: "rerun" | "abandon" }; /** Per older version, its mappings forwards: its args to the current args, its result to the current result. */ -export type JobAccepts = { +export type JobAccepts = { readonly [version: number]: { readonly args: (old: never) => Static; readonly result: (old: never) => Static }; }; @@ -66,8 +66,8 @@ export type JobError = /** Declares a job: owned work a call starts that outlives it (§15). */ export function defineJob< - const A extends TSchema, - const R extends TSchema, + const A extends Schema, + const R extends Schema, D extends StateDefinition | undefined = undefined, >(name: string, options: JobOptions): JobDefinition, Static, StateOf> { // Declaration policy: JavaScript callers reach every check below. diff --git a/packages/chord/src/next/link/README.md b/packages/chord/src/next/link/README.md index 806bb4f19..d68605b53 100644 --- a/packages/chord/src/next/link/README.md +++ b/packages/chord/src/next/link/README.md @@ -72,7 +72,7 @@ tab.connect(webSocketLink(new WebSocket(url)), { principal: { kind: "server" } } ## For Chord's layers (`internal.ts`) -- `Message` (`message.ts`): the protocol's one tagged union, from the TypeBox schema `MESSAGE`: `hello { protocol, +- `Message` (`message.ts`): the protocol's one tagged union, from the schema `MESSAGE`: `hello { protocol, incarnation, calls }` (`calls`: how many concurrent calls this side accepts per origin, its call window), `hold { hold, type, version, fingerprint, params, address, principal?, incarnation? }` (`address`: `{ kind, params }` the caller's `hostAddress` computed, it routes the hold; the provider places it by its params; `principal` and `incarnation`: whom it acts for, read only diff --git a/packages/chord/src/next/link/message.ts b/packages/chord/src/next/link/message.ts index 21ac9e0d4..745ce0ad7 100644 --- a/packages/chord/src/next/link/message.ts +++ b/packages/chord/src/next/link/message.ts @@ -1,7 +1,7 @@ -// Messages (§22, R1, R4, 2b): the one `Message` type every link carries, defined as a TypeBox schema and checked after -// every decode, and the codecs that turn it into bytes. +// Messages (§22, R1, R4, 2b): the one `Message` type every link carries, defined as a schema and checked after every +// decode, and the codecs that turn it into bytes. -import Type from "typebox"; +import { obj as openObj, type Schema } from "@earendil-works/pi-schema"; import type { Context } from "../../context/index.ts"; import { arr, int, lit, obj, oneOf, opt, type Static, str, union, unknown } from "../contracts/index.ts"; import { validatorFor } from "../data/internal.ts"; @@ -30,19 +30,19 @@ const wireError = union([ const failed = arr(obj({ member: index, error: wireError })); /** A part's envelope: the codec built it, so any value passes (its content is the owner's to parse). */ -const part = Type.Unsafe({}); +const part = unknown() as Schema; /** * The byte segments of a payload part whose method's schema has a `bytes()` leaf, referenced from its JSON by * `{"$bytes": i}`; absent for every other payload. Built by the codec, like the part. */ -const segments = opt(Type.Unsafe({})); +const segments = opt(unknown() as Schema); /** An address as a hold names it: its host kind and params, checked against the kind where it is routed. */ const address = obj({ kind: str(), params: unknown() }); /** A principal on a hold: JSON with a `kind`, honoured only on a parent link. */ -const principal = Type.Object({ kind: str() }); +const principal = openObj({ kind: str() }, { open: true }); /** A uuidv7 as text. */ const incarnation = str({ pattern: "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$" }); @@ -74,7 +74,7 @@ const MESSAGE = union([ principal: opt(principal), incarnation: opt(incarnation), // The instance ids of the activations waiting for this hold (Q5); read only on a trusted link. - chain: opt(Type.Array(incarnation, { maxItems: CHAIN })), + chain: opt(arr(incarnation, { max: CHAIN })), // The decoded limit per part of the link the hold was admitted on (Q2); read only on a parent link. limit: opt(int({ min: 0 })), }), @@ -185,7 +185,7 @@ const MESSAGE_SCHEMA = { schema: MESSAGE }; */ export function checkMessage(message: unknown, context: Context): Message { context.telemetry.metrics?.counter("chord.link.checks").add(1); - if (!validatorFor(MESSAGE_SCHEMA, context).Check(message)) throw new Error("message: fails the message schema"); + if (!validatorFor(MESSAGE_SCHEMA, context).check(message)) throw new Error("message: fails the message schema"); // The check above establishes the type. return message as Message; } diff --git a/packages/chord/src/next/store/archive.ts b/packages/chord/src/next/store/archive.ts index 1db668048..922057994 100644 --- a/packages/chord/src/next/store/archive.ts +++ b/packages/chord/src/next/store/archive.ts @@ -2,9 +2,21 @@ // are operations", "Writes carry their versions; archives are lossless"): the parts an export yields and an import // takes, their order, and the compiled schemas every imported part and every applied write is checked against first. -import Type, { type TInteger } from "typebox"; import type { Context } from "../../context/index.ts"; -import { arr, bool, int, lit, num, obj, opt, str, union, unknown } from "../contracts/index.ts"; +import { + arr, + bool, + type IntegerSchema, + int, + lit, + map, + num, + obj, + opt, + str, + union, + unknown, +} from "../contracts/index.ts"; import { invalid, validatorFor } from "../data/internal.ts"; import type { DocumentWrite } from "./engine.ts"; import type { TableWrite } from "./table-writes.ts"; @@ -34,7 +46,7 @@ export type ArchivePart = | { readonly kind: "end"; readonly writes: number }; /** An integer from `min` up to 2^53 - 1: seqs, ids, versions and counts stay exact (a seq after 2^53 would repeat). */ -function safe(min: number): TInteger { +function safe(min: number): IntegerSchema { return int({ min, max: Number.MAX_SAFE_INTEGER }); } @@ -48,14 +60,14 @@ const column = obj({ fingerprint: opt(str()), }); -const versions = Type.Record(Type.String(), version); +const versions = map(version); const TABLE = obj({ kind: lit("table"), table: str(), version, state: union([lit("ready"), lit("migrating")]), - columns: Type.Record(Type.String(), column), + columns: map(column), key: arr(str()), fingerprint: str(), }); @@ -63,7 +75,7 @@ const TABLE = obj({ const PUT = obj({ kind: lit("put"), table: str(), - row: Type.Record(Type.String(), unknown()), + row: map(unknown()), version, columns: versions, }); @@ -117,7 +129,7 @@ const edit = union([ obj({ kind: lit("json"), name: str(), column }), ]); -const key = Type.Record(Type.String(), union([str(), num(), bool()])); +const key = map(union([str(), num(), bool()])); /** One recorded write of a commit (a batch's); `where` and `set` nodes are checked by the applier. */ const WRITE = union([ @@ -158,7 +170,7 @@ const WRITE_SCHEMA = { schema: WRITE }; /** Checks one imported part against the archive schema (a trust boundary); `invalid/args` naming its position. */ export function checkPart(part: unknown, position: number, context: Context): ArchivePart { context.telemetry.metrics?.counter("chord.data.validations").add(1); - if (!validatorFor(PART_SCHEMA, context).Check(part)) { + if (!validatorFor(PART_SCHEMA, context).check(part)) { throw invalid("args", `import: part ${position}: not an archive part`); } // The check above establishes the type. @@ -168,7 +180,7 @@ export function checkPart(part: unknown, position: number, context: Context): Ar /** Checks one write of a batch's commit against the write schema (a trust boundary); `what` names its place. */ export function checkWrite(write: unknown, what: string, context: Context): StoreWrite { context.telemetry.metrics?.counter("chord.data.validations").add(1); - if (!validatorFor(WRITE_SCHEMA, context).Check(write)) throw invalid("args", `${what}: not a store write`); + if (!validatorFor(WRITE_SCHEMA, context).check(write)) throw invalid("args", `${what}: not a store write`); // The check above establishes the type, the nodes aside (the applier checks them). return write as StoreWrite; } diff --git a/packages/chord/src/next/store/columns.ts b/packages/chord/src/next/store/columns.ts index 8d4118262..180251efb 100644 --- a/packages/chord/src/next/store/columns.ts +++ b/packages/chord/src/next/store/columns.ts @@ -2,8 +2,8 @@ // subset of L1's helpers a table's `columns` take (`str int num bool`, `opt(...)` of them, `json(...)`, defaults), // their types for rows, keys and column references, and the column descriptors and records the store keeps. -import type { TBoolean, TInteger, TNumber, TOptional, TString } from "typebox"; -import type { Static, TSchema } from "../contracts/index.ts"; +import type { BooleanSchema, IntegerSchema, NumberSchema, Schema, Static, StringSchema } from "../contracts/index.ts"; +import { isOptional, type Optional } from "../contracts/schema.ts"; import { JsonColumn, type JsonColumnSchema } from "./json-column.ts"; import { storedText } from "./text.ts"; @@ -11,9 +11,9 @@ import { storedText } from "./text.ts"; export type ColumnSchema = | ScalarSchema | JsonColumnSchema - | TOptional>; + | Optional>; -type ScalarSchema = TString | TInteger | TNumber | TBoolean; +type ScalarSchema = StringSchema | IntegerSchema | NumberSchema | BooleanSchema; /** A table's `columns`: a flat record of column schemas. */ export type Columns = { readonly [name: string]: ColumnSchema }; @@ -22,23 +22,23 @@ export type Columns = { readonly [name: string]: ColumnSchema }; export type ColumnType = "str" | "int" | "num" | "bool" | "json"; /** The store type of column schema `S`. */ -export type ColumnTypeOf = S extends TOptional +export type ColumnTypeOf = S extends Optional ? ColumnTypeOf - : S extends TString + : S extends StringSchema ? "str" - : S extends TInteger + : S extends IntegerSchema ? "int" - : S extends TNumber + : S extends NumberSchema ? "num" - : S extends TBoolean + : S extends BooleanSchema ? "bool" : "json"; /** The value type of column schema `S` (without `undefined` for an `opt(...)` one). */ -export type ValueOf = S extends TOptional ? ValueOf : S extends TSchema ? Static : never; +export type ValueOf = S extends Optional ? ValueOf : S extends Schema ? Static : never; /** The names of the columns that are not `opt(...)`. */ -type RequiredNames = { [K in keyof C]: C[K] extends TOptional ? never : K }[keyof C]; +type RequiredNames = { [K in keyof C]: C[K] extends Optional ? never : K }[keyof C]; /** The names of the columns with a declared default. */ type DefaultedNames = { @@ -57,7 +57,7 @@ export type PutRowOf = { [K in Exclude, Defa /** The columns a key, an index or an `orderBy` may use: non-`opt` `str` or `int`. */ export type OrderColumn = { - [K in keyof C]: C[K] extends TString | TInteger ? (C[K] extends TOptional ? never : K) : never; + [K in keyof C]: C[K] extends Optional ? never : C[K] extends StringSchema | IntegerSchema ? K : never; }[keyof C] & string; @@ -112,15 +112,15 @@ const SCALAR_TYPES: { readonly [type: string]: "str" | "int" | "num" | "bool" } boolean: "bool", }; -function describeColumn(name: string, schema: ColumnSchema, what: string): Column { - const optional = Object.hasOwn(schema, "~optional"); +function describeColumn(name: string, column: ColumnSchema, what: string): Column { + const optional = isOptional(column); // Declaration policy: a JavaScript caller may pass any schema, so its fields are read as unknown data. - const fields = schema as unknown as { readonly [key: string]: unknown }; + const fields = (optional ? column.schema : column) as { readonly [key: string]: unknown }; const json = fields.json; if (json instanceof JsonColumn) return { name, type: "json", optional, json }; const type = SCALAR_TYPES[String(fields.type)]; if (type === undefined) throw new Error(`${what}: expected str(), int(), num(), bool(), opt(...) or json(...)`); - for (const key of Object.keys(schema)) { + for (const key of Object.keys(fields)) { if (key !== "type" && key !== "default") throw new Error(`${what}: "${key}" is not supported in a table column`); } const value = fields.default; diff --git a/packages/chord/src/next/store/expression.ts b/packages/chord/src/next/store/expression.ts index f62f47cf4..0cbc5885f 100644 --- a/packages/chord/src/next/store/expression.ts +++ b/packages/chord/src/next/store/expression.ts @@ -2,9 +2,8 @@ // form"): the builders plugins write `where` and `set` with, and their compilation at staging into one data-only // vocabulary (what the outbox and archives carry; `evaluate.ts` runs it in JavaScript, `sql.ts` renders it). -import type { TOptional } from "typebox"; import type { JsonValue } from "../../json.ts"; -import type { TSchema } from "../contracts/index.ts"; +import type { Optional } from "../contracts/index.ts"; import { invalid } from "../data/internal.ts"; import type { Column, Columns, ColumnType, ColumnTypeOf, ValueOf } from "./columns.ts"; import { checkScalar, encodeCell, type Scalar } from "./rows.ts"; @@ -89,7 +88,7 @@ export class Condition { * the column's type; `undefined` clears an `opt(...)` column. */ export type SetOf = { - readonly [P in Exclude]?: C[P] extends TOptional + readonly [P in Exclude]?: C[P] extends Optional ? ValueOf | Expression, ColumnTypeOf> | undefined : ValueOf | Expression, ColumnTypeOf>; }; diff --git a/packages/chord/src/next/store/json-column.ts b/packages/chord/src/next/store/json-column.ts index 8e8a0a1c7..42195a95e 100644 --- a/packages/chord/src/next/store/json-column.ts +++ b/packages/chord/src/next/store/json-column.ts @@ -2,10 +2,10 @@ // cell with its own version, typed when read, migrated on access (the result checked once, R8), never walked on write // (R9). -import Type, { type TUnsafe } from "typebox"; +import type { JsonSchema } from "@earendil-works/pi-schema"; import type { Context } from "../../context/index.ts"; import type { JsonValue } from "../../json.ts"; -import type { Static, TSchema } from "../contracts/index.ts"; +import type { Schema, Static } from "../contracts/index.ts"; import { fingerprintOf } from "../contracts/internal.ts"; import { snapshotSchema } from "../contracts/snapshot.ts"; import { ChordError } from "../data/index.ts"; @@ -20,24 +20,25 @@ export type JsonColumnOptions = { }; /** A `json(...)` column as `columns` takes it: a schema of the cells' type carrying the column. */ -export type JsonColumnSchema = TUnsafe & { readonly json: JsonColumn }; +export interface JsonColumnSchema extends Schema { + readonly json: JsonColumn; +} /** A column holding one JSON value of `schema` per row, at `version`; wrap it in `opt(...)` to make it nullable. */ -export function json(schema: S, options: JsonColumnOptions>): JsonColumnSchema> { - const column = new JsonColumn(schema, options.version, options.migrate); - // A TypeBox schema (so `opt(...)` and `Static` take it) whose enumerable `json` field `opt`'s clone keeps. - return Type.Unsafe>({ json: column }) as JsonColumnSchema>; +export function json(schema: S, options: JsonColumnOptions>): JsonColumnSchema> { + // Typed as the cells' schema, so `opt(...)` and `Static` take it; the table reads its `json` field. + return { json: new JsonColumn(schema, options.version, options.migrate) }; } /** A `json(...)` column's definition: its schema snapshot, version and fingerprint, and the read of a stored cell. */ export class JsonColumn { - readonly schema: TSchema; + readonly schema: JsonSchema; readonly version: number; /** The fingerprint of the schema's normal form, recorded with the table: the same version with another is `broken`. */ readonly fingerprint: string; readonly #migrate: ((old: JsonValue, from: number) => unknown) | undefined; - constructor(schema: TSchema, version: number, migrate: ((old: JsonValue, from: number) => unknown) | undefined) { + constructor(schema: Schema, version: number, migrate: ((old: JsonValue, from: number) => unknown) | undefined) { // Declaration policy: JavaScript callers reach this check. if (!Number.isSafeInteger(version) || version < 1) throw new Error(`json(...): version ${version} is not a positive integer`); @@ -65,7 +66,7 @@ export class JsonColumn { throw refused(`${what}: migrate from version ${version} threw`, error); } context.telemetry.metrics?.counter("chord.data.validations").add(1); - if (!validatorFor(this, context).Check(migrated)) { + if (!validatorFor(this, context).check(migrated)) { throw refused(`${what}: migrate from version ${version} does not match the schema`); } return migrated; diff --git a/packages/chord/src/next/store/table-evolve.ts b/packages/chord/src/next/store/table-evolve.ts index 3e31ff82e..d0d8723c5 100644 --- a/packages/chord/src/next/store/table-evolve.ts +++ b/packages/chord/src/next/store/table-evolve.ts @@ -3,9 +3,9 @@ // data step, `drop`, `index` and `dropIndex` after it) and track the columns and key through the chain; at runtime one // editor records the edits, checks each as it is made, and materializes the new version at the end. -import type { TOptional } from "typebox"; -import { opt, type TSchema } from "../contracts/index.ts"; +import { opt } from "../contracts/index.ts"; import { editWith, type Increment } from "../contracts/internal.ts"; +import { isOptional, type Optional } from "../contracts/schema.ts"; import { type ColumnSchema, type Columns, @@ -59,7 +59,7 @@ export interface TableEditor ext } /** A column `add` takes: `opt(...)` or one with a `default`. */ -export type AddedColumn = ColumnSchema & (TOptional | { readonly default: unknown }); +export type AddedColumn = ColumnSchema & (Optional | { readonly default: unknown }); /** A `rows` patch: new values of non-key columns; `undefined` clears an `opt(...)` one. */ export type PatchOf = { @@ -85,7 +85,7 @@ type JsonNames = { [P in keyof C]: ColumnTypeOf extends type WithJson = With< C, N, - C[N] extends TOptional ? TOptional : S + C[N] extends Optional ? Optional> : S >; /** `V + 1` for a literal version (contracts' `Increment`); `number` stays `number`. */ @@ -170,7 +170,7 @@ class Editor { this.#assertBefore("json"); const existing = describeColumns({ [name]: this.#assertColumn(name, "json") }, this.#what)[0]!; if (existing.type !== "json") throw new Error(`${this.#what}: json: column "${name}" is not a json(...) column`); - if (Object.hasOwn(column, "~optional")) { + if (isOptional(column)) { throw new Error(`${this.#what}: json: pass json(...) itself; the column keeps its nullability`); } const schema = existing.optional ? opt(column) : column; diff --git a/packages/chord/src/object.ts b/packages/chord/src/object.ts index 6bd1ee24f..d13df8475 100644 --- a/packages/chord/src/object.ts +++ b/packages/chord/src/object.ts @@ -1,5 +1,4 @@ -import type { Static, TSchema } from "typebox"; -import { Value } from "typebox/value"; +import type { Schema, Static } from "@earendil-works/pi-schema"; import { Cell, Derived, @@ -30,7 +29,7 @@ import { asChordError, ChordError, invalid, runAll } from "./errors.ts"; import { enqueue } from "./publish.ts"; import type { Registration, Registry } from "./registry.ts"; import type { RemoteHost } from "./remote.ts"; -import { assertValid, normalize } from "./schema.ts"; +import { assertValid, cleaned, normalize } from "./schema.ts"; import type { Ctx, Exposed, GetOptions, Readable, RefStatus, Shape, StateEvent } from "./types.ts"; const noop = (): void => {}; @@ -161,7 +160,7 @@ export interface Self

    { * it when its version rises, and refuses a schema change without a version bump. A cell a reload adds runs its `init` * while the reload prepares. */ - state( + state( name: string, schema: S, initial: Static | StateInit, P>, @@ -317,7 +316,7 @@ type TaskFn = (ctx: Ctx) => Promise; /** Everything one factory run declared, by kind and key. */ export class Decls { readonly cells = new Map>(); - readonly cellSchemas = new Map(); + readonly cellSchemas = new Map(); readonly cellVersions = new Map(); readonly cellMigrations = new Map unknown>(); readonly refs = new Map(); @@ -1297,7 +1296,7 @@ export class ObjectImpl { /** Every state source exposed as a remote state member gets that member's schema (checked per write in dev mode). */ refreshSchemas(): void { - const schemas = new Map, TSchema[]>(); + const schemas = new Map, Schema[]>(); for (const installed of this.implementations.values()) { for (const [source, list] of memberSchemas(installed.service, installed.implementation)) { schemas.set(source, [...(schemas.get(source) ?? []), ...list]); @@ -1592,8 +1591,8 @@ export class ObjectImpl { // The additive rule assumes a consumer sends only what its own schema knows, and a provider sees only what its // schema knows: strip with both before validating. const own = ref.contract.scope === "remote" && wanted?.kind === "method" ? wanted.args : undefined; - const sent = own === undefined ? args : Value.Clean(own, Value.Clone(args)); - input = validated?.args ? Value.Clean(validated.args, Value.Clone(sent)) : sent; + const sent = own === undefined ? args : cleaned(own, args); + input = validated?.args ? cleaned(validated.args, sent) : sent; if (validated?.args) assertValid("args", `${this.address}.${name} args`, validated.args, input); } this.lease(ctx); @@ -1731,8 +1730,8 @@ const isReadableLike = (value: unknown): boolean => export function memberSchemas( service: Service, implementation: Record, -): Map, TSchema[]> { - const schemas = new Map, TSchema[]>(); +): Map, Schema[]> { + const schemas = new Map, Schema[]>(); const contract = service.contract; if (contract.scope !== "remote") return schemas; const members = contract.description.variants[variantOf(implementation) ?? ""] ?? {}; @@ -1794,7 +1793,7 @@ class SelfImpl implements Self { if (map.has(name)) throw invalid("duplicate", `${this.#object.typeName}: duplicate ${kind} name ${name}`); } - state( + state( name: string, schema: S, initial: Static | StateInit>, diff --git a/packages/chord/src/remote.ts b/packages/chord/src/remote.ts index 73c1e1d24..47e8ffa77 100644 --- a/packages/chord/src/remote.ts +++ b/packages/chord/src/remote.ts @@ -1,4 +1,3 @@ -import { Value } from "typebox/value"; import { HYDRATION, type Hydration } from "./cell.ts"; import { type AnyContract, takesNoArgs } from "./contract.ts"; import { applyImmutable, type Decoder, decoder } from "./delta/index.ts"; @@ -13,6 +12,7 @@ import { refsByProxy, type StatusListener, } from "./object.ts"; +import { cleaned } from "./schema.ts"; import type { Ctx, Readable, RefStatus, Shape, StateEvent } from "./types.ts"; import { fromWireError, type RootRef, type WireError } from "./wire.ts"; @@ -344,7 +344,7 @@ export class RemoteBinding { throw new ChordError("not_in_variant", `${this.key}: ${name} is not a method of variant ${variant}`); } // Send only what the consumer's own schema knows (the additive rule's premise; see ObjectImpl.call). - const sent = description.args === undefined ? undefined : Value.Clean(description.args, Value.Clone(args)); + const sent = description.args === undefined ? undefined : cleaned(description.args, args); return hold.channel.call( hold.id, name, diff --git a/packages/chord/src/schema.ts b/packages/chord/src/schema.ts index 7883969e5..527accd9a 100644 --- a/packages/chord/src/schema.ts +++ b/packages/chord/src/schema.ts @@ -1,8 +1,7 @@ -import type { TSchema } from "typebox"; -import { Value } from "typebox/value"; +import { compile, type JsonSchema, type ValidationError, type Validator } from "@earendil-works/pi-schema"; import { type ChordErrorReason, invalid } from "./errors.ts"; -/** JSON Schema as data. TypeBox schemas are plain JSON Schema objects. */ +/** JSON Schema as data: pi-schema builders, or any JSON Schema object (TypeBox's included). */ type Schema = { readonly [key: string]: unknown }; const asSchema = (schema: unknown): Schema => @@ -34,23 +33,187 @@ export function deepEqual(a: unknown, b: unknown): boolean { return a === b || canonicalJson(a) === canonicalJson(b); } +const validators = new WeakMap(); + +/** The compiled validator of `schema`, compiled once per schema object; `format` is an annotation. */ +function validatorOf(schema: JsonSchema): Validator { + if (typeof schema === "boolean") return compile(schema); + let validator = validators.get(schema); + if (validator === undefined) { + validator = compile(schema); + validators.set(schema, validator); + } + return validator; +} + +/** Whether `value` matches `schema`. */ +export function isValid(schema: JsonSchema, value: unknown): boolean { + return validatorOf(schema).check(value); +} + /** Throws `invalid` (with `reason`) and the first schema errors when `value` does not match. */ -export function assertValid(reason: ChordErrorReason, what: string, schema: TSchema, value: unknown): void { - if (Value.Check(schema, value)) return; - const errors = [...Value.Errors(schema, value)].slice(0, 3); - const summary = errors.map((error) => `${error.instancePath || "/"} ${error.message}`).join("; "); +export function assertValid(reason: ChordErrorReason, what: string, schema: JsonSchema, value: unknown): void { + const validator = validatorOf(schema); + if (validator.check(value)) return; + const errors: ValidationError[] = validator.errors(value).slice(0, 3); + const summary = errors.map((error) => `${error.path || "/"} ${error.message}`).join("; "); throw invalid(reason, `${what}: ${summary}`, errors); } /** Applies defaults, strips unknown fields, validates. Returns a fresh value. */ -export function normalize(schema: TSchema, value: unknown, what: string): unknown { - const normalized = Value.Clean(schema, Value.Default(schema, Value.Clone(value ?? {}))); +export function normalize(schema: JsonSchema, value: unknown, what: string): unknown { + const normalized = cleanFrom(asSchema(schema), withDefaults(asSchema(schema), cloneData(value ?? {}))); assertValid("input", what, schema, normalized); return normalized; } +/** A copy of `value` with only what `schema` knows: unknown object fields dropped, recursively (TypeBox's `Clean`). */ +export function cleaned(schema: JsonSchema, value: unknown): unknown { + return cleanFrom(asSchema(schema), cloneData(value)); +} + +/** A deep copy of plain data: arrays, plain objects and bytes are copied, any other value is kept. */ +function cloneData(value: unknown): unknown { + if (Array.isArray(value)) return value.map(cloneData); + if (value instanceof Uint8Array) return value.slice(); + if (value === null || typeof value !== "object") return value; + const prototype = Object.getPrototypeOf(value); + if (prototype !== Object.prototype && prototype !== null) return value; + const copy: Record = {}; + for (const [key, item] of Object.entries(value)) copy[key] = cloneData(item); + return copy; +} + +const isPlainObject = (value: unknown): value is Record => + value !== null && typeof value === "object" && !Array.isArray(value) && !(value instanceof Uint8Array); + +const isSchemaObject = (value: unknown): value is Schema => + value !== null && typeof value === "object" && !Array.isArray(value); + +/** The object an `allOf` of object schemas evaluates to (TypeBox's `Evaluate`), or undefined for any other `allOf`. */ +function mergedAllOf(schema: Schema): Schema | undefined { + if (!Array.isArray(schema.allOf) || !schema.allOf.every((branch) => isSchemaObject(branch.properties))) + return undefined; + const properties: Record = {}; + const required = new Set(); + for (const branch of schema.allOf as Schema[]) { + Object.assign(properties, branch.properties); + for (const key of (branch.required as string[] | undefined) ?? []) required.add(key); + } + return { + type: "object", + properties, + required: [...required], + ...(schema.unevaluatedProperties === undefined ? {} : { additionalProperties: schema.unevaluatedProperties }), + }; +} + +/** The pattern and value schema of a record (`patternProperties` with one pattern). */ +function recordOf(schema: Schema): [RegExp, unknown] | undefined { + if (!isSchemaObject(schema.patternProperties) || isSchemaObject(schema.properties)) return undefined; + const [entry] = Object.entries(schema.patternProperties); + return entry === undefined ? undefined : [new RegExp(entry[0], "u"), entry[1]]; +} + +/** Whether `additional` (an object's `additionalProperties`) keeps an unknown field holding `value`. */ +const keepsUnknown = (additional: unknown, value: unknown): boolean => + additional === true || (isSchemaObject(additional) && isValid(additional, value)); + +// Mutates `value` (a fresh copy), as TypeBox's `Clean` does. +function cleanFrom(schema: Schema, value: unknown): unknown { + const merged = mergedAllOf(schema); + if (merged !== undefined) return cleanFrom(merged, value); + if (Array.isArray(schema.anyOf)) { + for (const branch of schema.anyOf) { + const candidate = cleanFrom(asSchema(branch), cloneData(value)); + if (isValid(asSchema(branch), candidate)) return candidate; + } + return value; + } + if (Array.isArray(value)) { + const prefix = Array.isArray(schema.items) + ? schema.items + : Array.isArray(schema.prefixItems) + ? schema.prefixItems + : undefined; + if (prefix !== undefined) { + const length = Math.min(value.length, prefix.length); + for (let index = 0; index < length; index++) value[index] = cleanFrom(asSchema(prefix[index]), value[index]); + return value.length > length ? value.slice(0, length) : value; + } + if (isSchemaObject(schema.items)) return value.map((item) => cleanFrom(schema.items as Schema, item)); + return value; + } + if (!isPlainObject(value)) return value; + const record = recordOf(schema); + const properties = isSchemaObject(schema.properties) ? schema.properties : undefined; + // An object schema names its fields, a record its key pattern, a map (pi-schema's) its value schema. + if (properties === undefined && record === undefined && !isSchemaObject(schema.additionalProperties)) return value; + for (const key of Object.keys(value)) { + const known = properties !== undefined && Object.hasOwn(properties, key) ? properties[key] : undefined; + const matched = known ?? (record?.[0].test(key) ? record[1] : undefined); + if (matched !== undefined) value[key] = cleanFrom(asSchema(matched), value[key]); + else if (keepsUnknown(schema.additionalProperties, value[key])) + value[key] = cleanFrom(asSchema(schema.additionalProperties), value[key]); + else delete value[key]; + } + return value; +} + +// Mutates `value` (a fresh copy), as TypeBox's `Default` does: a missing value takes its schema's default, a +// missing optional field never does. +function withDefaults(schema: Schema, value: unknown): unknown { + const defaulted = value === undefined && "default" in schema ? cloneData(schema.default) : value; + const merged = mergedAllOf(schema); + if (merged !== undefined) return withDefaults(merged, defaulted); + if (Array.isArray(schema.anyOf)) { + for (const branch of schema.anyOf) { + const candidate = withDefaults(asSchema(branch), cloneData(defaulted)); + if (isValid(asSchema(branch), candidate)) return candidate; + } + return defaulted; + } + if (Array.isArray(defaulted)) { + const prefix = Array.isArray(schema.items) + ? schema.items + : Array.isArray(schema.prefixItems) + ? schema.prefixItems + : undefined; + if (prefix !== undefined) { + for (let index = 0; index < prefix.length; index++) { + const item = withDefaults(asSchema(prefix[index]), defaulted[index]); + if (index < defaulted.length || item !== undefined) defaulted[index] = item; + } + return defaulted; + } + if (isSchemaObject(schema.items)) return defaulted.map((item) => withDefaults(schema.items as Schema, item)); + return defaulted; + } + if (!isPlainObject(defaulted)) return defaulted; + const properties = isSchemaObject(schema.properties) ? schema.properties : undefined; + const record = recordOf(schema); + if (properties !== undefined) { + const required = (schema.required as string[] | undefined) ?? []; + for (const [key, property] of Object.entries(properties)) { + const item = withDefaults(asSchema(property), defaulted[key]); + if (item === undefined && (!required.includes(key) || !("default" in asSchema(property)))) continue; + defaulted[key] = item; + } + } else if (record !== undefined && isSchemaObject(record[1]) && "default" in record[1]) { + for (const key of Object.keys(defaulted)) { + if (record[0].test(key)) defaulted[key] = withDefaults(record[1], defaulted[key]); + } + } + if (!isSchemaObject(schema.additionalProperties)) return defaulted; + for (const key of Object.keys(defaulted)) { + const known = properties !== undefined && Object.hasOwn(properties, key); + if (!known && !record?.[0].test(key)) defaulted[key] = withDefaults(schema.additionalProperties, defaulted[key]); + } + return defaulted; +} + /** Canonical identity of a normalized value: fields equal to their schema default are omitted. */ -export function canonicalIdentity(schema: TSchema, value: unknown): string { +export function canonicalIdentity(schema: JsonSchema, value: unknown): string { return canonicalJson(stripDefaults(asSchema(schema), value)); } diff --git a/packages/chord/src/wire.ts b/packages/chord/src/wire.ts index b9271f130..bd4bdf175 100644 --- a/packages/chord/src/wire.ts +++ b/packages/chord/src/wire.ts @@ -1,5 +1,21 @@ -import { type Static, type TSchema, Type } from "typebox"; -import { Value } from "typebox/value"; +import { + arr, + bool, + compile, + int, + lit, + map, + num, + obj, + oneOf, + opt, + type Properties, + type Schema, + type Static, + str, + union, + unknown, +} from "@earendil-works/pi-schema"; import { DescriptionSchema } from "./contract.ts"; import { CHORD_ERROR_CODES, ChordError } from "./errors.ts"; @@ -23,55 +39,57 @@ import { CHORD_ERROR_CODES, ChordError } from "./errors.ts"; */ export const PROTOCOL_VERSION = 4; -const Id = Type.Integer({ minimum: 1 }); -const Name = Type.String({ minLength: 1 }); -const Reload = Type.Integer({ minimum: 1 }); -/** Arrays a sender may pass as readonly. */ -const List = (item: T) => Type.Immutable(Type.Array(item)); +const Id = int({ minimum: 1 }); +const Name = str({ minLength: 1 }); +const Reload = int({ minimum: 1 }); +/** Arrays a sender may pass as readonly: the same schema, a readonly type. */ +const List = (item: Schema) => arr(item) as Schema; +/** An object that admits fields it does not name (a newer peer's), as every frame does. */ +const Open = (properties: P) => obj(properties, { open: true }); -const RootRefSchema = Type.Object({ rootId: Name, codeVersion: Type.Integer({ minimum: 1 }) }); +const RootRefSchema = Open({ rootId: Name, codeVersion: int({ minimum: 1 }) }); export type RootRef = Static; -const WireErrorSchema = Type.Object({ - code: Type.Enum(CHORD_ERROR_CODES), - message: Type.String(), - reason: Type.Optional(Type.String()), +const WireErrorSchema = Open({ + code: oneOf(CHORD_ERROR_CODES), + message: str(), + reason: opt(str()), }); export type WireError = Static; -const WireReloadOptionsSchema = Type.Object({ - remove: Type.Optional(List(Type.String())), - gateDeadline: Type.Optional(Type.Number()), - retireDeadline: Type.Optional(Type.Number()), +const WireReloadOptionsSchema = Open({ + remove: opt(List(str())), + gateDeadline: opt(num()), + retireDeadline: opt(num()), }); export type WireReloadOptions = Static; /** A child's prepared reload, sent with its vote (same fields as `ReloadReport`, without children). */ -const WireReloadReportSchema = Type.Object({ - codeVersion: Type.Integer(), - previous: Type.Integer(), - changed: List(Type.String()), - removed: List(Type.String()), - prepared: List(Type.String()), - gated: List(Type.String()), +const WireReloadReportSchema = Open({ + codeVersion: int(), + previous: int(), + changed: List(str()), + removed: List(str()), + prepared: List(str()), + gated: List(str()), }); export type WireReloadReport = Static; -const FrameSchema = Type.Union([ - Type.Object({ - t: Type.Literal("hello"), +const FrameSchema = union([ + Open({ + t: lit("hello"), /** The link protocol version (`PROTOCOL_VERSION`; absent: 1). Peers of another version close the link. */ - protocol: Type.Optional(Type.Integer({ minimum: 1 })), + protocol: opt(int({ minimum: 1 })), name: Name, - kind: Type.Union([Type.Literal("host"), Type.Literal("client")]), - codeVersion: Type.Integer({ minimum: 0 }), + kind: union([lit("host"), lit("client")]), + codeVersion: int({ minimum: 0 }), /** The sender acknowledges what it receives (`ack`). Flow control applies only when both hellos say so. */ - acks: Type.Optional(Type.Boolean()), + acks: opt(bool()), }), /** Every frame received so far except `ack` frames, counted from the link's start, once processed. */ - Type.Object({ t: Type.Literal("ack"), n: Type.Integer({ minimum: 1 }) }), - Type.Object({ - t: Type.Literal("hold"), + Open({ t: lit("ack"), n: int({ minimum: 1 }) }), + Open({ + t: lit("hold"), id: Id, /** * Target host name, resolved by the consumer from the token's `host` and `input`. Relays route on it after @@ -81,92 +99,92 @@ const FrameSchema = Type.Union([ /** The consumer's contract token; the provider checks that it covers it. Remote tokens carry `host`. */ token: DescriptionSchema, /** Normalized with the consumer's contract. */ - input: Type.Unknown(), + input: unknown(), /** A forwarded principal. Honored only on frames from the parent link. */ - principal: Type.Optional(Type.Unknown()), + principal: opt(unknown()), /** * The client connection the hold comes from: an opaque id the server assigned to it, forwarded by relays. Honored * only on frames from the parent link; present means the hold is a client's (not a reload participant). */ - origin: Type.Optional(Name), + origin: opt(Name), /** False: no state for this hold until `sub` (a method-only binding). */ - state: Type.Optional(Type.Literal(false)), + state: opt(lit(false)), /** The consumer already has values (a re-hold after a link loss): its unit is not urgent. */ - again: Type.Optional(Type.Literal(true)), + again: opt(lit(true)), }), - Type.Object({ t: Type.Literal("release"), id: Id }), - Type.Object({ - t: Type.Literal("shape"), + Open({ t: lit("release"), id: Id }), + Open({ + t: lit("shape"), id: Id, - variant: Type.Optional(Type.String()), - members: List(Type.String()), - contract: Type.String(), + variant: opt(str()), + members: List(str()), + contract: str(), /** The state members whose snapshots follow this frame; the consumer applies the shape once they are here. */ state: List(Name), }), - Type.Object({ t: Type.Literal("gone"), id: Id, error: WireErrorSchema }), - Type.Object({ - t: Type.Literal("call"), + Open({ t: lit("gone"), id: Id, error: WireErrorSchema }), + Open({ + t: lit("call"), id: Id, hold: Id, member: Name, - args: Type.Optional(Type.Unknown()), + args: opt(unknown()), /** Absent: the receiving host starts a root (browsers never carry roots). */ - root: Type.Optional(RootRefSchema), + root: opt(RootRefSchema), }), - Type.Object({ - t: Type.Literal("result"), + Open({ + t: lit("result"), id: Id, - value: Type.Optional(Type.Unknown()), - error: Type.Optional(WireErrorSchema), + value: opt(unknown()), + error: opt(WireErrorSchema), }), - Type.Object({ t: Type.Literal("cancel"), id: Id }), + Open({ t: lit("cancel"), id: Id }), /** Starts the state of a hold opened with `state: false` (answered with a unit). */ - Type.Object({ t: Type.Literal("sub"), hold: Id }), + Open({ t: lit("sub"), hold: Id }), /** Stops the state of a hold. */ - Type.Object({ t: Type.Literal("unsub"), hold: Id }), + Open({ t: lit("unsub"), hold: Id }), /** Sends member `m` again as a reset (a relay that had to drop what waited for it). */ - Type.Object({ t: Type.Literal("resync"), hold: Id, m: Name }), - Type.Object({ - t: Type.Literal("state"), + Open({ t: lit("resync"), hold: Id, m: Name }), + Open({ + t: lit("state"), hold: Id, m: Name, /** `snap`/`reset` without `value`: the value is `undefined`. */ - as: Type.Union([Type.Literal("snap"), Type.Literal("reset"), Type.Literal("ops"), Type.Literal("error")]), - value: Type.Optional(Type.Unknown()), + as: union([lit("snap"), lit("reset"), lit("ops"), lit("error")]), + value: opt(unknown()), /** Encoded with the stream's delta encoder; the decoder validates each op. */ - ops: Type.Optional(List(List(Type.Unknown()))), - error: Type.Optional(WireErrorSchema), + ops: opt(List(List(unknown()))), + error: opt(WireErrorSchema), }), - Type.Object({ t: Type.Literal("ping"), n: Type.Integer() }), - Type.Object({ t: Type.Literal("pong"), n: Type.Integer() }), + Open({ t: lit("ping"), n: int() }), + Open({ t: lit("pong"), n: int() }), /** Ends the link (consumed by the link itself). `reason: "protocol-version "`: the sender speaks version n. */ - Type.Object({ t: Type.Literal("bye"), reason: Type.Optional(Type.String()) }), + Open({ t: lit("bye"), reason: opt(str()) }), /** The root ended at its origin: its leases on objects end (sent on every link the root's calls used). */ - Type.Object({ t: Type.Literal("root_end"), rootId: Name }), - Type.Object({ t: Type.Literal("invalidate"), pattern: Type.Record(Type.String(), Type.Unknown()) }), - Type.Object({ - t: Type.Literal("prepare"), + Open({ t: lit("root_end"), rootId: Name }), + Open({ t: lit("invalidate"), pattern: map(unknown()) }), + Open({ + t: lit("prepare"), reload: Reload, - codeVersion: Type.Integer({ minimum: 1 }), - plugins: List(Type.String()), + codeVersion: int({ minimum: 1 }), + plugins: List(str()), options: WireReloadOptionsSchema, }), - Type.Object({ - t: Type.Literal("vote"), + Open({ + t: lit("vote"), reload: Reload, - error: Type.Optional(WireErrorSchema), - report: Type.Optional(WireReloadReportSchema), + error: opt(WireErrorSchema), + report: opt(WireReloadReportSchema), }), - Type.Object({ t: Type.Literal("commit"), reload: Reload }), + Open({ t: lit("commit"), reload: Reload }), /** The parent dropped `codeVersion`: roots of it may still come from there no longer. */ - Type.Object({ t: Type.Literal("retire"), codeVersion: Type.Integer({ minimum: 1 }) }), - Type.Object({ t: Type.Literal("abort"), reload: Reload }), + Open({ t: lit("retire"), codeVersion: int({ minimum: 1 }) }), + Open({ t: lit("abort"), reload: Reload }), ]); export type Frame = Static; /** A frame that passes validation; anything else is a protocol violation. */ -export const isFrame = (value: unknown): value is Frame => Value.Check(FrameSchema, value); +export const isFrame: (value: unknown) => value is Frame = compile(FrameSchema).check; export function toWireError(error: unknown): WireError { if (error instanceof ChordError) { diff --git a/packages/chord/test/awaiting.test.ts b/packages/chord/test/awaiting.test.ts index 655d66736..975cb31f8 100644 --- a/packages/chord/test/awaiting.test.ts +++ b/packages/chord/test/awaiting.test.ts @@ -1,7 +1,7 @@ // Entry 8: awaited gets. `host.get` and `ctx.get` resolve once the reference is ready; abort, the ctx closing and the // host's `readyTimeout` reject them and release only that reference. import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { num, obj } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { type Ctx, createHost, type Host, hostNamed, isChordError, memoryLinkPair, remote } from "../src/index.ts"; import { CountingLink } from "./fixtures/counting.ts"; @@ -14,9 +14,9 @@ async function until(check: () => boolean, attempts = 200): Promise { const Slow = remote("t.e8-slow", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), - variants: { main: { state: { n: Type.Number() } } }, + variants: { main: { state: { n: num() } } }, }); /** A server whose `Slow` object goes live when `open()` is called; a browser whose link counts frames. */ @@ -28,7 +28,7 @@ function setup(readyTimeout?: number) { const server = createHost({ self: { kind: "host" }, name: "server", onError: () => {} }); server.serve(Slow, (_input, _principal, self) => { self.resource("gate", { version: 1, open: () => opened }); - return { variant: "main", n: self.state("n", Type.Number(), 1) }; + return { variant: "main", n: self.state("n", num(), 1) }; }); const [down, up] = memoryLinkPair(); server.accept(down, { principal: { user: "u" } }); diff --git a/packages/chord/test/bindings.test.ts b/packages/chord/test/bindings.test.ts index 6e7d9b9b6..e3e642ebe 100644 --- a/packages/chord/test/bindings.test.ts +++ b/packages/chord/test/bindings.test.ts @@ -1,7 +1,7 @@ // Entry 8: one shared binding per host and key; a hold implies the state of its variant (units: shape, then // snapshots); method-only references; origins of client holds; protocol version 4. import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { arr, nul, num, obj, opt, str, unknown } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { HostImpl } from "../src/host.ts"; import { @@ -28,15 +28,15 @@ async function until(check: () => boolean, attempts = 200): Promise { assert.ok(check(), "condition not reached"); } -const NoteSchema = Type.Array(Type.String()); +const NoteSchema = arr(str()); const Notes = remote("t.e8-notes", { version: 1, - input: Type.Object({ list: Type.Optional(Type.String({ default: "main" })) }), + input: obj({ list: opt(str({ default: "main" })) }, { open: true }), host: hostNamed("server"), variants: { main: { - state: { notes: NoteSchema, title: Type.String() }, - methods: { add: method(Type.String()), rename: method(Type.String()) }, + state: { notes: NoteSchema, title: str() }, + methods: { add: method(str()), rename: method(str()) }, }, }, }); @@ -44,7 +44,7 @@ const Notes = remote("t.e8-notes", { function serveNotes(host: Host): void { host.serve(Notes, (_input, _principal, self) => { const notes = self.state("notes", NoteSchema, []); - const title = self.state("title", Type.String(), "untitled"); + const title = self.state("title", str(), "untitled"); return { variant: "main", notes, @@ -88,7 +88,7 @@ function setup( describe("shared bindings: traffic", () => { it("N references to one object: one hold, one snapshot per member, one frame per change, one release", async () => { const { server, browser, link } = setup(); - const Panel = local("t.e8-panel", { version: 1, input: Type.Object({}), members: { count: Type.Number() } }); + const Panel = local("t.e8-panel", { version: 1, input: obj({}, { open: true }), members: { count: num() } }); browser.serve(Panel, (_input, _principal, self) => { const notes = self.get(Notes, {}); return { count: self.derive([notes.notes], (list) => list.length) }; @@ -183,12 +183,12 @@ describe("shared bindings: identity", () => { }); const NotesV2 = remote("t.e8-notes", { version: 2, - input: Type.Object({ list: Type.Optional(Type.String({ default: "main" })) }), + input: obj({ list: opt(str({ default: "main" })) }, { open: true }), host: hostNamed("server"), variants: { main: { - state: { notes: NoteSchema, title: Type.String() }, - methods: { add: method(Type.String()), rename: method(Type.String()), clear: method() }, + state: { notes: NoteSchema, title: str() }, + methods: { add: method(str()), rename: method(str()), clear: method() }, }, }, }); @@ -196,7 +196,7 @@ describe("shared bindings: identity", () => { NotesV2, (_input, _principal, self) => { const notes = self.state("notes", NoteSchema, []); - const title = self.state("title", Type.String(), "untitled"); + const title = self.state("title", str(), "untitled"); return { variant: "main", notes, title, add: () => {}, rename: () => {}, clear: () => {} }; }, { accepts: { 1: (current) => current } }, @@ -230,12 +230,12 @@ describe("shared bindings: identity", () => { describe("hydration over a link", () => { const Matrix = remote("t.e8-matrix", { version: 1, - input: Type.Object({ variant: Type.String() }), + input: obj({ variant: str() }, { open: true }), host: hostNamed("server"), variants: { - values: { state: { none: Type.Null(), empty: Type.Array(Type.String()), maybe: Type.Unknown() } }, - bare: { methods: { ping: method(undefined, Type.String()) } }, - broken: { state: { n: Type.Number() } }, + values: { state: { none: nul(), empty: arr(str()), maybe: unknown() } }, + bare: { methods: { ping: method(undefined, str()) } }, + broken: { state: { n: num() } }, }, }); @@ -243,12 +243,12 @@ describe("hydration over a link", () => { host.serve(Matrix, ({ variant }, _principal, self) => { if (variant === "bare") return { variant: "bare", ping: () => "pong" }; if (variant === "broken") { - const source = self.state("source", Type.String(), "not a number"); + const source = self.state("source", str(), "not a number"); // A derivation exposed under a number schema: its snapshot fails validation. return { variant: "broken", n: self.derive([source], (text) => text as unknown as number) }; } - const none = self.state("none", Type.Null(), null); - const empty = self.state("empty", Type.Array(Type.String()), []); + const none = self.state("none", nul(), null); + const empty = self.state("empty", arr(str()), []); return { variant: "values", none, empty, maybe: self.derive([none], () => undefined as unknown) }; }); } @@ -332,9 +332,9 @@ describe("method-only references (state: false)", () => { describe("origins of client holds", () => { const Counter = remote("t.e8-counter", { version: 1, - input: Type.Object({ session: Type.String() }), + input: obj({ session: str() }, { open: true }), host: hostFamily("session:", "session"), - variants: { main: { state: { n: Type.Number() }, methods: { add: method(Type.Number()) } } }, + variants: { main: { state: { n: num() }, methods: { add: method(num()) } } }, }); function cluster() { @@ -354,7 +354,7 @@ describe("origins of client holds", () => { onError: () => {}, }); created.serve(Counter, (_input, _principal, self) => { - const n = self.state("n", Type.Number(), 0); + const n = self.state("n", num(), 0); return { variant: "main", n, diff --git a/packages/chord/test/boundary.test.ts b/packages/chord/test/boundary.test.ts index 8819a01c1..ae4f1beb4 100644 --- a/packages/chord/test/boundary.test.ts +++ b/packages/chord/test/boundary.test.ts @@ -8,10 +8,11 @@ const sourceDirectory = resolve(packageDirectory, "src"); const IMPORT_SPECIFIER = /(?:import|export)\s+(?:type\s+)?(?:[^;]*?\sfrom\s*)?["']([^"']+)["']/gu; describe("package boundary", () => { - // pi-telemetry is the one allowed Pi package: a leaf without dependencies, needed for Context.telemetry. - const ALLOWED = new Set(["@earendil-works/pi-telemetry"]); + // The allowed Pi packages are leaves without dependencies: pi-telemetry for Context.telemetry, pi-schema for schemas + // (decision "A schema package beneath pi-ai and Chord"). + const ALLOWED = new Set(["@earendil-works/pi-telemetry", "@earendil-works/pi-schema"]); - test("does not depend on Pi packages other than pi-telemetry, or on files outside Chord", async () => { + test("does not depend on Pi packages other than pi-telemetry and pi-schema, or on files outside Chord", async () => { const manifest = JSON.parse(await readFile(resolve(packageDirectory, "package.json"), "utf8")) as { dependencies?: Record; }; diff --git a/packages/chord/test/cluster.test.ts b/packages/chord/test/cluster.test.ts index 1547c3e1f..b0ad5dc97 100644 --- a/packages/chord/test/cluster.test.ts +++ b/packages/chord/test/cluster.test.ts @@ -1,5 +1,5 @@ import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { num, obj, str } from "@earendil-works/pi-schema"; import { describe, it, onTestFinished } from "vitest"; import { type Ctx, @@ -30,32 +30,32 @@ function latch() { const Greeter = remote("t.greeter", { version: 1, - input: Type.Object({ session: Type.String() }), + input: obj({ session: str() }, { open: true }), host: hostFamily("session:", "session"), variants: { main: { - state: { count: Type.Object({ n: Type.Number() }) }, - methods: { greet: method(Type.String(), Type.String()) }, + state: { count: obj({ n: num() }, { open: true }) }, + methods: { greet: method(str(), str()) }, }, }, }); const GreeterV2 = remote("t.greeter", { version: 2, - input: Type.Object({ session: Type.String() }), + input: obj({ session: str() }, { open: true }), host: hostFamily("session:", "session"), - variants: { main: { methods: { hello: method(Type.String(), Type.String()) } } }, + variants: { main: { methods: { hello: method(str(), str()) } } }, }); /** The server's own greeter: the greeter facet serves this one on the server, `Greeter` on session hosts. */ const ServerGreeter = remote("t.server-greeter", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), variants: { main: { - state: { count: Type.Object({ n: Type.Number() }) }, - methods: { greet: method(Type.String(), Type.String()) }, + state: { count: obj({ n: num() }, { open: true }) }, + methods: { greet: method(str(), str()) }, }, }, }); @@ -68,7 +68,7 @@ function greeterFacet(greeting: string, options: { fail?: boolean; stateVersion? if (options.fail) throw new Error("worker factory fails"); const count = self.state( "count", - Type.Object({ n: Type.Number() }), + obj({ n: num() }, { open: true }), { n: 0 }, { version: options.stateVersion ?? 1 }, ); @@ -105,9 +105,9 @@ function dropping(link: Link, drop: (frame: Frame) => boolean): Link { const Ping = remote("t.cluster-ping", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), - variants: { main: { methods: { ping: method(undefined, Type.String()) } } }, + variants: { main: { methods: { ping: method(undefined, str()) } } }, }); function setup() { @@ -314,7 +314,7 @@ describe("cross-host reload", () => { }, close: () => void log.push("closed"), }); - const count = self.state("count", Type.Object({ n: Type.Number() }), { n: 0 }); + const count = self.state("count", obj({ n: num() }, { open: true }), { n: 0 }); return { variant: "main", count, greet: (name: string) => `slow ${name}` }; }), }; @@ -569,7 +569,7 @@ describe("cross-host reload", () => { const never = latch(); const Slow = remote("t.cluster-slow", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("session:S1"), variants: { main: { methods: { wait: method() } } }, }); @@ -590,7 +590,7 @@ describe("cross-host reload", () => { resolve: async () => watched, principal: (name) => ({ host: name }), }); - const Store = local("t.cluster-store", { version: 1, input: Type.Object({}) }); + const Store = local("t.cluster-store", { version: 1, input: obj({}, { open: true }) }); const storeFacet = (version: number): Facet => ({ name: "store", setup: (api) => diff --git a/packages/chord/test/contract.test.ts b/packages/chord/test/contract.test.ts index df29a8a0b..0abcc1a67 100644 --- a/packages/chord/test/contract.test.ts +++ b/packages/chord/test/contract.test.ts @@ -1,25 +1,30 @@ import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { bool, num, obj, opt, str, unknown } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { bind, coverage, hostFamily, hostNamed, isChordError, local, method, opaque, remote } from "../src/index.ts"; import { canonicalIdentity, fits, normalize } from "../src/schema.ts"; -const View = Type.Object({ text: Type.String() }); +const View = obj({ text: str() }, { open: true }); const chatV1 = remote("t.chat", { version: 1, - input: Type.Object({ session: Type.String(), conversation: Type.Number({ default: 0 }) }), + input: obj({ session: str(), conversation: num({ default: 0 }) }, { open: true }), host: hostFamily("session:", "session"), variants: { viewer: { state: { view: View } }, participant: { state: { view: View }, - methods: { send: method(Type.Object({ text: Type.String() }), Type.Object({ id: Type.Number() })) }, + methods: { send: method(obj({ text: str() }, { open: true }), obj({ id: num() }, { open: true })) }, }, }, }); describe("contracts", () => { + it("treats `format` as an annotation: an input with format email accepts any string", () => { + const Mail = local("t.mail", { version: 1, input: obj({ to: str({ format: "email" }) }) }); + assert.deepEqual(normalize(Mail.input, { to: "nope" }, "input"), { to: "nope" }); + }); + it("normalizes input and omits defaults from the identity", () => { const a = normalize(chatV1.input, { session: "s", extra: 1 }, "input"); assert.deepEqual(a, { session: "s", conversation: 0 }); @@ -39,12 +44,12 @@ describe("contracts", () => { }); it("rejects reserved member names and opaque remote members", () => { - assert.throws(() => local("t.x", { version: 1, input: Type.Object({}), members: { status: method() } })); + assert.throws(() => local("t.x", { version: 1, input: obj({}, { open: true }), members: { status: method() } })); assert.throws(() => remote("t.x", { host: hostNamed("server"), version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), variants: { a: { state: { h: opaque("x@1") as never } } }, }), ); @@ -56,11 +61,11 @@ describe("contracts", () => { variants: { ...chatV1.def.variants, participant: { - state: { view: Type.Object({ text: Type.String(), extra: Type.Number() }) }, + state: { view: obj({ text: str(), extra: num() }, { open: true }) }, methods: { send: method( - Type.Object({ text: Type.String(), urgent: Type.Optional(Type.Boolean()) }), - Type.Object({ id: Type.Number(), at: Type.Number() }), + obj({ text: str(), urgent: opt(bool()) }, { open: true }), + obj({ id: num(), at: num() }, { open: true }), ), }, }, @@ -77,8 +82,8 @@ describe("contracts", () => { state: { view: View }, methods: { send: method( - Type.Object({ text: Type.String(), urgent: Type.Boolean() }), - Type.Object({ id: Type.Number() }), + obj({ text: str(), urgent: bool() }, { open: true }), + obj({ id: num() }, { open: true }), ), }, }, @@ -92,18 +97,18 @@ describe("contracts", () => { const retyped = remote("t.chat", { ...chatV1.def, - variants: { ...chatV1.def.variants, viewer: { state: { view: Type.Object({ text: Type.Number() }) } } }, + variants: { ...chatV1.def.variants, viewer: { state: { view: obj({ text: num() }, { open: true }) } } }, }); assert.match(coverage(retyped.description, chatV1.description) ?? "", /state/); }); it("methods without arguments or result: described without them; their presence must match, a void consumer accepts a result", () => { const of = (members: Parameters[1]["members"]) => - local("t.methods", { version: 1, input: Type.Object({}), members }).description; + local("t.methods", { version: 1, input: obj({}, { open: true }), members }).description; const none = of({ m: method() }); assert.deepEqual(none.variants[""]?.m, { kind: "method" }); - const withArgs = of({ m: method(Type.String()) }); - const withResult = of({ m: method(undefined, Type.Number()) }); + const withArgs = of({ m: method(str()) }); + const withResult = of({ m: method(undefined, num()) }); assert.equal(coverage(none, none), undefined); assert.match(coverage(withArgs, none) ?? "", /arguments/, "the consumer sends none, the provider expects some"); assert.match(coverage(none, withArgs) ?? "", /arguments/, "the consumer sends some, the provider takes none"); @@ -124,15 +129,25 @@ describe("contracts", () => { it("compares opaque members by name@version and function members structurally", () => { const a = local("t.idx", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), members: { harness: opaque("h@1"), - onChange: method(Type.Function([Type.String()], Type.Void())), + onChange: method({ type: "function", parameters: [str()], returnType: { type: "void" } }), }, }); const b = local("t.idx", { ...a.def, members: { ...a.def.members, harness: opaque("h@2") } }); assert.match(coverage(b.description, a.description) ?? "", /opaque/); - assert.ok(fits(Type.Function([Type.Unknown()], Type.Void()), Type.Function([Type.String()], Type.Void()))); - assert.ok(!fits(Type.Function([Type.String()], Type.Void()), Type.Function([Type.Unknown()], Type.Void()))); + assert.ok( + fits( + { type: "function", parameters: [unknown()], returnType: { type: "void" } }, + { type: "function", parameters: [str()], returnType: { type: "void" } }, + ), + ); + assert.ok( + !fits( + { type: "function", parameters: [str()], returnType: { type: "void" } }, + { type: "function", parameters: [unknown()], returnType: { type: "void" } }, + ), + ); }); }); diff --git a/packages/chord/test/fixtures/echo.ts b/packages/chord/test/fixtures/echo.ts index ebe12e0b9..20c81af77 100644 --- a/packages/chord/test/fixtures/echo.ts +++ b/packages/chord/test/fixtures/echo.ts @@ -1,11 +1,11 @@ -import { Type } from "typebox"; +import { num, obj, str } from "@earendil-works/pi-schema"; import { hostNamed, method, remote } from "../../src/index.ts"; export const Echo = remote("t.echo", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("w1"), variants: { - main: { methods: { echo: method(Type.String(), Type.String()), pid: method(undefined, Type.Number()) } }, + main: { methods: { echo: method(str(), str()), pid: method(undefined, num()) } }, }, }); diff --git a/packages/chord/test/fixtures/loader.ts b/packages/chord/test/fixtures/loader.ts index cded6d162..465114bca 100644 --- a/packages/chord/test/fixtures/loader.ts +++ b/packages/chord/test/fixtures/loader.ts @@ -2,11 +2,11 @@ import assert from "node:assert/strict"; import { mkdirSync, rmSync, writeFileSync } from "node:fs"; import { fileURLToPath } from "node:url"; -import { Type } from "typebox"; +import { bool, obj, str } from "@earendil-works/pi-schema"; import { createHost, local, method } from "../../src/index.ts"; import { nodeLoader } from "../../src/loader.ts"; -// Plugin modules live under the package so bare imports (typebox) resolve; removed afterwards. +// Plugin modules live under the package so bare imports (pi-schema) resolve; removed afterwards. const dir = fileURLToPath(new URL(`../.tmp-plugins-${process.pid}/`, import.meta.url)); // Separate from `dir`, which becomes a reload root once its facet loads. const plugins = fileURLToPath(new URL(`../.tmp-plugins-${process.pid}-order/`, import.meta.url)); @@ -15,19 +15,19 @@ const chord = new URL("../../src/index.ts", import.meta.url).href; // The test's own copy of the plugin's contract: binding is by description, not object identity. const Hello = local("t.hello", { version: 1, - input: Type.Object({}), - members: { greet: method(Type.String(), Type.String()) }, + input: obj({}, { open: true }), + members: { greet: method(str(), str()) }, }); const facetSource = ` -import { Type } from "typebox"; +import { obj, str } from "@earendil-works/pi-schema"; import { defineFacet, local, method } from ${JSON.stringify(chord)}; import { greeting } from "./greeting.ts"; const Hello = local("t.hello", { version: 1, - input: Type.Object({}), - members: { greet: method(Type.String(), Type.String()) }, + input: obj({}, { open: true }), + members: { greet: method(str(), str()) }, }); export default defineFacet((host) => { @@ -55,8 +55,8 @@ try { // A facet listed first that imports a later facet's directory shares that facet's copy of the module. const Shared = local("t.shared", { version: 1, - input: Type.Object({}), - members: { same: method(undefined, Type.Boolean()) }, + input: obj({}, { open: true }), + members: { same: method(undefined, bool()) }, }); mkdirSync(`${plugins}a`, { recursive: true }); mkdirSync(`${plugins}b`, { recursive: true }); @@ -67,8 +67,8 @@ try { ); writeFileSync( `${plugins}a/facet.ts`, - `import { Type } from "typebox";\nimport { defineFacet, local, method } from ${JSON.stringify(chord)};\nimport { state } from "../b/state.ts";\n` + - `const Shared = local("t.shared", { version: 1, input: Type.Object({}), members: { same: method(undefined, Type.Boolean()) } });\n` + + `import { bool, obj } from "@earendil-works/pi-schema";\nimport { defineFacet, local, method } from ${JSON.stringify(chord)};\nimport { state } from "../b/state.ts";\n` + + `const Shared = local("t.shared", { version: 1, input: obj({}, { open: true }), members: { same: method(undefined, bool()) } });\n` + `export default defineFacet((host) => host.serve(Shared, () => ({ same: async () => (await import(new URL("../b/facet.ts", import.meta.url).href)).own === state })));\n`, ); const ordered = createHost({ self: { kind: "host" }, loader: nodeLoader, onError: () => {} }); diff --git a/packages/chord/test/fixtures/schemas.ts b/packages/chord/test/fixtures/schemas.ts new file mode 100644 index 000000000..a21a9e09e --- /dev/null +++ b/packages/chord/test/fixtures/schemas.ts @@ -0,0 +1,14 @@ +// Schemas with the exact JSON TypeBox's `Type.Literal` and `Type.Record` built, so contracts declared with them keep +// the fingerprints they had under TypeBox (see the migration notes in CHANGELOG.md). + +import type { Schema } from "@earendil-works/pi-schema"; + +/** `Type.Literal(value)`: the literal with its JSON type (pi-schema's `lit` has no `type`). */ +export function typedLit(value: V): Schema { + return { type: typeof value, const: value }; +} + +/** `Type.Record(Type.String(), value)`: one pattern property for every key (pi-schema's `map` uses `additionalProperties`). */ +export function record(value: Schema): Schema> { + return { type: "object", patternProperties: { "^.*$": value } }; +} diff --git a/packages/chord/test/flow-units.test.ts b/packages/chord/test/flow-units.test.ts index ab578f178..0fd9c984f 100644 --- a/packages/chord/test/flow-units.test.ts +++ b/packages/chord/test/flow-units.test.ts @@ -1,7 +1,7 @@ // Entry 8: flow control with units. First units pass the window, so a get never waits behind other streams; waiting // ops are capped per link; relays ask downstream for a reset when they drop ops. import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { arr, num, obj, str } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { HostImpl } from "../src/host.ts"; import { @@ -26,12 +26,12 @@ async function until(check: () => boolean, attempts = 400): Promise { const Blob = remote("t.e8-blob", { version: 1, - input: Type.Object({ name: Type.String() }), + input: obj({ name: str() }, { open: true }), host: hostNamed("server"), variants: { main: { - state: { lines: Type.Array(Type.String()) }, - methods: { push: method(Type.String()), ping: method(undefined, Type.String()) }, + state: { lines: arr(str()) }, + methods: { push: method(str()), ping: method(undefined, str()) }, }, }, }); @@ -41,7 +41,7 @@ const lines = (kib: number, tag: string) => Array.from({ length: kib }, (_, i) = function serveBlob(host: Host, size: (name: string) => number): void { host.serve(Blob, ({ name }, _principal, self) => { - const state = self.state("lines", Type.Array(Type.String()), lines(size(name), name)); + const state = self.state("lines", arr(str()), lines(size(name), name)); return { variant: "main", lines: state, @@ -169,9 +169,9 @@ describe("flow control with units", () => { describe("relay topology", () => { const Counter = remote("t.e8-relayed", { version: 1, - input: Type.Object({ session: Type.String() }), + input: obj({ session: str() }, { open: true }), host: hostFamily("session:", "session"), - variants: { main: { state: { n: Type.Number() }, methods: { add: method(Type.Number()) } } }, + variants: { main: { state: { n: num() }, methods: { add: method(num()) } } }, }); function cluster() { @@ -191,7 +191,7 @@ describe("relay topology", () => { onError: () => {}, }); created.serve(Counter, (_input, _principal, self) => { - const n = self.state("n", Type.Number(), 0); + const n = self.state("n", num(), 0); return { variant: "main", n, diff --git a/packages/chord/test/flow.test.ts b/packages/chord/test/flow.test.ts index 80c8268a7..79835bb25 100644 --- a/packages/chord/test/flow.test.ts +++ b/packages/chord/test/flow.test.ts @@ -1,7 +1,7 @@ // Link flow control: state waits while the peer's acknowledgements lag a window behind, coalesced per subscription; // everything else is sent at once. import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { num, obj } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { type Ctx, @@ -65,12 +65,12 @@ class AckGate implements Link { const counterOn = (name: string, host: string) => remote(name, { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed(host), variants: { main: { - state: { count: Type.Object({ n: Type.Number() }) }, - methods: { add: method(Type.Number(), Type.Number()), wait: method() }, + state: { count: obj({ n: num() }, { open: true }) }, + methods: { add: method(num(), num()), wait: method() }, }, }, }); @@ -89,7 +89,7 @@ const waiting = () => function serveCounter(host: Host, type = Counter): void { host.serve(type, (_input, _principal, self) => { - const count = self.state("count", Type.Object({ n: Type.Number() }), { n: 0 }); + const count = self.state("count", obj({ n: num() }, { open: true }), { n: 0 }); return { variant: "main", count, diff --git a/packages/chord/test/generations.test.ts b/packages/chord/test/generations.test.ts index 3f20b95cc..7496cd1fa 100644 --- a/packages/chord/test/generations.test.ts +++ b/packages/chord/test/generations.test.ts @@ -4,7 +4,7 @@ import assert from "node:assert/strict"; import { setFlagsFromString } from "node:v8"; import { runInNewContext } from "node:vm"; -import { Type } from "typebox"; +import { num, obj } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { createHost, defineRegistry, type Facet, local, method } from "../src/index.ts"; @@ -41,8 +41,8 @@ function tracker() { const Store = local("t.gen-store", { version: 1, - input: Type.Object({}), - members: { get: method(undefined, Type.Number()), count: Type.Number() }, + input: obj({}, { open: true }), + members: { get: method(undefined, num()), count: num() }, }); const Routes = defineRegistry<() => number>("t.gen-routes"); const RELOADS = 6; @@ -57,7 +57,7 @@ describe("generation GC", () => { name: "store", setup: (api) => api.serve(Store, (_input, _principal, self) => { - const count = self.state("count", Type.Number(), 0); + const count = self.state("count", num(), 0); return { count, get: () => sentinel.generation }; }), }; @@ -84,7 +84,7 @@ describe("generation GC", () => { api.serve( Store, (_input, _principal, self) => { - const count = self.state("count", Type.Number(), 0); + const count = self.state("count", num(), 0); // Kept across reloads (same version): adopts each new declaration's close. self.resource("conn", { version: 1, @@ -123,7 +123,7 @@ describe("generation GC", () => { api.serve( Store, (_input, _principal, self) => { - const count = self.state("count", Type.Number(), 0); + const count = self.state("count", num(), 0); self.task(async (ctx) => { await new Promise((resolve) => ctx.signal.addEventListener("abort", () => resolve())); void sentinel.generation; @@ -153,8 +153,8 @@ describe("generation GC", () => { const objects = tracker(); const Item = local("t.gen-item", { version: 1, - input: Type.Object({ id: Type.Number() }), - members: { id: method(undefined, Type.Number()) }, + input: obj({ id: num() }, { open: true }), + members: { id: method(undefined, num()) }, }); const host = createHost({ self: { kind: "host" }, onError: () => {} }); host.serve(Item, (input) => { diff --git a/packages/chord/test/host-objects-reload.test.ts b/packages/chord/test/host-objects-reload.test.ts index 6e762e701..d01912102 100644 --- a/packages/chord/test/host-objects-reload.test.ts +++ b/packages/chord/test/host-objects-reload.test.ts @@ -1,5 +1,5 @@ import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { bool, num, obj, str } from "@earendil-works/pi-schema"; import { onTestFinished, test } from "vitest"; import { HostImpl } from "../src/host.ts"; import { @@ -12,15 +12,16 @@ import { method, type Self, } from "../src/index.ts"; +import { record } from "./fixtures/schemas.ts"; -const input = Type.Object({}); -const flags = Type.Record(Type.String(), Type.Boolean()); +const input = obj({}, { open: true }); +const flags = record(bool()); const SwitchV1 = local("t.eager-switch", { version: 1, input, members: { - enabled: Type.Boolean(), - set: method(Type.Boolean()), + enabled: bool(), + set: method(bool()), }, }); const SwitchV2 = local("t.eager-switch", { @@ -28,7 +29,7 @@ const SwitchV2 = local("t.eager-switch", { input, members: { enabled: flags, - set: method(Type.Object({ conversation: Type.Number(), enabled: Type.Boolean() })), + set: method(obj({ conversation: num(), enabled: bool() }, { open: true })), }, }); const SwitchV3 = local("t.eager-switch", { @@ -36,7 +37,7 @@ const SwitchV3 = local("t.eager-switch", { input, members: { enabled: flags, - update: method(Type.Object({ conversation: Type.Number(), enabled: Type.Boolean() })), + update: method(obj({ conversation: num(), enabled: bool() }, { open: true })), }, }); const Read = defineRegistry<() => unknown>("t.eager-switch-read"); @@ -60,7 +61,7 @@ function switchFacets(log: string[]) { SwitchV1, (_input, _principal, self) => { declareResource(self); - const enabledState = self.state("enabled", Type.Boolean(), false); + const enabledState = self.state("enabled", bool(), false); self.register(Read, { current: () => enabledState.value }); return { enabled: enabledState, @@ -171,13 +172,13 @@ test("the host's reference to a host object follows contract bumps at commit wit const ReadV1 = local("t.eager-read", { version: 1, - input: Type.Object({}), - members: { read: method(undefined, Type.String()) }, + input: obj({}, { open: true }), + members: { read: method(undefined, str()) }, }); const ReadV2 = local("t.eager-read", { version: 2, - input: Type.Object({}), - members: { fetch: method(undefined, Type.String()) }, + input: obj({}, { open: true }), + members: { fetch: method(undefined, str()) }, }); const readerV1: Facet = { name: "read", @@ -219,8 +220,8 @@ test("object-owned old-version references still block a host object's upgrade un onTestFinished(() => host.close()); const Consumer = local("t.eager-consumer", { version: 1, - input: Type.Object({}), - members: { read: method(undefined, Type.String()) }, + input: obj({}, { open: true }), + members: { read: method(undefined, str()) }, }); const oldConsumer: Facet = { name: "consumer", @@ -254,8 +255,8 @@ test("a forgotten contract bump remains refused even when the only consumer is t await host.load([readerV1]); const incompatible = local(ReadV1.name, { version: 1, - input: Type.Object({}), - members: { fetch: method(undefined, Type.String()) }, + input: obj({}, { open: true }), + members: { fetch: method(undefined, str()) }, }); await assert.rejects( host.reload([ @@ -310,7 +311,7 @@ test("a cluster contract upgrade retokens the server's and worker's references t test("a reload that changes a type's lifetime creates its host object, or lets it go", async () => { const host = createHost({ self: { kind: "host" }, onError: () => {} }); - const Kept = local("t.lifetime-change", { version: 1, input: Type.Object({}) }); + const Kept = local("t.lifetime-change", { version: 1, input: obj({}, { open: true }) }); const log: string[] = []; const facet = (lifetime: "host" | "referenced"): Facet => ({ name: "kept", diff --git a/packages/chord/test/host.test.ts b/packages/chord/test/host.test.ts index eb68b5f6b..6a0d34896 100644 --- a/packages/chord/test/host.test.ts +++ b/packages/chord/test/host.test.ts @@ -1,5 +1,5 @@ import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { bool, num, obj, opt, str } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { type Ctx, @@ -27,10 +27,10 @@ async function until(check: () => boolean): Promise { const Counter = local("t.counter", { version: 1, - input: Type.Object({ name: Type.String() }), + input: obj({ name: str() }, { open: true }), members: { - count: Type.Object({ n: Type.Number() }), - add: method(Type.Number(), Type.Number()), + count: obj({ n: num() }, { open: true }), + add: method(num(), num()), }, }); @@ -39,7 +39,7 @@ function serveCounter(host: Host, log: string[] = [], options: { linger?: number Counter, ({ name }, _principal, self) => { log.push(`factory ${name}`); - const count = self.state("count", Type.Object({ n: Type.Number() }), { n: 0 }); + const count = self.state("count", obj({ n: num() }, { open: true }), { n: 0 }); const conn = self.resource("conn", { version: 1, open: async () => { @@ -123,7 +123,7 @@ describe("host: activation and lifetime", () => { const host = quiet(); const log: string[] = []; let release = () => {}; - const Slow = local("t.slow", { version: 1, input: Type.Object({}) }); + const Slow = local("t.slow", { version: 1, input: obj({}, { open: true }) }); host.serve(Slow, (_input, _principal, self) => { self.resource("r", { version: 1, @@ -156,8 +156,8 @@ describe("host: activation and lifetime", () => { it("propagates activation failures to dependents and retries on the next get", async () => { const host = quiet(); let fail = true; - const Leaf = local("t.leaf", { version: 1, input: Type.Object({}) }); - const Parent = local("t.parent", { version: 1, input: Type.Object({}) }); + const Leaf = local("t.leaf", { version: 1, input: obj({}, { open: true }) }); + const Parent = local("t.parent", { version: 1, input: obj({}, { open: true }) }); host.serve(Leaf, (_input, _principal, self) => { self.resource("r", { version: 1, @@ -184,8 +184,8 @@ describe("host: activation and lifetime", () => { it("unwinds declarations when a factory throws and rejects async factories", async () => { const host = quiet(); - const Leaf = local("t.leaf2", { version: 1, input: Type.Object({}) }); - const Bad = local("t.bad", { version: 1, input: Type.Object({ async: Type.Boolean() }) }); + const Leaf = local("t.leaf2", { version: 1, input: obj({}, { open: true }) }); + const Bad = local("t.bad", { version: 1, input: obj({ async: bool() }, { open: true }) }); let leafLive = 0; host.serve(Leaf, (_input, _principal, self) => { self.resource("r", { @@ -213,10 +213,10 @@ describe("host: activation and lifetime", () => { it("rejects duplicate declaration names", async () => { const host = quiet(); - const Dup = local("t.dup", { version: 1, input: Type.Object({}) }); + const Dup = local("t.dup", { version: 1, input: obj({}, { open: true }) }); host.serve(Dup, (_input, _principal, self) => { - self.state("a", Type.Object({}), {}); - self.state("a", Type.Object({}), {}); + self.state("a", obj({}, { open: true }), {}); + self.state("a", obj({}, { open: true }), {}); return {}; }); await assert.rejects(host.hold(Dup, {}).ready, /duplicate state name a/); @@ -224,8 +224,8 @@ describe("host: activation and lifetime", () => { it("detects hold cycles", async () => { const host = quiet(); - const A = local("t.cycle-a", { version: 1, input: Type.Object({}) }); - const B = local("t.cycle-b", { version: 1, input: Type.Object({}) }); + const A = local("t.cycle-a", { version: 1, input: obj({}, { open: true }) }); + const B = local("t.cycle-b", { version: 1, input: obj({}, { open: true }) }); host.serve(A, (_input, _principal, self) => { self.get(B, {}); return {}; @@ -241,7 +241,7 @@ describe("host: activation and lifetime", () => { const errors: unknown[] = []; const host = quiet(errors); const log: string[] = []; - const Multi = local("t.multi", { version: 1, input: Type.Object({}) }); + const Multi = local("t.multi", { version: 1, input: obj({}, { open: true }) }); host.serve(Multi, (_input, _principal, self) => { const a = self.resource("a", { version: 1, open: () => "a", close: () => void log.push("close a") }); self.resource("b", { @@ -291,15 +291,15 @@ describe("host: activation and lifetime", () => { describe("host: declarations and methods", () => { it("refuses a second declaration of one name per kind; kinds have their own names", async () => { const host = quiet(); - const Twice = local("t.twice", { version: 1, input: Type.Object({ kind: Type.String() }) }); + const Twice = local("t.twice", { version: 1, input: obj({ kind: str() }, { open: true }) }); host.serve(Twice, ({ kind }, _principal, self) => { const open = { version: 1, open: () => null }; - self.state("x", Type.Number(), 0); + self.state("x", num(), 0); self.resource("x", open); - self.on("x", self.state("y", Type.Number(), 0), () => {}); - if (kind === "state") self.state("x", Type.Number(), 1); + self.on("x", self.state("y", num(), 0), () => {}); + if (kind === "state") self.state("x", num(), 1); if (kind === "resource") self.resource("x", open); - if (kind === "listener") self.on("x", self.state("z", Type.Number(), 0), () => {}); + if (kind === "listener") self.on("x", self.state("z", num(), 0), () => {}); return {}; }); await host.hold(Twice, { kind: "none" }).ready; @@ -318,11 +318,11 @@ describe("host: declarations and methods", () => { const host = quiet(); const Tool = local("t.tool", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), members: { ping: method(), - count: method(undefined, Type.Number()), - note: method(Type.String()), + count: method(undefined, num()), + note: method(str()), }, }); const seen: unknown[] = []; @@ -348,8 +348,8 @@ describe("host: declarations and methods", () => { const host = quiet(); const Doubled = local("t.doubled", { version: 1, - input: Type.Object({}), - members: { set: method(Type.Number()) }, + input: obj({}, { open: true }), + members: { set: method(num()) }, }); const seen: number[] = []; const facet = (label: string): Facet => ({ @@ -358,7 +358,7 @@ describe("host: declarations and methods", () => { api.serve( Doubled, (_input, _principal, self) => { - const n = self.state("n", Type.Number(), 1); + const n = self.state("n", num(), 1); const doubled = self.derive([n], (value) => value * 2); self.on("doubled", doubled, (value) => { seen.push(value); @@ -389,7 +389,7 @@ describe("host: declarations and methods", () => { describe("host: lifetime", () => { const Store = local("t.store", { version: 1, - input: Type.Object({ scope: Type.Optional(Type.String({ default: "all" })) }), + input: obj({ scope: opt(str({ default: "all" })) }, { open: true }), members: { seen: opaque("t.seen@1") }, }); @@ -397,11 +397,11 @@ describe("host: lifetime", () => { const host = quiet(); const Remote = remote("t.remote-store", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("host"), variants: { main: {} }, }); - const Keyed = local("t.keyed", { version: 1, input: Type.Object({ id: Type.String() }) }); + const Keyed = local("t.keyed", { version: 1, input: obj({ id: str() }, { open: true }) }); // Each refused call is also a type error; the runtime check is the backstop for untyped callers. const refused: [() => void, RegExp][] = [ // @ts-expect-error: a remote type is never of lifetime "host" @@ -415,7 +415,7 @@ describe("host: lifetime", () => { ]; for (const [serve, error] of refused) assert.throws(serve, error); // Allowed: a local type whose input has only optional fields, and any type with the default lifetime. - const Optional = local("t.optional", { version: 1, input: Type.Object({ id: Type.Optional(Type.String()) }) }); + const Optional = local("t.optional", { version: 1, input: obj({ id: opt(str()) }, { open: true }) }); host.serve(Optional, () => ({}), { lifetime: "host" }); host.serve(Keyed, () => ({}), { lifetime: "referenced", linger: 10 }); host.serve(Remote, () => ({ variant: "main" }), { linger: 10 }); @@ -462,13 +462,13 @@ describe("host: lifetime", () => { describe("host: principals, variants and disposal", () => { const Doc = remote("t.doc", { version: 1, - input: Type.Object({ id: Type.String() }), + input: obj({ id: str() }, { open: true }), host: hostNamed("host"), variants: { - viewer: { state: { view: Type.Object({ text: Type.String() }) } }, + viewer: { state: { view: obj({ text: str() }, { open: true }) } }, editor: { - state: { view: Type.Object({ text: Type.String() }) }, - methods: { edit: method(Type.Object({ text: Type.String() }), Type.Object({ ok: Type.Boolean() })) }, + state: { view: obj({ text: str() }, { open: true }) }, + methods: { edit: method(obj({ text: str() }, { open: true }), obj({ ok: bool() }, { open: true })) }, }, }, }); @@ -476,7 +476,7 @@ describe("host: principals, variants and disposal", () => { function serveDoc(host: Host, principals: unknown[] = []) { host.serve(Doc, (_input, principal, self) => { principals.push(principal); - const view = self.state("view", Type.Object({ text: Type.String() }), { text: "" }); + const view = self.state("view", obj({ text: str() }, { open: true }), { text: "" }); if ((principal as { role?: string }).role !== "editor") return { variant: "viewer", view }; return { variant: "editor", @@ -531,7 +531,7 @@ describe("host: principals, variants and disposal", () => { let seenAbort = false; const Slow = local("t.inflight", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), members: { wait: method() }, }); host.serve(Slow, () => ({ @@ -585,8 +585,8 @@ describe("host: registries", () => { it('only objects of lifetime "host" register; tables follow registration order', async () => { const host = quiet(); - const RoutesA = local("t.routes-a", { version: 1, input: Type.Object({}) }); - const RoutesB = local("t.routes-b", { version: 1, input: Type.Object({}) }); + const RoutesA = local("t.routes-a", { version: 1, input: obj({}, { open: true }) }); + const RoutesB = local("t.routes-b", { version: 1, input: obj({}, { open: true }) }); host.serve( RoutesA, (_input, _principal, self) => { @@ -611,7 +611,7 @@ describe("host: registries", () => { assert.equal(host.registry(Routes).get("GET /b"), "b"); assert.deepEqual(tables.at(-1), ["GET /a", "GET /c", "GET /b"]); - const Lazy = local("t.lazy-register", { version: 1, input: Type.Object({}) }); + const Lazy = local("t.lazy-register", { version: 1, input: obj({}, { open: true }) }); host.serve(Lazy, (_input, _principal, self) => { self.register(Routes, { x: "x" }); return {}; @@ -624,7 +624,7 @@ describe("host: registries", () => { const host = quiet(errors); for (const name of ["t.conflict-a", "t.conflict-b"]) { host.serve( - local(name, { version: 1, input: Type.Object({}) }), + local(name, { version: 1, input: obj({}, { open: true }) }), (_input, _principal, self) => { self.register(Routes, { same: name }); return {}; @@ -641,19 +641,19 @@ describe("host: registries", () => { describe("host: composition", () => { const Source = local("t.source", { version: 1, - input: Type.Object({}), - members: { n: Type.Number(), set: method(Type.Number()) }, + input: obj({}, { open: true }), + members: { n: num(), set: method(num()) }, }); const Mirror = local("t.mirror", { version: 1, - input: Type.Object({}), - members: { copy: Type.Number(), doubled: Type.Number() }, + input: obj({}, { open: true }), + members: { copy: num(), doubled: num() }, }); it("reference state members are sources for listeners and derivations", async () => { const host = quiet(); host.serve(Source, (_input, _principal, self) => { - const n = self.state("n", Type.Number(), 1); + const n = self.state("n", num(), 1); return { n, set: (value: number, ctx: Ctx) => { @@ -663,7 +663,7 @@ describe("host: composition", () => { }); host.serve(Mirror, (_input, _principal, self) => { const source = self.get(Source, {}); - const copy = self.state("copy", Type.Number(), 0); + const copy = self.state("copy", num(), 0); self.on("follow", source.n, (value, ctx) => copy.replace(ctx, value ?? 0)); const doubled = self.derive([source.n], (n) => (n ?? 0) * 2); return { copy, doubled }; @@ -688,12 +688,12 @@ describe("host: composition", () => { const seen: unknown[] = []; const Who = local("t.who", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), members: { ping: method() }, }); const Front = local("t.front", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), members: { relay: method() }, }); host.serve(Who, (_input, principal) => { @@ -711,10 +711,10 @@ describe("host: composition", () => { it("calling a reference inside a factory throws not_ready", async () => { const host = quiet(); host.serve(Source, (_input, _principal, self) => ({ - n: self.state("n", Type.Number(), 0), + n: self.state("n", num(), 0), set: () => {}, })); - const Eager = local("t.eager-call", { version: 1, input: Type.Object({}) }); + const Eager = local("t.eager-call", { version: 1, input: obj({}, { open: true }) }); host.serve(Eager, (_input, _principal, self) => { const source = self.get(Source, {}); void source.set(1); @@ -725,8 +725,8 @@ describe("host: composition", () => { it("a ChordError from a factory keeps its code through dependents", async () => { const host = quiet(); - const Secret = local("t.secret", { version: 1, input: Type.Object({}) }); - const Page = local("t.page", { version: 1, input: Type.Object({}) }); + const Secret = local("t.secret", { version: 1, input: obj({}, { open: true }) }); + const Page = local("t.page", { version: 1, input: obj({}, { open: true }) }); host.serve(Secret, () => { throw forbidden("not for you"); }); diff --git a/packages/chord/test/hydration.test.ts b/packages/chord/test/hydration.test.ts index 2c80255cc..77b3806ac 100644 --- a/packages/chord/test/hydration.test.ts +++ b/packages/chord/test/hydration.test.ts @@ -1,7 +1,7 @@ // Entry 8: a local object goes live only with its state, also where it forwards or derives from a remote reference's // member; activation waits at most `readyTimeout` (a cycle across hosts fails with a hint instead of hanging). import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { num, obj, str } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { type Ctx, @@ -20,22 +20,22 @@ const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); /** Lives on the worker `w`; its value is set by `set`. */ const Source = remote("t.e8-source", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("w"), - variants: { main: { state: { v: Type.String() }, methods: { set: method(Type.String()) } } }, + variants: { main: { state: { v: str() }, methods: { set: method(str()) } } }, }); /** Lives on the server; forwards (and derives from) the worker's `Source`. */ const Forward = remote("t.e8-forward", { version: 1, - input: Type.Object({ how: Type.String() }), + input: obj({ how: str() }, { open: true }), host: hostNamed("server"), - variants: { main: { state: { v: Type.String(), size: Type.Number() } } }, + variants: { main: { state: { v: str(), size: num() } } }, }); function serveSource(host: Host): void { host.serve(Source, (_input, _principal, self) => { - const v = self.state("v", Type.String(), "from the worker"); + const v = self.state("v", str(), "from the worker"); return { variant: "main", v, @@ -105,9 +105,9 @@ describe("local hydration", () => { it("a member read inside a factory throws not_ready; it is a plain value once live", async () => { const host = createHost({ self: { kind: "host" }, onError: () => {} }); - const Inner = local("t.e8-inner", { version: 1, input: Type.Object({}), members: { n: Type.Number() } }); - const Outer = local("t.e8-outer", { version: 1, input: Type.Object({}), members: { m: Type.Number() } }); - host.serve(Inner, (_input, _principal, self) => ({ n: self.state("n", Type.Number(), 7) })); + const Inner = local("t.e8-inner", { version: 1, input: obj({}, { open: true }), members: { n: num() } }); + const Outer = local("t.e8-outer", { version: 1, input: obj({}, { open: true }), members: { m: num() } }); + host.serve(Inner, (_input, _principal, self) => ({ n: self.state("n", num(), 7) })); const seen: unknown[] = []; host.serve(Outer, (_input, _principal, self) => { const inner = self.get(Inner, {}); @@ -127,15 +127,15 @@ describe("local hydration", () => { it("a host object of a child holding the parent's object goes live once the parent admits the child", async () => { const Settings = remote("t.e8-settings", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), - variants: { main: { state: { theme: Type.String() } } }, + variants: { main: { state: { theme: str() } } }, }); - const Local = local("t.e8-boot", { version: 1, input: Type.Object({}), members: { theme: Type.String() } }); + const Local = local("t.e8-boot", { version: 1, input: obj({}, { open: true }), members: { theme: str() } }); const c = cluster({ delay: 20 }); c.server.serve(Settings, (_input, _principal, self) => ({ variant: "main", - theme: self.state("theme", Type.String(), "dark"), + theme: self.state("theme", str(), "dark"), })); await c.worker.install( { @@ -163,13 +163,13 @@ describe("local hydration", () => { it("a reference cycle across hosts fails at readyTimeout with a cycle hint", async () => { const A = remote("t.e8-cycle-a", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), variants: { main: {} }, }); const B = remote("t.e8-cycle-b", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("w"), variants: { main: {} }, }); diff --git a/packages/chord/test/link.test.ts b/packages/chord/test/link.test.ts index 31c0c9229..eee67c8b4 100644 --- a/packages/chord/test/link.test.ts +++ b/packages/chord/test/link.test.ts @@ -1,5 +1,5 @@ import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { arr, num, obj, str, unknown } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { type Ctx, @@ -25,24 +25,24 @@ const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); const counterVariants = { main: { - state: { count: Type.Object({ n: Type.Number() }) }, + state: { count: obj({ n: num() }, { open: true }) }, methods: { - add: method(Type.Number(), Type.Number()), - who: method(undefined, Type.Unknown()), + add: method(num(), num()), + who: method(undefined, unknown()), }, }, }; /** A counter on the server. */ const Counter = remote("t.counter", { version: 1, - input: Type.Object({ name: Type.String() }), + input: obj({ name: str() }, { open: true }), host: hostNamed("server"), variants: counterVariants, }); /** A counter on a session host (`session:` + its `session` field). */ const SessionCounter = remote("t.session-counter", { version: 1, - input: Type.Object({ session: Type.String(), name: Type.String() }), + input: obj({ session: str(), name: str() }, { open: true }), host: hostFamily("session:", "session"), variants: counterVariants, }); @@ -52,7 +52,7 @@ const SessionCounter = remote("t.session-counter", { function serveCounter(host: Host, principals: unknown[] = []) { const counter = (principal: unknown, self: Self) => { principals.push(principal); - const count = self.state("count", Type.Object({ n: Type.Number() }), { n: 0 }); + const count = self.state("count", obj({ n: num() }, { open: true }), { n: 0 }); return { variant: "main" as const, count, @@ -199,7 +199,7 @@ function cluster(options: { pingInterval?: number; pingTimeout?: number } = {}): open: () => void log.push(`open ${input.name}`), close: () => void log.push(`close ${input.name}`), }); - const count = self.state("count", Type.Object({ n: Type.Number() }), { n: 0 }); + const count = self.state("count", obj({ n: num() }, { open: true }), { n: 0 }); return { variant: "main", count, @@ -346,9 +346,9 @@ describe("links: relay, stamping, lifetime", () => { it("checks contract tokens across links: contract_mismatch for hosts, upgrade_required for browsers", async () => { const CounterV2 = remote("t.counter", { version: 2, - input: Type.Object({ name: Type.String() }), + input: obj({ name: str() }, { open: true }), host: hostNamed("server"), - variants: { main: { methods: { inc: method(undefined, Type.Number()) } } }, + variants: { main: { methods: { inc: method(undefined, num()) } } }, }); const { server, worker } = serverWithWorker(); server.serve(CounterV2, () => ({ variant: "main", inc: () => 1 })); @@ -408,12 +408,12 @@ describe("links: relay, stamping, lifetime", () => { const Big = remote("t.big", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("session:S1"), variants: { main: { - state: { notes: Type.Array(Type.String()) }, - methods: { echo: method(Type.Array(Type.String()), Type.Array(Type.String())) }, + state: { notes: arr(str()) }, + methods: { echo: method(arr(str()), arr(str())) }, }, }, }); @@ -425,10 +425,10 @@ describe("links: methods without arguments or result", () => { it("travel without args and without a value, and resolve to the result or undefined", async () => { const Tool = remote("t.tool", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), variants: { - main: { methods: { ping: method(), count: method(undefined, Type.Number()), note: method(Type.String()) } }, + main: { methods: { ping: method(), count: method(undefined, num()), note: method(str()) } }, }, }); const server = createHost({ self: { kind: "host" }, name: "server", onError: () => {} }); @@ -551,7 +551,7 @@ describe("links: packed payloads", () => { onError: () => {}, }); worker.serve(Big, (_input, _principal, self) => { - const notesState = self.state("notes", Type.Array(Type.String()), large("a")); + const notesState = self.state("notes", arr(str()), large("a")); return { variant: "main", notes: notesState, echo: (notes: string[]) => notes }; }); return down; diff --git a/packages/chord/test/links-review.test.ts b/packages/chord/test/links-review.test.ts index c607d673f..f6a456afb 100644 --- a/packages/chord/test/links-review.test.ts +++ b/packages/chord/test/links-review.test.ts @@ -1,6 +1,6 @@ // Regression tests for Astra's phase 1b review (findings #1-#11) and the shared-worker test gap. import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { num, obj, str, unknown } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { type Ctx, @@ -66,14 +66,14 @@ class WatchedLink implements Link { const Counter = remote("t.counter", { version: 1, - input: Type.Object({ name: Type.String() }), + input: obj({ name: str() }, { open: true }), host: hostNamed("session:S1"), variants: { main: { - state: { count: Type.Object({ n: Type.Number() }) }, + state: { count: obj({ n: num() }, { open: true }) }, methods: { - add: method(Type.Number(), Type.Number()), - who: method(undefined, Type.Unknown()), + add: method(num(), num()), + who: method(undefined, unknown()), slow: method(), }, }, @@ -138,7 +138,7 @@ function cluster() { open: () => void log.push(`open ${input.name}`), close: () => void log.push(`close ${input.name}`), }); - const count = self.state("count", Type.Object({ n: Type.Number() }), { n: 0 }); + const count = self.state("count", obj({ n: num() }, { open: true }), { n: 0 }); return { variant: "main", count, @@ -190,15 +190,15 @@ describe("links review: identity and stamping", () => { const server = createHost({ self: { kind: "host" }, name: "server", onError: () => {} }); const Echo = remote("t.echo", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), - variants: { main: { methods: { root: method(undefined, Type.String()) } } }, + variants: { main: { methods: { root: method(undefined, str()) } } }, }); const facet = (version: number): Facet => ({ name: "echo", setup: (api) => api.serve(Echo, (_input, _principal, self) => { - self.state("n", Type.Number(), 0, { version }); + self.state("n", num(), 0, { version }); return { variant: "main", root: (ctx: Ctx) => ctx.rootId }; }), }); @@ -342,22 +342,22 @@ describe("links review: relay ownership", () => { it("#11 snapshots through an accepts adapter are validated against the consumer's schema", async () => { const DocV1 = remote("t.doc", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), - variants: { main: { state: { view: Type.Object({ text: Type.String() }) } } }, + variants: { main: { state: { view: obj({ text: str() }, { open: true }) } } }, }); const DocV2 = remote("t.doc", { version: 2, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), - variants: { main: { state: { view: Type.Object({ body: Type.String() }) } } }, + variants: { main: { state: { view: obj({ body: str() }, { open: true }) } } }, }); const server = createHost({ self: { kind: "host" }, name: "server", onError: () => {} }); server.serve( DocV2, (_input, _principal, self) => ({ variant: "main", - view: self.state("view", Type.Object({ body: Type.String() }), { body: "hi" }), + view: self.state("view", obj({ body: str() }, { open: true }), { body: "hi" }), }), { accepts: { @@ -402,9 +402,9 @@ describe("links review: relay ownership", () => { describe("links review: cross-host reload", () => { const Greeter = remote("t.greeter", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("session:S1"), - variants: { main: { methods: { greet: method(Type.String(), Type.String()) } } }, + variants: { main: { methods: { greet: method(str(), str()) } } }, }); const greeterFacet = (greeting: string): Facet => ({ name: "greeter", @@ -468,7 +468,7 @@ describe("links review: cross-host reload", () => { const { server, modules } = setup(); await tick(); const results: unknown[] = []; - const Caller = local("t.caller", { version: 1, input: Type.Object({}) }); + const Caller = local("t.caller", { version: 1, input: obj({}, { open: true }) }); const callerFacet = (active: boolean): Facet => ({ name: "caller", setup: (api) => @@ -517,13 +517,13 @@ describe("links review: cross-host reload", () => { await tick(); const Echo = remote("t.echo2", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), variants: { main: { methods: { ping: method() } } }, }); const Bouncer = remote("t.bouncer", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("session:S1"), variants: { main: { methods: { bounce: method() } } }, }); @@ -531,7 +531,7 @@ describe("links review: cross-host reload", () => { name: "echo", setup: (api) => api.serve(Echo, (_input, _principal, self) => { - self.state("n", Type.Number(), 0, { version }); + self.state("n", num(), 0, { version }); return { variant: "main", ping: () => {} }; }), }); diff --git a/packages/chord/test/next/contracts/declare.test.ts b/packages/chord/test/next/contracts/declare.test.ts index 25e4e0811..9f17d301c 100644 --- a/packages/chord/test/next/contracts/declare.test.ts +++ b/packages/chord/test/next/contracts/declare.test.ts @@ -1,4 +1,3 @@ -import Type from "typebox"; import { describe, expect, it } from "vitest"; import { arr, @@ -143,7 +142,12 @@ describe("declaration checks", () => { [ "open arguments", "n", - { version: 1, params: {}, hostAddress, variants: { a: { methods: { m: method(Type.Object({})) } } } }, + { + version: 1, + params: {}, + hostAddress, + variants: { a: { methods: { m: method({ type: "object", properties: {} }) } } }, + }, "n.a.methods.m.args: argument objects must be closed", ], [ @@ -166,7 +170,7 @@ describe("declaration checks", () => { ], [ "open arguments", - { version: 1, params: {}, members: { m: method(Type.Object({})) } }, + { version: 1, params: {}, members: { m: method({ type: "object", properties: {} }) } }, "i.members.m.args: argument objects must be closed", ], ])("local refuses %s", (_case, declaration, message) => { @@ -181,7 +185,13 @@ describe("declaration checks", () => { }); it("does not check state for closed objects", () => { - expect(() => local("i", { version: 1, params: {}, members: { s: Type.Object({ a: str() }) } })).not.toThrow(); + expect(() => + local("i", { + version: 1, + params: {}, + members: { s: { type: "object", properties: { a: str() }, required: ["a"] } }, + }), + ).not.toThrow(); }); }); diff --git a/packages/chord/test/next/contracts/evolve.test.ts b/packages/chord/test/next/contracts/evolve.test.ts index 4ddfb78ff..0c4a17354 100644 --- a/packages/chord/test/next/contracts/evolve.test.ts +++ b/packages/chord/test/next/contracts/evolve.test.ts @@ -1,4 +1,3 @@ -import Type from "typebox"; import { describe, expect, expectTypeOf, it } from "vitest"; import type { Context } from "../../../src/context/index.ts"; import { type Contract, contractData, type ShapeOf } from "../../../src/next/contracts/contract.ts"; @@ -136,16 +135,18 @@ describe("edits", () => { }); it("checks every edit: closed arguments and the vocabulary", () => { - expect(() => evolve(NotesV1, (e) => e.args("add", Type.Object({ text: str() })))).toThrow( + expect(() => evolve(NotesV1, (e) => e.args("add", { type: "object", properties: { text: str() } }))).toThrow( "notes v2.editor.methods.add.args: argument objects must be closed", ); - expect(() => evolve(InfoV1, (e) => e.addMethod("m", Type.Object({}), nul()))).toThrow( + expect(() => evolve(InfoV1, (e) => e.addMethod("m", { type: "object", properties: {} }, nul()))).toThrow( "info v2.members.m.args: argument objects must be closed", ); expect(() => evolve(NotesV1, (e) => e.state("editor", "count", { not: {} } as never))).toThrow( 'notes v2.editor.state.count: unsupported keyword "not"', ); - expect(() => evolve(NotesV1, (e) => e.state("editor", "count", Type.Object({})))).not.toThrow(); + expect(() => + evolve(NotesV1, (e) => e.state("editor", "count", { type: "object", properties: {} })), + ).not.toThrow(); }); }); diff --git a/packages/chord/test/next/contracts/fingerprint.test.ts b/packages/chord/test/next/contracts/fingerprint.test.ts index f159b7856..a31434198 100644 --- a/packages/chord/test/next/contracts/fingerprint.test.ts +++ b/packages/chord/test/next/contracts/fingerprint.test.ts @@ -1,13 +1,23 @@ -import Type, { type TSchema } from "typebox"; import { describe, expect, it } from "vitest"; -import { evolve, hostKind, int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { + evolve, + hostKind, + int, + local, + method, + obj, + remote, + type Schema, + str, + union, +} from "../../../src/next/contracts/index.ts"; import { NORMAL_FORM_VERSION } from "../../../src/next/contracts/internal.ts"; import type { Params } from "../../../src/next/contracts/params.ts"; const Server = hostKind("server"); /** Fingerprint of a one-variant remote contract with state `s` and params `params`. */ -function fingerprintOf(state: TSchema, params: Params = {}): string { +function fingerprintOf(state: Schema, params: Params = {}): string { return remote("n", { version: 1, params, hostAddress: () => Server(), variants: { a: { state: { s: state } } } }) .fingerprint; } @@ -47,12 +57,12 @@ describe("fingerprints", () => { }); it.each([ - ["defaults", Type.String(), Type.String({ default: "x" })], - ["bounds", Type.String(), Type.String({ maxLength: 3 })], - ["formats", Type.String(), Type.String({ format: "email" })], - ["union order", Type.Union([Type.String(), Type.Integer()]), Type.Union([Type.Integer(), Type.String()])], - ["additionalProperties", Type.Object({}), Type.Object({}, { additionalProperties: false })], - ["readOnly", Type.String(), Type.String({ readOnly: true })], + ["defaults", str(), str({ default: "x" })], + ["bounds", str(), str({ max: 3 })], + ["formats", str(), { type: "string", format: "email" }], + ["union order", union([str(), int()]), union([int(), str()])], + ["additionalProperties", { type: "object", properties: {} }, obj({})], + ["readOnly", str(), { type: "string", readOnly: true }], ])("are sensitive to %s", (_case, a, b) => { expect(fingerprintOf(a)).not.toBe(fingerprintOf(b)); }); diff --git a/packages/chord/test/next/contracts/params.test.ts b/packages/chord/test/next/contracts/params.test.ts index 96187b2b0..e760a9c58 100644 --- a/packages/chord/test/next/contracts/params.test.ts +++ b/packages/chord/test/next/contracts/params.test.ts @@ -1,4 +1,3 @@ -import Type from "typebox"; import { describe, expect, it } from "vitest"; import { arr, @@ -48,7 +47,7 @@ describe("params policy", () => { ["arrays", arr(str())], ["bytes", bytes()], ["numeric literals", oneOf(1, 2)], - ["defaults", Type.String({ default: "x" })], + ["defaults", str({ default: "x" })], ])("refuses %s, in the types and at declaration", (_case, schema) => { // @ts-expect-error: not a params field schema expect(() => remote("n", { version: 1, params: { p: schema }, hostAddress: () => Server(), variants })).toThrow( diff --git a/packages/chord/test/next/contracts/schema.test.ts b/packages/chord/test/next/contracts/schema.test.ts index f509d0262..ee6a05691 100644 --- a/packages/chord/test/next/contracts/schema.test.ts +++ b/packages/chord/test/next/contracts/schema.test.ts @@ -1,4 +1,6 @@ -import { Compile } from "typebox/compile"; +// Tests of the schema helpers: the JSON Schema they build over pi-schema's builders, their types, `bytes()`, `method`. + +import { compile } from "@earendil-works/pi-schema"; import { describe, expect, expectTypeOf, it } from "vitest"; import { arr, @@ -22,7 +24,7 @@ import { Method } from "../../../src/next/contracts/internal.ts"; import type { Equal } from "../support.ts"; describe("schema helpers", () => { - it("produce ordinary TypeBox schemas with bounds under their JSON Schema names", () => { + it("produce plain JSON Schema with bounds under their JSON Schema names", () => { expect(str({ min: 1, max: 3, pattern: "^a" })).toEqual({ type: "string", minLength: 1, @@ -58,34 +60,39 @@ describe("schema helpers", () => { properties: { a: str(), b: int() }, additionalProperties: false, }); - expect(Compile(schema).Check({ a: "x" })).toBe(true); - expect(Compile(schema).Check({ a: "x", c: 1 })).toBe(false); + expect(compile(schema).check({ a: "x" })).toBe(true); + expect(compile(schema).check({ a: "x", c: 1 })).toBe(false); }); - it("Static and TypeBox's compiler work unchanged", () => { + it("Static gives the values' type; pi-schema's compiler checks them", () => { const schema = obj({ kind: oneOf("chat", "agent"), n: opt(int()), tags: arr(str()), data: bytes() }); type Expected = { kind: "chat" | "agent"; n?: number; tags: string[]; data: Uint8Array }; expectTypeOf, Expected>>().toEqualTypeOf(); - expect(Compile(schema).Check({ kind: "chat", tags: [], data: new Uint8Array(1) })).toBe(true); + expect(compile(schema).check({ kind: "chat", tags: [], data: new Uint8Array(1) })).toBe(true); }); }); describe("bytes", () => { - it("is the data marker with the check attached non-enumerably", () => { + it("is plain data: the marker and nothing else", () => { const schema = bytes({ min: 1, max: 2 }); - expect(JSON.parse(JSON.stringify(schema))).toEqual({ type: "bytes", minLength: 1, maxLength: 2 }); - expect(Object.keys(schema)).toEqual(["type", "minLength", "maxLength"]); - expect(Object.getOwnPropertyDescriptor(schema, "~refine")?.enumerable).toBe(false); + expect(Reflect.ownKeys(schema)).toEqual(["type", "minLength", "maxLength"]); + expect(schema).toEqual({ type: "bytes", minLength: 1, maxLength: 2 }); }); it("checks instanceof Uint8Array and the length bounds", () => { - const validator = Compile(bytes({ min: 1, max: 2 })); - expect(validator.Check(new Uint8Array(1))).toBe(true); - expect(validator.Check(new Uint8Array(2))).toBe(true); - expect(validator.Check(new Uint8Array(0))).toBe(false); - expect(validator.Check(new Uint8Array(3))).toBe(false); - expect(validator.Check([1])).toBe(false); - expect(validator.Check("ab")).toBe(false); + const validator = compile(bytes({ min: 1, max: 2 })); + expect(validator.check(new Uint8Array(1))).toBe(true); + expect(validator.check(new Uint8Array(2))).toBe(true); + expect(validator.check(new Uint8Array(0))).toBe(false); + expect(validator.check(new Uint8Array(3))).toBe(false); + expect(validator.check([1])).toBe(false); + expect(validator.check("ab")).toBe(false); + }); + + it("checks inside a record (slices.md L1 carry-over)", () => { + const validator = compile(map(bytes({ max: 1 }))); + expect(validator.check({ a: new Uint8Array(1) })).toBe(true); + expect(validator.check({ a: new Uint8Array(2) })).toBe(false); }); }); diff --git a/packages/chord/test/next/contracts/snapshot.test.ts b/packages/chord/test/next/contracts/snapshot.test.ts index d90e5a284..e3d6d76b8 100644 --- a/packages/chord/test/next/contracts/snapshot.test.ts +++ b/packages/chord/test/next/contracts/snapshot.test.ts @@ -1,27 +1,35 @@ -import Type from "typebox"; -import { Compile } from "typebox/compile"; +// Tests of schema snapshots: the vocabulary, closed arguments, copies and their normal forms, bytes leaves. + +import { compile } from "@earendil-works/pi-schema"; import { describe, expect, it } from "vitest"; -import { arr, bytes, int, map, obj, opt, str } from "../../../src/next/contracts/schema.ts"; +import { arr, bytes, int, map, obj, opt, str, union } from "../../../src/next/contracts/schema.ts"; import { snapshotArguments, snapshotSchema } from "../../../src/next/contracts/snapshot.ts"; describe("vocabulary", () => { it("accepts primitives, arrays, tuples, objects, records, unions, intersections, $defs/$ref, constraints", () => { - const recursive = Type.Cyclic( - { Node: Type.Object({ value: Type.Number(), next: Type.Union([Type.Null(), Type.Ref("Node")]) }) }, - "Node", - ); + const recursive = { + $defs: { + Node: { + type: "object", + properties: { value: { type: "number" }, next: { anyOf: [{ type: "null" }, { $ref: "#/$defs/Node" }] } }, + required: ["value", "next"], + }, + }, + $ref: "#/$defs/Node", + }; const schemas = [ - Type.Tuple([Type.String(), Type.Integer()]), - Type.Intersect([Type.Object({ a: Type.String() }), Type.Object({ b: Type.String() })]), - Type.Enum(["a", "b"]), + { type: "array", prefixItems: [{ type: "string" }, { type: "integer" }], items: false }, + { allOf: [obj({ a: str() }), obj({ b: str() })] }, + { enum: ["a", "b"] }, recursive, str({ min: 1, pattern: "^x" }), - Type.String({ format: "email", default: "a@b.c", readOnly: true, title: "t", description: "d" }), - Type.Number({ multipleOf: 2, exclusiveMinimum: 0 }), - Type.Array(Type.String(), { uniqueItems: true }), - Type.Object({}, { minProperties: 1, maxProperties: 2, $comment: "c", deprecated: true, examples: [{}] }), + { type: "string", format: "email", default: "a@b.c", readOnly: true, title: "t", description: "d" }, + { type: "number", multipleOf: 2, exclusiveMinimum: 0 }, + { type: "array", items: { type: "string" }, uniqueItems: true }, + { type: "object", minProperties: 1, maxProperties: 2, $comment: "c", deprecated: true, examples: [{}] }, map(int()), bytes({ max: 4 }), + map(bytes()), ]; for (const schema of schemas) expect(() => snapshotSchema(schema, "s")).not.toThrow(); }); @@ -43,18 +51,15 @@ describe("vocabulary", () => { ["definitions", { definitions: {} }, 'unsupported keyword "definitions"'], ["oneOf", { oneOf: [{ type: "string" }] }, 'unsupported keyword "oneOf"'], ["unknown keyword", { type: "string", flavour: "x" }, 'unsupported keyword "flavour"'], - ["non-JSON type", Type.BigInt(), 'unsupported type "bigint"'], - ["undefined", Type.Undefined(), 'unsupported type "undefined"'], - ["function", Type.Function([], Type.Null()), "s: unsupported"], - ["Unsafe", Type.Unsafe({ type: "string" }), "unsafe types are not supported"], - ["refinement", Type.Refine(Type.String(), () => true), "refinements"], + ["non-JSON type", { type: "bigint" }, 'unsupported type "bigint"'], + ["undefined", { type: "undefined" }, 'unsupported type "undefined"'], + ["function", { type: "function", parameters: [], returnType: { type: "null" } }, "s: unsupported"], [ - "codec", - Type.Codec(Type.String()) - .Decode((v) => v) - .Encode((v) => v), - "codecs", + "non-enumerable key (a TypeBox refinement)", + Object.defineProperty({ type: "string" }, "~refine", { value: [{ check: () => true }] }), + "s.~refine: non-enumerable keys are not supported", ], + ["opt(...) outside an object", opt(str()), "s: expected a schema"], ["non-JSON data", { type: "number", default: Number.NaN }, "s.default: expected JSON data"], ["undefined data", { type: "number", maximum: undefined }, "s.maximum: expected JSON data"], ["foreign prototype", new (class Schema {})(), "s: expected a schema"], @@ -86,62 +91,62 @@ describe("vocabulary", () => { describe("closed arguments", () => { it("requires every object, nested ones included, to be closed", () => { expect(() => snapshotArguments(obj({ a: obj({ b: str() }) }), "m.args")).not.toThrow(); - expect(() => snapshotArguments(obj({ a: Type.Object({ b: str() }) }), "m.args")).toThrow( + const open = (properties: object) => ({ type: "object", properties }); + expect(() => snapshotArguments(obj({ a: open({ b: str() }) }), "m.args")).toThrow( "m.args.properties.a: argument objects must be closed (additionalProperties: false)", ); - expect(() => snapshotArguments(Type.Object({}), "m.args")).toThrow("m.args: argument objects must be closed"); - expect(() => snapshotArguments(obj({ list: arr(Type.Object({})) }), "m.args")).toThrow( + expect(() => snapshotArguments(open({}), "m.args")).toThrow("m.args: argument objects must be closed"); + expect(() => snapshotArguments(obj({ list: arr(open({})) }), "m.args")).toThrow( "m.args.properties.list.items: argument objects must be closed", ); expect(() => snapshotArguments({ type: "object" }, "m.args")).toThrow("argument objects must be closed"); }); - it("accepts a raw closed Type.Object and records", () => { - expect(() => snapshotArguments(Type.Object({ a: str() }, { additionalProperties: false }), "m")).not.toThrow(); + it("accepts a raw closed object schema and records", () => { + const closed = { type: "object", properties: { a: str() }, additionalProperties: false }; + expect(() => snapshotArguments(closed, "m")).not.toThrow(); expect(() => snapshotArguments(obj({ tags: map(str()) }), "m")).not.toThrow(); }); it("checks objects inside $defs, unions and tuples", () => { - const open = Type.Object({ a: str() }); - expect(() => snapshotArguments(Type.Union([open, str()]), "m")).toThrow("m.anyOf[0]: argument objects"); - expect(() => snapshotArguments(Type.Tuple([open]), "m")).toThrow("m.items[0]: argument objects"); + const open = { type: "object", properties: { a: str() } }; + expect(() => snapshotArguments(union([open, str()]), "m")).toThrow("m.anyOf[0]: argument objects"); + expect(() => snapshotArguments({ type: "array", items: [open] }, "m")).toThrow("m.items[0]: argument objects"); expect(() => snapshotArguments({ $defs: { A: open }, $ref: "A" }, "m")).toThrow("m.$defs.A: argument objects"); }); it("does not apply to state or results", () => { - expect(() => snapshotSchema(Type.Object({ a: str() }), "state")).not.toThrow(); + expect(() => snapshotSchema({ type: "object", properties: { a: str() } }, "state")).not.toThrow(); }); }); describe("snapshots", () => { it("are copies, independent of later changes to the app's schema objects", () => { - const app = obj({ text: str(), tags: arr(str()) }); + const app = obj({ text: str(), tags: arr(str()) }) as { properties: { text: object }; required: string[] }; const { schema } = snapshotSchema(app, "s"); (app.properties.text as { minLength?: number }).minLength = 5; - (app.required as string[]).push("x"); + app.required.push("x"); expect(schema).not.toBe(app); - expect(Compile(schema).Check({ text: "", tags: [] })).toBe(true); + expect(compile(schema).check({ text: "", tags: [] })).toBe(true); }); it("copy data values (defaults, consts)", () => { const fallback = { a: [1] }; - const { schema } = snapshotSchema(Type.Object({}, { default: fallback, additionalProperties: false }), "s"); + const { schema } = snapshotSchema({ type: "object", default: fallback, additionalProperties: false }, "s"); fallback.a.push(2); const copied = (schema as { default: { a: number[] } }).default; expect(copied).toEqual({ a: [1] }); }); - it("rebuild the bytes check from the marker; no app callback is copied", () => { - const app = bytes({ max: 2 }); - const appCheck = Object.getOwnPropertyDescriptor(app, "~refine")!.value; - const { schema } = snapshotSchema(app, "s"); - expect(Object.getOwnPropertyDescriptor(schema, "~refine")!.value).not.toBe(appCheck); - expect(Compile(schema).Check(new Uint8Array(2))).toBe(true); - expect(Compile(schema).Check(new Uint8Array(3))).toBe(false); - // A hand-written marker works too: the check comes from the marker, not from the app. - const marker = snapshotSchema({ type: "bytes", minLength: 1 }, "s").schema; - expect(Compile(marker).Check(new Uint8Array(0))).toBe(false); - expect(Compile(marker).Check(new Uint8Array(1))).toBe(true); + it("copy a bytes leaf as data: the compiled check comes from its keywords", () => { + const { schema } = snapshotSchema(bytes({ max: 2 }), "s"); + expect(schema).toEqual({ type: "bytes", maxLength: 2 }); + expect(compile(schema).check(new Uint8Array(2))).toBe(true); + expect(compile(schema).check(new Uint8Array(3))).toBe(false); + // A record of bytes (slices.md L1 carry-over: refused while bytes carried a TypeBox refinement). + const record = snapshotSchema(map(bytes({ min: 1 })), "s").schema; + expect(compile(record).check({ a: new Uint8Array(1) })).toBe(true); + expect(compile(record).check({ a: new Uint8Array(0) })).toBe(false); }); it("refuse a bytes marker with other keywords or bad bounds", () => { @@ -173,10 +178,9 @@ describe("snapshots", () => { }); it("normal form keeps union and tuple order", () => { - const ab = snapshotSchema(Type.Union([str(), int()]), "s").normal; - const ba = snapshotSchema(Type.Union([int(), str()]), "s").normal; + const ab = snapshotSchema(union([str(), int()]), "s").normal; + const ba = snapshotSchema(union([int(), str()]), "s").normal; expect(ab).not.toEqual(ba); - expect(snapshotSchema(opt(str()), "s").normal).toEqual({ type: "string" }); }); }); @@ -184,7 +188,7 @@ describe("bytes", () => { it("mark a snapshot with a bytes leaf anywhere inside", () => { expect(snapshotSchema(str(), "s").bytes).toBe(false); expect(snapshotSchema(obj({ a: arr(obj({ b: bytes() })) }), "s").bytes).toBe(true); - expect(snapshotSchema(Type.Union([str(), bytes()]), "s").bytes).toBe(true); - expect(snapshotSchema(Type.Object({ a: str() }, { default: { type: "bytes" } }), "s").bytes).toBe(false); + expect(snapshotSchema(union([str(), bytes()]), "s").bytes).toBe(true); + expect(snapshotSchema({ type: "object", default: { type: "bytes" } }, "s").bytes).toBe(false); }); }); diff --git a/packages/chord/test/next/data/args.test.ts b/packages/chord/test/next/data/args.test.ts index 896861fc0..bdd26d32b 100644 --- a/packages/chord/test/next/data/args.test.ts +++ b/packages/chord/test/next/data/args.test.ts @@ -147,10 +147,20 @@ describe("checkArgs", () => { const local1 = bindProvider(Info, {}); expect(checkArgs(local1, "save", { title: "t" }, 1, BACKGROUND_CONTEXT).args).toEqual({ title: "t" }); expect(refusal(() => checkArgs(local1, "save", { title: 1 }, 1, BACKGROUND_CONTEXT))).toMatchObject({ - message: "args/title: must be string", + message: "args/title: expected string, got integer", }); }); + it("treats `format` as an annotation: a contract with format email accepts any string", () => { + const Mail = local("mail", { + version: 1, + params: {}, + members: { send: method(obj({ to: { type: "string", format: "email" } }), nul()) }, + }); + const mail = bindProvider(Mail, {}); + expect(checkArgs(mail, "send", { to: "nope" }, 1, BACKGROUND_CONTEXT).args).toEqual({ to: "nope" }); + }); + it("refuses an unknown method, an unaccepted version and a refused method", () => { expect(refusal(() => checkArgs(provider, "missing", null, 3, BACKGROUND_CONTEXT))).toEqual({ code: "invalid", diff --git a/packages/chord/test/next/data/params.test.ts b/packages/chord/test/next/data/params.test.ts index 5848b0378..20777e981 100644 --- a/packages/chord/test/next/data/params.test.ts +++ b/packages/chord/test/next/data/params.test.ts @@ -59,10 +59,10 @@ describe("checkParams", () => { details: { path: "/extra" }, }); expect(refusal({ ...valid, page: "2" })).toMatchObject({ - message: "params/page: must be integer", + message: "params/page: expected integer, got string", details: { path: "/page" }, }); - expect(refusal("S1")).toMatchObject({ message: "params: must be object", details: { path: "/" } }); + expect(refusal("S1")).toMatchObject({ message: "params: expected object, got string", details: { path: "/" } }); }); it("enforces literal membership and constraints", () => { diff --git a/packages/chord/test/next/data/validator.test.ts b/packages/chord/test/next/data/validator.test.ts index b78b55c58..d1c09ec4c 100644 --- a/packages/chord/test/next/data/validator.test.ts +++ b/packages/chord/test/next/data/validator.test.ts @@ -1,4 +1,5 @@ -import { GetExternal, GetFunctions } from "typebox/schema"; +// Tests of compiled checks: pi-schema validators of Chord-owned schemas and the `invalid` error of a failed check. + import { describe, expect, it } from "vitest"; import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; import { arr, bytes, int, map, obj, str } from "../../../src/next/contracts/schema.ts"; @@ -31,17 +32,21 @@ describe("failed checks", () => { expect(failure(valid)).toBe("passed"); }); - it("report TypeBox's first error with its JSON Pointer path", () => { + it("report the first error with its JSON Pointer path", () => { expect(failure({ ...valid, text: 1 })).toEqual({ reason: "args", - message: "args/text: must be string", + message: "args/text: expected string, got integer", details: { path: "/text" }, }); expect(failure({ ...valid, items: [{ n: "x" }] })).toMatchObject({ details: { path: "/items/0/n" } }); - expect(failure(null)).toEqual({ reason: "args", message: "args: must be object", details: { path: "/" } }); + expect(failure(null)).toEqual({ + reason: "args", + message: "args: expected object, got null", + details: { path: "/" }, + }); }); - it("report a missing field at its own path (TypeBox reports it at the parent)", () => { + it("report a missing field at its own path", () => { const { text: _, ...missing } = valid; expect(failure(missing)).toMatchObject({ message: "args/text: missing", @@ -60,14 +65,22 @@ describe("failed checks", () => { }); it("check bytes by instanceof and length", () => { - expect(failure({ ...valid, blob: [1] })).toMatchObject({ - message: "args/blob: expected bytes of the declared length", + expect(failure({ ...valid, blob: [1] })).toMatchObject({ message: "args/blob: expected bytes, got array" }); + expect(failure({ ...valid, blob: new Uint8Array(3) })).toMatchObject({ + message: "args/blob: expected at most 2 bytes", + details: { path: "/blob" }, }); - expect(failure({ ...valid, blob: new Uint8Array(3) })).toMatchObject({ details: { path: "/blob" } }); }); - it("leave keys containing `/` or `~` unescaped (TypeBox's instance path, ambiguous)", () => { - expect(failure({ ...valid, tags: { "a/b": 1 } })).toMatchObject({ details: { path: "/tags/a/b" } }); + it("treat `format` as an annotation: no format is checked", () => { + const owner = snapshotArguments(obj({ mail: { type: "string", format: "email" } }), "m"); + const validator = validatorFor(owner, BACKGROUND_CONTEXT); + expect(validator.check({ mail: "a@b.c" })).toBe(true); + expect(validator.check({ mail: "nope" })).toBe(true); + }); + + it("escape `~` and `/` in keys (JSON Pointer), so a path names one place", () => { + expect(failure({ ...valid, tags: { "a/b": 1, "c~d": "x" } })).toMatchObject({ details: { path: "/tags/a~1b" } }); }); }); @@ -86,20 +99,11 @@ describe("validators", () => { ]); }); - it("leave nothing in TypeBox's build accumulators", () => { - validatorFor(snapshotArguments(obj({ blob: bytes(), n: int() }), "m"), BACKGROUND_CONTEXT); - expect(GetExternal().variables).toEqual([]); - expect(GetFunctions()).toEqual([]); - }); - - it("are collected with their schema owner; TypeBox retains no build state", async () => { + it("are collected with their schema owner, the check function too", async () => { const refs = (() => { - const owner = snapshotArguments(obj({ blob: bytes() }), "m"); + const owner = snapshotArguments(obj({ blob: bytes(), n: int() }), "m"); const validator = validatorFor(owner, BACKGROUND_CONTEXT); - const blob = (owner.schema as { properties: { blob: object } }).properties.blob; - // TypeBox's external variables would hold the bytes check if the build state were retained. - const [check] = Object.getOwnPropertyDescriptor(blob, "~refine")!.value as object[]; - return [new WeakRef(owner), new WeakRef(validator), new WeakRef(check!)]; + return [new WeakRef(owner), new WeakRef(validator), new WeakRef(validator.check)]; })(); await expectCollected(...refs); }); diff --git a/packages/chord/test/next/examples/37-plugin-imports/plugins/shapes/units.ts b/packages/chord/test/next/examples/37-plugin-imports/plugins/shapes/units.ts index e2fc0bfb5..c5a8a13b9 100644 --- a/packages/chord/test/next/examples/37-plugin-imports/plugins/shapes/units.ts +++ b/packages/chord/test/next/examples/37-plugin-imports/plugins/shapes/units.ts @@ -1,7 +1,7 @@ // A file of the shapes plugin that imports a package (not a host package): fine for the shapes plugin itself, but a // plugin importing this file would copy it, and a copied file may not import packages (the ruler plugin shows it). -import { Type } from "typebox"; +import { num } from "@earendil-works/pi-schema"; /** A length in millimetres. */ -export const Millimetres = Type.Number({ minimum: 0 }); +export const Millimetres = num({ minimum: 0 }); diff --git a/packages/chord/test/next/schema/declarations.ts b/packages/chord/test/next/schema/declarations.ts new file mode 100644 index 000000000..cf0a329c0 --- /dev/null +++ b/packages/chord/test/next/schema/declarations.ts @@ -0,0 +1,180 @@ +// The declarations of the schema switch's proofs (TypeBox to pi-schema): one of every schema helper and option in a +// contract, persisted documents, tables with defaults, an optional and a `json(...)` column, an evolved table and a +// persisted job. Their fingerprints were recorded with the TypeBox helpers; the stored data in `stored/` was written +// by that code (`write-stored.ts`). + +import { + arr, + bool, + bytes, + hostKind, + int, + keyed, + lit, + local, + map, + method, + nul, + num, + obj, + oneOf, + opt, + remote, + str, + union, + unknown, +} from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { defineJob } from "../../../src/next/jobs/index.ts"; +import { defineTable, evolveTable, json } from "../../../src/next/store/index.ts"; + +/** A schema with every helper and option a declaration can use. */ +export const everything = obj({ + name: str({ min: 1, max: 40, pattern: "^[a-z ]+$" }), + nick: str({ default: "x" }), + age: int({ min: 0, max: 150 }), + count: int({ default: 0 }), + score: num({ min: 0, max: 100 }), + ratio: num({ default: 0.5 }), + active: bool(), + verified: bool({ default: false }), + role: oneOf("admin", "user", "guest"), + level: oneOf(1, 2, 3), + kind: lit("person"), + one: lit(1), + yes: lit(true), + tags: arr(str(), { min: 0, max: 5 }), + points: arr(int()), + extra: map(int()), + note: opt(str()), + nothing: nul(), + choice: union([str(), int(), obj({ deep: arr(obj({ x: num() })) })]), + nested: obj({ inner: opt(obj({ v: str() })) }), +}); + +/** A local contract with state, keyed state and methods of every argument shape. */ +export const Everything = local("fixture.switch.everything", { + version: 1, + params: { id: str(), shard: int() }, + members: { + profile: everything, + notes: keyed(obj({ body: str() })), + plain: method(), + args: method(obj({ a: str(), b: opt(int()), any: unknown() })), + result: method(undefined, arr(str())), + both: method(obj({ when: int({ min: 0 }) }), union([lit("ok"), obj({ error: str() })])), + blob: method(obj({ data: bytes({ max: 1024 }) }), bytes()), + }, +}); + +const Server = hostKind("fixture.switch.server", { region: str() }); + +/** A remote contract with variants and a host address. */ +export const Account = remote("fixture.switch.account", { + version: 1, + params: { user: str({ min: 1 }) }, + hostAddress: () => Server({ region: "eu" }), + variants: { + owner: { state: { balance: num() }, methods: { pay: method(obj({ amount: num({ min: 0 }) }), bool()) } }, + guest: { state: { balance: num() } }, + }, +}); + +/** A persisted singleton document of `everything`. */ +export const Profile = defineState("fixture.switch.profile", { + schema: everything, + version: 1, + initial: () => ({ + name: "ada", + nick: "x", + age: 36, + count: 0, + score: 1, + ratio: 0.5, + active: true, + verified: false, + role: "admin" as const, + level: 1 as const, + kind: "person" as const, + one: 1 as const, + yes: true as const, + tags: [], + points: [], + extra: {}, + nothing: null, + choice: "a", + nested: {}, + }), + persist: true, + checkpointWhen: (_value, _ops, { deltasSinceBase }) => deltasSinceBase >= 2, +}); + +/** A persisted keyed document. */ +export const Notes = defineState("fixture.switch.notes", { + schema: obj({ body: str(), tags: opt(arr(str())) }), + version: 1, + initial: () => ({ body: "" }), + keyed: true, + persist: true, + checkpointWhen: () => false, +}); + +/** A table with defaults, an optional column and a `json(...)` column. */ +export const Rows = defineTable("fixture.switch.rows", { + version: 1, + columns: { + id: str(), + n: int({ default: 0 }), + flag: bool({ default: false }), + label: opt(str()), + data: json(obj({ a: int(), b: opt(arr(str())) }), { version: 1 }), + maybe: opt(json(map(num()), { version: 1 })), + }, + key: ["id"], + indexes: { by_n: ["n", "id"] }, +}); + +/** A table evolved over three versions: a rename, an added column with a default, a json column, a drop. */ +export const EditsV1 = defineTable("fixture.switch.edits", { + version: 1, + columns: { id: str(), file: str(), at: int() }, + key: ["id"], +}); + +export const EditsV2 = evolveTable(EditsV1, (t) => + t + .rename("file", "path") + .add("dir", str({ default: "" })) + .rows((row) => ({ dir: row.path.slice(0, row.path.lastIndexOf("/")) })), +); + +export const EditsV3 = evolveTable(EditsV2, (t) => t.drop("dir").index("by_path", ["path", "at"])); + +/** A persisted job with a state document. */ +export const ImportState = defineState("fixture.switch.import.state", { + schema: obj({ done: int() }), + version: 1, + initial: () => ({ done: 0 }), + persist: true, + checkpointWhen: () => false, +}); + +export const ImportJob = defineJob("fixture.switch.import", { + version: 1, + args: obj({ path: str() }), + result: obj({ count: int() }), + state: ImportState, + persist: { resume: "rerun" }, +}); + +/** The object owning the persisted documents and the tables (a local type with empty params, as tables need). */ +export const Owner = local("fixture.switch.owner", { + version: 1, + params: {}, + members: { + profile: everything, + write: method(obj({ id: str(), body: str() })), + read: method(str(), str()), + dump: method(obj({}), unknown()), + }, +}); diff --git a/packages/chord/test/next/schema/owner.ts b/packages/chord/test/next/schema/owner.ts new file mode 100644 index 000000000..88928df5e --- /dev/null +++ b/packages/chord/test/next/schema/owner.ts @@ -0,0 +1,85 @@ +// The object of the stored-data proof: it owns the persisted documents, the tables and the job of `declarations.ts`, +// writes them (`write-stored.ts`, with the TypeBox helpers) and reads them back (the test, after the switch). + +import type { Context } from "../../../src/context/index.ts"; +import { createHost, defineFacet, type Host } from "../../../src/next/host/index.ts"; +import type { TableDefinition } from "../../../src/next/store/index.ts"; +import { openStore, query, type Store, sqliteBackend } from "../../../src/next/store/index.ts"; +import { openNodeSqliteDatabase } from "../../../src/sqlite/node.ts"; +import { ImportJob, Notes, Owner, Profile, Rows } from "./declarations.ts"; + +/** What the job's `run` does in this process: wait until its context ends, or finish at once. */ +export type RunBehaviour = "wait" | "finish"; + +/** The finished runs' results, in order. */ +export const finished: unknown[] = []; + +/** The owner's facet over `edits` (the table version this process declares). */ +export function ownerFacet(edits: TableDefinition, behaviour: RunBehaviour) { + return defineFacet((host) => { + host.serve(Owner, (_, self) => { + const profile = self.state(Profile); + const notes = self.state(Notes); + const rows = self.table(Rows); + const editsTable = self.table(edits); + const imports = self.job(ImportJob, { + run: async ({ path }, ctx) => { + await ctx.state.change((draft) => { + draft.done += path.length; + }, ctx); + if (behaviour === "wait") { + await new Promise((resolve) => ctx.abortSignal?.addEventListener("abort", resolve)); + } + return { count: ctx.state.value.done }; + }, + finish: async (outcome) => { + finished.push(outcome.kind === "done" ? outcome.result : outcome.kind); + }, + }); + return { + profile, + write: async ({ id, body }, ctx) => { + await self.tx(async (tx) => { + (await tx.draft(notes, id)).body = body; + const draft = await tx.draft(profile); + draft.count += 1; + draft.tags.push(id); + draft.extra[id] = body.length; + draft.choice = { deep: [{ x: draft.count }] }; + draft.nested = { inner: { v: body } }; + tx.put(rows, { + id, + label: body, + data: { a: body.length, b: [id] }, + ...(id === "a" ? { maybe: { n: 1.5 } } : {}), + }); + tx.put(editsTable, { id, file: `/notes/${id}.md`, at: body.length } as never); + }, ctx); + await imports.start({ path: id }, ctx); + return null; + }, + read: async (id, ctx) => (await notes.get(id, ctx))?.body ?? "", + dump: async (_, ctx) => ({ + profile: profile.value, + notes: { a: (await notes.get("a", ctx)) ?? null, b: (await notes.get("b", ctx)) ?? null }, + rows: (await rows.page(query(Rows), 10, ctx)).items, + edits: (await editsTable.page(query(edits as TableDefinition), 10, ctx)).items, + runs: imports.running.value.length, + }), + }; + }); + }); +} + +/** A host serving the owner on the SQLite store at `path`. */ +export async function ownerHost( + path: string, + edits: TableDefinition, + behaviour: RunBehaviour, + context: Context, +): Promise<{ store: Store; host: Host }> { + const store = await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context); + const host = createHost({ store, onError: () => {}, closeTimeout: 100 }); + await host.load([{ name: "@fixture/owner", facet: ownerFacet(edits, behaviour) }], context); + return { store, host }; +} diff --git a/packages/chord/test/next/schema/proofs.ts b/packages/chord/test/next/schema/proofs.ts new file mode 100644 index 000000000..6f285111e --- /dev/null +++ b/packages/chord/test/next/schema/proofs.ts @@ -0,0 +1,113 @@ +// The values the schema switch must keep (`recorded.json`, made with the TypeBox helpers): the fingerprints and normal +// forms of `declarations.ts`, and what a host reads back from the stored data in `stored/`. + +import { createReadStream } from "node:fs"; +import { copyFile, mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import type { Context } from "../../../src/context/index.ts"; +import type { JsonValue } from "../../../src/json.ts"; +import type { Contract } from "../../../src/next/contracts/index.ts"; +import { contractData } from "../../../src/next/contracts/internal.ts"; +import { snapshotSchema } from "../../../src/next/contracts/snapshot.ts"; +import { decodeJsonl, importStore, sqliteBackend } from "../../../src/next/store/index.ts"; +import { openNodeSqliteDatabase } from "../../../src/sqlite/node.ts"; +import * as declared from "./declarations.ts"; +import { finished, ownerHost } from "./owner.ts"; + +/** Every fingerprint and normal form of the declarations. */ +export function fingerprints(): JsonValue { + const contract = (C: Contract) => { + const data = contractData(C); + return { + params: data.params.normal, + versions: data.versions.map((version) => ({ + fingerprint: version.fingerprint, + variants: Object.fromEntries( + [...version.variants].map(([name, variant]) => [ + name, + { + state: Object.fromEntries([...variant.state].map(([m, s]) => [m, s.normal])), + keyed: Object.fromEntries([...variant.keyed].map(([m, s]) => [m, s.normal])), + methods: Object.fromEntries( + [...variant.methods].map(([m, { args, result }]) => [ + m, + { args: args.normal, result: result.normal }, + ]), + ), + }, + ]), + ), + })), + }; + }; + const table = (T: { readonly fingerprint: string; readonly record: unknown }) => ({ + fingerprint: T.fingerprint, + record: T.record as unknown as JsonValue, + }); + return { + contracts: { + everything: contract(declared.Everything), + account: contract(declared.Account), + owner: contract(declared.Owner), + }, + documents: { + profile: { + fingerprint: declared.Profile.fingerprint, + normal: snapshotSchema(declared.everything, "p").normal, + }, + notes: declared.Notes.fingerprint, + importState: declared.ImportState.fingerprint, + }, + tables: { + rows: table(declared.Rows), + editsV1: table(declared.EditsV1), + editsV2: table(declared.EditsV2), + editsV3: table(declared.EditsV3), + }, + job: declared.ImportJob.fingerprint, + }; +} + +const stored = fileURLToPath(new URL("./stored/", import.meta.url)); + +/** + * What a host of the current code reads from the stored data: from a copy of the SQLite store, and from a store + * imported from the archive (both written at table version 1, read at version 3; the stopped job runs resume). + */ +export async function storedReads(context: Context): Promise { + const directory = await mkdtemp(join(tmpdir(), "chord-schema-switch-")); + try { + const fromStore = join(directory, "store.db"); + await copyFile(join(stored, "store.db"), fromStore); + const store = await read(fromStore, context); + const imported = await importStore( + sqliteBackend(await openNodeSqliteDatabase(join(directory, "imported.db"))), + decodeJsonl(createReadStream(join(stored, "archive.jsonl"))), + context, + ); + await imported.close(context); + const archive = await read(join(directory, "imported.db"), context); + return { store, archive }; + } finally { + await rm(directory, { recursive: true, force: true }); + } +} + +/** The owner's dump on the store at `path`, after its stopped runs finished. */ +async function read(path: string, context: Context): Promise { + finished.length = 0; + const { store, host } = await ownerHost(path, declared.EditsV3, "finish", context); + const owner = await host.get(declared.Owner, {}, context); + for (let i = 0; i < 100 && finished.length < 3; i++) await new Promise((resolve) => setTimeout(resolve, 10)); + const dump = await owner.dump({}, context); + const result = { + dump: dump as JsonValue, + finished: [...finished] as JsonValue, + note: await owner.read("b", context), + }; + await host.close(context); + await store.close(context); + return result; +} diff --git a/packages/chord/test/next/schema/recorded.json b/packages/chord/test/next/schema/recorded.json new file mode 100644 index 000000000..00c0a3588 --- /dev/null +++ b/packages/chord/test/next/schema/recorded.json @@ -0,0 +1,1143 @@ +{ + "fingerprints": { + "contracts": { + "everything": { + "params": [ + [ + "id", + { + "type": "string" + } + ], + [ + "shard", + { + "type": "integer" + } + ] + ], + "versions": [ + { + "fingerprint": "3861c5a457ef4aa3", + "variants": { + "members": { + "state": { + "profile": { + "additionalProperties": false, + "properties": { + "active": { + "type": "boolean" + }, + "age": { + "maximum": 150, + "minimum": 0, + "type": "integer" + }, + "choice": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "integer" + }, + { + "additionalProperties": false, + "properties": { + "deep": { + "items": { + "additionalProperties": false, + "properties": { + "x": { + "type": "number" + } + }, + "required": [ + "x" + ], + "type": "object" + }, + "type": "array" + } + }, + "required": [ + "deep" + ], + "type": "object" + } + ] + }, + "count": { + "default": 0, + "type": "integer" + }, + "extra": { + "patternProperties": { + "^.*$": { + "type": "integer" + } + }, + "type": "object" + }, + "kind": { + "const": "person", + "type": "string" + }, + "level": { + "anyOf": [ + { + "const": 1, + "type": "number" + }, + { + "const": 2, + "type": "number" + }, + { + "const": 3, + "type": "number" + } + ] + }, + "name": { + "maxLength": 40, + "minLength": 1, + "pattern": "^[a-z ]+$", + "type": "string" + }, + "nested": { + "additionalProperties": false, + "properties": { + "inner": { + "additionalProperties": false, + "properties": { + "v": { + "type": "string" + } + }, + "required": [ + "v" + ], + "type": "object" + } + }, + "type": "object" + }, + "nick": { + "default": "x", + "type": "string" + }, + "note": { + "type": "string" + }, + "nothing": { + "type": "null" + }, + "one": { + "const": 1, + "type": "number" + }, + "points": { + "items": { + "type": "integer" + }, + "type": "array" + }, + "ratio": { + "default": 0.5, + "type": "number" + }, + "role": { + "anyOf": [ + { + "const": "admin", + "type": "string" + }, + { + "const": "user", + "type": "string" + }, + { + "const": "guest", + "type": "string" + } + ] + }, + "score": { + "maximum": 100, + "minimum": 0, + "type": "number" + }, + "tags": { + "items": { + "type": "string" + }, + "maxItems": 5, + "minItems": 0, + "type": "array" + }, + "verified": { + "default": false, + "type": "boolean" + }, + "yes": { + "const": true, + "type": "boolean" + } + }, + "required": [ + "active", + "age", + "choice", + "count", + "extra", + "kind", + "level", + "name", + "nested", + "nick", + "nothing", + "one", + "points", + "ratio", + "role", + "score", + "tags", + "verified", + "yes" + ], + "type": "object" + } + }, + "keyed": { + "notes": { + "additionalProperties": false, + "properties": { + "body": { + "type": "string" + } + }, + "required": [ + "body" + ], + "type": "object" + } + }, + "methods": { + "plain": { + "args": { + "type": "null" + }, + "result": { + "type": "null" + } + }, + "args": { + "args": { + "additionalProperties": false, + "properties": { + "a": { + "type": "string" + }, + "any": {}, + "b": { + "type": "integer" + } + }, + "required": [ + "a", + "any" + ], + "type": "object" + }, + "result": { + "type": "null" + } + }, + "result": { + "args": { + "type": "null" + }, + "result": { + "items": { + "type": "string" + }, + "type": "array" + } + }, + "both": { + "args": { + "additionalProperties": false, + "properties": { + "when": { + "minimum": 0, + "type": "integer" + } + }, + "required": [ + "when" + ], + "type": "object" + }, + "result": { + "anyOf": [ + { + "const": "ok", + "type": "string" + }, + { + "additionalProperties": false, + "properties": { + "error": { + "type": "string" + } + }, + "required": [ + "error" + ], + "type": "object" + } + ] + } + }, + "blob": { + "args": { + "additionalProperties": false, + "properties": { + "data": { + "maxLength": 1024, + "type": "bytes" + } + }, + "required": [ + "data" + ], + "type": "object" + }, + "result": { + "type": "bytes" + } + } + } + } + } + } + ] + }, + "account": { + "params": [ + [ + "user", + { + "minLength": 1, + "type": "string" + } + ] + ], + "versions": [ + { + "fingerprint": "821be6a79fbe350a", + "variants": { + "owner": { + "state": { + "balance": { + "type": "number" + } + }, + "keyed": {}, + "methods": { + "pay": { + "args": { + "additionalProperties": false, + "properties": { + "amount": { + "minimum": 0, + "type": "number" + } + }, + "required": [ + "amount" + ], + "type": "object" + }, + "result": { + "type": "boolean" + } + } + } + }, + "guest": { + "state": { + "balance": { + "type": "number" + } + }, + "keyed": {}, + "methods": {} + } + } + } + ] + }, + "owner": { + "params": [], + "versions": [ + { + "fingerprint": "b01eec2d4e2311b0", + "variants": { + "members": { + "state": { + "profile": { + "additionalProperties": false, + "properties": { + "active": { + "type": "boolean" + }, + "age": { + "maximum": 150, + "minimum": 0, + "type": "integer" + }, + "choice": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "integer" + }, + { + "additionalProperties": false, + "properties": { + "deep": { + "items": { + "additionalProperties": false, + "properties": { + "x": { + "type": "number" + } + }, + "required": [ + "x" + ], + "type": "object" + }, + "type": "array" + } + }, + "required": [ + "deep" + ], + "type": "object" + } + ] + }, + "count": { + "default": 0, + "type": "integer" + }, + "extra": { + "patternProperties": { + "^.*$": { + "type": "integer" + } + }, + "type": "object" + }, + "kind": { + "const": "person", + "type": "string" + }, + "level": { + "anyOf": [ + { + "const": 1, + "type": "number" + }, + { + "const": 2, + "type": "number" + }, + { + "const": 3, + "type": "number" + } + ] + }, + "name": { + "maxLength": 40, + "minLength": 1, + "pattern": "^[a-z ]+$", + "type": "string" + }, + "nested": { + "additionalProperties": false, + "properties": { + "inner": { + "additionalProperties": false, + "properties": { + "v": { + "type": "string" + } + }, + "required": [ + "v" + ], + "type": "object" + } + }, + "type": "object" + }, + "nick": { + "default": "x", + "type": "string" + }, + "note": { + "type": "string" + }, + "nothing": { + "type": "null" + }, + "one": { + "const": 1, + "type": "number" + }, + "points": { + "items": { + "type": "integer" + }, + "type": "array" + }, + "ratio": { + "default": 0.5, + "type": "number" + }, + "role": { + "anyOf": [ + { + "const": "admin", + "type": "string" + }, + { + "const": "user", + "type": "string" + }, + { + "const": "guest", + "type": "string" + } + ] + }, + "score": { + "maximum": 100, + "minimum": 0, + "type": "number" + }, + "tags": { + "items": { + "type": "string" + }, + "maxItems": 5, + "minItems": 0, + "type": "array" + }, + "verified": { + "default": false, + "type": "boolean" + }, + "yes": { + "const": true, + "type": "boolean" + } + }, + "required": [ + "active", + "age", + "choice", + "count", + "extra", + "kind", + "level", + "name", + "nested", + "nick", + "nothing", + "one", + "points", + "ratio", + "role", + "score", + "tags", + "verified", + "yes" + ], + "type": "object" + } + }, + "keyed": {}, + "methods": { + "write": { + "args": { + "additionalProperties": false, + "properties": { + "body": { + "type": "string" + }, + "id": { + "type": "string" + } + }, + "required": [ + "body", + "id" + ], + "type": "object" + }, + "result": { + "type": "null" + } + }, + "read": { + "args": { + "type": "string" + }, + "result": { + "type": "string" + } + }, + "dump": { + "args": { + "additionalProperties": false, + "properties": {}, + "type": "object" + }, + "result": {} + } + } + } + } + } + ] + } + }, + "documents": { + "profile": { + "fingerprint": "f345152aff1bb993", + "normal": { + "additionalProperties": false, + "properties": { + "active": { + "type": "boolean" + }, + "age": { + "maximum": 150, + "minimum": 0, + "type": "integer" + }, + "choice": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "integer" + }, + { + "additionalProperties": false, + "properties": { + "deep": { + "items": { + "additionalProperties": false, + "properties": { + "x": { + "type": "number" + } + }, + "required": [ + "x" + ], + "type": "object" + }, + "type": "array" + } + }, + "required": [ + "deep" + ], + "type": "object" + } + ] + }, + "count": { + "default": 0, + "type": "integer" + }, + "extra": { + "patternProperties": { + "^.*$": { + "type": "integer" + } + }, + "type": "object" + }, + "kind": { + "const": "person", + "type": "string" + }, + "level": { + "anyOf": [ + { + "const": 1, + "type": "number" + }, + { + "const": 2, + "type": "number" + }, + { + "const": 3, + "type": "number" + } + ] + }, + "name": { + "maxLength": 40, + "minLength": 1, + "pattern": "^[a-z ]+$", + "type": "string" + }, + "nested": { + "additionalProperties": false, + "properties": { + "inner": { + "additionalProperties": false, + "properties": { + "v": { + "type": "string" + } + }, + "required": [ + "v" + ], + "type": "object" + } + }, + "type": "object" + }, + "nick": { + "default": "x", + "type": "string" + }, + "note": { + "type": "string" + }, + "nothing": { + "type": "null" + }, + "one": { + "const": 1, + "type": "number" + }, + "points": { + "items": { + "type": "integer" + }, + "type": "array" + }, + "ratio": { + "default": 0.5, + "type": "number" + }, + "role": { + "anyOf": [ + { + "const": "admin", + "type": "string" + }, + { + "const": "user", + "type": "string" + }, + { + "const": "guest", + "type": "string" + } + ] + }, + "score": { + "maximum": 100, + "minimum": 0, + "type": "number" + }, + "tags": { + "items": { + "type": "string" + }, + "maxItems": 5, + "minItems": 0, + "type": "array" + }, + "verified": { + "default": false, + "type": "boolean" + }, + "yes": { + "const": true, + "type": "boolean" + } + }, + "required": [ + "active", + "age", + "choice", + "count", + "extra", + "kind", + "level", + "name", + "nested", + "nick", + "nothing", + "one", + "points", + "ratio", + "role", + "score", + "tags", + "verified", + "yes" + ], + "type": "object" + } + }, + "notes": "bd44e8ea54b8f283", + "importState": "53d035820123516d" + }, + "tables": { + "rows": { + "fingerprint": "a8a7a46b1d1b5058", + "record": { + "id": { + "type": "str" + }, + "n": { + "type": "int", + "default": 0 + }, + "flag": { + "type": "bool", + "default": false + }, + "label": { + "type": "str", + "optional": true + }, + "data": { + "type": "json", + "version": 1, + "fingerprint": "595acb546ede10d5" + }, + "maybe": { + "type": "json", + "optional": true, + "version": 1, + "fingerprint": "27fc51c99dad9a88" + } + } + }, + "editsV1": { + "fingerprint": "b6a64f82180e9b7f", + "record": { + "id": { + "type": "str" + }, + "file": { + "type": "str" + }, + "at": { + "type": "int" + } + } + }, + "editsV2": { + "fingerprint": "2fbd0130077d5eed", + "record": { + "id": { + "type": "str" + }, + "path": { + "type": "str" + }, + "at": { + "type": "int" + }, + "dir": { + "type": "str", + "default": "" + } + } + }, + "editsV3": { + "fingerprint": "303055ac505946c3", + "record": { + "id": { + "type": "str" + }, + "path": { + "type": "str" + }, + "at": { + "type": "int" + } + } + } + }, + "job": "c280624818bc83ab" + }, + "stored": { + "store": { + "dump": { + "profile": { + "name": "ada", + "nick": "x", + "age": 36, + "count": 3, + "score": 1, + "ratio": 0.5, + "active": true, + "verified": false, + "role": "admin", + "level": 1, + "kind": "person", + "one": 1, + "yes": true, + "tags": [ + "a", + "b", + "c" + ], + "points": [], + "extra": { + "a": 10, + "b": 6, + "c": 9 + }, + "nothing": null, + "choice": { + "deep": [ + { + "x": 3 + } + ] + }, + "nested": { + "inner": { + "v": "third one" + } + } + }, + "notes": { + "a": { + "body": "first note" + }, + "b": { + "body": "second" + } + }, + "rows": [ + { + "id": "a", + "n": 0, + "flag": false, + "label": "first note", + "data": { + "a": 10, + "b": [ + "a" + ] + }, + "maybe": { + "n": 1.5 + } + }, + { + "id": "b", + "n": 0, + "flag": false, + "label": "second", + "data": { + "a": 6, + "b": [ + "b" + ] + } + }, + { + "id": "c", + "n": 0, + "flag": false, + "label": "third one", + "data": { + "a": 9, + "b": [ + "c" + ] + } + } + ], + "edits": [ + { + "id": "a", + "path": "/notes/a.md", + "at": 10 + }, + { + "id": "b", + "path": "/notes/b.md", + "at": 6 + }, + { + "id": "c", + "path": "/notes/c.md", + "at": 9 + } + ], + "runs": 0 + }, + "finished": [ + { + "count": 2 + }, + { + "count": 2 + }, + { + "count": 2 + } + ], + "note": "second" + }, + "archive": { + "dump": { + "profile": { + "name": "ada", + "nick": "x", + "age": 36, + "count": 3, + "score": 1, + "ratio": 0.5, + "active": true, + "verified": false, + "role": "admin", + "level": 1, + "kind": "person", + "one": 1, + "yes": true, + "tags": [ + "a", + "b", + "c" + ], + "points": [], + "extra": { + "a": 10, + "b": 6, + "c": 9 + }, + "nothing": null, + "choice": { + "deep": [ + { + "x": 3 + } + ] + }, + "nested": { + "inner": { + "v": "third one" + } + } + }, + "notes": { + "a": { + "body": "first note" + }, + "b": { + "body": "second" + } + }, + "rows": [ + { + "id": "a", + "n": 0, + "flag": false, + "label": "first note", + "data": { + "a": 10, + "b": [ + "a" + ] + }, + "maybe": { + "n": 1.5 + } + }, + { + "id": "b", + "n": 0, + "flag": false, + "label": "second", + "data": { + "a": 6, + "b": [ + "b" + ] + } + }, + { + "id": "c", + "n": 0, + "flag": false, + "label": "third one", + "data": { + "a": 9, + "b": [ + "c" + ] + } + } + ], + "edits": [ + { + "id": "a", + "path": "/notes/a.md", + "at": 10 + }, + { + "id": "b", + "path": "/notes/b.md", + "at": 6 + }, + { + "id": "c", + "path": "/notes/c.md", + "at": 9 + } + ], + "runs": 0 + }, + "finished": [ + { + "count": 2 + }, + { + "count": 2 + }, + { + "count": 2 + } + ], + "note": "second" + } + } +} diff --git a/packages/chord/test/next/schema/schema-switch.test.ts b/packages/chord/test/next/schema/schema-switch.test.ts new file mode 100644 index 000000000..830593422 --- /dev/null +++ b/packages/chord/test/next/schema/schema-switch.test.ts @@ -0,0 +1,22 @@ +// Tests of the schema switch from TypeBox to pi-schema (decision "A schema package beneath pi-ai and Chord"): the same +// declarations give byte-identical fingerprints and normal forms, and data stored by the code before the switch loads +// after it (`recorded.json` and `stored/` were made with the TypeBox helpers, see `write-stored.ts`). + +import { readFileSync } from "node:fs"; +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { fingerprints, storedReads } from "./proofs.ts"; + +const recorded = JSON.parse(readFileSync(new URL("./recorded.json", import.meta.url), "utf8")); + +describe("fingerprints", () => { + it("keeps every fingerprint and normal form of contracts, documents, tables, json columns and jobs", () => { + expect(fingerprints()).toEqual(recorded.fingerprints); + }); +}); + +describe("stored data", () => { + it("reads a store and an archive written before the switch: documents, tables, a migrated table, stopped runs", async () => { + expect(await storedReads(context)).toEqual(recorded.stored); + }); +}); diff --git a/packages/chord/test/next/schema/stored/archive.jsonl b/packages/chord/test/next/schema/stored/archive.jsonl new file mode 100644 index 000000000..7fcbd5972 --- /dev/null +++ b/packages/chord/test/next/schema/stored/archive.jsonl @@ -0,0 +1,36 @@ +{"kind":"manifest","format":1,"schema":2,"lastSeq":13,"nextId":12} +{"kind":"table","table":"fixture.switch.owner:fixture.switch.edits","version":1,"state":"ready","columns":{"id":{"type":"str"},"file":{"type":"str"},"at":{"type":"int"}},"key":["id"],"fingerprint":"b6a64f82180e9b7f"} +{"kind":"put","table":"fixture.switch.owner:fixture.switch.edits","row":{"id":"a","file":"/notes/a.md","at":10},"version":1,"columns":{}} +{"kind":"put","table":"fixture.switch.owner:fixture.switch.edits","row":{"id":"b","file":"/notes/b.md","at":6},"version":1,"columns":{}} +{"kind":"put","table":"fixture.switch.owner:fixture.switch.edits","row":{"id":"c","file":"/notes/c.md","at":9},"version":1,"columns":{}} +{"kind":"table","table":"fixture.switch.owner:fixture.switch.rows","version":1,"state":"ready","columns":{"id":{"type":"str"},"n":{"type":"int","default":0},"flag":{"type":"bool","default":false},"label":{"type":"str","optional":true},"data":{"type":"json","version":1,"fingerprint":"595acb546ede10d5"},"maybe":{"type":"json","optional":true,"version":1,"fingerprint":"27fc51c99dad9a88"}},"key":["id"],"fingerprint":"a8a7a46b1d1b5058"} +{"kind":"put","table":"fixture.switch.owner:fixture.switch.rows","row":{"id":"a","n":0,"flag":false,"label":"first note","data":{"a":10,"b":["a"]},"maybe":{"n":1.5}},"version":1,"columns":{"data":1,"maybe":1}} +{"kind":"put","table":"fixture.switch.owner:fixture.switch.rows","row":{"id":"b","n":0,"flag":false,"label":"second","data":{"a":6,"b":["b"]}},"version":1,"columns":{"data":1}} +{"kind":"put","table":"fixture.switch.owner:fixture.switch.rows","row":{"id":"c","n":0,"flag":false,"label":"third one","data":{"a":9,"b":["c"]}},"version":1,"columns":{"data":1}} +{"kind":"index","table":"fixture.switch.owner:fixture.switch.rows","name":"by_n","columns":["n","id"]} +{"kind":"document.create","record":{"id":2,"owner":"fixture.switch.owner","namespace":"chord","kind":"fixture.switch.notes","key":"a","history":"none","createdAt":2,"revision":1,"version":1,"fingerprint":"bd44e8ea54b8f283"}} +{"kind":"document.revision","id":2,"seq":2,"content":{"kind":"base","version":1,"value":{"body":"first note"}}} +{"kind":"document.create","record":{"id":3,"owner":"fixture.switch.owner","namespace":"chord","kind":"fixture.switch.profile","history":"none","createdAt":2,"revision":3,"version":1,"fingerprint":"f345152aff1bb993"}} +{"kind":"document.revision","id":3,"seq":2,"content":{"kind":"base","version":1,"value":{"name":"ada","nick":"x","age":36,"count":1,"score":1,"ratio":0.5,"active":true,"verified":false,"role":"admin","level":1,"kind":"person","one":1,"yes":true,"tags":["a"],"points":[],"extra":{"a":10},"nothing":null,"choice":{"deep":[{"x":1}]},"nested":{"inner":{"v":"first note"}}}}} +{"kind":"document.revision","id":3,"seq":5,"content":{"kind":"delta","version":1,"ops":[["s",["count"],2],["s",["choice"],{"deep":[{"x":2}]}],["s",["nested"],{"inner":{"v":"second"}}],["p",["tags"],1,0,["b"]],["s",["extra","b"],6]]}} +{"kind":"document.revision","id":3,"seq":9,"content":{"kind":"delta","version":1,"ops":[["s",["count"],3],["s",["choice"],{"deep":[{"x":3}]}],["s",["nested"],{"inner":{"v":"third one"}}],["p",["tags"],2,0,["c"]],["s",["extra","c"],9]]}} +{"kind":"document.create","record":{"id":4,"owner":"fixture.switch.owner","namespace":"chord.jobs","kind":"fixture.switch.import","key":"01a1943d-176c-71f3-a32c-b9cbc67247a3","history":"none","createdAt":3,"revision":2,"version":1,"fingerprint":"chord.job-run.1"}} +{"kind":"document.revision","id":4,"seq":4,"content":{"kind":"base","version":1,"value":{"version":1,"fingerprint":"c280624818bc83ab","args":{"path":"a"},"phase":"running","restarts":0,"finishFailures":0}}} +{"kind":"document.create","record":{"id":5,"owner":"fixture.switch.owner","namespace":"chord.job-states","kind":"fixture.switch.import","key":"01a1943d-176c-71f3-a32c-b9cbc67247a3","history":"none","createdAt":3,"revision":2,"version":1,"fingerprint":"53d035820123516d"}} +{"kind":"document.revision","id":5,"seq":3,"content":{"kind":"base","version":1,"value":{"done":0}}} +{"kind":"document.revision","id":5,"seq":6,"content":{"kind":"delta","version":1,"ops":[["s",["done"],1]]}} +{"kind":"document.create","record":{"id":6,"owner":"fixture.switch.owner","namespace":"chord","kind":"fixture.switch.notes","key":"b","history":"none","createdAt":5,"revision":1,"version":1,"fingerprint":"bd44e8ea54b8f283"}} +{"kind":"document.revision","id":6,"seq":5,"content":{"kind":"base","version":1,"value":{"body":"second"}}} +{"kind":"document.create","record":{"id":7,"owner":"fixture.switch.owner","namespace":"chord.jobs","kind":"fixture.switch.import","key":"01a1943d-176e-71f3-a32c-b9cc7f5426c6","history":"none","createdAt":7,"revision":2,"version":1,"fingerprint":"chord.job-run.1"}} +{"kind":"document.revision","id":7,"seq":8,"content":{"kind":"base","version":1,"value":{"version":1,"fingerprint":"c280624818bc83ab","args":{"path":"b"},"phase":"running","restarts":0,"finishFailures":0}}} +{"kind":"document.create","record":{"id":8,"owner":"fixture.switch.owner","namespace":"chord.job-states","kind":"fixture.switch.import","key":"01a1943d-176e-71f3-a32c-b9cc7f5426c6","history":"none","createdAt":7,"revision":2,"version":1,"fingerprint":"53d035820123516d"}} +{"kind":"document.revision","id":8,"seq":7,"content":{"kind":"base","version":1,"value":{"done":0}}} +{"kind":"document.revision","id":8,"seq":10,"content":{"kind":"delta","version":1,"ops":[["s",["done"],1]]}} +{"kind":"document.create","record":{"id":9,"owner":"fixture.switch.owner","namespace":"chord","kind":"fixture.switch.notes","key":"c","history":"none","createdAt":9,"revision":1,"version":1,"fingerprint":"bd44e8ea54b8f283"}} +{"kind":"document.revision","id":9,"seq":9,"content":{"kind":"base","version":1,"value":{"body":"third one"}}} +{"kind":"document.create","record":{"id":10,"owner":"fixture.switch.owner","namespace":"chord.jobs","kind":"fixture.switch.import","key":"01a1943d-176f-71f3-a32c-b9cf6bb98dc7","history":"none","createdAt":11,"revision":2,"version":1,"fingerprint":"chord.job-run.1"}} +{"kind":"document.revision","id":10,"seq":12,"content":{"kind":"base","version":1,"value":{"version":1,"fingerprint":"c280624818bc83ab","args":{"path":"c"},"phase":"running","restarts":0,"finishFailures":0}}} +{"kind":"document.create","record":{"id":11,"owner":"fixture.switch.owner","namespace":"chord.job-states","kind":"fixture.switch.import","key":"01a1943d-176f-71f3-a32c-b9cf6bb98dc7","history":"none","createdAt":11,"revision":2,"version":1,"fingerprint":"53d035820123516d"}} +{"kind":"document.revision","id":11,"seq":11,"content":{"kind":"base","version":1,"value":{"done":0}}} +{"kind":"document.revision","id":11,"seq":13,"content":{"kind":"delta","version":1,"ops":[["s",["done"],1]]}} +{"kind":"end","writes":34} diff --git a/packages/chord/test/next/schema/stored/store.db b/packages/chord/test/next/schema/stored/store.db new file mode 100644 index 0000000000000000000000000000000000000000..f9b1e70836f843f0967cdb76f7890de36f441247 GIT binary patch literal 57344 zcmeI5U2Ggz6@X`dXJ`MWR1GVHO2$*%*ktYO|Bq3iX|ips1gCWz(kfcq`P+Eg_0BRg zPHb8GV5bTaFBL+9!V3tgZ-@t85EA04C{=}2UO+$*NJt=r1QJqJiSoj^cV@@4>s@E< zEVOmb${x?&-*dim?%kd7xp%HUf4OOCLPIy!B}>RNk1&2e^OPVk4C9C2B>eVX2{;M% z-oRg<=X}(upPB#qqoiUY&|>&F|_RH84+~2v+MZOt%jD0oq{opSk3tvd!*b~_O za)M2*uKMn5Taw(;8qG~>!_el<+f7SZo7Zo*HDh5QQB#|iIgmEGVDkKO?c7RDSULCf z<(eRlT~8Ed#HK0=E43F^grzI+d+qY&G-;rv?N5~~QCM7BsXbd;b_)yVFV@aqf@VnO z)3ufBwc3&}SIuO~*=jXcER+kGYBejITe=_&CeA`fpS`ue$+Ym?^5U1yEx#ySs=a6n zI6Es`U0Gf{zmgWNFRom?a&1LezH)u>!p;kEHg&V^+u3TiRc&(|y$$`gxgc+?w#O}; zaM&&`U8ub<9$U8$QMj@+eidSkwg^stWlcBKRYSYgG@E+cT$LqLGY1l5 zecm{bY1be=H;^IBsJgPTuC=YzrkWN^?W?oG^^3LTn(zwrESwcik-koKIa{1*k2t-G z=Blaa9nEgoO^gl^$IY~B?1|%M2s3mlOABpjT{An9qNRnVI=k~&giU=Oa9kPMzoe>$ zW;#IJwzvZ&aO#XH1A&niUeUJFf?{ZrrKziuHM?_`WmD7BzTLBQB-#Y@4NKNHz2}j> z5#`MhW}+OtfVw+Z+-u)EpiXPTO#(|gA;HS%zAvRS6f3!bcY*Ibt&DB=(*5Ggm zJM^wCEk1v(2BMOWjA8W->yWb)fjwjm&1xFdHHF2i^b`Be#UQot-Df=(b_E^iKQzR` zZdu2|9$)j&vvJa)JB+hCn1^NVx4YdM47Op#*P5oK8v|KAPw8b+XKCwg?G!jm3x=y{ zkuyWP4da-6(Q9vP+c!0%V>Acd)a!yf z;Q*a%cCL6#Nl%@L&pr=@vQ*pV>C$mSkG1qoErE_KJ9)Ly(sg5B-9Gy~<6~3VtZ%pM!9sjhgHO9PZQa4+P4N%Gh1&I%bai}cJ*uFZY?2Mqk+3ke_rB!C2v01`j~NB{{S0VIF~kN^@m zb_9aH0ORx9*Z=P`{QJjFK~x9{AOR$R1dsp{Kmter2_OL^fCP}hLrxo|}nB$A9r zfrLL1V-m6CBM&(gR38Z-0VIF~kN^@u0!RP}Ac2R6z-}P8dZ#+h$~n3!b6cdL-{Qg{ z)>)=SCk|U&$fiXIskbz!4{MCnEio-BdTV37ZHf!qWa%Fqt*s8pH!VZ#rtQ_3`%+=S zrbSrJAZ@gOL52__YXtXb$hzJF_1q#2sby+V%w7(-SB~&a zi+YDFijZ33f@N%wVsxG0Ua6N&9ad%b8L*#hv09WAxmYM^s+P^DMPf61>EOOb`ujSl z(OkKq6thaTs!D2As#NZ0Ua2CLr9w&0s#&?1DOUQ-E0P7T#4(5>#5ua%L`-0!(Ki_W zM~QDF{*t`QbMa5df0p=S^o`i-56{p?O_2Z+Kmter2_OL^fCN5L1h%3HrZ`o;b^^AX z@^Pn!U7gV7$n!7j^5E7d&Gn9MSedMptrqg?T((?l%$2i^{G61}DRXjFX_REST2YlU zSwbEQxDCx28}0dQ{9DXsanA0V5~0oxQcu3bJlRuzH1!=OGI>S^%DIoxa~w#JD)PiL zM+<7G1Af!iVV44hsLsi1p`cYXsaTLJja((a93U-L6Sl99cDR8SJBChnYa&hh1C4y4 zm@Vd{Mk5QpUCnO=CuvEg-;%OYD(4DiDNp(i3|f-EF`gwulegr7a_+Ze_~@3DiRz(2 zOTrX-wP(pg_=eY#gNA$#;`NW}ZW8s4K_SWi|D*AD8UF3$cM{*@znOd|@$2}zd?C4z zcR`T)W7TgQ?Zc>i_H2%Bz$@mXq*;sb; zFvA=qfCP{L5{1K|3`1SnqT+6-vQl7c9xJpk#(d6*tYbXQXBeo?%0hy0Q0| zzfuvR@dp<{@rnTdiEl#?^H?G%wg|Bai_p>9Diz^m*yl@e*;s-;mf*$>(G=4+L>2x4 z2(i}LdVOaK43Xk?^8v1}graXiaCSYNuS0QK2QEUJ3aPaXA<9|@;&QjeO>rULt#{or zZ9op}NttFFo)1K#n{R&XIhno?pT_oL*xRdfPw)n`-K`g}aM`Xm59!A->+i&E0yZH^rx(LTz1b`WT_BFEV( z$~uJD_i@|#jBTj7K&Y>Yo}qFsz7dAXjbkWNgGW-RL#&jgR+5KU3DVLfaHQUdW+#AN z_711Rq7MaZhCTE!zs*zs2|5|xMGgHi@i5mo_KUFnjX3rTh5O*|+3#^`zcb-bpZ$6` zcc4Zos)cE@4PZP=zdDfcrOVWR_WIZ?sw77i=ach^`D3g$s)huR01`j~ zNB{{S0VIF~kif^6K!c^R{-cAY$<`1Lx<5lK_7Y2@{SWTu=;8jcXqlz){-Zk>9`Mgi zRR57N`H>0fKe7Y%Nczj6@I>_=6`3ChO-TQd9Rw?2e~A4b+x|}Qzdvv!aFX9oum6*G z8U9cF&-qKq-}0^GzhM`^*ZJw>k3YWC0ZNVpkN^@u0!RP}AOR$R1dsp{Kms340;fa8 zxdSeGJvVP+vFT88>ON)7XjBLlQwNmdxbeG;M8@foW$)8vI6Qurq0l&827~wMG7z{= z0q6Rkf0N-q;NRol=Kse3lm9dS8~zXculP418DB^M2_OL^fCP{L5G5j`sC`fQSG8zr+6!{@@D< zAOR$R1dsp{Kmter2_OL^fCP{L68N7Hn2z$1$jLJtJVa7)xAvsl0#?(a44Va#tvkCS zDfKB@O13vbq(xwJLs|s(jqGMfk*O?qlH;VkZT_4M2Ay)S4Pkd%Y{Qm=^TjTS{}1s0 zV)*~?@53K_Apsndj_$xf0C ONntxla*`xSivAa5Xn)TD literal 0 HcmV?d00001 diff --git a/packages/chord/test/next/schema/write-stored.ts b/packages/chord/test/next/schema/write-stored.ts new file mode 100644 index 000000000..e40248d32 --- /dev/null +++ b/packages/chord/test/next/schema/write-stored.ts @@ -0,0 +1,40 @@ +// Writes the stored data of the schema switch's proof (`stored/`): a SQLite store and its export archive, made by the +// code before the switch (TypeBox helpers). Kept as it was written; rerun only to make a new fixture on purpose. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/schema/write-stored.ts + +import { createWriteStream } from "node:fs"; +import { copyFile, mkdir, mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { pipeline } from "node:stream/promises"; +import { fileURLToPath } from "node:url"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { encodeJsonl } from "../../../src/next/store/index.ts"; +import { EditsV1, Owner } from "./declarations.ts"; +import { ownerHost } from "./owner.ts"; + +const stored = fileURLToPath(new URL("./stored/", import.meta.url)); +const directory = await mkdtemp(join(tmpdir(), "chord-schema-switch-")); +const path = join(directory, "store.db"); +try { + const { store, host } = await ownerHost(path, EditsV1, "wait", context); + const owner = await host.get(Owner, {}, context); + for (const [id, body] of [ + ["a", "first note"], + ["b", "second"], + ["c", "third one"], + ] as const) { + await owner.write({ id, body }, context); + } + await new Promise((resolve) => setTimeout(resolve, 200)); + await mkdir(stored, { recursive: true }); + await pipeline(encodeJsonl(store.export(context)), createWriteStream(join(stored, "archive.jsonl"))); + // A clean close: the waiting runs are stored `stopped`, resumed by the next host. + await host.close(context); + await store.close(context); + await copyFile(path, join(stored, "store.db")); + console.log("written to", stored); +} finally { + await rm(directory, { recursive: true, force: true }); +} diff --git a/packages/chord/test/placement.test.ts b/packages/chord/test/placement.test.ts index 869bfa655..3e3a7bd8e 100644 --- a/packages/chord/test/placement.test.ts +++ b/packages/chord/test/placement.test.ts @@ -1,5 +1,5 @@ import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { num, obj, opt, str } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { bind, @@ -22,17 +22,17 @@ const tick = () => new Promise((resolve) => setImmediate(resolve)); /** Lives on `session:` + its input's `session`. */ const Note = remote("t.note", { version: 1, - input: Type.Object({ session: Type.String(), id: Type.String() }), + input: obj({ session: str(), id: str() }, { open: true }), host: hostFamily("session:", "session"), - variants: { main: { methods: { where: method(undefined, Type.String()) } } }, + variants: { main: { methods: { where: method(undefined, str()) } } }, }); /** Lives on `server`. */ const Catalog = remote("t.catalog", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), - variants: { main: { methods: { where: method(undefined, Type.String()) } } }, + variants: { main: { methods: { where: method(undefined, str()) } } }, }); /** A server with two session hosts as children; every host serves what it may and reports its name. */ @@ -126,9 +126,9 @@ describe("placement", () => { it("the host comes from the normalized input: unknown fields and defaults do not change it", async () => { const Doc = remote("t.doc-default", { version: 1, - input: Type.Object({ session: Type.String({ default: "S1" }) }), + input: obj({ session: str({ default: "S1" }) }, { open: true }), host: hostFamily("session:", "session"), - variants: { main: { methods: { where: method(undefined, Type.String()) } } }, + variants: { main: { methods: { where: method(undefined, str()) } } }, }); const { server, s1, close } = cluster(); s1.serve(Doc, () => ({ variant: "main", where: () => "session:S1" })); @@ -143,7 +143,7 @@ describe("placement", () => { const at = (host: string) => remote("t.moved", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed(host), variants: { main: { methods: { ping: method() } } }, }); @@ -155,7 +155,7 @@ describe("placement", () => { api.serve( remote("t.moved", { version, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed(place), variants: { main: { methods: { ping: method() } } }, }), @@ -170,7 +170,7 @@ describe("placement", () => { // A new version that `accepts` version 1 but lives elsewhere: no adapter makes version 1 consumers bind. const moved = remote("t.moved", { version: 2, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("session:S1"), variants: { main: { methods: { ping: method() } } }, }); @@ -182,7 +182,7 @@ describe("placement", () => { }); it("types: the family's field is a required string of the input, and remote contracts need a host", () => { - const input = Type.Object({ session: Type.String(), count: Type.Number(), note: Type.Optional(Type.String()) }); + const input = obj({ session: str(), count: num(), note: opt(str()) }, { open: true }); const variants = { main: {} }; remote("t.ok", { version: 1, input, host: hostFamily("session:", "session"), variants }); // @ts-expect-error: no such field diff --git a/packages/chord/test/principals.test.ts b/packages/chord/test/principals.test.ts index 4aef22cb3..aab011dab 100644 --- a/packages/chord/test/principals.test.ts +++ b/packages/chord/test/principals.test.ts @@ -1,5 +1,5 @@ import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { obj, str } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { type Ctx, @@ -26,8 +26,8 @@ const principalKey = (principal: Principal): string => const Who = local("t.who", { version: 1, - input: Type.Object({}), - members: { who: method(undefined, Type.String()) }, + input: obj({}, { open: true }), + members: { who: method(undefined, str()) }, }); describe("principals", () => { @@ -86,9 +86,9 @@ describe("principals", () => { it("a child's parent acts as the principal the child gave `connect`; relayed callers keep theirs", async () => { const Echo = remote("t.echo", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("worker"), - variants: { main: { methods: { who: method(undefined, Type.String()) } } }, + variants: { main: { methods: { who: method(undefined, str()) } } }, }); const server = createHost({ name: "server", self: { kind: "host" }, principalKey }); const [down, up] = memoryLinkPair(); diff --git a/packages/chord/test/provider-cost.test.ts b/packages/chord/test/provider-cost.test.ts index 470c74b20..50a8bf533 100644 --- a/packages/chord/test/provider-cost.test.ts +++ b/packages/chord/test/provider-cost.test.ts @@ -1,7 +1,7 @@ // Entry 8: what a change costs the provider. The diff is computed once per change; each feed (one per connection and // member) encodes it with its own stateful encoder and its link serializes it. import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { arr, obj, str } from "@earendil-works/pi-schema"; import { describe, it, vi } from "vitest"; import type * as delta from "../src/delta/index.ts"; import { type Ctx, createHost, hostNamed, memoryLinkPair, method, remote } from "../src/index.ts"; @@ -30,16 +30,16 @@ const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); const List = remote("t.e8-cost", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), - variants: { main: { state: { items: Type.Array(Type.String()) }, methods: { add: method(Type.String()) } } }, + variants: { main: { state: { items: arr(str()) }, methods: { add: method(str()) } } }, }); describe("provider cost", () => { it("one change to K clients: K encodes (one per feed), one frame per link, the same ops on every link", async () => { const server = createHost({ self: { kind: "host" }, name: "server", onError: () => {} }); server.serve(List, (_input, _principal, self) => { - const items = self.state("items", Type.Array(Type.String()), []); + const items = self.state("items", arr(str()), []); return { variant: "main", items, diff --git a/packages/chord/test/readiness.test.ts b/packages/chord/test/readiness.test.ts index a781a6e34..b34b2540c 100644 --- a/packages/chord/test/readiness.test.ts +++ b/packages/chord/test/readiness.test.ts @@ -1,5 +1,5 @@ import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { obj } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { createHost, @@ -15,8 +15,8 @@ import { const tick = () => new Promise((resolve) => setImmediate(resolve)); const unexpectedError = (error: unknown) => assert.fail(String(error)); -const Eager = local("ready.eager", { version: 1, input: Type.Object({}) }); -const Lazy = local("ready.lazy", { version: 1, input: Type.Object({}) }); +const Eager = local("ready.eager", { version: 1, input: obj({}, { open: true }) }); +const Lazy = local("ready.lazy", { version: 1, input: obj({}, { open: true }) }); // Startup must not report ready while an eager object has failed. describe("host: eager readiness", () => { @@ -162,7 +162,7 @@ describe("host: eager readiness", () => { it("ready includes direct host serves, but not later serves", async () => { const host = createHost({ self: { kind: "host" }, onError: () => {} }); - const First = local("ready.first", { version: 1, input: Type.Object({}) }); + const First = local("ready.first", { version: 1, input: obj({}, { open: true }) }); let created = 0; try { host.serve( @@ -201,7 +201,7 @@ describe("host: eager readiness", () => { }); const Status = remote("ready.status", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), variants: { default: {} }, }); diff --git a/packages/chord/test/reload.test.ts b/packages/chord/test/reload.test.ts index f2931f0c9..b9998b316 100644 --- a/packages/chord/test/reload.test.ts +++ b/packages/chord/test/reload.test.ts @@ -1,5 +1,5 @@ import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { num, obj, opt, type Schema, str, unknown } from "@earendil-works/pi-schema"; import { describe, it, onTestFinished, vi } from "vitest"; import { type Ctx, @@ -30,15 +30,15 @@ function latch() { const Greeter = local("t.greeter", { version: 1, - input: Type.Object({ id: Type.Optional(Type.String({ default: "a" })) }), - members: { greet: method(Type.String(), Type.String()), count: Type.Unknown() }, + input: obj({ id: opt(str({ default: "a" })) }, { open: true }), + members: { greet: method(str(), str()), count: unknown() }, }); -const CountV1 = Type.Object({ n: Type.Number() }); +const CountV1 = obj({ n: num() }, { open: true }); interface GreeterSpec { readonly greeting: string; - readonly schema?: typeof CountV1 | ReturnType; + readonly schema?: typeof CountV1 | Schema; readonly initial?: object; readonly bump?: (draft: Record) => void; readonly options?: ServeOptions; @@ -129,7 +129,7 @@ describe("reload", () => { const writing = greeterFacet({ greeting: "hi", extra: (self) => { - const probe = self.state("probe", Type.Object({}), {}); + const probe = self.state("probe", obj({}, { open: true }), {}); probe.replace(captured!, {}); }, }); @@ -164,7 +164,7 @@ describe("reload", () => { await host.hold(Greeter, {}).ready; const changed = greeterFacet({ greeting: "hi", - schema: Type.Object({ total: Type.Number() }), + schema: obj({ total: num() }, { open: true }), initial: { total: 0 }, }); await assert.rejects( @@ -191,7 +191,7 @@ describe("reload", () => { const froms: number[] = []; const v2 = greeterFacet({ greeting: "hi", - schema: Type.Object({ total: Type.Number() }), + schema: obj({ total: num() }, { open: true }), initial: { total: 0 }, bump: (draft) => { draft.total = (draft.total ?? 0) + 1; @@ -218,16 +218,16 @@ describe("reload", () => { const host = quiet(); const Pair = local("t.pair", { version: 1, - input: Type.Object({}), - members: { a: Type.Unknown(), b: Type.Unknown(), bump: method() }, + input: obj({}, { open: true }), + members: { a: unknown(), b: unknown(), bump: method() }, }); const migrated: string[] = []; const facet = (version: number): Facet => ({ name: "pair", setup: (api) => api.serve(Pair, (_input, _principal, self) => { - const a = self.state("a", Type.Number(), 1); - const b = self.state("b", Type.Number(), 1, { + const a = self.state("a", num(), 1); + const b = self.state("b", num(), 1, { version, migrate: (old, from) => { migrated.push(`b ${from}->${version}`); @@ -294,7 +294,7 @@ describe("reload", () => { await new Promise((resolve) => ctx.signal.addEventListener("abort", () => resolve())); log.push(`stop ${greeting}`); }); - const count = self.state("count", Type.Number(), 0, { version }); + const count = self.state("count", num(), 0, { version }); return { count, greet: (name: string) => `${greeting} ${name}` }; }), }); @@ -340,7 +340,7 @@ describe("reload", () => { open: () => void log.push(`open cache ${rev}`), close: () => void log.push(`close cache ${rev}`), }); - const count = self.state("count", Type.Number(), 0); + const count = self.state("count", num(), 0); return { count, greet: (name: string) => `${lock.value} ${name}` }; }), }); @@ -523,21 +523,21 @@ describe("reload", () => { it("checks contracts at reload: forgotten bumps, live consumers, accepts adapters", async () => { const DocV1 = remote("t.rdoc", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("host"), - variants: { main: { methods: { read: method(undefined, Type.Object({ text: Type.String() })) } } }, + variants: { main: { methods: { read: method(undefined, obj({ text: str() }, { open: true })) } } }, }); const DocBroken = remote("t.rdoc", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("host"), - variants: { main: { methods: { read: method(undefined, Type.Object({ text: Type.Number() })) } } }, + variants: { main: { methods: { read: method(undefined, obj({ text: num() }, { open: true })) } } }, }); const DocV2 = remote("t.rdoc", { version: 2, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("host"), - variants: { main: { methods: { fetch: method(undefined, Type.Object({ body: Type.String() })) } } }, + variants: { main: { methods: { fetch: method(undefined, obj({ body: str() }, { open: true })) } } }, }); const host = quiet(); host.install({ @@ -590,14 +590,14 @@ describe("reload", () => { const log: string[] = []; const Sum = local("t.sum", { version: 1, - input: Type.Object({}), - members: { sum: Type.Object({ v: Type.Number() }) }, + input: obj({}, { open: true }), + members: { sum: obj({ v: num() }, { open: true }) }, }); const facet = (factor: number): Facet => ({ name: "sum", setup: (api) => api.serve(Sum, (_input, _principal, self) => { - const sum = self.state("sum", Type.Object({ v: Type.Number() }), { v: 0 }); + const sum = self.state("sum", obj({ v: num() }, { open: true }), { v: 0 }); self.on( "add", events, @@ -632,7 +632,7 @@ describe("reload", () => { it("keeps registry tables per code version", async () => { const host = quiet(); const Routes = defineRegistry("t.reload-routes"); - const Router = local("t.router", { version: 1, input: Type.Object({}) }); + const Router = local("t.router", { version: 1, input: obj({}, { open: true }) }); const facet = (value: string): Facet => ({ name: "router", setup: (api) => @@ -672,8 +672,8 @@ describe("reload", () => { await ref.ready; const Clock = local("t.clock", { version: 1, - input: Type.Object({}), - members: { now: method(undefined, Type.Number()) }, + input: obj({}, { open: true }), + members: { now: method(undefined, num()) }, }); const clockFacet: Facet = { name: "clock", setup: (api) => api.serve(Clock, () => ({ now: () => 42 })) }; const consumer = (fail: boolean): Facet => ({ diff --git a/packages/chord/test/requests.test.ts b/packages/chord/test/requests.test.ts index fef1f2a30..5e3cc5abb 100644 --- a/packages/chord/test/requests.test.ts +++ b/packages/chord/test/requests.test.ts @@ -1,6 +1,6 @@ // Regression tests for resolved application requests (#1-#10), numbered as in their test names. import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { arr, num, obj, str, unknown } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { applyImmutable } from "../src/delta/index.ts"; import { @@ -36,19 +36,19 @@ function latch() { return { promise, open }; } -const View = Type.Object({ text: Type.String() }); +const View = obj({ text: str() }, { open: true }); describe("application requests", () => { it("#1 a held object's state member can be returned as this object's state member", async () => { const Conv = remote("t.conv", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), - variants: { main: { state: { view: View }, methods: { set: method(Type.String()) } } }, + variants: { main: { state: { view: View }, methods: { set: method(str()) } } }, }); const Chat = remote("t.chat", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), variants: { viewer: { state: { view: View } } }, }); @@ -95,11 +95,11 @@ describe("application requests", () => { it("#2 facets reach their own host through FacetApi.root and FacetApi.registry", async () => { const Config = defineRegistry("t.config"); - const Settings = local("t.settings", { version: 1, input: Type.Object({}) }); + const Settings = local("t.settings", { version: 1, input: obj({}, { open: true }) }); const Reader = local("t.reader", { version: 1, - input: Type.Object({}), - members: { read: method(undefined, Type.String()) }, + input: obj({}, { open: true }), + members: { read: method(undefined, str()) }, }); const captured: FacetApi[] = []; const facet = (value: string): Facet => ({ @@ -135,8 +135,8 @@ describe("application requests", () => { it("#3 a listener keeps its subscription across reloads when resource and select are unchanged", async () => { const Watcher = local("t.watcher", { version: 1, - input: Type.Object({}), - members: { seen: Type.Array(Type.String()) }, + input: obj({}, { open: true }), + members: { seen: arr(str()) }, }); // A source that accepts one subscriber at a time (like a Durable watch). let subscribers = 0; @@ -155,7 +155,7 @@ describe("application requests", () => { name: "watcher", setup: (api) => api.serve(Watcher, (_input, _principal, self) => { - const seen = self.state("seen", Type.Array(Type.String()), []); + const seen = self.state("seen", arr(str()), []); const watchResource = self.resource("watch", { version: 1, open: () => single }); // A new adapter object per call, and a new arrow per factory run. self.on( @@ -181,7 +181,7 @@ describe("application requests", () => { }); it("#3 a changed select resubscribes", async () => { - const Watcher = local("t.watcher2", { version: 1, input: Type.Object({}) }); + const Watcher = local("t.watcher2", { version: 1, input: obj({}, { open: true }) }); let subscribes = 0; const source = { subscribe: () => { @@ -210,10 +210,10 @@ describe("application requests", () => { it("#4 replace diffs a new revision that shares structure; unrelated values replace", async () => { const Mirror = local("t.mirror", { version: 1, - input: Type.Object({}), - members: { doc: Type.Unknown(), set: method(Type.Unknown()) }, + input: obj({}, { open: true }), + members: { doc: unknown(), set: method(unknown()) }, }); - const Doc = Type.Object({ entries: Type.Array(Type.String()), status: Type.String() }); + const Doc = obj({ entries: arr(str()), status: str() }, { open: true }); let lastOptions: { diff?: boolean } | undefined; const host = createHost({ self: { kind: "host" }, onError: () => {} }); host.serve(Mirror, (_input, _principal, self) => { @@ -256,9 +256,9 @@ describe("application requests", () => { it("#5 roots of a child that connected behind the server are admitted in the server's code version", async () => { const Echo = remote("t.echo", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), - variants: { main: { methods: { codeVersion: method(undefined, Type.Number()) } } }, + variants: { main: { methods: { codeVersion: method(undefined, num()) } } }, }); const server = createHost({ self: { kind: "host" }, name: "server", onError: () => {} }); const echoFacet: Facet = { @@ -293,7 +293,7 @@ describe("application requests", () => { it("#6 self.expose: an opaque member backed by a resource, readable once live", async () => { const Store = local("t.store", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), members: { db: opaque<{ readonly queries: string[] }>("t.db@1") }, }); const gate = latch(); @@ -315,7 +315,7 @@ describe("application requests", () => { await ref.ready; assert.equal(ref.db, native); - const Other = local("t.other", { version: 1, input: Type.Object({}) }); + const Other = local("t.other", { version: 1, input: obj({}, { open: true }) }); host.serve(Other, (_input, _principal, self) => { self.expose({ key: "x", value: 1 }); return {}; @@ -326,9 +326,9 @@ describe("application requests", () => { it("#7 a cluster reload reports what each child prepared", async () => { const Greeter = remote("t.greeter", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("session:S1"), - variants: { main: { methods: { greet: method(Type.String(), Type.String()) } } }, + variants: { main: { methods: { greet: method(str(), str()) } } }, }); const greeter = (text: string): Facet => ({ name: "greeter", @@ -362,13 +362,13 @@ describe("application requests", () => { it("#8 a held remote reference keeps its state without listeners; a later subscriber sees the current value", async () => { const Counter = remote("t.count8", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), - variants: { main: { state: { n: Type.Number() }, methods: { set: method(Type.Number()) } } }, + variants: { main: { state: { n: num() }, methods: { set: method(num()) } } }, }); const server = createHost({ self: { kind: "host" }, name: "server", onError: () => {} }); server.serve(Counter, (_input, _principal, self) => { - const n = self.state("n", Type.Number(), 0); + const n = self.state("n", num(), 0); return { variant: "main", n, @@ -404,7 +404,7 @@ describe("application requests", () => { it("#9 references report status changes: live, loading while a link reconnects, gone, released", async () => { const Thing = remote("t.thing9", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), variants: { main: {} }, }); @@ -449,9 +449,9 @@ describe("application requests", () => { const log: string[] = []; const Remote = remote("t.remote10", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), - variants: { main: { methods: { note: method(Type.String()) } } }, + variants: { main: { methods: { note: method(str()) } } }, }); const server = createHost({ self: { kind: "host" }, name: "server", onError: () => {} }); server.serve(Remote, () => ({ @@ -462,10 +462,10 @@ describe("application requests", () => { })); const Proc = local("t.proc", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), members: { tracker: opaque>("t.tracker@1") }, }); - const Chat = local("t.chat10", { version: 1, input: Type.Object({}) }); + const Chat = local("t.chat10", { version: 1, input: obj({}, { open: true }) }); const [down, up] = memoryLinkPair(); server.accept(down, { principal: { worker: "w" }, kind: "host", name: "w" }); const worker = createHost({ diff --git a/packages/chord/test/review.test.ts b/packages/chord/test/review.test.ts index 554a24097..418d592ea 100644 --- a/packages/chord/test/review.test.ts +++ b/packages/chord/test/review.test.ts @@ -1,7 +1,6 @@ // Regression tests for Sol's phase 1a review (findings #1-#11) and the requested race tests. import assert from "node:assert/strict"; -import { Type } from "typebox"; -import { Value } from "typebox/value"; +import { arr, bool, compile, num, obj, opt, str, unknown } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { type Ctx, @@ -18,6 +17,7 @@ import { } from "../src/index.ts"; import { fits } from "../src/schema.ts"; import { CountingLink } from "./fixtures/counting.ts"; +import { typedLit } from "./fixtures/schemas.ts"; const tick = () => new Promise((resolve) => setImmediate(resolve)); const ticks = async (n = 10) => { @@ -51,8 +51,8 @@ function emitter(options: { failSubscribe?: boolean } = {}) { const Item = local("t.item", { version: 1, - input: Type.Object({ id: Type.String() }), - members: { version: method(undefined, Type.String()) }, + input: obj({ id: str() }, { open: true }), + members: { version: method(undefined, str()) }, }); describe("review: reload commit", () => { @@ -82,15 +82,15 @@ describe("review: reload commit", () => { const host = createHost({ self: { kind: "host" }, dev: true, onError: () => {} }); const V1 = remote("t.counted", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("host"), - variants: { main: { state: { count: Type.Object({ n: Type.Number() }) } } }, + variants: { main: { state: { count: obj({ n: num() }, { open: true }) } } }, }); const V2 = remote("t.counted", { version: 2, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("host"), - variants: { main: { state: { count: Type.Object({ total: Type.Number() }) } } }, + variants: { main: { state: { count: obj({ total: num() }, { open: true }) } } }, }); host.install({ name: "counted", @@ -99,7 +99,7 @@ describe("review: reload commit", () => { V1, (_input, _principal, self) => ({ variant: "main", - count: self.state("count", Type.Object({ n: Type.Number() }), { n: 3 }), + count: self.state("count", obj({ n: num() }, { open: true }), { n: 3 }), }), { linger: 60_000 }, ), @@ -117,7 +117,7 @@ describe("review: reload commit", () => { variant: "main", count: self.state( "count", - Type.Object({ total: Type.Number() }), + obj({ total: num() }, { open: true }), { total: 0 }, { version: 2, migrate: (old) => ({ total: (old as { n: number }).n }) }, ), @@ -160,8 +160,8 @@ describe("review: reload commit", () => { const host = createHost({ self: { kind: "host" }, onError: () => {} }); const Other = local("t.other", { version: 1, - input: Type.Object({}), - members: { version: method(undefined, Type.String()) }, + input: obj({}, { open: true }), + members: { version: method(undefined, str()) }, }); const facet = (version: string): Facet => ({ name: "pair", @@ -184,8 +184,8 @@ describe("review: reload commit", () => { describe("review: leases, invocations, activation", () => { const Store = local("t.store", { version: 1, - input: Type.Object({}), - members: { count: Type.Unknown() }, + input: obj({}, { open: true }), + members: { count: unknown() }, }); it("#4 a running listener holds a lease: the gate waits for it before migrating", async () => { @@ -197,7 +197,7 @@ describe("review: leases, invocations, activation", () => { name: "store", setup: (api) => api.serve(Store, (_input, _principal, self) => { - const count = self.state("count", Type.Object({ n: Type.Number() }), { n: 1 }); + const count = self.state("count", obj({ n: num() }, { open: true }), { n: 1 }); self.on("slow", events, async () => { await go.promise; seen.push(count.value); @@ -218,7 +218,7 @@ describe("review: leases, invocations, activation", () => { api.serve(Store, (_input, _principal, self) => ({ count: self.state( "count", - Type.Object({ total: Type.Number() }), + obj({ total: num() }, { open: true }), { total: 0 }, { version: 2, migrate: (old) => ({ total: (old as { n: number }).n }) }, ), @@ -238,8 +238,8 @@ describe("review: leases, invocations, activation", () => { it("#5 disposal waits for running methods before closing resources (released and forced)", async () => { const Worker = local("t.busy", { version: 1, - input: Type.Object({ id: Type.String() }), - members: { work: method(undefined, Type.Boolean()) }, + input: obj({ id: str() }, { open: true }), + members: { work: method(undefined, bool()) }, }); const log: string[] = []; let go = latch(); @@ -291,7 +291,9 @@ describe("review: leases, invocations, activation", () => { const host = createHost({ self: { kind: "host" }, onError: (error) => errors.push(error) }); const Routes = defineRegistry("t.review-routes"); const log: string[] = []; - const types = ["t.eager-a", "t.eager-b"].map((name) => local(name, { version: 1, input: Type.Object({}) })); + const types = ["t.eager-a", "t.eager-b"].map((name) => + local(name, { version: 1, input: obj({}, { open: true }) }), + ); for (const type of types) { host.serve( type, @@ -321,7 +323,7 @@ describe("review: leases, invocations, activation", () => { let runs = 0; host.serve(Store, (_input, _principal, self) => { runs++; - const count = self.state("count", Type.Number(), 0); + const count = self.state("count", num(), 0); self.resource("r", { version: 1, open: () => void log.push("open"), @@ -345,8 +347,8 @@ describe("review: leases, invocations, activation", () => { describe("review: prepare isolation and contracts", () => { const Db = local("t.db", { version: 1, - input: Type.Object({}), - members: { read: method(undefined, Type.Unknown()) }, + input: obj({}, { open: true }), + members: { read: method(undefined, unknown()) }, }); it("#7 a prepare rerun cannot touch retained resources, even open ones", async () => { @@ -372,7 +374,7 @@ describe("review: prepare isolation and contracts", () => { it("#7 objects of types a reload adds start listeners and tasks only at commit", async () => { const host = createHost({ self: { kind: "host" }, onError: () => {} }); const log: string[] = []; - const Fresh = local("t.fresh", { version: 1, input: Type.Object({}) }); + const Fresh = local("t.fresh", { version: 1, input: obj({}, { open: true }) }); host.install({ name: "item", setup: (api) => api.serve(Item, () => ({ version: () => "v1" })) }); await Promise.all(["a", "b"].map((id) => host.hold(Item, { id }).ready)); const plugins = (fail: boolean): Facet[] => [ @@ -408,20 +410,20 @@ describe("review: prepare isolation and contracts", () => { const host = createHost({ self: { kind: "host" }, onError: () => {} }); const P1 = remote("t.provider", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("host"), - variants: { main: { methods: { get: method(undefined, Type.String()) } } }, + variants: { main: { methods: { get: method(undefined, str()) } } }, }); const P2 = remote("t.provider", { version: 2, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("host"), - variants: { main: { methods: { fetch: method(undefined, Type.String()) } } }, + variants: { main: { methods: { fetch: method(undefined, str()) } } }, }); const Consumer = local("t.consumer", { version: 1, - input: Type.Object({}), - members: { ask: method(undefined, Type.String()) }, + input: obj({}, { open: true }), + members: { ask: method(undefined, str()) }, }); host.install({ name: "both", @@ -493,7 +495,7 @@ describe("review: prepare isolation and contracts", () => { it("#10 retirement closes dropped resources before releasing dropped holds", async () => { const host = createHost({ self: { kind: "host" }, onError: () => {} }); const log: string[] = []; - const Dep = local("t.dep", { version: 1, input: Type.Object({}) }); + const Dep = local("t.dep", { version: 1, input: obj({}, { open: true }) }); const facet = (withDecls: boolean): Facet => ({ name: "item", setup: (api) => { @@ -535,48 +537,49 @@ describe("review: prepare isolation and contracts", () => { assert.deepEqual(log, ["own closed", "dep closed"]); }); - it("#11 fits has no false positives for unmodelled keywords; counterexamples checked with Value.Check", () => { + it("#11 fits has no false positives for unmodelled keywords; counterexamples checked with compile", () => { const cases = [ - [Type.String(), Type.String({ not: { const: "x" } }), "x"], - [Type.Array(Type.Number()), Type.Array(Type.Number(), { contains: Type.Literal(1) }), [2]], + [str(), { ...str(), not: { const: "x" } }, "x"], + [arr(num()), { ...arr(num()), contains: typedLit(1) }, [2]], [ - Type.Object({ a: Type.String() }), - Type.Object({ a: Type.String() }, { additionalProperties: Type.Number() }), + obj({ a: str() }, { open: true }), + { ...obj({ a: str() }, { open: true }), additionalProperties: num() }, { a: "s", b: "t" }, ], [ - Type.Object({ x: Type.String() }), - Type.Object({}, { patternProperties: { "^x$": Type.Number() } }), + obj({ x: str() }, { open: true }), + { ...obj({}, { open: true }), patternProperties: { "^x$": num() } }, { x: "s" }, ], - [Type.Number(), Type.Intersect([Type.Number(), Type.Number({ minimum: 1 })]), 0], + [num(), { allOf: [num(), num({ minimum: 1 })] }, 0], [ - Type.Object({ a: Type.String() }, { additionalProperties: Type.Number() }), - Type.Object({ a: Type.String(), b: Type.Optional(Type.String()) }), + { ...obj({ a: str() }, { open: true }), additionalProperties: num() }, + obj({ a: str(), b: opt(str()) }, { open: true }), { a: "", b: 1 }, ], ] as const; for (const [sub, sup, value] of cases) { - assert.ok(Value.Check(sub, value) && !Value.Check(sup, value), `bad counterexample ${JSON.stringify(value)}`); + assert.ok( + compile(sub).check(value) && !compile(sup).check(value), + `bad counterexample ${JSON.stringify(value)}`, + ); assert.equal(fits(sub, sup), false, `fits(${JSON.stringify(sub)}, ${JSON.stringify(sup)})`); } // Additive changes still fit. - assert.ok( - fits(Type.Object({ a: Type.String() }), Type.Object({ a: Type.String(), b: Type.Optional(Type.Number()) })), - ); - assert.ok(fits(Type.Object({ a: Type.String(), extra: Type.Number() }), Type.Object({ a: Type.String() }))); + assert.ok(fits(obj({ a: str() }, { open: true }), obj({ a: str(), b: opt(num()) }, { open: true }))); + assert.ok(fits(obj({ a: str(), extra: num() }, { open: true }), obj({ a: str() }, { open: true }))); }); it("#11 arguments are stripped to what the consumer knows, then to what the provider knows", async () => { const host = createHost({ self: { kind: "host" }, onError: () => {} }); const Strict = remote("t.strict", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("host"), variants: { main: { methods: { - echo: method(Type.Object({ a: Type.String() }, { additionalProperties: false }), Type.Unknown()), + echo: method(obj({ a: str() }), unknown()), }, }, }, @@ -590,13 +593,13 @@ describe("review: prepare isolation and contracts", () => { describe("review: ctx races", () => { const Target = local("t.target", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), members: { hit: method() }, }); it("an aborted task ctx cannot start fresh calls", async () => { const host = createHost({ self: { kind: "host" }, onError: () => {} }); - const Runner = local("t.runner", { version: 1, input: Type.Object({}) }); + const Runner = local("t.runner", { version: 1, input: obj({}, { open: true }) }); let hits = 0; const go = latch(); let outcome: unknown; @@ -650,7 +653,7 @@ describe("review: ctx races", () => { it("a remote reference refuses a foreign ctx and a ctx not made by Chord, without calling", async () => { const RemoteTarget = remote("t.remote-target", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), variants: { main: { state: {}, methods: { hit: method() } } }, }); diff --git a/packages/chord/test/state-init.test.ts b/packages/chord/test/state-init.test.ts index 77f101d5e..24d086aac 100644 --- a/packages/chord/test/state-init.test.ts +++ b/packages/chord/test/state-init.test.ts @@ -1,5 +1,5 @@ import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { arr, bool, num, obj, str } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { type Ctx, createHost, type Facet, isChordError, local, method } from "../src/index.ts"; @@ -19,16 +19,16 @@ function latch() { const Notes = local("t.notes", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), members: { - notes: Type.Array(Type.String()), - count: Type.Number(), - has: method(Type.String(), Type.Boolean()), - add: method(Type.String()), + notes: arr(str()), + count: num(), + has: method(str(), bool()), + add: method(str()), }, }); -const NotesSchema = Type.Array(Type.String()); +const NotesSchema = arr(str()); describe("state init", () => { it("runs after the resources are open and before the object goes live: nothing sees another value", async () => { @@ -124,7 +124,7 @@ describe("state init", () => { const host = createHost({ self: { kind: "host" }, onError: () => {} }); const Source = local("t.source", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), members: { all: method(undefined, NotesSchema) }, }); host.serve(Source, () => ({ all: () => ["from source"] })); @@ -155,7 +155,7 @@ describe("state init", () => { return ["loaded"]; }, }); - const counter = self.state("counter", Type.Number(), 0, { version: spec.countVersion ?? 1 }); + const counter = self.state("counter", num(), 0, { version: spec.countVersion ?? 1 }); const extra = spec.extra; const more = extra === undefined diff --git a/packages/chord/test/state.test.ts b/packages/chord/test/state.test.ts index 656ac0601..2f48c8799 100644 --- a/packages/chord/test/state.test.ts +++ b/packages/chord/test/state.test.ts @@ -1,5 +1,5 @@ import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { arr, num, obj, str } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { applyImmutable } from "../src/delta/index.ts"; import { type Ctx, createHost, isChordError, local, method, type StateEvent } from "../src/index.ts"; @@ -22,15 +22,15 @@ function emitter() { }; } -const Item = Type.Object({ id: Type.Number(), label: Type.String() }); -const Items = Type.Object({ items: Type.Array(Item), count: Type.Number() }); +const Item = obj({ id: num(), label: str() }, { open: true }); +const Items = obj({ items: arr(Item), count: num() }, { open: true }); const List = local("t.list", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), members: { list: Items, - labels: Type.Array(Type.String()), + labels: arr(str()), push: method(Item), set: method(Items), }, @@ -84,11 +84,11 @@ describe("state cells", () => { const host = createHost({ self: { kind: "host" }, onError: () => {} }); const Counter = local("t.prim", { version: 1, - input: Type.Object({}), - members: { n: Type.Number(), inc: method(undefined, Type.Number()) }, + input: obj({}, { open: true }), + members: { n: num(), inc: method(undefined, num()) }, }); host.serve(Counter, (_input, _principal, self) => { - const nState = self.state("n", Type.Number(), 0); + const nState = self.state("n", num(), 0); return { n: nState, inc: (ctx: Ctx) => { @@ -117,11 +117,11 @@ describe("state cells", () => { const host = createHost({ self: { kind: "host" }, onError: () => {} }); const Box = local("t.box", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), members: { put: method() }, }); host.serve(Box, (_input, _principal, self) => { - const list = self.state("list", Type.Array(Type.Number()), []); + const list = self.state("list", arr(num()), []); return { put: (ctx: Ctx) => { // The classic slip: an expression-bodied arrow returns push()'s length. @@ -133,10 +133,10 @@ describe("state cells", () => { }); it("validates initial values always and writes only in dev mode", async () => { - const Bad = local("t.bad-initial", { version: 1, input: Type.Object({}) }); + const Bad = local("t.bad-initial", { version: 1, input: obj({}, { open: true }) }); const host = createHost({ self: { kind: "host" }, onError: () => {} }); host.serve(Bad, (_input, _principal, self) => { - self.state("x", Type.Object({ n: Type.Number() }), { n: "no" } as never); + self.state("x", obj({ n: num() }, { open: true }), { n: "no" } as never); return {}; }); await assert.rejects(host.hold(Bad, {}).ready, /initial value of state x/); @@ -157,13 +157,13 @@ describe("state cells", () => { it("rejects writes from inside a factory and placeholder use before activation", async () => { const host = createHost({ self: { kind: "host" }, onError: () => {} }); - const Writer = local("t.writer", { version: 1, input: Type.Object({}) }); + const Writer = local("t.writer", { version: 1, input: obj({}, { open: true }) }); let captured: Ctx | undefined; await host.root((ctx) => { captured = ctx; }); host.serve(Writer, (_input, _principal, self) => { - const x = self.state("x", Type.Object({}), {}); + const x = self.state("x", obj({}, { open: true }), {}); const r = self.resource("r", { version: 1, open: () => 1 }); assert.throws( () => r.value, @@ -180,13 +180,13 @@ describe("state cells", () => { const events = emitter(); const Watcher = local("t.watcher", { version: 1, - input: Type.Object({}), - members: { seen: Type.Object({ sum: Type.Number() }) }, + input: obj({}, { open: true }), + members: { seen: obj({ sum: num() }, { open: true }) }, }); const roots = new Set(); let taskAborted = false; host.serve(Watcher, (_input, _principal, self) => { - const seen = self.state("seen", Type.Object({ sum: Type.Number() }), { sum: 0 }); + const seen = self.state("seen", obj({ sum: num() }, { open: true }), { sum: 0 }); const sourceResource = self.resource("source", { version: 1, open: () => events }); self.on( "onEvent", @@ -223,11 +223,11 @@ describe("state cells", () => { const host = createHost({ self: { kind: "host" }, onError: () => {} }); const Echo = local("t.reentrant", { version: 1, - input: Type.Object({}), - members: { n: Type.Number(), d: Type.Number(), set: method(Type.Number()) }, + input: obj({}, { open: true }), + members: { n: num(), d: num(), set: method(num()) }, }); host.serve(Echo, (_input, _principal, self) => { - const nState = self.state("n", Type.Number(), 0); + const nState = self.state("n", num(), 0); const d = self.derive([nState], (n) => n * 2); // Subscribed first: re-enters the write while revision 1 is being published. self.on("bump", nState, (n, ctx) => { diff --git a/packages/chord/test/tutorial.test.ts b/packages/chord/test/tutorial.test.ts index 55c0dfe17..72f98e1dd 100644 --- a/packages/chord/test/tutorial.test.ts +++ b/packages/chord/test/tutorial.test.ts @@ -1,7 +1,7 @@ // example:imports import assert from "node:assert/strict"; import { readFileSync } from "node:fs"; -import { type Static, Type } from "typebox"; +import { arr, bool, int, num, obj, type Static, str } from "@earendil-works/pi-schema"; import { it, onTestFinished } from "vitest"; import { type Ctx, @@ -51,23 +51,23 @@ function checkExamples(section: string): void { // example:02-contract const TodoLabel = local("tutorial.todos", { version: 1, - input: Type.Object({ list: Type.String() }), - members: { label: method(undefined, Type.String()) }, + input: obj({ list: str() }, { open: true }), + members: { label: method(undefined, str()) }, }); // end-example:02-contract // example:03-contract -const Todo = Type.Object({ text: Type.String(), done: Type.Boolean() }); -const Rows = Type.Array(Todo); +const Todo = obj({ text: str(), done: bool() }, { open: true }); +const Rows = arr(Todo); const TodoStore = local("tutorial.todos", { version: 1, - input: Type.Object({ list: Type.String() }), + input: obj({ list: str() }, { open: true }), members: { - label: method(undefined, Type.String()), + label: method(undefined, str()), rows: Rows, - openCount: Type.Number(), - add: method(Type.Object({ text: Type.String() })), - finish: method(Type.Object({ index: Type.Integer() })), + openCount: num(), + add: method(obj({ text: str() }, { open: true })), + finish: method(obj({ index: int() }, { open: true })), reset: method(Rows), }, }); @@ -107,13 +107,13 @@ const todoFactory = todoStore(); // example:05-views const Todos = remote("tutorial.todo-view", { version: 1, - input: Type.Object({ list: Type.String() }), + input: obj({ list: str() }, { open: true }), host: hostNamed("server"), variants: { viewer: { state: { rows: Rows } }, editor: { state: { rows: Rows }, - methods: { add: method(Type.Object({ text: Type.String() })) }, + methods: { add: method(obj({ text: str() }, { open: true })) }, }, }, }); @@ -190,13 +190,13 @@ it("04: holds, asynchronous resources and release", { timeout: 5000 }, async () // example:04-database const Database = local("tutorial.database", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), members: { db: opaque<{ saved: string[] }>("tutorial.database@1") }, }); const SavedTodos = local("tutorial.saved-todos", { version: 1, - input: Type.Object({ list: Type.String() }), - members: { add: method(Type.Object({ text: Type.String() })) }, + input: obj({ list: str() }, { open: true }), + members: { add: method(obj({ text: str() }, { open: true })) }, }); const host = createHost({ self: "server" }); let opens = 0; @@ -306,8 +306,8 @@ it("07: listeners react and tasks start fresh calls", { timeout: 5000 }, async ( // example:07 const Activity = local("tutorial.activity", { version: 1, - input: Type.Object({}), - members: { seen: Type.Number() }, + input: obj({}, { open: true }), + members: { seen: num() }, }); const host = createHost({ self: "server" }); host.serve(TodoStore, todoFactory); @@ -316,7 +316,7 @@ it("07: listeners react and tasks start fresh calls", { timeout: 5000 }, async ( const stopped = Promise.withResolvers(); host.serve(Activity, (_input, _principal, self) => { const todos = self.get(TodoStore, { list: "family" }); - const seen = self.state("seen", Type.Number(), 0); + const seen = self.state("seen", num(), 0); self.on("rows", todos.rows, (rows, ctx) => { seen.replace(ctx, rows?.length ?? 0); if (rows?.length === 1) observed.resolve(); @@ -345,7 +345,7 @@ it("08: plugins contribute registry entries and expose callbacks", { timeout: 50 onTestFinished(() => host.close()); // example:08 const Actions = defineRegistry<(ctx: Ctx) => Promise>("tutorial.actions"); - const Menu = local("tutorial.menu", { version: 1, input: Type.Object({}) }); + const Menu = local("tutorial.menu", { version: 1, input: obj({}, { open: true }) }); let click: (() => Promise) | undefined; const storeFacet: Facet = { name: "todo-store", @@ -444,7 +444,7 @@ it("09: compatible reload, migration and an old-contract adapter", { timeout: 50 const RenamedStore = local("tutorial.todos", { version: 2, input: TodoStore.def.input, - members: { ...members, heading: method(undefined, Type.String()) }, + members: { ...members, heading: method(undefined, str()) }, }); await host.reload([ { diff --git a/packages/chord/test/types.test.ts b/packages/chord/test/types.test.ts index f64f0ab98..9df86f431 100644 --- a/packages/chord/test/types.test.ts +++ b/packages/chord/test/types.test.ts @@ -1,25 +1,25 @@ // Entry 8: reference types. State members are plain values (a reference you hold has its state); a remote reference is a // union discriminated by `variant`, so another variant's members are reachable only after narrowing. Checked by tsc. -import { Type } from "typebox"; +import { num, obj, str } from "@earendil-works/pi-schema"; import { describe, expectTypeOf, it } from "vitest"; import { type Ctx, type Host, hostNamed, local, method, type Readable, type Ref, remote } from "../src/index.ts"; const Doc = remote("t.e8-types-doc", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), variants: { - reader: { state: { text: Type.String() } }, - editor: { state: { text: Type.String() }, methods: { edit: method(Type.String()) } }, + reader: { state: { text: str() } }, + editor: { state: { text: str() }, methods: { edit: method(str()) } }, }, }); const Single = remote("t.e8-types-single", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), - variants: { main: { state: { n: Type.Number() }, methods: { bump: method() } } }, + variants: { main: { state: { n: num() }, methods: { bump: method() } } }, }); -const Local = local("t.e8-types-local", { version: 1, input: Type.Object({}), members: { n: Type.Number() } }); +const Local = local("t.e8-types-local", { version: 1, input: obj({}, { open: true }), members: { n: num() } }); describe("reference types", () => { it("state members are T; other variants' members need narrowing; gets are awaited", () => { diff --git a/packages/chord/test/variants.test.ts b/packages/chord/test/variants.test.ts index 363f50361..3b8c2359d 100644 --- a/packages/chord/test/variants.test.ts +++ b/packages/chord/test/variants.test.ts @@ -1,7 +1,7 @@ // Entry 8: a binding applies a shape only once its new state members have snapshots; members outside the variant // throw `not_in_variant`; a re-hold after a link loss keeps the last values readable and sends one fresh unit. import assert from "node:assert/strict"; -import { Type } from "typebox"; +import { num, obj, str } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { type Ctx, @@ -24,12 +24,12 @@ async function until(check: () => boolean, attempts = 200): Promise { const Doc = remote("t.e8-doc", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), variants: { - a: { state: { x: Type.Number() }, methods: { bump: method() } }, - b: { state: { x: Type.Number(), y: Type.String() }, methods: { bump: method() } }, - c: { state: { y: Type.String() }, methods: { bump: method() } }, + a: { state: { x: num() }, methods: { bump: method() } }, + b: { state: { x: num(), y: str() }, methods: { bump: method() } }, + c: { state: { y: str() }, methods: { bump: method() } }, }, }); @@ -39,8 +39,8 @@ function docFacet(variant: "a" | "b" | "c"): Facet { name: "doc", setup: (api) => api.serve(Doc, (_input, _principal, self) => { - const x = self.state("x", Type.Number(), 1); - const y = self.state("y", Type.String(), "why"); + const x = self.state("x", num(), 1); + const y = self.state("y", str(), "why"); const bump = (ctx: Ctx) => { x.replace(ctx, x.value + 1); }; diff --git a/packages/chord/test/websocket.test.ts b/packages/chord/test/websocket.test.ts index 8cb1e95c2..b5bc4389b 100644 --- a/packages/chord/test/websocket.test.ts +++ b/packages/chord/test/websocket.test.ts @@ -1,6 +1,6 @@ import assert from "node:assert/strict"; import type { IncomingMessage } from "node:http"; -import { Type } from "typebox"; +import { arr, obj, str, unknown } from "@earendil-works/pi-schema"; import { describe, it } from "vitest"; import { WebSocketServer } from "ws"; import { @@ -17,12 +17,12 @@ import { const Board = remote("t.board", { version: 1, - input: Type.Object({}), + input: obj({}, { open: true }), host: hostNamed("server"), variants: { main: { - state: { notes: Type.Array(Type.String()) }, - methods: { add: method(Type.String()), who: method(undefined, Type.Unknown()) }, + state: { notes: arr(str()) }, + methods: { add: method(str()), who: method(undefined, unknown()) }, }, }, }); @@ -31,7 +31,7 @@ describe("WebSocket links", () => { it("serves a browser-style client over a real socket, stamped by the authenticator", async () => { const server = createHost({ self: { kind: "host" }, name: "server", onError: () => {} }); server.serve(Board, (_input, principal, self) => { - const notes = self.state("notes", Type.Array(Type.String()), []); + const notes = self.state("notes", arr(str()), []); return { variant: "main", notes, @@ -112,7 +112,7 @@ describe("WebSocket links: packed payloads", () => { const notes = Array.from({ length: 40 }, (_, i) => `${i}:${"x".repeat(4000)}`); const server = createHost({ self: { kind: "host" }, name: "server", onError: () => {} }); server.serve(Board, (_input, _principal, self) => { - const state = self.state("notes", Type.Array(Type.String()), notes); + const state = self.state("notes", arr(str()), notes); return { variant: "main", notes: state, add: () => {}, who: () => notes }; }); const wss = new WebSocketServer({ port: 0, host: "127.0.0.1" }); From a6bde34bb61499cfcd9ef70545d5750416a632f7 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sun, 1 Nov 2026 10:22:05 +0000 Subject: [PATCH 116/175] feat(schema): benchmark against ata-validator, Ajv and Zod --- package-lock.json | 222 +++++++++++++++++++++- packages/schema/CHANGELOG.md | 2 +- packages/schema/README.md | 128 +++++++++---- packages/schema/package.json | 6 +- packages/schema/test/bench/all.ts | 6 +- packages/schema/test/bench/candidates.ts | 106 ++++++++++- packages/schema/test/bench/conformance.ts | 4 +- packages/schema/test/bench/size.ts | 13 +- packages/schema/test/bench/speed.ts | 31 ++- packages/schema/test/bench/zod.ts | 187 ++++++++++++++++++ 10 files changed, 652 insertions(+), 53 deletions(-) create mode 100644 packages/schema/test/bench/zod.ts diff --git a/package-lock.json b/package-lock.json index 4bc15ba5c..8abc3354d 100644 --- a/package-lock.json +++ b/package-lock.json @@ -50,6 +50,116 @@ } } }, + "node_modules/@ata-validator/native-darwin-arm64": { + "version": "1.61.3", + "resolved": "https://registry.npmjs.org/@ata-validator/native-darwin-arm64/-/native-darwin-arm64-1.61.3.tgz", + "integrity": "sha512-DC+iozKe5mNpsmT4MLuMxG2pIeGPVR3YBYuBmops0AF5UG6eSxg/X0484hW/Nmc2dil54+b4N41hIhS5U9rExg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@ata-validator/native-darwin-x64": { + "version": "1.61.3", + "resolved": "https://registry.npmjs.org/@ata-validator/native-darwin-x64/-/native-darwin-x64-1.61.3.tgz", + "integrity": "sha512-q63+k48mWlvrNaUXmtGVxp2NWTl3wXlkkWkc2q4MerHzjIfeFBca30tfTRcyVXtDXPCj8BRJkrI69aJJXk+pLg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@ata-validator/native-linux-arm64-gnu": { + "version": "1.61.3", + "resolved": "https://registry.npmjs.org/@ata-validator/native-linux-arm64-gnu/-/native-linux-arm64-gnu-1.61.3.tgz", + "integrity": "sha512-D9OG2oKJd/hW9F8IFoqIfK+SDfyryzSGr5YRVJB39LwvvWNNfh6pxsYpjthEjwqGuJdoBtYIqOUFnjgBqDWh9w==", + "cpu": [ + "arm64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@ata-validator/native-linux-arm64-musl": { + "version": "1.61.3", + "resolved": "https://registry.npmjs.org/@ata-validator/native-linux-arm64-musl/-/native-linux-arm64-musl-1.61.3.tgz", + "integrity": "sha512-L1NhRd5/i6v7VQw28gR5L53+Zlo09CnYfXBapgGUC9I3uNppOw56/dihU5cPrd5T/k9VeljosllOP+dWA9cLpA==", + "cpu": [ + "arm64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@ata-validator/native-linux-x64-gnu": { + "version": "1.61.3", + "resolved": "https://registry.npmjs.org/@ata-validator/native-linux-x64-gnu/-/native-linux-x64-gnu-1.61.3.tgz", + "integrity": "sha512-Dlp+hcvQqKkAQEyrr7RohCbYoLYHdSjeB5puU3Ju92kSeH4MaF451q0VqwOm3StaMQB3LrLA03GDb4/G3b5gsA==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@ata-validator/native-linux-x64-musl": { + "version": "1.61.3", + "resolved": "https://registry.npmjs.org/@ata-validator/native-linux-x64-musl/-/native-linux-x64-musl-1.61.3.tgz", + "integrity": "sha512-2TutVxyhzbpUyBFzuGNp4fjyC3Suhi1idFfbk883tPdfcmp2/j0Q5QT5ySk4LPLGrNzV2haqjESZvha/QKctkw==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@ata-validator/native-win32-x64": { + "version": "1.61.3", + "resolved": "https://registry.npmjs.org/@ata-validator/native-win32-x64/-/native-win32-x64-1.61.3.tgz", + "integrity": "sha512-Ac2fqD70b/lQSuLDdz03xiGPhn2FK7oKXdQ7JbCq9ZYiJBbymhFazMKkRaI+3Y5XKIV3zalAfNGzUJAzMUsXow==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, "node_modules/@aws-sdk/client-bedrock-runtime": { "version": "3.1127.0", "resolved": "https://registry.npmjs.org/@aws-sdk/client-bedrock-runtime/-/client-bedrock-runtime-3.1127.0.tgz", @@ -3001,6 +3111,41 @@ "node": ">= 14" } }, + "node_modules/ajv": { + "version": "8.20.0", + "resolved": "https://registry.npmjs.org/ajv/-/ajv-8.20.0.tgz", + "integrity": "sha512-Thbli+OlOj+iMPYFBVBfJ3OmCAnaSyNn4M1vz9T6Gka5Jt9ba/HIR56joy65tY6kx/FCF5VXNB819Y7/GUrBGA==", + "dev": true, + "license": "MIT", + "dependencies": { + "fast-deep-equal": "^3.1.3", + "fast-uri": "^3.0.1", + "json-schema-traverse": "^1.0.0", + "require-from-string": "^2.0.2" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/epoberezkin" + } + }, + "node_modules/ajv-formats": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/ajv-formats/-/ajv-formats-3.0.1.tgz", + "integrity": "sha512-8iUql50EUR+uUcdRQ3HDqa6EVyo3docL8g5WJ3FNcWmu62IbkGUue/pEyLBW8VGKKucTPgqeks4fIU1DA4yowQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "ajv": "^8.0.0" + }, + "peerDependencies": { + "ajv": "^8.0.0" + }, + "peerDependenciesMeta": { + "ajv": { + "optional": true + } + } + }, "node_modules/asn1": { "version": "0.2.6", "resolved": "https://registry.npmjs.org/asn1/-/asn1-0.2.6.tgz", @@ -3020,6 +3165,36 @@ "node": ">=12" } }, + "node_modules/ata-validator": { + "version": "1.61.3", + "resolved": "https://registry.npmjs.org/ata-validator/-/ata-validator-1.61.3.tgz", + "integrity": "sha512-6o3V8lpiVQzzHe6blbQZj9saD0Qxmhdv2TTyO5BgPcUWeP3EZj572f9w1b93E6c+BmUce0QOf7GvwOffRUf3tw==", + "dev": true, + "license": "MIT", + "bin": { + "ata": "bin/ata.js" + }, + "engines": { + "node": ">=20.0.0" + }, + "optionalDependencies": { + "@ata-validator/native-darwin-arm64": "1.61.3", + "@ata-validator/native-darwin-x64": "1.61.3", + "@ata-validator/native-linux-arm64-gnu": "1.61.3", + "@ata-validator/native-linux-arm64-musl": "1.61.3", + "@ata-validator/native-linux-x64-gnu": "1.61.3", + "@ata-validator/native-linux-x64-musl": "1.61.3", + "@ata-validator/native-win32-x64": "1.61.3" + }, + "peerDependencies": { + "yaml": "^2.0.0" + }, + "peerDependenciesMeta": { + "yaml": { + "optional": true + } + } + }, "node_modules/balanced-match": { "version": "4.0.4", "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-4.0.4.tgz", @@ -3451,12 +3626,36 @@ "node": ">=18" } }, + "node_modules/fast-deep-equal": { + "version": "3.1.3", + "resolved": "https://registry.npmjs.org/fast-deep-equal/-/fast-deep-equal-3.1.3.tgz", + "integrity": "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q==", + "dev": true, + "license": "MIT" + }, "node_modules/fast-sha256": { "version": "1.3.0", "resolved": "https://registry.npmjs.org/fast-sha256/-/fast-sha256-1.3.0.tgz", "integrity": "sha512-n11RGP/lrWEFI/bWdygLxhI+pVeo1ZYIVwvvPkW7azl/rOy+F3HYRZ2K5zeE9mmkhQppyv9sQFx0JM9UabnpPQ==", "license": "Unlicense" }, + "node_modules/fast-uri": { + "version": "3.1.9", + "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.9.tgz", + "integrity": "sha512-hS/0Wt3Ya043oKXs0sIxrMpKQITXhMLIrVAKKXi4gozGC9v7YRWVnXIZzjNPTO02M9BdxLCWqK1yRXgNziVyJQ==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/fastify" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/fastify" + } + ], + "license": "BSD-3-Clause" + }, "node_modules/fdir": { "version": "6.5.0", "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.5.0.tgz", @@ -3774,6 +3973,13 @@ "node": ">=16" } }, + "node_modules/json-schema-traverse": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/json-schema-traverse/-/json-schema-traverse-1.0.0.tgz", + "integrity": "sha512-NM8/P9n3XjXhIZn1lLhkFaACTOURQXjWhV4BA/RnOv8xvgqtqpAX9IO4mRQxSx1Rlo4tqzeqb0sOlruaOy3dug==", + "dev": true, + "license": "MIT" + }, "node_modules/jwa": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/jwa/-/jwa-2.0.1.tgz", @@ -4590,6 +4796,16 @@ "node": ">= 6" } }, + "node_modules/require-from-string": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz", + "integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, "node_modules/retry": { "version": "0.13.1", "resolved": "https://registry.npmjs.org/retry/-/retry-0.13.1.tgz", @@ -5778,8 +5994,12 @@ "devDependencies": { "@cfworker/json-schema": "4.1.1", "@oh-my-pi/omptype": "18.13.3", + "ajv": "8.20.0", + "ajv-formats": "3.0.1", + "ata-validator": "1.61.3", "typebox": "1.3.27", - "vitest": "4.1.12" + "vitest": "4.1.12", + "zod": "4.6.7" }, "engines": { "node": ">=22.19.0" diff --git a/packages/schema/CHANGELOG.md b/packages/schema/CHANGELOG.md index 69650940a..02f86d449 100644 --- a/packages/schema/CHANGELOG.md +++ b/packages/schema/CHANGELOG.md @@ -9,4 +9,4 @@ - `compile(schema, { coerce: true })` adds `coerce(value)` and `parse(value)` for loosely typed input such as a model's tool call arguments: optional `null`s dropped, TypeBox 1.3.27's `Value.Convert` rules keyed on JSON Schema keywords (numeric strings, `"true"`, single values to arrays, `const`/`enum` members, union branches), then Ajv-style primitive coercion by `type`. `coerce` copies only the paths it changes and returns the input itself when nothing changes; `parse` coerces, then checks. Validators compiled without the option have neither; the README lists every rule. - `unresolvedRefs: "reject"` compiles a `$ref` that names nothing as `false` (the error is `unresolved reference `) instead of throwing, for schemas whose references may dangle. - `formats` also checks `url` and `json-pointer-uri-fragment`, as TypeBox and ajv-formats do. -- `npm run bench` compares conformance, speed, setup cost, memory and bundle size with TypeBox, @cfworker/json-schema and omptype. +- `npm run bench` compares conformance, speed (with and without eval), setup cost, memory and bundle size with ata-validator (generated code and its interpreter), Ajv, Zod (native schemas, and `z.fromJSONSchema` for the test suite), TypeBox, @cfworker/json-schema and omptype. diff --git a/packages/schema/README.md b/packages/schema/README.md index 100186dd9..6654fe54d 100644 --- a/packages/schema/README.md +++ b/packages/schema/README.md @@ -1,7 +1,7 @@ # @earendil-works/pi-schema JSON Schema builders and validation without `eval`. A schema compiles once into plain closures, so it checks values -about as fast as validators that generate code, also under a strict Content Security Policy, and needs no +at close to the speed of validators that generate code, also under a strict Content Security Policy, and needs no dependencies. - Drafts 2020-12 and 7. Passes 2,222 of the 2,223 core tests of the official @@ -181,53 +181,107 @@ coerced by any pass. ## Benchmarks -Node 24.4.1 on one core of an i9-13900, against TypeBox 1.3.27, @cfworker/json-schema 4.1.1 and @oh-my-pi/omptype -18.13.3 (`fromJsonSchema`). +Node 24.4.1 on one core of an i9-13900. The other libraries: -| test suite (tests passed) | pi-schema | TypeBox | cfworker | omptype | -|---|---:|---:|---:|---:| -| draft 2020-12 core (1,298) | 1,297 | 1,276 | 1,249 | 863 | -| draft 7 core (925) | 925 | 924 | 918 | 659 | -| draft 2020-12 format (895), with `formats` | 895 | 895 | 740 | 412 | +- ata-validator 1.61.3: its generated code, and its eval-free interpreter chosen with `engine: "interpreter"`; +- Ajv 8.20.0, with ajv-formats 3.0.1 for the format tests; +- Zod 4.6.7: native Zod schemas for speed, setup and memory, and `z.fromJSONSchema` for the test suite; +- TypeBox 1.3.27; +- @cfworker/json-schema 4.1.1; +- @oh-my-pi/omptype 18.13.3, through `fromJsonSchema`. -| check, valid value | pi-schema | TypeBox | cfworker | omptype | -|---|---:|---:|---:|---:| -| nested object with tagged unions | 409 ns | 474 ns | 44 µs | 689 ns | -| union of 5 messages | 89 ns | 95 ns | 10 µs | 93 ns | -| tool schema (read) | 31 ns | 9 ns | 1.9 µs | 9 ns | -| MCP tool schema with `$ref` | 162 ns | 89 ns | 13 µs | 1.8 µs | -| recursive `$ref` schema | 753 ns | 1.2 µs | 41 µs | 7.4 µs | +| test suite (tests passed) | pi-schema | ata | ata interpreter | Ajv | Zod | TypeBox | cfworker | omptype | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| draft 2020-12 core (1,298) | 1,297 | 1,298 | 1,298 | 1,236 | 693 | 1,276 | 1,249 | 863 | +| draft 7 core (925) | 925 | 925 | 925 | 917 | 632 | 924 | 918 | 659 | +| draft 2020-12 format (895), with format checks | 895 | 814 | 814 | 750 | 412 | 895 | 740 | 412 | -| same, without eval (strict CSP) | pi-schema | TypeBox | cfworker | omptype | -|---|---:|---:|---:|---:| -| nested object with tagged unions | 407 ns | 42 µs | 43 µs | 2.6 µs | -| MCP tool schema with `$ref` | 162 ns | 36 µs | 12 µs | 2.0 µs | -| recursive `$ref` schema | 750 ns | 124 µs | 41 µs | 9.0 µs | +| check, valid value | pi-schema | ata | ata interpreter | Ajv | Zod | TypeBox | cfworker | omptype | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| nested object with tagged unions | 410 ns | 180 ns | 836 ns | 627 ns | 1.3 µs | 475 ns | 43 µs | 730 ns | +| union of 5 messages | 86 ns | 37 ns | 212 ns | 70 ns | 297 ns | 95 ns | 11 µs | 96 ns | +| tool schema (read) | 31 ns | 10 ns | 41 ns | 13 ns | 41 ns | 9 ns | 2.0 µs | 9 ns | +| MCP tool schema with `$ref` | 161 ns | 64 ns | 314 ns | 136 ns | 927 ns | 87 ns | 15 µs | 1.9 µs | +| recursive `$ref` schema | 755 ns | 307 ns | 1.9 µs | 378 ns | 2.7 µs | 1.2 µs | 41 µs | 7.5 µs | -| cost | pi-schema | TypeBox | cfworker | omptype | -|---|---:|---:|---:|---:| -| compile a new schema | 21 µs | 203 µs | 20 µs | 26 µs + 106 µs on its first check | -| memory, 3,200 distinct schemas | 7.1 MB | 54.7 MB | 7.6 MB | 20.0 MB | -| bundle, minified + gzip | 8.5 KB | 35.2 KB | 5.8 KB | 38.8 KB | +| same, without eval (strict CSP) | pi-schema | ata | ata interpreter | Ajv | Zod | TypeBox | cfworker | omptype | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| nested object with tagged unions | 411 ns | 1.6 µs | 844 ns | fails | 3.4 µs | 44 µs | 44 µs | 2.7 µs | +| union of 5 messages | 87 ns | 396 ns | 209 ns | fails | 652 ns | 10 µs | 11 µs | 501 ns | +| MCP tool schema with `$ref` | 164 ns | 484 ns | 326 ns | fails | 2.0 µs | 39 µs | 13 µs | 2.2 µs | +| recursive `$ref` schema | 746 ns | 2.8 µs | 1.9 µs | fails | 7.3 µs | 130 µs | 42 µs | 9.0 µs | -Reproduce with `npm run bench` in `packages/schema` (about 5 minutes). It runs every library through its fastest -boolean path and keeps the object each returns. The scripts are in `test/bench/`. +| cost | pi-schema | ata | ata interpreter | Ajv | Zod | TypeBox | cfworker | omptype | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| compile a new schema | 20 µs | 4.9 µs | 1.3 µs | 1.4 ms | 151 µs | 210 µs | 22 µs | 28 µs | +| its first check | 0.8 µs | 335 µs | 87 µs | 195 µs | 466 µs | 41 µs | 7.1 µs | 109 µs | +| memory, 3,200 distinct schemas | 7.1 MB | 36.0 MB | 51.3 MB | 59.7 MB | 170.8 MB | 54.7 MB | 7.6 MB | 19.9 MB | +| bundle, minified + gzip | 8.5 KB | 134.0 KB | 134.0 KB | 39.7 KB | 24.4 KB | 35.2 KB | 5.8 KB | 38.8 KB | -- **conformance:** every library on the vendored test suite, with the remote documents and meta-schemas the tests - reference. -- **speed:** realistic schemas, in ns per check after warm-up; the median of 15 rounds, with the libraries' - order rotated in each round. -- **speed without eval:** the same, run with `--disallow-code-generation-from-strings`. -- **setup:** compiling 1,000 schemas a library has not seen, and their first checks. -- **memory:** 3,200 distinct tool-like schemas, each library in its own process. -- **size:** esbuild bundles of what a validator needs, minified and gzipped. +Where code generation is unavailable: -Where code generation is unavailable, TypeBox falls back to its interpreter, and omptype throws `EvalError` once per -schema, then validates through its interpreter. +- **ata** answers through its interpreted engine without being told. That automatic fallback measured slower than the + engine chosen with `engine: "interpreter"`. +- **Ajv** cannot compile at all: `compile()` throws `EvalError`. +- **Zod** notices and parses objects without its generated code. +- **TypeBox** falls back to its interpreter. +- **omptype** throws `EvalError` once per schema, then validates through its interpreter. + +ata, Zod and omptype compile on a schema's first check, so their compile cost shows there. ata's bundle is for the +browser without code splitting; with splitting, its first load is about 53 KB plus a 76 KB generator chunk loaded +later. + +Zod is not a JSON Schema validator: + +- **Speed, setup and memory:** Zod gets the same schemas written as native Zod schemas (`z.strictObject`, + `z.discriminatedUnion`, `z.lazy`), with `.describe()` for the corpus descriptions, and `safeParse` as the check. + Its memory includes building the schemas, which the other libraries receive as JSON. Without the descriptions it + is 131 MB. +- **Test suite:** through `z.fromJSONSchema`, which is semi-experimental. It refuses `unevaluatedProperties`, + `unevaluatedItems`, `dependentSchemas`, `dependentRequired`, `if`/`then`/`else`, `not`, and every `$ref` that is + not local. A refused schema fails all its tests: 405 of the 1,298 draft 2020-12 core tests and 107 of the 925 draft + 7 core tests. +- **Bundle:** measured with Zod's documented `import * as z from "zod"`. A named `import { z }` keeps every locale + (about 90 KB gzipped), and `z.fromJSONSchema` brings it to 40.0 KB; `zod/mini` is about 6 KB. Flat objects of simple types check in 13–80 ns, against 7–15 ns for TypeBox. Without code generation, reading a property whose name comes from a list costs a few nanoseconds. +### Reproduce + +`npm run bench` in `packages/schema` takes about 12 minutes. It runs every library through its fastest boolean path +(ata's `isValidObject()`) and keeps the object each returns. ata runs with `ATA_NO_NATIVE=1`: its optional native +addon only serves its buffer APIs. The scripts are in `test/bench/`. + +- **Conformance:** every library on the vendored test suite, with the remote documents and meta-schemas the tests + reference (Zod through `z.fromJSONSchema`, which takes no remote documents). ata reads the draft only from `$schema`, so draft 7 schemas without one get it declared. ata's defaults + (`useDefaults`, `assertFormat`) are turned off to match the other libraries; formats are checked only in the + format tests, by each library's own checks. +- **Speed:** realistic schemas, in ns per check after warm-up. Each result is the median of 15 rounds, with the + libraries' order rotated in each round. +- **Speed without eval:** the same, run with `--disallow-code-generation-from-strings`. +- **Setup:** compiling 1,000 schemas a library has not seen, and their first checks. +- **Memory:** 3,200 distinct tool-like schemas, compiled and each checked 3 times, each library in its own process. +- **Size:** esbuild browser bundles of what a validator needs, minified and gzipped. + +### Code generated at build time + +Ajv (`ajv/dist/standalone`) and ata (`ata build`, `ata compile`, `toStandaloneModule()`) can also generate their +validation code when the app is built. The app then imports plain modules and evaluates nothing at run time, so those +checks run at code-generation speed under a strict CSP. This is not in the benchmark, because it is a different +deal: + +- **A build step per schema.** Every schema must be known when the app is built. Schemas that arrive at run time + (MCP servers, extensions, user input) cannot be validated this way. +- **Code per schema.** In a separate measurement on the same 3,200 schemas: + - Ajv: 1.8 ms to build and 8.9 KB of JavaScript per schema; + - ata: 1.1 ms and 13 KB per schema, or 0.3–0.8 KB gzipped when only its `isValid` export is bundled; + - importing all 3,200 generated validators took 40 MB of heap (Ajv, one module) and 68 MB (ata, one module per + schema). + +pi-schema compiles at run time in 25 µs per schema, generates no code, and covers schemas known at build time and +schemas that arrive later. + ## Development `npm test` runs the official test suite (drafts 2020-12 and 7, including the optional and format tests, vendored in diff --git a/packages/schema/package.json b/packages/schema/package.json index f6e94e1d2..3fdb1b14f 100644 --- a/packages/schema/package.json +++ b/packages/schema/package.json @@ -50,7 +50,11 @@ "devDependencies": { "@cfworker/json-schema": "4.1.1", "@oh-my-pi/omptype": "18.13.3", + "ajv": "8.20.0", + "ajv-formats": "3.0.1", + "ata-validator": "1.61.3", "typebox": "1.3.27", - "vitest": "4.1.12" + "vitest": "4.1.12", + "zod": "4.6.7" } } diff --git a/packages/schema/test/bench/all.ts b/packages/schema/test/bench/all.ts index bb8421fc2..96ce485c4 100644 --- a/packages/schema/test/bench/all.ts +++ b/packages/schema/test/bench/all.ts @@ -1,5 +1,6 @@ // Runs every benchmark, each in its own process, and writes the results as JSON next to the printed tables. // Speed runs twice: normally and with code generation from strings disabled, as under a strict CSP. +// The children run with ATA_NO_NATIVE=1 (see candidates.ts). // Run: npm run bench [-- --out ] (default: pi-schema-bench-results.json in the temp directory) import { execFileSync } from "node:child_process"; @@ -28,7 +29,10 @@ const results: Record = { for (const [key, file, flags] of runs) { const json = join(tmpdir(), `pi-schema-bench-${key}.json`); console.log(`\n## ${key}${flags.length > 0 ? ` (${flags.join(" ")})` : ""}\n`); - execFileSync(process.execPath, [...flags, join(dir, file), "--json", json], { stdio: "inherit" }); + execFileSync(process.execPath, [...flags, join(dir, file), "--json", json], { + stdio: "inherit", + env: { ...process.env, ATA_NO_NATIVE: "1" }, + }); results[key] = JSON.parse(readFileSync(json, "utf8")); rmSync(json); } diff --git a/packages/schema/test/bench/candidates.ts b/packages/schema/test/bench/candidates.ts index 8b231c855..ce0c6d4cf 100644 --- a/packages/schema/test/bench/candidates.ts +++ b/packages/schema/test/bench/candidates.ts @@ -1,18 +1,28 @@ // The validators the benchmarks compare, each behind the same interface: compile a schema, then check values. // Every library is used through its fastest boolean path and keeps the object it returns, as an application would. +// Run the benchmarks through `npm run bench`, which sets ATA_NO_NATIVE=1: ata-validator's optional native addon only +// serves its buffer APIs, and the verdicts measured here are JavaScript either way. import { Validator as CfValidator } from "@cfworker/json-schema"; import { fromJsonSchema } from "@oh-my-pi/omptype"; +import Ajv7Module from "ajv"; +import Ajv2020Module from "ajv/dist/2020.js"; +import draft07MetaSchema from "ajv/dist/refs/json-schema-draft-07.json" with { type: "json" }; +import addFormatsModule from "ajv-formats"; +import { Validator as AtaValidator } from "ata-validator"; import { Compile } from "typebox/compile"; import { ResetExternal, ResetFunctions } from "typebox/schema"; import { Settings } from "typebox/system"; import { compile, type FormatCheck, type JsonSchema } from "../../src/index.ts"; +import { nativeCases, nativeZod, zodFromJsonSchema } from "./zod.ts"; /** How a benchmark compiles a schema. */ export interface PrepareOptions { readonly draft: "2020-12" | "7"; readonly schemas?: Readonly>; readonly formats?: Readonly>; + /** The schema is a JSON-Schema-Test-Suite schema (for libraries that take native schemas otherwise). */ + readonly suite?: boolean; } /** A compiled schema: its boolean check and the object the library returned (kept alive by the caller). */ @@ -92,8 +102,102 @@ const omptype: Candidate = { }, }; +// Ajv's CommonJS modules export the class as module.exports and as `default`; the latter is what its types describe. +const Ajv2020 = Ajv2020Module.default; +const Ajv7 = Ajv7Module.default; +const addFormats = addFormatsModule.default; +type AjvInstance = InstanceType; + +// One Ajv per draft, as an application keeps it. Ajv2020 also gets the draft-07 meta-schema, so tool schemas that +// declare draft-07 compile. strict: false so schemas written for other tools compile; logger: false keeps Ajv from +// printing generated code when compiling fails without eval. +function newAjv(draft: "2020-12" | "7", formats: boolean): AjvInstance { + const options = { strict: false, validateFormats: formats, logger: false as const }; + const ajv = draft === "7" ? new Ajv7(options) : new Ajv2020(options); + if (draft !== "7") ajv.addMetaSchema(draft07MetaSchema); + if (formats) addFormats(ajv); + return ajv; +} + +const sharedAjv = new Map(); + +// Ajv compiles with new Function and has no interpreter: without eval, compile() throws EvalError. +const ajv: Candidate = { + name: noEval ? "ajv (no eval: fails)" : "ajv", + prepare(schema, options) { + let instance: AjvInstance; + if (options.schemas === undefined) { + const key = `${options.draft} ${options.formats !== undefined}`; + instance = sharedAjv.get(key) ?? newAjv(options.draft, options.formats !== undefined); + sharedAjv.set(key, instance); + } else { + // The suite reuses $ids across groups, which one shared instance would reject: a fresh one per schema. + instance = newAjv(options.draft, options.formats !== undefined); + for (const [uri, remote] of Object.entries(options.schemas)) { + // Ajv ships its own meta-schemas; adding them again under the same id throws. + if (instance.getSchema(uri) === undefined) { + try { + instance.addSchema(remote as object, uri); + } catch { + // A remote whose $id Ajv already registered under another key. + } + } + } + } + const validator = instance.compile(schema as object); + return { check: (v) => validator(v) as boolean, validator }; + }, +}; + +// ata-validator: its defaults fill `default` values into the input and assert `format`; both are turned off to match +// the other candidates (format checks only in the format run, with its own checks). It reads the draft only from +// $schema, so a draft 7 schema without one gets it declared. +function ataValidator(schema: JsonSchema, options: PrepareOptions, engine: "interpreter" | undefined) { + const declared = + options.draft === "7" && typeof schema === "object" && !("$schema" in schema) + ? { $schema: "http://json-schema.org/draft-07/schema#", ...schema } + : schema; + return new AtaValidator(declared, { + useDefaults: false, + assertFormat: options.formats !== undefined, + schemas: (options.schemas ?? {}) as Record, + logger: false, + ...(engine === undefined ? {} : { engine }), + }); +} + +// Generated code, or its interpreted engine where code generation is blocked. +const ata: Candidate = { + name: noEval ? "ata (no eval: interpreter)" : "ata", + prepare(schema, options) { + const validator = ataValidator(schema, options, undefined); + return { check: (v) => validator.isValidObject(v), validator }; + }, +}; + +// Its eval-free engine, a tree of closures, chosen explicitly: what it runs on for schemas from outside. +const ataInterpreter: Candidate = { + name: "ata interpreter", + prepare(schema, options) { + const validator = ataValidator(schema, options, "interpreter"); + return { check: (v) => validator.isValidObject(v), validator }; + }, +}; + +// Zod: native Zod schemas for the speed cases and the corpus, z.fromJSONSchema only for the suite. Its object parser +// generates code with new Function when it can and parses without it otherwise; safeParse is its boolean path. +const zod: Candidate = { + name: noEval ? "zod (no eval: no JIT)" : "zod", + prepare(schema, options) { + const validator = options.suite + ? zodFromJsonSchema(schema, options.draft) + : (nativeCases.get(schema) ?? nativeZod(schema as Record)); + return { check: (v) => validator.safeParse(v).success, validator }; + }, +}; + /** Every candidate, ours first. */ -export const candidates: readonly Candidate[] = [piSchema, typebox, cfworker, omptype]; +export const candidates: readonly Candidate[] = [piSchema, typebox, ajv, ata, ataInterpreter, zod, cfworker, omptype]; /** Candidates by name, including TypeBox's interpreter (what TypeBox runs when eval is unavailable). */ export const byName: ReadonlyMap = new Map( diff --git a/packages/schema/test/bench/conformance.ts b/packages/schema/test/bench/conformance.ts index d0cc72a7e..9d8977d76 100644 --- a/packages/schema/test/bench/conformance.ts +++ b/packages/schema/test/bench/conformance.ts @@ -1,6 +1,7 @@ // Conformance of every candidate on the vendored JSON-Schema-Test-Suite (drafts 2020-12 and 7), by section: // core, optional and optional/format. Every candidate gets the remote documents and meta-schemas; format tests run -// with pi-schema's format checks (the other libraries check formats on their own or not at all). +// with pi-schema's format checks (the other libraries check formats on their own or not at all). Zod goes through +// z.fromJSONSchema, which refuses some keywords and every remote $ref: a schema it refuses fails all its tests. // Run: node test/bench/conformance.ts [--json ] import { writeFileSync } from "node:fs"; @@ -29,6 +30,7 @@ for (const file of suiteFiles()) { draft: file.draft, schemas: suiteSchemas(), formats: kind === "format" ? formats : undefined, + suite: true, }).check; } catch { continue; // A schema the library cannot compile fails all its tests. diff --git a/packages/schema/test/bench/size.ts b/packages/schema/test/bench/size.ts index b6996af60..46a09e3d3 100644 --- a/packages/schema/test/bench/size.ts +++ b/packages/schema/test/bench/size.ts @@ -1,4 +1,5 @@ -// Bundle size of what a validator needs from each library: esbuild bundles and minifies the import, then gzip -9. +// Bundle size of what a validator needs from each library, as a browser app ships it: esbuild bundles and minifies the +// import for the browser platform (one file, no code splitting), then gzip -9. // Run: node test/bench/size.ts [--json ] import { writeFileSync } from "node:fs"; @@ -12,6 +13,13 @@ const entries: Record = { "pi-schema + formats": `export { compile } from ${JSON.stringify(`${src}index.ts`)}; export { formats } from ${JSON.stringify(`${src}formats/index.ts`)};`, typebox: `export { Compile } from "typebox/compile";`, cfworker: `export { Validator } from "@cfworker/json-schema";`, + ajv: `export { default } from "ajv/dist/2020.js";`, + "ajv + ajv-formats": `export { default } from "ajv/dist/2020.js"; export { default as addFormats } from "ajv-formats";`, + "ata-validator": `export { Validator } from "ata-validator";`, + // Zod's documented `import * as z` (a named `{ z }` import keeps every locale, about 90 KB gzipped). Its core with + // native schemas, and with z.fromJSONSchema for JSON Schema. + zod: `import * as z from "zod"; export const check = (v: unknown) => z.object({ a: z.string() }).safeParse(v).success;`, + "zod + fromJSONSchema": `import * as z from "zod"; export const fromJSONSchema = z.fromJSONSchema;`, omptype: `export { fromJsonSchema } from "@oh-my-pi/omptype";`, }; @@ -25,8 +33,7 @@ for (const [name, contents] of Object.entries(entries)) { bundle: true, minify: true, format: "esm", - platform: "neutral", - mainFields: ["module", "main"], + platform: "browser", write: false, logLevel: "silent", }); diff --git a/packages/schema/test/bench/speed.ts b/packages/schema/test/bench/speed.ts index 6f470951b..ba2f17b23 100644 --- a/packages/schema/test/bench/speed.ts +++ b/packages/schema/test/bench/speed.ts @@ -20,21 +20,37 @@ function median(values: number[]): number { return [...values].sort((a, b) => a - b)[values.length >> 1]; } -/** Median ns per check, by case name, input ("valid"/"invalid") and candidate name. */ -export type SpeedResults = Record>>; +/** + * Median ns per check, by case name, input ("valid"/"invalid") and candidate name; null where the candidate cannot + * compile the schema (Ajv without eval). + */ +export type SpeedResults = Record>>; const results: SpeedResults = {}; for (const c of cases) { results[c.name] = { valid: {}, invalid: {} }; for (const input of ["valid", "invalid"] as const) { const value = c[input]; - const runs = candidates.map((candidate) => { - const { check } = candidate.prepare(c.schema, { draft: "2020-12" }); + const runs: { + candidate: (typeof candidates)[number]; + check: (v: unknown) => boolean; + n: number; + samples: number[]; + }[] = []; + for (const candidate of candidates) { + let check: (v: unknown) => boolean; + try { + check = candidate.prepare(c.schema, { draft: "2020-12" }).check; + } catch (error) { + if (!(error instanceof EvalError)) throw error; + results[c.name][input][candidate.name] = null; + continue; + } if (check(value) !== (input === "valid")) throw new Error(`${candidate.name} gets ${c.name} (${input}) wrong`); let n = 1000; while (time(check, value, n) * n < ROUND_MS * 1e6 && n < 1e8) n *= 2; - return { candidate, check, n, samples: [] as number[] }; - }); + runs.push({ candidate, check, n, samples: [] }); + } for (let round = 0; round < ROUNDS; round++) { for (let i = 0; i < runs.length; i++) { const run = runs[(i + round) % runs.length]; @@ -57,6 +73,7 @@ const out = process.argv.indexOf("--json"); if (out >= 0) writeFileSync(process.argv[out + 1], JSON.stringify(results, null, "\t")); if (sink < 0) console.log(sink); -function format(ns: number): string { +function format(ns: number | null): string { + if (ns === null) return "fails"; return ns >= 1000 ? `${(ns / 1000).toFixed(ns >= 10000 ? 0 : 1)} µs` : `${ns.toFixed(0)} ns`; } diff --git a/packages/schema/test/bench/zod.ts b/packages/schema/test/bench/zod.ts new file mode 100644 index 000000000..c0d4d8371 --- /dev/null +++ b/packages/schema/test/bench/zod.ts @@ -0,0 +1,187 @@ +// Zod in the comparison. Zod is not a JSON Schema validator: the speed cases are written here as the native Zod +// schemas a developer would write (z.strictObject for closed objects, z.discriminatedUnion for tagged unions, z.lazy +// for recursion), and the corpus schemas are built into the same native calls by `nativeZod`. Only the conformance run +// goes through z.fromJSONSchema, Zod's (semi-experimental) JSON Schema import. + +import * as z from "zod"; +import type { JsonSchema } from "../../src/index.ts"; +import { cases } from "./cases.ts"; + +const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/; + +const block = z.discriminatedUnion("kind", [ + z.strictObject({ kind: z.literal("text"), text: z.string() }), + z.strictObject({ + kind: z.literal("image"), + src: z.string(), + width: z.number().int().min(0), + height: z.number().int().min(0), + alt: z.string().optional(), + }), + z.strictObject({ kind: z.literal("code"), lang: z.enum(["ts", "js", "py"]), source: z.string() }), +]); + +const id = z.number().int().min(0); + +const range = z.strictObject({ start: z.number().int().min(1), end: z.number().int().min(1) }); +const replace = z.strictObject({ oldText: z.string(), newText: z.string() }); +const lineEdit = z.strictObject({ range, text: z.string() }); + +const richText = z.strictObject({ + type: z.enum(["text", "mention"]), + content: z.string().max(2000), + annotations: z.strictObject({ bold: z.boolean(), italic: z.boolean(), color: z.string() }).optional(), +}); + +type PageBlock = { + type: "paragraph" | "heading_1" | "heading_2" | "bulleted_list_item" | "to_do"; + rich_text: z.infer[]; + checked?: boolean; + children?: PageBlock[]; +}; + +const pageBlock: z.ZodType = z.lazy(() => + z.strictObject({ + type: z.enum(["paragraph", "heading_1", "heading_2", "bulleted_list_item", "to_do"]), + rich_text: z.array(richText), + checked: z.boolean().optional(), + children: z.array(pageBlock).optional(), + }), +); + +/** The speed cases as native Zod schemas, by case name (the JSON Schema cases in cases.ts, one for one). */ +const native: Record = { + "small closed object": z.strictObject({ text: z.string() }), + "nested object with tagged unions": z.strictObject({ + id: z.string().regex(uuid), + title: z.string().min(1).max(200), + tags: z.array(z.string()).max(16), + author: z.strictObject({ name: z.string(), email: z.string().optional() }), + blocks: z.array(block), + score: z.number().min(0).max(1).optional(), + draft: z.boolean().optional(), + }), + "message union of 5": z.discriminatedUnion("kind", [ + z.strictObject({ + kind: z.literal("hello"), + protocol: z.number().int().min(0), + incarnation: z.string().regex(uuid), + calls: z.number().int().min(1), + }), + z.strictObject({ + kind: z.literal("hold"), + hold: id, + type: z.string(), + version: z.number().int().min(1), + fingerprint: z.string().regex(/^[0-9a-f]{16}$/), + params: z.unknown(), + address: z.strictObject({ kind: z.string(), params: z.unknown() }), + chain: z.array(z.string().regex(uuid)).max(64).optional(), + }), + z.strictObject({ kind: z.literal("release"), hold: id }), + z.strictObject({ kind: z.literal("call"), call: id, hold: id, method: id, args: z.unknown().optional() }), + z.strictObject({ kind: z.literal("abort"), call: id }), + ]), + "tool: read": z.object({ path: z.string(), offset: z.number().optional(), limit: z.number().optional() }), + "tool: edit": z.object({ path: z.string(), edits: z.array(z.object({ oldText: z.string(), newText: z.string() })) }), + "tool: bash": z.object({ command: z.string(), timeout: z.number().optional() }), + "MCP create_issue (pattern, anyOf, enum)": z.strictObject({ + owner: z + .string() + .min(1) + .regex(/^[A-Za-z0-9-]+$/), + repo: z + .string() + .min(1) + .regex(/^[A-Za-z0-9._-]+$/), + title: z.string().min(1), + body: z.string().optional(), + labels: z.array(z.string()).optional(), + assignees: z.array(z.string()).optional(), + milestone: z.number().nullable().optional(), + state: z.enum(["open", "closed"]).optional(), + }), + "MCP edit_file ($ref, anyOf)": z.strictObject({ + path: z.string().regex(/^(\/|\.\/|[A-Za-z]:\\)?[^\u0000]+$/), + edits: z.array(z.union([replace, lineEdit])).min(1), + dryRun: z.boolean().optional(), + }), + "MCP create_page (recursive $ref, oneOf)": z.strictObject({ + parent: z.union([ + z.strictObject({ page_id: z.string().regex(uuid) }), + z.strictObject({ database_id: z.string().regex(uuid) }), + ]), + title: z.string().min(1), + due: z + .string() + .regex(/^\d{4}-\d{2}-\d{2}$/) + .optional(), + children: z.array(pageBlock).max(100).optional(), + }), +}; + +/** The native Zod schema of each speed case's JSON Schema. */ +export const nativeCases: ReadonlyMap = new Map( + cases.map((c) => { + const schema = native[c.name]; + if (schema === undefined) throw new Error(`No native Zod schema for the case ${c.name}`); + return [c.schema, schema]; + }), +); + +type Json = Record; + +/** + * The native Zod schema a developer would write for a corpus schema (corpus.ts): objects (closed ones strict), + * strings with bounds, patterns and enums, bounded integers, numbers, booleans, string arrays, descriptions, and + * $defs reused through their constant. Throws for any other shape. + */ +export function nativeZod(schema: Json, root: Json = schema): z.ZodType { + if (typeof schema.$ref === "string") { + const name = schema.$ref.replace("#/$defs/", ""); + return nativeZod((root.$defs as Record)[name], root); + } + const built = nativeType(schema, root); + return typeof schema.description === "string" ? built.describe(schema.description) : built; +} + +function nativeType(schema: Json, root: Json): z.ZodType { + switch (schema.type) { + case "object": { + const required = new Set(schema.required as string[]); + const shape: Record = {}; + for (const [key, value] of Object.entries(schema.properties as Record)) { + const property = nativeZod(value, root); + shape[key] = required.has(key) ? property : property.optional(); + } + return schema.additionalProperties === false ? z.strictObject(shape) : z.object(shape); + } + case "string": { + if (Array.isArray(schema.enum)) return z.enum(schema.enum as [string, ...string[]]); + let string = z.string(); + if (typeof schema.minLength === "number") string = string.min(schema.minLength); + if (typeof schema.maxLength === "number") string = string.max(schema.maxLength); + if (typeof schema.pattern === "string") string = string.regex(new RegExp(schema.pattern)); + return string; + } + case "integer": + return z + .number() + .int() + .min(schema.minimum as number) + .max(schema.maximum as number); + case "number": + return z.number(); + case "boolean": + return z.boolean(); + case "array": + return z.array(z.string()).max(schema.maxItems as number); + default: + throw new Error(`No native Zod builder for ${JSON.stringify(schema).slice(0, 80)}`); + } +} + +/** z.fromJSONSchema with the draft given; it accepts only local $refs, so remote documents are not passed. */ +export function zodFromJsonSchema(schema: JsonSchema, draft: "2020-12" | "7"): z.ZodType { + return z.fromJSONSchema(schema as never, { defaultTarget: draft === "7" ? "draft-7" : "draft-2020-12" }); +} From 6291b9bcc3ee9480c2caba8676fc2f6cd321eaa9 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sun, 1 Nov 2026 11:12:44 +0000 Subject: [PATCH 117/175] feat(agent): pi-schema's Static types TypeBox-built schemas too, without depending on TypeBox --- packages/agent/CHANGELOG.md | 2 +- packages/agent/src/types.ts | 4 +- .../agent/test/agent-tool-typebox.test.ts | 65 ++++++++++++++++ .../src/core/tools/tool-definition-wrapper.ts | 4 +- packages/durable/CHANGELOG.md | 2 +- packages/durable/src/harness/types.ts | 4 +- .../durable/test/tool-args-typebox.test.ts | 61 +++++++++++++++ packages/schema/CHANGELOG.md | 1 + packages/schema/README.md | 6 ++ packages/schema/src/builders.ts | 17 ++++- packages/schema/src/typebox-static.ts | 76 +++++++++++++++++++ packages/schema/test/typebox-static.test.ts | 72 ++++++++++++++++++ 12 files changed, 304 insertions(+), 10 deletions(-) create mode 100644 packages/agent/test/agent-tool-typebox.test.ts create mode 100644 packages/durable/test/tool-args-typebox.test.ts create mode 100644 packages/schema/src/typebox-static.ts create mode 100644 packages/schema/test/typebox-static.test.ts diff --git a/packages/agent/CHANGELOG.md b/packages/agent/CHANGELOG.md index f095f4f4f..96b4b9428 100644 --- a/packages/agent/CHANGELOG.md +++ b/packages/agent/CHANGELOG.md @@ -4,7 +4,7 @@ ### Breaking Changes -- `AgentTool` takes any JSON Schema object as `parameters` (`AgentTool`) and types `params` and `prepareArguments` through `@earendil-works/pi-schema` (`AgentToolParams`): pi-schema's builders give their static type, other schemas (TypeBox included) give `unknown`, `AgentTool` stays `any`. `outputSchema` is any JSON Schema object. `typebox` is no longer a dependency. Migration: build parameters with pi-schema (`obj({ path: str() })` instead of `Type.Object({ path: Type.String() })`), or keep TypeBox schemas and narrow `params` yourself. +- `AgentTool` takes any JSON Schema object as `parameters` (`AgentTool`) and types `params` and `prepareArguments` through `@earendil-works/pi-schema` (`AgentToolParams`): pi-schema's builders give their static type, TypeBox schemas give TypeBox's static type for the common kinds (String, Number, Integer, Boolean, Null, Literal, Array, Object with optional and readonly properties, Union, Record, Unsafe, Unknown, Any; other kinds such as Enum, Tuple and Intersect give `unknown`), other schemas give `unknown`, `AgentTool` stays `any`. `outputSchema` is any JSON Schema object. `typebox` is no longer a dependency. Migration: build parameters with pi-schema (`obj({ path: str() })` instead of `Type.Object({ path: Type.String() })`), or keep TypeBox schemas (narrow `params` yourself only where they use other kinds). ### Fixed diff --git a/packages/agent/src/types.ts b/packages/agent/src/types.ts index 539fee1ae..24bdd7531 100644 --- a/packages/agent/src/types.ts +++ b/packages/agent/src/types.ts @@ -463,8 +463,8 @@ export interface AgentToolCallOutcome { export type AgentToolUpdateCallback = (partialResult: AgentToolResult) => void; /** - * The arguments type of a tool with parameter schema `TParameters`: `Static` for schemas built with - * `@earendil-works/pi-schema`'s builders, `unknown` for other schemas, and `any` for `AgentTool`. + * The arguments type of a tool with parameter schema `TParameters`: pi-schema's `Static` (the builders' + * type, TypeBox's for the common TypeBox kinds, else `unknown`), and `any` for `AgentTool`. */ export type AgentToolParams = IsAny extends true ? any : Static; diff --git a/packages/agent/test/agent-tool-typebox.test.ts b/packages/agent/test/agent-tool-typebox.test.ts new file mode 100644 index 000000000..1c859b7ae --- /dev/null +++ b/packages/agent/test/agent-tool-typebox.test.ts @@ -0,0 +1,65 @@ +// AgentTool params typed from a TypeBox-built parameter schema. pi-agent-core does not depend on TypeBox: the schema +// below has TypeBox 1.x's type shapes as its d.mts declare them; pi-schema's own tests check the shapes against TypeBox. + +import { describe, expectTypeOf, it } from "vitest"; +import type { AgentTool } from "../src/types.ts"; + +interface TString { + "~kind": "String"; + type: "string"; +} + +interface TLiteral { + "~kind": "Literal"; + const: Value; +} + +interface TUnion { + "~kind": "Union"; + anyOf: Branches; +} + +interface TObject { + "~kind": "Object"; + type: "object"; + properties: Properties; + required: string[]; +} + +type TOptional = Schema & { "~optional": true }; + +const parameters: TObject<{ command: TString; mode: TOptional, TLiteral<"deep">]>> }> = { + "~kind": "Object", + type: "object", + properties: { + command: { "~kind": "String", type: "string" }, + mode: { + "~kind": "Union", + anyOf: [ + { "~kind": "Literal", const: "fast" }, + { "~kind": "Literal", const: "deep" }, + ], + "~optional": true, + }, + }, + required: ["command"], +}; + +type Params = { command: string; mode?: "fast" | "deep" }; + +describe("AgentTool params of a TypeBox schema", () => { + it("are typed by the schema in execute() and prepareArguments()", () => { + const tool: AgentTool = { + name: "bash", + label: "Bash", + description: "Runs a command", + parameters, + async execute(_toolCallId, params) { + expectTypeOf(params).toEqualTypeOf(); + return { content: [], details: undefined }; + }, + }; + expectTypeOf[1]>().toEqualTypeOf(); + expectTypeOf>>().toEqualTypeOf(); + }); +}); diff --git a/packages/coding-agent/src/core/tools/tool-definition-wrapper.ts b/packages/coding-agent/src/core/tools/tool-definition-wrapper.ts index 320030fab..700dfd35c 100644 --- a/packages/coding-agent/src/core/tools/tool-definition-wrapper.ts +++ b/packages/coding-agent/src/core/tools/tool-definition-wrapper.ts @@ -4,8 +4,8 @@ import type { ExtensionToolContext, ToolDefinition } from "../extensions/types.t /** * An `AgentTool` with a TypeBox parameter schema whose `execute` and `prepareArguments` are typed by TypeBox's - * `Static`, as the built-in tools are (pi-agent-core types `params` through pi-schema, which gives `unknown` for - * TypeBox schemas). Spreads into a `ToolDefinition` of the same schema. + * `Static`, as the built-in tools are (pi-agent-core types `params` through pi-schema, which maps only the common + * TypeBox kinds). Spreads into a `ToolDefinition` of the same schema. */ export type TypeBoxAgentTool = Omit< AgentTool, diff --git a/packages/durable/CHANGELOG.md b/packages/durable/CHANGELOG.md index e852e8fa8..6e9dcdb21 100644 --- a/packages/durable/CHANGELOG.md +++ b/packages/durable/CHANGELOG.md @@ -4,7 +4,7 @@ ### Breaking Changes -- Tool schemas are JSON Schema typed through `@earendil-works/pi-schema`: `defineTool()` and `ToolRegistration` take any JSON Schema object as `parameters` and `structuredOutputSchema`, and type `args` with pi-schema's `Static` (`ToolArgs`; other schemas give `unknown`). The built-in tools (`read`, `write`, `edit`, `bash`, `powershell`, `codemode`, `tool_search`, the model tools) are built with pi-schema, with the same JSON Schema up to key order. `structuredOutput` is validated with pi-schema, so `invalid_structured_output` messages use its wording. `typebox` is no longer a dependency. Migration: replace `Type.Object({ n: Type.Number() })` from `@earendil-works/pi-ai` with `obj({ n: num() })` from `@earendil-works/pi-schema` (closed by default; `{ open: true }` allows other properties). +- Tool schemas are JSON Schema typed through `@earendil-works/pi-schema`: `defineTool()` and `ToolRegistration` take any JSON Schema object as `parameters` and `structuredOutputSchema`, and type `args` with pi-schema's `Static` (`ToolArgs`; TypeBox schemas give TypeBox's static type for the common kinds, as pi-schema's `Static` describes; other schemas give `unknown`). The built-in tools (`read`, `write`, `edit`, `bash`, `powershell`, `codemode`, `tool_search`, the model tools) are built with pi-schema, with the same JSON Schema up to key order. `structuredOutput` is validated with pi-schema, so `invalid_structured_output` messages use its wording. `typebox` is no longer a dependency. Migration: replace `Type.Object({ n: Type.Number() })` from `@earendil-works/pi-ai` with `obj({ n: num() })` from `@earendil-works/pi-schema` (closed by default; `{ open: true }` allows other properties). - `SqliteExecutor` (re-exported from Chord) gains `runCounted(sql, ...params)`, resolving with the rows the statement changed; custom `SqliteDatabase` implementations add it. - `Storage.commit()` stores a group of commits in one call: `commit(batches, receive, context)` resolves with a `CommitResult` (`{ stored, refused? }`) and replaces `commit(writes, adopt, context)`. Each batch is one atomic commit with its own seq, checked against the state the batches before it left. The first batch Durable's own checks refuse (global IDs, document actions, copy sources) is not stored, nor anything after it; the batches before it are, and the call resolves with their number in `stored` and the refusal in `refused`. Any other failure (a driver or I/O error, a failed `COMMIT`) rejects the whole call as an unknown outcome and stores nothing the backend could roll back. `receive(index, seq)` is called once per stored batch, in order, synchronously, once they are durable and before the next call starts; receipts made before a rejection stand. Every batch has at least one write. The optional `publishing(publish, context)` runs each commit's publication inside the storage's own scope (`StoreStorage`: the attached host's batch). Memory, SQLite, JSONL, IndexedDB and `StoreStorage` store a call in one transaction or append, and the storage conformance suite checks the contract. Migration for a custom Storage: diff --git a/packages/durable/src/harness/types.ts b/packages/durable/src/harness/types.ts index c0f03e65d..e312758b9 100644 --- a/packages/durable/src/harness/types.ts +++ b/packages/durable/src/harness/types.ts @@ -333,8 +333,8 @@ export interface ToolExecutionApi extend } /** - * The arguments type of a tool with parameter schema `TParameters`: `Static` for schemas built with - * `@earendil-works/pi-schema`'s builders, `unknown` for other JSON Schema, and `any` for `ToolRegistration`. + * The arguments type of a tool with parameter schema `TParameters`: pi-schema's `Static` (the builders' + * type, TypeBox's for the common TypeBox kinds, else `unknown`), and `any` for `ToolRegistration`. */ export type ToolArgs = IsAny extends true ? any : Static; diff --git a/packages/durable/test/tool-args-typebox.test.ts b/packages/durable/test/tool-args-typebox.test.ts new file mode 100644 index 000000000..7a606cf96 --- /dev/null +++ b/packages/durable/test/tool-args-typebox.test.ts @@ -0,0 +1,61 @@ +// Tool arguments typed from a TypeBox-built parameter schema. Durable does not depend on TypeBox: the schema below +// has TypeBox 1.x's type shapes as its d.mts declare them; pi-schema's own tests check the shapes against TypeBox. + +import { defineTool, type ToolRegistration } from "@earendil-works/pi-durable"; +import { describe, expectTypeOf, it } from "vitest"; + +interface TString { + "~kind": "String"; + type: "string"; +} + +interface TInteger { + "~kind": "Integer"; + type: "integer"; +} + +interface TArray { + "~kind": "Array"; + type: "array"; + items: Items; +} + +interface TObject { + "~kind": "Object"; + type: "object"; + properties: Properties; + required: string[]; +} + +type TOptional = Schema & { "~optional": true }; + +const parameters: TObject<{ path: TString; offset: TOptional; tags: TArray }> = { + "~kind": "Object", + type: "object", + properties: { + path: { "~kind": "String", type: "string" }, + offset: { "~kind": "Integer", type: "integer", "~optional": true }, + tags: { "~kind": "Array", type: "array", items: { "~kind": "String", type: "string" } }, + }, + required: ["path", "tags"], +}; + +type Args = { path: string; offset?: number; tags: string[] }; + +describe("tool arguments of a TypeBox schema", () => { + it("are typed by the schema in execute() and prepareArguments()", () => { + const tool = defineTool({ + name: "read", + description: "Reads a file", + parameters, + async execute(args) { + expectTypeOf(args).toEqualTypeOf(); + return {}; + }, + }); + expectTypeOf[0]>().toEqualTypeOf(); + expectTypeOf< + ReturnType["prepareArguments"]>> + >().toEqualTypeOf(); + }); +}); diff --git a/packages/schema/CHANGELOG.md b/packages/schema/CHANGELOG.md index 02f86d449..404e93c57 100644 --- a/packages/schema/CHANGELOG.md +++ b/packages/schema/CHANGELOG.md @@ -7,6 +7,7 @@ - Initial release: `compile(schema, options)` validates JSON Schema drafts 2020-12 and 7 without `eval` or `new Function`, compiling a schema once into plain closures. `Validator.check` is the boolean check (a type guard for schemas built with the builders), `Validator.errors` reports each failure with a JSON Pointer path. Format checks are in the optional `@earendil-works/pi-schema/formats` entry point. - Schema builders (`str`, `num`, `int`, `bool`, `nul`, `unknown`, `lit`, `oneOf`, `obj`, `opt`, `arr`, `tuple`, `map`, `union`, `nullable`, `bytes`) that return plain JSON Schema typed with the values it accepts (`Static`). - `compile(schema, { coerce: true })` adds `coerce(value)` and `parse(value)` for loosely typed input such as a model's tool call arguments: optional `null`s dropped, TypeBox 1.3.27's `Value.Convert` rules keyed on JSON Schema keywords (numeric strings, `"true"`, single values to arrays, `const`/`enum` members, union branches), then Ajv-style primitive coercion by `type`. `coerce` copies only the paths it changes and returns the input itself when nothing changes; `parse` coerces, then checks. Validators compiled without the option have neither; the README lists every rule. +- `Static` also types TypeBox-built schemas, from TypeBox 1.x's type shapes and without importing TypeBox: String, Number, Integer, Boolean, Null, Literal, Array, Object (with `Type.Optional` and `Type.Readonly` properties), Union, Record, Unsafe, Unknown and Any give TypeBox's static type; other TypeBox kinds give `unknown`. - `unresolvedRefs: "reject"` compiles a `$ref` that names nothing as `false` (the error is `unresolved reference `) instead of throwing, for schemas whose references may dangle. - `formats` also checks `url` and `json-pointer-uri-fragment`, as TypeBox and ajv-formats do. - `npm run bench` compares conformance, speed (with and without eval), setup cost, memory and bundle size with ata-validator (generated code and its interpreter), Ajv, Zod (native schemas, and `z.fromJSONSchema` for the test suite), TypeBox, @cfworker/json-schema and omptype. diff --git a/packages/schema/README.md b/packages/schema/README.md index 6654fe54d..f1ff34f4d 100644 --- a/packages/schema/README.md +++ b/packages/schema/README.md @@ -77,6 +77,12 @@ compile(Block).errors({ kind: "video" }); // [{ path: "/kind", keyword: "anyOf", message: 'expected one of ["text","image"]' }] ``` +`Static` also types schemas built with TypeBox 1.x, read from TypeBox's type shapes (pi-schema does not import +TypeBox): `String`, `Number`, `Integer`, `Boolean`, `Null`, `Literal`, `Array`, `Object` (with `Type.Optional` and +`Type.Readonly` properties), `Union`, `Record`, `Unsafe`, `Unknown` and `Any` give TypeBox's own static type, so +`Static` is `{ path: string }`. Other TypeBox kinds (`Enum`, `Tuple`, +`Intersect`, `Cyclic`, ...) give `unknown` where they stand. Other JSON Schema objects give `unknown`. + ### Options ```typescript diff --git a/packages/schema/src/builders.ts b/packages/schema/src/builders.ts index 1f4e21a13..2dc61a520 100644 --- a/packages/schema/src/builders.ts +++ b/packages/schema/src/builders.ts @@ -2,14 +2,27 @@ // they accept, so `Static` is the TypeScript type a valid value has. The objects are ordinary data: // they serialize with JSON.stringify and compile with compile() like any other schema. +import type { TypeBoxStatic } from "./typebox-static.ts"; + declare const STATIC: unique symbol; declare const OPTIONAL_TYPE: unique symbol; /** A JSON Schema object whose valid values have type `T` (a type-level brand; nothing extra exists at run time). */ export type Schema = { readonly [keyword: string]: unknown } & { readonly [STATIC]?: T }; -/** The type of the values `S` accepts; `unknown` for schemas not built with these functions. */ -export type Static = S extends { readonly [STATIC]?: infer T } ? (unknown extends T ? unknown : T) : unknown; +// TypeBox is tested first and in its own branch: a builder brand's fallback through TypeBoxStatic would leave +// `Static` of a generic `S` (constrained to builder schemas) unresolvable to its constraint's types. +/** + * The type of the values `S` accepts: the type a builder recorded; for a TypeBox schema (`'~kind'` or `'~unsafe'`), + * TypeBox's static type for the common kinds (see `TypeBoxStatic`); else `unknown`. + */ +export type Static = S extends { readonly "~kind": unknown } | { readonly "~unsafe": unknown } + ? TypeBoxStatic + : S extends { readonly [STATIC]?: infer T } + ? unknown extends T + ? unknown + : T + : unknown; /** Annotations every builder accepts: they describe the schema and never change what it accepts. */ export interface Annotations { diff --git a/packages/schema/src/typebox-static.ts b/packages/schema/src/typebox-static.ts new file mode 100644 index 000000000..ce9f29b51 --- /dev/null +++ b/packages/schema/src/typebox-static.ts @@ -0,0 +1,76 @@ +// The static types of TypeBox-built schemas, read from the type shapes TypeBox 1.x declares (`'~kind'`, `'~optional'`, +// `'~readonly'`, `'~unsafe'`) without importing TypeBox, so `Static` also types tool arguments of TypeBox schemas. +// Only the common kinds are mapped; every other kind (Enum, Tuple, Intersect, Cyclic, Ref, ...) is `unknown`. + +import type { Static } from "./builders.ts"; + +/** The value type of a TypeBox schema type by its `'~kind'`, or its `Type.Unsafe` type; `unknown` for other kinds. */ +export type TypeBoxStatic = S extends { readonly "~kind": infer K } + ? KindStatic + : S extends { readonly "~unsafe": infer T } + ? T + : unknown; + +interface Primitives { + String: string; + Number: number; + Integer: number; + Boolean: boolean; + Null: null; + Unknown: unknown; + Any: any; +} + +type KindStatic = K extends keyof Primitives + ? Primitives[K] + : K extends "Literal" + ? S extends { readonly const: infer V } + ? V + : unknown + : K extends "Array" + ? S extends { readonly items: infer I } + ? Static[] + : unknown + : K extends "Object" + ? S extends { readonly properties: infer P } + ? ObjectStatic

    + : unknown + : K extends "Union" + ? S extends { readonly anyOf: readonly (infer B)[] } + ? Static + : unknown + : K extends "Record" + ? S extends { readonly patternProperties: infer P } + ? Record, Static> + : unknown + : unknown; + +type Optional = { readonly "~optional": true }; + +type Readonly = { readonly "~readonly": true }; + +// As TypeBox: no properties is `object`; `'~optional'` properties may be omitted, `'~readonly'` ones are readonly. +type ObjectStatic

    = keyof P extends never + ? object + : Simplify< + { -readonly [K in keyof P as P[K] extends Optional | Readonly ? never : K]: Static } & { + -readonly [K in keyof P as P[K] extends Optional ? (P[K] extends Readonly ? never : K) : never]?: Static< + P[K] + >; + } & { + readonly [K in keyof P as P[K] extends Readonly ? (P[K] extends Optional ? never : K) : never]: Static< + P[K] + >; + } & { + readonly [K in keyof P as P[K] extends Optional ? (P[K] extends Readonly ? K : never) : never]?: Static< + P[K] + >; + } + >; + +type Simplify = { [K in keyof T]: T[K] } & {}; + +// TypeBox's key patterns for integer and number keys; every other pattern types its keys as strings. +type RecordKey = Pattern extends "^-?(?:0|[1-9][0-9]*)$" | "^-?(?:0|[1-9][0-9]*)(?:\\.[0-9]+)?$" + ? number + : string; diff --git a/packages/schema/test/typebox-static.test.ts b/packages/schema/test/typebox-static.test.ts new file mode 100644 index 000000000..c62ec03f9 --- /dev/null +++ b/packages/schema/test/typebox-static.test.ts @@ -0,0 +1,72 @@ +// Static for TypeBox-built schemas: the common kinds give TypeBox's own static type, without pi-schema importing +// TypeBox; other kinds give `unknown`. Type-level tests (they pass when this file type-checks), against TypeBox 1.3.27. + +import { Type, type Static as TypeBoxOwn } from "typebox"; +import { describe, expectTypeOf, it } from "vitest"; +import { compile, type Static } from "../src/index.ts"; + +describe("Static of TypeBox schemas", () => { + it("maps the primitive kinds, literals and Unsafe", () => { + expectTypeOf>>().toEqualTypeOf(); + expectTypeOf>>().toEqualTypeOf(); + expectTypeOf>>().toEqualTypeOf(); + expectTypeOf>>().toEqualTypeOf(); + expectTypeOf>>().toEqualTypeOf(); + expectTypeOf>>().toEqualTypeOf(); + const any = Type.Any(); + expectTypeOf>().toBeAny(); + const literal = Type.Literal("add"); + expectTypeOf>().toEqualTypeOf<"add">(); + const unsafe = Type.Unsafe<"a" | "b">({ type: "string", enum: ["a", "b"] }); + expectTypeOf>().toEqualTypeOf<"a" | "b">(); + }); + + it("maps objects with optional and readonly properties as TypeBox does", () => { + const schema = Type.Object({ + path: Type.String(), + offset: Type.Optional(Type.Integer()), + id: Type.Readonly(Type.String()), + tag: Type.Readonly(Type.Optional(Type.Literal("x"))), + }); + type Expected = { path: string; offset?: number; readonly id: string; readonly tag?: "x" }; + expectTypeOf>().toEqualTypeOf(); + expectTypeOf>().toEqualTypeOf>(); + const empty = Type.Object({}); + expectTypeOf>().toEqualTypeOf>(); + }); + + it("maps arrays, unions and records, nested", () => { + const schema = Type.Object({ + edits: Type.Array(Type.Object({ oldText: Type.String(), newText: Type.String() })), + mode: Type.Union([Type.Literal("text"), Type.Literal("binary"), Type.Null()]), + env: Type.Record(Type.String(), Type.Union([Type.String(), Type.Number()])), + counts: Type.Record(Type.Integer(), Type.Boolean()), + }); + type Expected = { + edits: { oldText: string; newText: string }[]; + mode: "text" | "binary" | null; + env: Record; + counts: Record; + }; + expectTypeOf>().toEqualTypeOf(); + expectTypeOf>().toEqualTypeOf>(); + }); + + it("gives unknown for kinds it does not map", () => { + const enumeration = Type.Enum(["a", "b"]); + const tuple = Type.Tuple([Type.String()]); + const intersect = Type.Intersect([Type.Object({ a: Type.String() }), Type.Object({ b: Type.String() })]); + expectTypeOf>().toEqualTypeOf(); + expectTypeOf>().toEqualTypeOf(); + expectTypeOf>().toEqualTypeOf(); + // Inside a mapped kind, an unmapped kind is unknown only where it stands. + const object = Type.Object({ pair: Type.Tuple([Type.String()]), name: Type.String() }); + expectTypeOf>().toEqualTypeOf<{ pair: unknown; name: string }>(); + }); + + it("types compile() and parse() for TypeBox schemas", () => { + const schema = Type.Object({ n: Type.Number(), s: Type.Optional(Type.String()) }); + const result = compile(schema, { coerce: true }).parse({ n: "1" }); + if (result.ok) expectTypeOf(result.value).toEqualTypeOf<{ n: number; s?: string }>(); + }); +}); From 0f21a3319d83d3a17176cf0850f69b3d9cc3f5ca Mon Sep 17 00:00:00 2001 From: Armin Ronacher Date: Sun, 1 Nov 2026 14:40:17 +0100 Subject: [PATCH 118/175] fix(coding-agent,tui): reuse syntax highlighting and rendered Markdown blocks Codemode and write rows keep their last highlighting in a CodeHighlighter that checks theme and grammar changes itself. Markdown reuses the top-level blocks of its last render, held weakly, and assistant messages reuse their Markdown instances while streaming. Text.setText() keeps its wrapped lines for unchanged text. closes #10971 --- packages/coding-agent/CHANGELOG.md | 1 + .../src/core/tools/renderers/write.ts | 41 ++++-- .../src/extensions/codemode/renderer.ts | 14 +- .../components/assistant-message.ts | 85 +++++++---- .../components/markdown-transform.ts | 5 +- .../interactive/components/user-message.ts | 2 +- .../src/modes/interactive/theme/theme.ts | 64 ++++++--- .../coding-agent/src/utils/highlight-js.d.ts | 1 + .../src/utils/syntax-highlight.ts | 8 ++ .../test/assistant-message.test.ts | 34 ++++- .../test/code-highlighter.test.ts | 71 +++++++++ packages/tui/CHANGELOG.md | 2 + packages/tui/src/components/markdown.ts | 135 ++++++++++++------ packages/tui/src/components/text.ts | 2 + packages/tui/test/markdown.test.ts | 59 ++++++++ packages/tui/test/text.test.ts | 12 ++ 16 files changed, 420 insertions(+), 116 deletions(-) create mode 100644 packages/coding-agent/test/code-highlighter.test.ts create mode 100644 packages/tui/test/text.test.ts diff --git a/packages/coding-agent/CHANGELOG.md b/packages/coding-agent/CHANGELOG.md index ae5a6f324..d391a4f56 100644 --- a/packages/coding-agent/CHANGELOG.md +++ b/packages/coding-agent/CHANGELOG.md @@ -51,6 +51,7 @@ - Fixed `tool_search` and codemode `searchTools()` ranking a tool below tools that only mention its name in their descriptions, for example `peekaboo app` ranking `mcp__peekaboo__app` 14th. A tool matching more of the query's words now ranks above one matching fewer, a query word in a tool's name scores more than any number of mentions in its description, a tool whose whole name is in the query gets a large bonus, and long server instructions no longer lower the scores of that server's tools ([#11331](https://github.com/earendil-works/pi/issues/11331)) - Fixed `tool_search` and codemode `searchTools()` finding nothing for Chinese, Japanese, Thai, and other non-ASCII queries, because the tokenizer kept only ASCII letters and digits. Words in any script now match, text without spaces between words such as `搜索记录` is split into words, and canonically equivalent text such as `café` and `cafe\u0301` matches ([#10816](https://github.com/earendil-works/pi/issues/10816), [#11037](https://github.com/earendil-works/pi/pull/11037) by [@haoqixu](https://github.com/haoqixu)) - Fixed `pi update` stopping with `git clean -fdx failed with code 1` on Windows when another pi session had a git extension's native module or executable open. Windows cannot delete such files, so the update left the new code with the old dependencies. pi now warns, installs the dependencies anyway, and retries the clean on the next update ([#11597](https://github.com/earendil-works/pi/issues/11597)) +- Fixed tool rows and streamed answers highlighting the same code again on every redraw. A running codemode row highlighted its whole script on every progress update and timer tick, a complete `write` row its whole file on every update and expand toggle, a streamed `write` its first 50 lines on every delta, and a streamed answer every earlier code block on every delta. Rows now reuse their last highlighting until the code, theme, or loaded grammars change, and assistant messages keep their rendered Markdown blocks while streaming ([#10971](https://github.com/earendil-works/pi/issues/10971)) ### Removed diff --git a/packages/coding-agent/src/core/tools/renderers/write.ts b/packages/coding-agent/src/core/tools/renderers/write.ts index 7c8f32636..8e2338a7d 100644 --- a/packages/coding-agent/src/core/tools/renderers/write.ts +++ b/packages/coding-agent/src/core/tools/renderers/write.ts @@ -8,7 +8,12 @@ import { Container, Text } from "@earendil-works/pi-tui"; import { keyHint } from "../../../modes/interactive/components/keybinding-hints.ts"; -import { getLanguageFromPath, highlightCode, type Theme } from "../../../modes/interactive/theme/theme.ts"; +import { + CodeHighlighter, + getLanguageFromPath, + highlightCode, + type Theme, +} from "../../../modes/interactive/theme/theme.ts"; import type { ToolDefinition, ToolRenderResultOptions } from "../../extensions/types.ts"; import { normalizeDisplayText, renderToolPath, replaceTabs, str } from "../render-utils.ts"; @@ -20,7 +25,12 @@ type WriteHighlightCache = { highlightedLines: string[]; }; class WriteCallRenderComponent extends Text { + // Line by line highlighting of a call whose content is still streaming. cache?: WriteHighlightCache; + // A streamed call highlights its first lines on every delta, which stop changing once the file is longer, and a + // complete call highlights the whole file on every update and expand toggle. Either way the row reuses the last + // highlighting; it also holds the only copy of a complete call's displayed source. + readonly highlighter = new CodeHighlighter(); constructor() { super("", 0, 0); @@ -31,11 +41,11 @@ function highlightSingleLine(line: string, lang: string): string { const highlighted = highlightCode(line, lang); return highlighted[0] ?? ""; } -function refreshWriteHighlightPrefix(cache: WriteHighlightCache): void { +function refreshWriteHighlightPrefix(cache: WriteHighlightCache, highlighter: CodeHighlighter): void { const prefixCount = Math.min(WRITE_PARTIAL_FULL_HIGHLIGHT_LINES, cache.normalizedLines.length); if (prefixCount === 0) return; const prefixSource = cache.normalizedLines.slice(0, prefixCount).join("\n"); - const prefixHighlighted = highlightCode(prefixSource, cache.lang); + const prefixHighlighted = highlighter.highlight(prefixSource, cache.lang); for (let i = 0; i < prefixCount; i++) { cache.highlightedLines[i] = prefixHighlighted[i] ?? highlightSingleLine(cache.normalizedLines[i] ?? "", cache.lang); @@ -58,12 +68,14 @@ function updateWriteHighlightCacheIncremental( cache: WriteHighlightCache | undefined, rawPath: string | null, fileContent: string, + highlighter: CodeHighlighter, ): WriteHighlightCache | undefined { const lang = rawPath ? getLanguageFromPath(rawPath) : undefined; if (!lang) return undefined; if (!cache) return rebuildWriteHighlightCacheFull(rawPath, fileContent); - if (cache.lang !== lang || cache.rawPath !== rawPath) return rebuildWriteHighlightCacheFull(rawPath, fileContent); - if (!fileContent.startsWith(cache.rawContent)) return rebuildWriteHighlightCacheFull(rawPath, fileContent); + if (cache.lang !== lang || cache.rawPath !== rawPath || !fileContent.startsWith(cache.rawContent)) { + return rebuildWriteHighlightCacheFull(rawPath, fileContent); + } if (fileContent.length === cache.rawContent.length) return cache; const deltaRaw = fileContent.slice(cache.rawContent.length); @@ -83,10 +95,10 @@ function updateWriteHighlightCacheIncremental( cache.normalizedLines.push(segments[i]); cache.highlightedLines.push(highlightSingleLine(segments[i], cache.lang)); } - refreshWriteHighlightPrefix(cache); + refreshWriteHighlightPrefix(cache, highlighter); return cache; } -function trimTrailingEmptyLines(lines: string[]): string[] { +function trimTrailingEmptyLines(lines: readonly string[]): string[] { let end = lines.length; while (end > 0 && lines[end - 1] === "") { end--; @@ -98,6 +110,7 @@ function formatWriteCall( options: ToolRenderResultOptions, theme: Theme, cache: WriteHighlightCache | undefined, + highlighter: CodeHighlighter, cwd: string, ): string { const rawPath = str(args?.file_path ?? args?.path); @@ -110,7 +123,7 @@ function formatWriteCall( } else if (fileContent) { const lang = rawPath ? getLanguageFromPath(rawPath) : undefined; const renderedLines = lang - ? (cache?.highlightedLines ?? highlightCode(replaceTabs(normalizeDisplayText(fileContent)), lang)) + ? (cache?.highlightedLines ?? highlighter.highlight(replaceTabs(normalizeDisplayText(fileContent)), lang)) : normalizeDisplayText(fileContent).split("\n"); const lines = trimTrailingEmptyLines(renderedLines); const totalLines = lines.length; @@ -149,19 +162,17 @@ export const writeRenderers: Pick, "renderCall" | "rend const fileContent = str(renderArgs?.content); const component = (context.lastComponent as WriteCallRenderComponent | undefined) ?? new WriteCallRenderComponent(); - if (fileContent !== null) { - component.cache = context.argsComplete - ? rebuildWriteHighlightCacheFull(rawPath, fileContent) - : updateWriteHighlightCacheIncremental(component.cache, rawPath, fileContent); - } else { - component.cache = undefined; - } + component.cache = + fileContent !== null && !context.argsComplete + ? updateWriteHighlightCacheIncremental(component.cache, rawPath, fileContent, component.highlighter) + : undefined; component.setText( formatWriteCall( renderArgs, { expanded: context.expanded, isPartial: context.isPartial }, theme, component.cache, + component.highlighter, context.cwd, ), ); diff --git a/packages/coding-agent/src/extensions/codemode/renderer.ts b/packages/coding-agent/src/extensions/codemode/renderer.ts index 8d2937351..660803e68 100644 --- a/packages/coding-agent/src/extensions/codemode/renderer.ts +++ b/packages/coding-agent/src/extensions/codemode/renderer.ts @@ -12,7 +12,7 @@ import type { ToolDefinition } from "../../core/extensions/types.ts"; import { getTextOutput, replaceTabs, startToolTimer, str, toolTimerText } from "../../core/tools/render-utils.ts"; import { keyHint } from "../../modes/interactive/components/keybinding-hints.ts"; import { VisualLinePreview } from "../../modes/interactive/components/visual-truncate.ts"; -import { highlightCode, type Theme } from "../../modes/interactive/theme/theme.ts"; +import { CodeHighlighter, type Theme } from "../../modes/interactive/theme/theme.ts"; import type { CodemodeNestedCall, CodemodeToolDetails } from "./tool.ts"; const CODE_PREVIEW_LINES = 10; @@ -21,6 +21,12 @@ const OUTPUT_PREVIEW_LINES = 5; const COLLAPSED_ARGS_CHARS = 80; const SCRIPT_HEADER = /^Script (completed|failed)\nWall time [\d.]+ seconds\nOutput:\n$/; +/** Renderer state of a codemode row. */ +interface CodemodeRenderState { + // Progress, timer ticks and expanding redraw the call, so the row keeps the script's highlighting. + scriptHighlighter?: CodeHighlighter; +} + function expandHint(theme: Theme, hidden: number, noun: string): string { return `${theme.fg("muted", `... (${hidden} more ${noun},`)} ${keyHint("app.tools.expand", "to expand")}${theme.fg("muted", ")")}`; } @@ -79,7 +85,11 @@ export const codemodeRenderers: Pick< } component.addChild(new Text(title, 0, 0)); if (code) { - const highlighted = highlightCode(replaceTabs(code.replace(/\r/g, "").trimEnd()), "javascript").join("\n"); + const state = context.state as CodemodeRenderState; + state.scriptHighlighter ??= new CodeHighlighter(); + const highlighted = state.scriptHighlighter + .highlight(replaceTabs(code.replace(/\r/g, "").trimEnd()), "javascript") + .join("\n"); component.addChild( context.expanded ? new Text(highlighted, 0, 0) diff --git a/packages/coding-agent/src/modes/interactive/components/assistant-message.ts b/packages/coding-agent/src/modes/interactive/components/assistant-message.ts index 67bb71a3f..7cb296031 100644 --- a/packages/coding-agent/src/modes/interactive/components/assistant-message.ts +++ b/packages/coding-agent/src/modes/interactive/components/assistant-message.ts @@ -16,6 +16,15 @@ const OSC133_ZONE_START = "\x1b]133;A\x07"; const OSC133_ZONE_END = "\x1b]133;B\x07"; const OSC133_ZONE_FINAL = "\x1b]133;C\x07"; +/** + * A Markdown block of the last update. The next update reuses it at the same position for the same kind of content, so + * a streamed answer keeps the blocks it already rendered, also once the stream ends. + */ +interface MarkdownSlot { + kind: "text" | "thinking"; + markdown: Markdown; +} + /** * Component that renders a complete assistant message */ @@ -32,6 +41,7 @@ export class AssistantMessageComponent extends Container { private messageId?: string; private timestamp?: string; private thinkingVisibilityOverrides = new Map(); + private markdownSlots: MarkdownSlot[] = []; constructor( message?: AssistantMessage, @@ -86,6 +96,8 @@ export class AssistantMessageComponent extends Container { setOutputPad(padding: number): void { this.outputPad = padding; + // Markdown takes its padding at construction. + this.markdownSlots = []; if (this.lastMessage) { this.updateContent(this.lastMessage); } @@ -108,6 +120,15 @@ export class AssistantMessageComponent extends Container { // Clear content container this.contentContainer.clear(); + const previousSlots = this.markdownSlots; + this.markdownSlots = []; + const markdownBlock = (kind: MarkdownSlot["kind"], text: string, create: () => Markdown): Markdown => { + const slot = previousSlots[this.markdownSlots.length]; + const markdown = slot?.kind === kind ? slot.markdown : create(); + markdown.setText(text); + this.markdownSlots.push({ kind, markdown }); + return markdown; + }; // Spacers are added lazily, right before the next visible content block. This puts a blank // line before the first visible block and after each visible thinking run that is followed @@ -128,16 +149,22 @@ export class AssistantMessageComponent extends Container { if (content.type === "text" && content.text.trim()) { // Assistant text messages with no background - trim the text // Set paddingY=0 to avoid extra spacing before tool executions + const text = content.text.trim(); addVisibleChild( - new Markdown(content.text.trim(), this.outputPad, 0, this.markdownTheme, undefined, { - transform: createMarkdownTransform( - "assistant", - this.isStreaming, - this.markdownTransformers, - this.messageId, - this.timestamp, - ), - }), + markdownBlock( + "text", + text, + () => + new Markdown(text, this.outputPad, 0, this.markdownTheme, undefined, { + transform: createMarkdownTransform( + "assistant", + () => this.isStreaming, + this.markdownTransformers, + this.messageId, + this.timestamp, + ), + }), + ), ); } else if (content.type === "thinking") { const thinkingBlocks: string[] = []; @@ -163,26 +190,32 @@ export class AssistantMessageComponent extends Container { if (hidden && !this.hiddenThinkingLabel.trim()) { continue; } + const thinking = thinkingBlocks.join("\n\n"); const thinkingComponent = hidden ? new Text(theme.italic(theme.fg("thinkingText", this.hiddenThinkingLabel)), this.outputPad, 0) - : new Markdown( - thinkingBlocks.join("\n\n"), - this.outputPad, - 0, - this.markdownTheme, - { - color: (text: string) => theme.fg("thinkingText", text), - italic: true, - }, - { - transform: createMarkdownTransform( - "assistant-thinking", - this.isStreaming, - this.markdownTransformers, - this.messageId, - this.timestamp, + : markdownBlock( + "thinking", + thinking, + () => + new Markdown( + thinking, + this.outputPad, + 0, + this.markdownTheme, + { + color: (text: string) => theme.fg("thinkingText", text), + italic: true, + }, + { + transform: createMarkdownTransform( + "assistant-thinking", + () => this.isStreaming, + this.markdownTransformers, + this.messageId, + this.timestamp, + ), + }, ), - }, ); addVisibleChild( new MouseRegion(thinkingComponent, (event) => { diff --git a/packages/coding-agent/src/modes/interactive/components/markdown-transform.ts b/packages/coding-agent/src/modes/interactive/components/markdown-transform.ts index 4ff7685c2..58bae9094 100644 --- a/packages/coding-agent/src/modes/interactive/components/markdown-transform.ts +++ b/packages/coding-agent/src/modes/interactive/components/markdown-transform.ts @@ -1,8 +1,9 @@ import type { MarkdownTransformContext, MarkdownTransformer } from "../../../core/extensions/types.ts"; +/** Applies extension Markdown transformers. `isStreaming` is read per transform, so a reused component sees a stream end. */ export function createMarkdownTransform( messageType: MarkdownTransformContext["messageType"], - isStreaming: boolean, + isStreaming: () => boolean, transformers: readonly MarkdownTransformer[], messageId?: string, timestamp?: string, @@ -10,7 +11,7 @@ export function createMarkdownTransform( return (markdown, availableWidth) => applyMarkdownTransformers( markdown, - { messageType, isStreaming, availableWidth, messageId, timestamp }, + { messageType, isStreaming: isStreaming(), availableWidth, messageId, timestamp }, transformers, ); } diff --git a/packages/coding-agent/src/modes/interactive/components/user-message.ts b/packages/coding-agent/src/modes/interactive/components/user-message.ts index d8ca373cc..e313e0242 100644 --- a/packages/coding-agent/src/modes/interactive/components/user-message.ts +++ b/packages/coding-agent/src/modes/interactive/components/user-message.ts @@ -60,7 +60,7 @@ export class UserMessageComponent extends Container { preserveBackslashEscapes: true, transform: createMarkdownTransform( "user", - false, + () => false, this.markdownTransformers, this.messageId, this.timestamp, diff --git a/packages/coding-agent/src/modes/interactive/theme/theme.ts b/packages/coding-agent/src/modes/interactive/theme/theme.ts index 876ba9e86..922b23d9f 100644 --- a/packages/coding-agent/src/modes/interactive/theme/theme.ts +++ b/packages/coding-agent/src/modes/interactive/theme/theme.ts @@ -26,7 +26,7 @@ import chalk from "chalk"; import { getCustomThemesDir, getThemesDir } from "../../../config.ts"; import type { SourceInfo } from "../../../core/source-info.ts"; import { closeWatcher, watchWithErrorHandler } from "../../../utils/fs-watch.ts"; -import { highlight, supportsLanguage } from "../../../utils/syntax-highlight.ts"; +import { getHighlightLanguageCount, highlight, supportsLanguage } from "../../../utils/syntax-highlight.ts"; import { stripBom } from "../../../utils/text.ts"; import { generateSystemThemeColors, SYSTEM_THEME_NAME, terminalAppearance } from "./system-theme.ts"; import { @@ -658,6 +658,11 @@ export const theme: Theme = new Proxy({} as Theme, { }, }); +/** The theme instance behind the `theme` proxy, whose own identity never changes. */ +function getGlobalTheme(): Theme | undefined { + return (globalThis as Record)[THEME_KEY]; +} + function setGlobalTheme(t: Theme): void { (globalThis as Record)[THEME_KEY] = t; (globalThis as Record)[THEME_KEY_OLD] = t; @@ -944,7 +949,41 @@ export function highlightCode(code: string, lang?: string): string[] { try { return highlight(code, opts).split("\n"); } catch { - return code.split("\n"); + return code.split("\n").map((line) => theme.fg("mdCodeBlock", line)); + } +} + +/** + * Highlights code for one owner, such as a tool row, and reuses the last result while the code, the language, the + * theme and the registered grammars stay the same. Each owner keeps its own highlighter, so cached results live as long + * as the code they show instead of in a process-wide cache. + */ +export class CodeHighlighter { + private last?: { + code: string; + lang: string | undefined; + theme: Theme | undefined; + languageCount: number; + lines: readonly string[]; + }; + + /** Like {@link highlightCode}. The result is shared with later calls, so callers must not modify it. */ + highlight(code: string, lang: string | undefined): readonly string[] { + // Renderers cannot tell a theme change from any other redraw, so the highlighter checks what its result depends on. + const currentTheme = getGlobalTheme(); + const languageCount = getHighlightLanguageCount(); + const last = this.last; + if ( + last?.code === code && + last.lang === lang && + last.theme === currentTheme && + last.languageCount === languageCount + ) { + return last.lines; + } + const lines = highlightCode(code, lang); + this.last = { code, lang, theme: currentTheme, languageCount, lines }; + return lines; } } @@ -1035,26 +1074,7 @@ export function getMarkdownTheme(): MarkdownTheme { italic: (text: string) => theme.italic(text), underline: (text: string) => theme.underline(text), strikethrough: (text: string) => theme.strikethrough(text), - highlightCode: (code: string, lang?: string): string[] => { - // Validate language before highlighting to avoid stderr spam from cli-highlight - const validLang = lang && supportsLanguage(lang) ? lang : undefined; - // Skip highlighting when no valid language is specified. cli-highlight's - // auto-detection is unreliable and can misidentify prose as AppleScript, - // LiveCodeServer, etc., coloring random English words as keywords. - if (!validLang) { - return code.split("\n").map((line) => theme.fg("mdCodeBlock", line)); - } - const opts = { - language: validLang, - ignoreIllegals: true, - theme: getCliHighlightTheme(theme), - }; - try { - return highlight(code, opts).split("\n"); - } catch { - return code.split("\n").map((line) => theme.fg("mdCodeBlock", line)); - } - }, + highlightCode, }; } diff --git a/packages/coding-agent/src/utils/highlight-js.d.ts b/packages/coding-agent/src/utils/highlight-js.d.ts index d4b71174e..c4c951fce 100644 --- a/packages/coding-agent/src/utils/highlight-js.d.ts +++ b/packages/coding-agent/src/utils/highlight-js.d.ts @@ -17,6 +17,7 @@ interface HighlightJsApi { highlight(code: string, options: HighlightJsOptions): HighlightJsResult; highlightAuto(code: string, languageSubset?: string[]): HighlightJsResult; getLanguage(name: string): HighlightJsLanguageDefinition | undefined; + listLanguages(): string[]; registerLanguage(name: string, language: HighlightJsLanguageFactory): void; } diff --git a/packages/coding-agent/src/utils/syntax-highlight.ts b/packages/coding-agent/src/utils/syntax-highlight.ts index 1ee1986b9..91375bb01 100644 --- a/packages/coding-agent/src/utils/syntax-highlight.ts +++ b/packages/coding-agent/src/utils/syntax-highlight.ts @@ -217,3 +217,11 @@ export function highlight(code: string, options: HighlightOptions = {}): string export function supportsLanguage(name: string): boolean { return hljs.getLanguage(name) !== undefined; } + +/** + * Number of registered grammars. It grows when the deferred grammars load, which can change the highlighting of + * code that embeds another language, such as HTML in a JavaScript template literal. + */ +export function getHighlightLanguageCount(): number { + return hljs.listLanguages().length; +} diff --git a/packages/coding-agent/test/assistant-message.test.ts b/packages/coding-agent/test/assistant-message.test.ts index 42f6e496d..2cf368915 100644 --- a/packages/coding-agent/test/assistant-message.test.ts +++ b/packages/coding-agent/test/assistant-message.test.ts @@ -1,10 +1,11 @@ import type { AssistantMessage } from "@earendil-works/pi-ai"; import type { TuiMouseEvent } from "@earendil-works/pi-tui"; -import { describe, expect, test } from "vitest"; +import { afterEach, describe, expect, test, vi } from "vitest"; import { AssistantMessageComponent } from "../src/modes/interactive/components/assistant-message.ts"; import { UserMessageComponent } from "../src/modes/interactive/components/user-message.ts"; import { initTheme } from "../src/modes/interactive/theme/theme.ts"; import { stripAnsi } from "../src/utils/ansi.ts"; +import * as syntaxHighlight from "../src/utils/syntax-highlight.ts"; const OSC133_ZONE_START = "\x1b]133;A\x07"; const OSC133_ZONE_END = "\x1b]133;B\x07"; @@ -34,6 +35,37 @@ function createAssistantMessage( } describe("AssistantMessageComponent", () => { + afterEach(() => vi.restoreAllMocks()); + + test("reuses its Markdown across streaming and its end, and renders like a new component after a theme change", () => { + initTheme("dark"); + const highlight = vi.spyOn(syntaxHighlight, "highlight"); + const streaming: boolean[] = []; + const code = "```ts\nconst a = 1;\n```"; + const content = (text: string): AssistantMessage["content"] => [ + { type: "thinking", thinking: "Thinking with **bold** text" }, + { type: "text", text }, + ]; + const component = new AssistantMessageComponent(undefined, false, undefined, "Thinking...", 1, [ + (markdown, { isStreaming }) => { + streaming.push(isStreaming); + return markdown; + }, + ]); + component.updateContent(createAssistantMessage(content(code)), true); + component.render(80); + const message = createAssistantMessage(content(`${code}\n\nThe end.`)); + component.updateContent(message, false); + component.render(80); + expect(highlight).toHaveBeenCalledTimes(1); + expect(streaming.at(-1)).toBe(false); + + initTheme("light"); + component.invalidate(); + expect(component.render(80)).toEqual(new AssistantMessageComponent(message).render(80)); + initTheme("dark"); + }); + test("adds OSC 133 zone markers to assistant messages without tool calls", () => { initTheme("dark"); diff --git a/packages/coding-agent/test/code-highlighter.test.ts b/packages/coding-agent/test/code-highlighter.test.ts new file mode 100644 index 000000000..b0d0433b4 --- /dev/null +++ b/packages/coding-agent/test/code-highlighter.test.ts @@ -0,0 +1,71 @@ +import type { TUI } from "@earendil-works/pi-tui"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { writeRenderers } from "../src/core/tools/renderers/write.ts"; +import { codemodeRenderers } from "../src/extensions/codemode/renderer.ts"; +import { ToolExecutionComponent } from "../src/modes/interactive/components/tool-execution.ts"; +import { CodeHighlighter, highlightCode, initTheme } from "../src/modes/interactive/theme/theme.ts"; +import * as syntaxHighlight from "../src/utils/syntax-highlight.ts"; + +const tui = { requestRender: () => {} } as unknown as TUI; + +describe("CodeHighlighter", () => { + beforeEach(() => initTheme("dark")); + afterEach(() => { + vi.restoreAllMocks(); + vi.useRealTimers(); + }); + + it("reuses its last result until the code, the theme or the grammars change", async () => { + const code = 'const page = html`

    ok
    `;'; + const highlighter = new CodeHighlighter(); + const first = highlighter.highlight(code, "javascript"); + expect(highlighter.highlight(code, "javascript")).toBe(first); + expect(highlighter.highlight("const a = 1;", "javascript")).not.toBe(first); + + initTheme("light"); + expect(highlighter.highlight(code, "javascript")).toEqual(highlightCode(code, "javascript")); + const light = highlighter.highlight(code, "javascript"); + await syntaxHighlight.loadAllHighlightLanguages(); + expect(highlighter.highlight(code, "javascript")).not.toEqual(light); + }); + + it("codemode highlights the script once across progress, timer ticks, expanding and completion", () => { + vi.useFakeTimers(); + const highlight = vi.spyOn(syntaxHighlight, "highlight"); + const row = new ToolExecutionComponent( + "codemode", + "c", + { code: "await tools.read({});" }, + {}, + codemodeRenderers, + tui, + "/", + ); + row.setArgsComplete(); + row.markExecutionStarted(); + row.updateResult({ content: [], details: { calls: [] }, isError: false }, true); + vi.advanceTimersByTime(2_000); + row.setExpanded(true); + row.updateResult({ content: [{ type: "text", text: "done" }], details: { calls: [] }, isError: false }, false); + row.render(80); + expect(highlight).toHaveBeenCalledTimes(1); + }); + + it("write highlights unchanged first lines and a complete file only once", () => { + const content = Array.from({ length: 60 }, (_, i) => `const v${i} = ${i};`).join("\n"); + const row = new ToolExecutionComponent("write", "w", { path: "a.ts", content }, {}, writeRenderers, tui, "/"); + row.updateArgs({ path: "a.ts", content: `${content} //` }); + row.render(80); + const highlight = vi.spyOn(syntaxHighlight, "highlight"); + // Only the changed last line is highlighted again. + row.updateArgs({ path: "a.ts", content: `${content} // more` }); + row.render(80); + expect(highlight).toHaveBeenCalledTimes(1); + + row.setArgsComplete(); + row.setExpanded(true); + row.setExpanded(false); + row.render(80); + expect(highlight).toHaveBeenCalledTimes(2); + }); +}); diff --git a/packages/tui/CHANGELOG.md b/packages/tui/CHANGELOG.md index d03c3cfc8..1b13cf0a7 100644 --- a/packages/tui/CHANGELOG.md +++ b/packages/tui/CHANGELOG.md @@ -18,6 +18,8 @@ - Fixed display math wider than the content width being wrapped row by row, which split fractions and sums apart. It now falls back to the raw LaTeX source ([#11483](https://github.com/earendil-works/pi/issues/11483)) - Fixed `tui.select.pageUp` and `tui.select.pageDown` being ignored by `SelectList`, `SettingsList`, and editor autocomplete ([#7629](https://github.com/earendil-works/pi/issues/7629), based on [#7680](https://github.com/earendil-works/pi/pull/7680) by [@muyiyr](https://github.com/muyiyr)) - Fixed `wrapTextWithAnsi()` starting the next line with a space when a styled word ended exactly at the wrap width, and strikethrough continuing into the right padding of wrapped lines ([#10925](https://github.com/earendil-works/pi/issues/10925)) +- Fixed `Markdown` styling, highlighting, and wrapping every block on each render. It now reuses the top-level blocks of its last render, so a streaming message renders only its changed last block, and a resize reuses highlighted code. Reused blocks are held weakly and dropped by `invalidate()`, which a theme change must call because `MarkdownTheme.highlightCode` results are reused until then ([#10971](https://github.com/earendil-works/pi/issues/10971)) +- Fixed `Text.setText()` with unchanged text dropping the wrapped lines, so renderers that set the same text on every update wrapped it again ([#10971](https://github.com/earendil-works/pi/issues/10971)) ## [1.4.0] - 2026-10-21 diff --git a/packages/tui/src/components/markdown.ts b/packages/tui/src/components/markdown.ts index bacfe29f8..a21f75181 100644 --- a/packages/tui/src/components/markdown.ts +++ b/packages/tui/src/components/markdown.ts @@ -1,4 +1,4 @@ -import { Marked, type Token, Tokenizer, type TokenizerExtension, type Tokens } from "marked"; +import { Marked, type Token, Tokenizer, type TokenizerExtension, type Tokens, type TokensList } from "marked"; import markedCjkFriendly from "marked-cjk-friendly"; import { renderLatex } from "../latex.ts"; import { getCapabilities, hyperlink, isImageLine } from "../terminal-image.ts"; @@ -244,6 +244,7 @@ export interface MarkdownTheme { italic: (text: string) => string; strikethrough: (text: string) => string; underline: (text: string) => string; + /** Highlighted lines of a code block. A Markdown component reuses the result for the same code until invalidate(). */ highlightCode?: (code: string, lang?: string) => string[]; /** Prefix applied to each rendered code block line (default: " ") */ codeBlockIndent?: string; @@ -265,6 +266,16 @@ interface InlineStyleContext { stylePrefix: string; } +/** A top-level block of the last render, keyed by its source, reused by the next render. */ +interface RenderedBlock { + /** Width and type of the following block (which decides the spacing) that the lines were rendered for. */ + width: number; + nextTokenType: string | undefined; + lines: string[]; + /** Highlighted lines of the block's code, keyed by language and source; they survive a resize or new spacing. */ + highlightedCode: Map; +} + export class Markdown implements Component { private text: string; private paddingX: number; // Left/right padding @@ -281,7 +292,14 @@ export class Markdown implements Component { // Parsed tokens depend only on the source, so they survive theme and width invalidation. Held weakly: a token tree is // about ten times the size of its source, and every message of a long transcript keeps a Markdown component. The // tokens survive a burst of re-renders, such as a theme preview, and are collected afterwards. - private cachedTokens?: WeakRef<{ source: string; tokens: Token[] }>; + private cachedTokens?: WeakRef<{ source: string; tokens: TokensList }>; + // The last render's top-level blocks. While a message streams only its last block changes, so a render styles and + // wraps only that block. Each render keeps only the blocks it used, which drops a streamed block's earlier versions. + // Held weakly like the tokens: the blocks repeat the source in their keys and the code in their highlighting, and + // only a burst of renders, such as streaming or a resize, needs them. + private blocks?: WeakRef>; + // Highlighted code of the block being rendered, only set while renderBlock runs. + private blockHighlightedCode?: Map; constructor( text: string, @@ -301,10 +319,17 @@ export class Markdown implements Component { setText(text: string): void { this.text = text; - this.invalidate(); + this.clearRenderCache(); } invalidate(): void { + this.clearRenderCache(); + // The theme may have changed, which changes every styled line and the default style's prefix. + this.blocks = undefined; + this.defaultStylePrefix = undefined; + } + + private clearRenderCache(): void { this.cachedText = undefined; this.cachedWidth = undefined; this.cachedLines = undefined; @@ -322,6 +347,7 @@ export class Markdown implements Component { // Don't render anything if there's no actual text if (!text || text.trim() === "") { + this.blocks = undefined; const result: string[] = []; // Update cache this.cachedText = this.text; @@ -342,65 +368,39 @@ export class Markdown implements Component { this.cachedTokens = new WeakRef({ source: normalizedText, tokens }); } - // Convert tokens to styled terminal output - const renderedLines: string[] = []; - + // A reference link resolves through a definition anywhere in the text, so blocks with the same source can render + // differently; such texts are rare in practice and render without reusing blocks. + const reuseBlocks = Object.keys(tokens.links).length === 0; + const previousBlocks = this.blocks?.deref(); + const blocks = new Map(); + this.blocks = new WeakRef(blocks); + const contentLines: string[] = []; for (let i = 0; i < tokens.length; i++) { const token = tokens[i]; - const nextToken = tokens[i + 1]; - const tokenLines = this.renderToken(token, contentWidth, nextToken?.type); - for (const tokenLine of tokenLines) { - renderedLines.push(tokenLine); + const nextTokenType = tokens[i + 1]?.type; + const reused = reuseBlocks ? (blocks.get(token.raw) ?? previousBlocks?.get(token.raw)) : undefined; + const block = reused ?? { width: -1, nextTokenType, lines: [], highlightedCode: new Map() }; + if (block.width !== width || block.nextTokenType !== nextTokenType) { + this.renderBlock(block, token, nextTokenType, width, contentWidth); } - } - - // Wrap lines (NO padding, NO background yet) - const wrappedLines: string[] = []; - for (const line of renderedLines) { - if (isImageLine(line)) { - wrappedLines.push(line); - } else { - for (const wrappedLine of wrapTextWithAnsi(line, contentWidth)) { - wrappedLines.push(wrappedLine); - } - } - } - - // Add margins and background to each wrapped line - const leftMargin = " ".repeat(this.paddingX); - const rightMargin = " ".repeat(this.paddingX); - const bgFn = this.defaultTextStyle?.bgColor; - const contentLines: string[] = []; - - for (const line of wrappedLines) { - if (isImageLine(line)) { + if (reuseBlocks) blocks.set(token.raw, block); + for (const line of block.lines) { contentLines.push(line); - continue; - } - - const lineWithMargins = leftMargin + line + rightMargin; - - if (bgFn) { - contentLines.push(applyBackgroundToLine(lineWithMargins, width, bgFn)); - } else { - // No background - just pad to width - const visibleLen = visibleWidth(lineWithMargins); - const paddingNeeded = Math.max(0, width - visibleLen); - contentLines.push(lineWithMargins + " ".repeat(paddingNeeded)); } } // Add top/bottom padding (empty lines) + const bgFn = this.defaultTextStyle?.bgColor; const emptyLine = " ".repeat(width); const emptyLines: string[] = []; for (let i = 0; i < this.paddingY; i++) { const line = bgFn ? applyBackgroundToLine(emptyLine, width, bgFn) : emptyLine; emptyLines.push(line); } + flattenLines(emptyLines); // Combine top padding, content, and bottom padding const result = emptyLines.concat(contentLines, emptyLines); - flattenLines(result); // Update cache this.cachedText = this.text; @@ -410,6 +410,42 @@ export class Markdown implements Component { return result.length > 0 ? result : [""]; } + /** Render one top-level block to output lines: wrapped, with margins, and padded or backgrounded to the width. */ + private renderBlock( + block: RenderedBlock, + token: Token, + nextTokenType: string | undefined, + width: number, + contentWidth: number, + ): void { + this.blockHighlightedCode = block.highlightedCode; + const leftMargin = " ".repeat(this.paddingX); + const rightMargin = " ".repeat(this.paddingX); + const bgFn = this.defaultTextStyle?.bgColor; + const lines: string[] = []; + for (const line of this.renderToken(token, contentWidth, nextTokenType)) { + if (isImageLine(line)) { + lines.push(line); + continue; + } + for (const wrappedLine of wrapTextWithAnsi(line, contentWidth)) { + const lineWithMargins = leftMargin + wrappedLine + rightMargin; + if (bgFn) { + lines.push(applyBackgroundToLine(lineWithMargins, width, bgFn)); + } else { + // No background - just pad to width + const paddingNeeded = Math.max(0, width - visibleWidth(lineWithMargins)); + lines.push(lineWithMargins + " ".repeat(paddingNeeded)); + } + } + } + this.blockHighlightedCode = undefined; + flattenLines(lines); + block.lines = lines; + block.width = width; + block.nextTokenType = nextTokenType; + } + /** * Apply default text style to a string. * This is the base styling applied to all text content. @@ -567,8 +603,13 @@ export class Markdown implements Component { const indent = this.theme.codeBlockIndent ?? " "; lines.push(this.theme.codeBlockBorder(`\`\`\`${token.lang || ""}`)); if (this.theme.highlightCode) { - const highlightedLines = this.theme.highlightCode(token.text, token.lang); - for (const hlLine of highlightedLines) { + const key = `${token.lang ?? ""}\0${token.text}`; + let highlighted = this.blockHighlightedCode?.get(key); + if (!highlighted) { + highlighted = this.theme.highlightCode(token.text, token.lang); + this.blockHighlightedCode?.set(key, highlighted); + } + for (const hlLine of highlighted) { lines.push(`${indent}${hlLine}`); } } else { diff --git a/packages/tui/src/components/text.ts b/packages/tui/src/components/text.ts index 024748912..c9aea3d13 100644 --- a/packages/tui/src/components/text.ts +++ b/packages/tui/src/components/text.ts @@ -23,6 +23,8 @@ export class Text implements Component { } setText(text: string): void { + // Renderers often set the same text on every update; keep the wrapped lines then. + if (text === this.text) return; this.text = text; this.cachedText = undefined; this.cachedWidth = undefined; diff --git a/packages/tui/test/markdown.test.ts b/packages/tui/test/markdown.test.ts index c1615df66..e0f5b4dd5 100644 --- a/packages/tui/test/markdown.test.ts +++ b/packages/tui/test/markdown.test.ts @@ -1,5 +1,7 @@ import assert from "node:assert"; import { afterEach, describe, it } from "node:test"; +import { setFlagsFromString } from "node:v8"; +import { runInNewContext } from "node:vm"; import type { Terminal as XtermTerminalType } from "@xterm/headless"; import { Chalk } from "chalk"; import { Markdown, type MarkdownTheme } from "../src/components/markdown.ts"; @@ -65,6 +67,63 @@ describe("Markdown component", () => { }); }); + describe("Block reuse", () => { + afterEach(() => resetCapabilitiesCache()); + + function countingTheme(calls: string[]): MarkdownTheme { + return { + ...defaultMarkdownTheme, + highlightCode: (code, lang) => { + calls.push(`${lang}:${code}`); + return code.split("\n").map((line) => chalk.green(line)); + }, + }; + } + + it("highlights a code block once across streamed text and resizes, and again after invalidate", () => { + const calls: string[] = []; + const markdown = new Markdown("```ts\nx\n```", 0, 0, countingTheme(calls)); + markdown.render(80); + markdown.setText("```ts\nx\n```\n\nafter"); + markdown.render(40); + assert.deepStrictEqual(calls, ["ts:x"]); + markdown.invalidate(); + markdown.render(40); + assert.deepStrictEqual(calls, ["ts:x", "ts:x"]); + }); + + it("holds rendered blocks weakly, so a full garbage collection releases them", async () => { + setFlagsFromString("--expose-gc"); + const gc = runInNewContext("gc") as () => void; + const calls: string[] = []; + const markdown = new Markdown("```ts\nx\n```", 0, 0, countingTheme(calls)); + markdown.render(80); + // A WeakRef keeps its target alive until the current job ends. + await new Promise((resolve) => setImmediate(resolve)); + gc(); + markdown.render(40); + assert.deepStrictEqual(calls, ["ts:x", "ts:x"]); + }); + + it("renders a block again when the block after it changes its spacing", () => { + const markdown = new Markdown("# Title", 0, 0, defaultMarkdownTheme); + markdown.render(80); + markdown.setText("# Title\nfoo"); + assert.deepStrictEqual( + markdown.render(80).map((line) => stripAnsi(line).trim()), + ["Title", "", "foo"], + ); + }); + + it("follows changed reference link definitions", () => { + setCapabilities({ images: null, trueColor: false, hyperlinks: false }); + const markdown = new Markdown("[a][x]\n\n[x]: https://one.example", 0, 0, defaultMarkdownTheme); + markdown.render(80); + markdown.setText("[a][x]\n\n[x]: https://two.example"); + assert.ok(stripAnsi(markdown.render(80).join("\n")).includes("https://two.example")); + }); + }); + describe("Lists", () => { it("should render simple nested list", () => { const markdown = new Markdown( diff --git a/packages/tui/test/text.test.ts b/packages/tui/test/text.test.ts new file mode 100644 index 000000000..9c2e2bd61 --- /dev/null +++ b/packages/tui/test/text.test.ts @@ -0,0 +1,12 @@ +import assert from "node:assert"; +import { describe, it } from "node:test"; +import { Text } from "../src/components/text.ts"; + +describe("Text", () => { + it("keeps its wrapped lines when the same text is set again", () => { + const text = new Text("hello world", 0, 0); + const lines = text.render(5); + text.setText("hello world"); + assert.strictEqual(text.render(5), lines); + }); +}); From c09c98d7d2caf659b4bbe1e2427adb1bdac4b114 Mon Sep 17 00:00:00 2001 From: Armin Ronacher Date: Sun, 1 Nov 2026 14:47:36 +0100 Subject: [PATCH 119/175] fix(chord): admit a routed host even when the relay that spawned it is gone A relayed hold released (or its principal refused or revoked) while the route resolved made #resolve throw before adding the connection, so the spawned host's link was never read and its own upstream references failed unavailable/ready_timeout after readyTimeout. The connection is now added right after resolve; the checks only fail the relay. --- packages/chord/CHANGELOG.md | 1 + packages/chord/docs/spec.md | 2 +- packages/chord/src/host.ts | 10 ++++++---- packages/chord/test/links-review.test.ts | 23 +++++++++++++++++++++++ 4 files changed, 31 insertions(+), 5 deletions(-) diff --git a/packages/chord/CHANGELOG.md b/packages/chord/CHANGELOG.md index 4f50cc098..1a1382eb7 100644 --- a/packages/chord/CHANGELOG.md +++ b/packages/chord/CHANGELOG.md @@ -37,6 +37,7 @@ ### Fixed +- A host a route resolves is admitted even when the relayed reference that asked for it is released, its client disconnects, or its principal is refused or revoked while the route resolves. Before, the parent never read the new host's link: a spawned worker's own references to its parent (its `lifetime: "host"` objects' upstream references) waited until they failed `unavailable/ready_timeout`, reported as a possible reference cycle across hosts. - The Node SQLite driver keeps the 256 most recently used prepared statements per connection instead of every statement it ever prepared, so callers generating SQL of unbounded shapes (a query builder) no longer grow memory without bound. - Fixed append ops from the delta tracker and `diffRevisions()` keeping the whole string they were cut from alive (V8 represents a slice as a view of its source), so a delta chain, a watch backlog or a replica built from appends to a growing text held one copy of the text per commit ([#9265](https://github.com/earendil-works/pi/issues/9265)) diff --git a/packages/chord/docs/spec.md b/packages/chord/docs/spec.md index ca910d758..4c261cb54 100644 --- a/packages/chord/docs/spec.md +++ b/packages/chord/docs/spec.md @@ -279,7 +279,7 @@ The tests cover at least: lifetime/references/linger/forced disposal/gone; activ - **Two-version rule.** An object of a changed type created after a commit has only new code; a previous-code-version root calling it gets `superseded`. - **Validation.** Remote calls are validated at reference dispatch, in-host too: a direct binding against the provider's schema, a binding through an adapter against the consumer's. A provider validates each snapshot and reset of a remote state member once per (re)subscription, against the consumer's contract through an `accepts` adapter and the provider's otherwise; ops are validated only in dev mode. At reload, the new provider's description must cover the old provider's when the version is unchanged, in addition to the check against live consumers and facets' `uses`. - **Contracts.** Local contracts list their members, so a loading reference knows whether a member is a method, state or opaque. Canonical identity omits input fields equal to their default; use `opt(str({ default }))` (pi-schema) to let callers omit them. Provider variants must cover consumer variants; members the consumer's contract does not list are hidden. Schema change detection compares canonical JSON, so reordering `anyOf` branches counts as a change. Fingerprints use a non-cryptographic 64-bit hash: synchronous and browser-safe; contracts are trusted. -- **Principals.** A root ctx at ingress acts as its principal (`host.root(fn, { principal })`); contexts inside objects act as `host`. The acceptor decides a link's kind and name (`accept(link, { principal, kind, name })`); only configured host links are addressable, carry roots and may vote, and control frames are accepted only from the parent. Clients never carry roots or code versions. Relaying needs a matching `route`; `authorize(principal, hostName)` runs before `resolve` and again after, and revoking or re-stamping a principal closes its relays at once. `invalidatePrincipal(match)` re-stamps or closes matching links and propagates pattern matches to child hosts. +- **Principals.** A root ctx at ingress acts as its principal (`host.root(fn, { principal })`); contexts inside objects act as `host`. The acceptor decides a link's kind and name (`accept(link, { principal, kind, name })`); only configured host links are addressable, carry roots and may vote, and control frames are accepted only from the parent. Clients never carry roots or code versions. Relaying needs a matching `route`; `authorize(principal, hostName)` runs before `resolve` and again after, and revoking or re-stamping a principal closes its relays at once. The host `resolve` returns is admitted before that second check, so a refused, revoked or released relay never leaves a spawned host's link unread. `invalidatePrincipal(match)` re-stamps or closes matching links and propagates pattern matches to child hosts. - **Cells and readables.** A value returned from `change` replaces the value and is always validated; `replace` transfers ownership without copying; by default (`{ diff }` overrides) it publishes a diff when the new value reuses a container of the old one at the same key, else `r`. `subscribe(fn)` delivers values, starting with the current one; `events(fn)` delivers `snap`, then `ops`, or `reset`. Reference state members are stable readables that throw `not_ready` while loading and `not_in_variant` outside the variant, and reset when their source moves; a listener added before they are readable gets its snapshot then. Remote state members stay hydrated while the reference is held. - **References and calls.** A reference's identity is its address; a second `self.get` of the same address in one run returns the same reference. A reference call without ctx, or with a task ctx, starts a fresh root in the current code version; a ctx whose call or root already settled is refused (`aborted/expired`), one from another host `invalid/foreign_ctx`. Calling a reference inside a factory throws `not_ready`. - **Resources and disposal.** New and replaced exclusive resources open at the gate, and so do non-exclusive resources of the same object that depend on them. A new object for an address that is still being disposed waits for that disposal. A resource open that completes after its signal aborted closes its value at once. diff --git a/packages/chord/src/host.ts b/packages/chord/src/host.ts index 9f96a4909..fe94fd9be 100644 --- a/packages/chord/src/host.ts +++ b/packages/chord/src/host.ts @@ -1136,7 +1136,7 @@ export class HostImpl implements Host, HostCore { return best?.[1]; } - /** Authorize, resolve (maybe spawn), authorize again; returns the synced connection to the host. */ + /** Authorize, resolve (maybe spawn) and admit the host, authorize again; returns the synced connection to the host. */ async #resolve( route: Route, hostName: string, @@ -1148,15 +1148,17 @@ export class HostImpl implements Host, HostCore { if (!(await route.authorize(principal, hostName))) throw refuse(); if (!still()) throw revoked(); const link = await route.resolve(hostName); - if (!still()) throw revoked(); - if (!(await route.authorize(principal, hostName))) throw refuse(); - if (!still()) throw revoked(); + // Admitted before the checks: a host the route spawned for a request that is gone or refused meanwhile still runs, + // and its own references to this host would otherwise wait on a link nobody reads until they time out. let connection = [...this.connections].find((candidate) => candidate.link === link); connection ??= this.#addConnection(link, { role: "downstream", principal: route.principal(hostName), peer: { kind: "host", name: hostName }, }); + if (!still()) throw revoked(); + if (!(await route.authorize(principal, hostName))) throw refuse(); + if (!still()) throw revoked(); await connection.whenSynced(); if (!still()) throw revoked(); return connection; diff --git a/packages/chord/test/links-review.test.ts b/packages/chord/test/links-review.test.ts index f6a456afb..f5d0f7173 100644 --- a/packages/chord/test/links-review.test.ts +++ b/packages/chord/test/links-review.test.ts @@ -311,6 +311,29 @@ describe("links review: relay ownership", () => { assert.ok(!(c.workers.get("session:S1")?.link.sent ?? []).some((frame) => frame.t === "hold")); }); + it("a release while the route resolves still admits the resolved worker: its references upstream go live", async () => { + const Feed = remote("t.feed", { + version: 1, + input: obj({}, { open: true }), + host: hostNamed("server"), + variants: { main: { methods: { ping: method(undefined, str()) } } }, + }); + const c = cluster(); + c.server.serve(Feed, () => ({ variant: "main", ping: () => "pong" })); + const gate = latch(); + c.resolveGate = gate.promise; + const { raw } = c.raw({ guest: "g" }); + raw.send({ t: "hold", id: 1, at: "session:S1", token: Counter.description, input: { name: "a" } }); + await sleep(5); + // The client goes away while the worker starts: the route still hands over the worker it spawned. + raw.send({ t: "release", id: 1 }); + gate.open(); + await until(() => c.workers.has("session:S1")); + const worker = c.workers.get("session:S1")!.host; + const feed = await worker.get(Feed, {}, { signal: AbortSignal.timeout(500) }); + assert.equal(await feed.ping(), "pong"); + }); + it("#9 an origin disconnect removes its relayed holds and subscriptions downstream", async () => { const c = cluster(); const browser = c.browser({ guest: "g" }); From b556cfe5b0300a91f39c7bc5369a65a25fb7fcd5 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sun, 1 Nov 2026 15:20:46 +0000 Subject: [PATCH 120/175] docs(chord): schemas are pi-schema's in the design --- packages/chord/docs/next/design.md | 13 +++++++------ packages/chord/docs/next/layer-1.md | 2 +- packages/chord/docs/next/slice-3a.md | 4 ++-- packages/chord/docs/next/slices.md | 4 ---- 4 files changed, 10 insertions(+), 13 deletions(-) diff --git a/packages/chord/docs/next/design.md b/packages/chord/docs/next/design.md index d40d82677..517276db3 100644 --- a/packages/chord/docs/next/design.md +++ b/packages/chord/docs/next/design.md @@ -442,7 +442,8 @@ drops, folds or resets. - **JSON plus `Uint8Array` leaves**, with a bytes schema type, defined independently of the transport. Every link carries the same `Message` through its codec (section 22); parts are encoded at their origin; byte segments travel raw, never base64. Same-host calls (no link) pass values by reference. -- Schemas are TypeBox and must describe data (no callbacks, no native types, no runtime codecs). Method arguments are +- Schemas are plain JSON Schema built with `@earendil-works/pi-schema` (decision "The schema switch") and must + describe data (no callbacks, no native types, no runtime codecs); `format` is an annotation, never checked. Method arguments are validated at the provider for every caller against closed schemas (section 5, R7, Q12, Q15); results and state are never validated (section 30). Values are immutable by convention once handed over (Delta's rule): nobody mutates a value it got, and nothing is copied or frozen to enforce it. There is no development mode (R9): one behavior @@ -572,7 +573,7 @@ host.serve(Notes, ({ session }, principal, self) => (`invalid/member` outside it). - **Params** (owner, 2026-10-21/22): a flat record of required fields, each `str()`, `int()`, `bool()` or `oneOf(...)` of string literals, constraints allowed (`str({ pattern })`, `int({ min: 0 })`); no defaults, optional, nested or - unknown fields. Checked for every caller with one compiled closed-object TypeBox check (a missing or extra field + unknown fields. Checked for every caller with one compiled closed-object pi-schema check (a missing or extra field fails `invalid/params`). The identity key is the field values in declared order, each escaped, joined with `|`. Params are never copied or frozen: the object keeps the caller's object, immutable by convention. - **Arguments (R7, Q12, Q15):** always validated with the provider's schema, whoever the caller is (trusted code is @@ -592,8 +593,8 @@ host.serve(Notes, ({ session }, principal, self) => link protocol (adding one bumps the protocol version; a receiver rejects unknown reasons; a test pins the list): `invalid` (`args`, `params`, `member`, `host_address`, `reentrant`, `read_after_bulk`, `cycle`), `contract_mismatch` (`updating`), `forbidden` (`revoked`), `gone`, `not_ready`, `released` (`ended`, `closed`, `settled`, `version`, `discarded`; none for a holder's own release), `too_large` (`part`), `unavailable` (`locked`, - `park_timeout`, `busy`, `storage_failed`, `undeclared`: a registry no plugin serves here, slice 8), `aborted`. A failed check names TypeBox's instance path ("unknown field", "missing", else TypeBox's - message; keys containing `/` or `~` make the path ambiguous, documented). Plugins throw `PluginError(code, message, + `park_timeout`, `busy`, `storage_failed`, `undeclared`: a registry no plugin serves here, slice 8), `aborted`. A failed check names pi-schema's JSON Pointer path ("unknown field", "missing", else + pi-schema's message; `~` and `/` in keys escaped as `~0`, `~1`). Plugins throw `PluginError(code, message, details?)` for codes of their own; anything else crosses a link as the plugin code `internal` with its message, never a stack or a cause; plugin errors never use Chord's codes. An error rebuilt from the wire is not copied and only its envelope is checked; a malformed envelope is a protocol error. @@ -1997,7 +1998,7 @@ validation, no clone of snapshots or ops. - **The protocol version** travels in `hello`; hosts of different versions refuse each other (a web page reloads for a newer server). Error reasons and message kinds are part of it. Until p3 ships, changes keep `PROTOCOL_VERSION` 1 (nothing is deployed); after that every message or reason change bumps it. -- **One typed `Message`** (owner, 2026-10-22, "Wire protocol"): a TypeBox schema `MESSAGE`, a tagged union over +- **One typed `Message`** (owner, 2026-10-22, "Wire protocol"): a pi-schema schema `MESSAGE`, a tagged union over `kind`, from which the protocol types derive; every decoded message is checked against it (compiled once) unless the link is trusted (section 3). Kinds: @@ -2416,7 +2417,7 @@ const accepted = checkAccepts(NotesV2, { providers fail at declaration naming the member; unknown keys and wrong shapes are rejected. - **The fingerprint** is a hash (`hash64`, portable, not a security boundary) of a schema-aware, data-only normal form (`NORMAL_FORM_VERSION`, tied to the protocol): annotations dropped only on schema nodes, `required`/`enum` sorted, - union and intersection order kept (it is meaning: defaults), a whitelist of TypeBox keywords, executable features + union and intersection order kept (it is meaning: defaults), a whitelist of JSON Schema keywords, executable features rejected; bytes are a `{ type: "bytes" }` data marker. Schemas are deep-copied into a private history (never frozen; immutable by convention). - **Schema helpers:** `str num int bool nul lit oneOf union obj arr map opt bytes unknown` and `type Static`. - Chord wraps state mappings as a derive and diffs per object and version (section 22); members an old consumer does not know do diff --git a/packages/chord/docs/next/layer-1.md b/packages/chord/docs/next/layer-1.md index 505fe7294..0129181df 100644 --- a/packages/chord/docs/next/layer-1.md +++ b/packages/chord/docs/next/layer-1.md @@ -14,7 +14,7 @@ checked, what a principal key is, what an error is. Pure functions, types and we Source: `packages/chord/src/next/contracts/` (declarations) and `packages/chord/src/next/data/` (per-call checks, principals, errors), split into files by concept. Everything else under `src/next/` is the frozen old kernel: never imported, never edited. Reused: `src/context/` (`Context`, `BACKGROUND_CONTEXT`, `context.telemetry`) and -`src/json.ts` (`JsonValue`), both unchanged. Schemas are TypeBox 1.3 (`typebox` in `node_modules`). +`src/json.ts` (`JsonValue`), both unchanged. Schemas were TypeBox 1.3 when L1 was built; since the schema switch (2026-11-01) they are pi-schema's plain JSON Schema (design §3 "Values"). The TypeBox details below are historical. ## 1. Values diff --git a/packages/chord/docs/next/slice-3a.md b/packages/chord/docs/next/slice-3a.md index 4605bb95c..a6dba100b 100644 --- a/packages/chord/docs/next/slice-3a.md +++ b/packages/chord/docs/next/slice-3a.md @@ -89,7 +89,7 @@ if (notes.variant === "reader") console.log(notes.notes.value); // narrowed by | u32 header length, big-endian | header JSON (UTF-8) | part 0 | part 1 | … | ``` -- **One format on every link.** The header is a JSON object, a tagged union over `kind`, defined as a TypeBox schema; +- **One format on every link.** The header is a JSON object, a tagged union over `kind`, defined as a schema (TypeBox then, pi-schema since the schema switch); every received header is checked with the compiled schema (the wire is a trust boundary; skipping it on trusted IPC links comes with IPC in slice 4). The header lists its parts: `parts: [{ json, bytes }]`, the JSON text's byte length and the byte lengths of the segments that follow it (`bytes: []` for state parts and arguments without bytes). @@ -125,7 +125,7 @@ if (notes.variant === "reader") console.log(notes.notes.value); // narrowed by Replaces the JSON-header framing of section 2 (`| u32 header length | header JSON | parts |`); the message kinds, ids, races and rules of section 2 stay. -- **`Message`** is a tagged union defined once as a TypeBox schema (`MESSAGE`; `type Message = Static`): `hello { protocol, incarnation }`, `hold { hold, type, version, fingerprint, params }`, `release { hold }`, `call { call, hold, method, args? }`, `cancel { call }`, `bound { hold, variant, members: [{ member, value }], failed: [{ member, error }] }`, `state { holds: [{ hold, members: [{ member, ops }], failed: [...] }] }`, `failed { diff --git a/packages/chord/docs/next/slices.md b/packages/chord/docs/next/slices.md index 452528592..28e094cd5 100644 --- a/packages/chord/docs/next/slices.md +++ b/packages/chord/docs/next/slices.md @@ -170,15 +170,11 @@ deleted (the slice's spec and code are the record). Moved here from `carry-over. slice 3b.) - **The park deadline's per-type override** is not built: a parked call waits the reference's `readyTimeout`. Build it only if a real case needs it. (Slice 3b spec.) -- Failed-check paths are TypeBox's instance paths (ambiguous for keys containing `/` or `~`); documented, not fixed. - (L1.) - No CBOR; revisit only if benchmarks show encoding as a bottleneck (slice 3a's and 4a's do not). (L2 decisions.) - **One N on every host:** a relay forwards calls at once, so a worker announcing a smaller N than the relay's could see more running calls of one origin than it accepts (a protocol error). Every host announces 256; per-link N, if ever needed, must bound the relay's forwarding too. (Slice 4a rework.) - **The Durable conversation view's size on re-hold** is a known cost until pi #11327 (windowed view) lands. (Slice 4.) -- **`map(bytes())` is refused at declaration** ("unsupported keyword `~refine`"): TypeBox's `Record` copies the bytes - schema's check as an enumerable key, which the snapshot walk refuses. An L1 bug found in slice 3a, not fixed yet. ### Later: reload (when a case needs it) From e1669ee4b841fc053de7c4fb0b872f41f8d9a898 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sun, 1 Nov 2026 23:11:16 +0000 Subject: [PATCH 121/175] docs(chord): contracts come from where the plugin's code came from --- packages/chord/docs/next/decisions.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 7dd53c73b..843af1a7c 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2063,7 +2063,7 @@ validation and copies the v2 kernel added around Delta. and TUI, decided point by point; the slice 10 components are its inside. Inputs from the owner: user management is part of it ("user management will also need to be a thing we support"), and app shapes without workers (server and clients only: "a chat app doesn't need workers"). -- **Plugins import each other by name (owner, 2026-10-31: "i like it"; changes 7c's cross-plugin rule):** +- **Plugins import each other by name (owner, 2026-10-31: "i like it"; changes 7c's cross-plugin rule; **superseded** by "Contracts come from where the plugin's code came from", 2026-11-01):** `import { Notes } from "@app/notes/contract"` resolves to the discovered plugin named `@app/notes`, its file `contract.ts`, wherever its directory is (any root), copied into the importer's script as before (contracts and side-effect-free files only; they may import only host packages and files of their own plugin); a missing plugin is @@ -2140,6 +2140,19 @@ validation and copies the v2 kernel added around Delta. annotation in Chord** ("no format check": no format module in Chord's or Durable's entries; pi-ai still checks formats on tool arguments). **Chord's envelope check costs +10 to +100 ns per message** from untrusted links (pi-schema reads fields by name from a list; no eval): accepted ("negligible"), no hand-written check. +- **Contracts come from where the plugin's code came from (owner, 2026-11-01: "ok"; plugin model point 1, primer + `docs/next/plugin-primer/`; supersedes "Plugins import each other by name"):** a plugin carries the contract it was + written against, never the running app's newest file. The problem: by name, a third-party plugin's next build copied + the provider's newest `contract.ts` (version 2), so its version-1 code claimed version 2, `accepts` was never used + and it computed wrong results silently. Rule: (A) plugins that change together, the app's own in its repository or a + user's in one root, import each other by relative path (`../notes/contract.ts`), copied into the importer's script + and listed in `copied`; a copied file may import only host packages and files of its own plugin; a relative path + leaving its root (into base code or another root) is a build error pointing to the API or an npm dependency. (B) + Plugins that do not change together take contracts as an npm dependency at a fixed version, bundled at the installed + version (decision "A plugin's npm dependencies are bundled"); the provider serves it through `accepts` or the + consumer waits "contract mismatch". Which package carries the app's contracts is point 5. The by-name lookup and the + `tsconfig` path aliases go; built in the plugin slice (examples 35-38, fixtures, design §21, the plugins README, the + guide's chapter "Plugins Using Plugins"). - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From c9f4c7b9fcd0f3e6b5bbc35c7c9b6cda35580eb8 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sun, 1 Nov 2026 23:11:22 +0000 Subject: [PATCH 122/175] docs(chord): fold the cross-plugin import rule into design and slices --- packages/chord/docs/next/design.md | 5 ++++- packages/chord/docs/next/slices.md | 7 +++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/packages/chord/docs/next/design.md b/packages/chord/docs/next/design.md index 517276db3..c11cb89e4 100644 --- a/packages/chord/docs/next/design.md +++ b/packages/chord/docs/next/design.md @@ -1962,7 +1962,10 @@ type ReloadReport = { - **A plugin** is a directory with a `package.json` (its `name` is the plugin's name, npm style) and one file per facet kind the app names; plugins come from roots found by directory listing (no source parsing; the first root wins a duplicate name). Dependencies between plugins resolve by contract name at runtime; names are prefixed - (`.`; tools `_`). + (`.`; tools `_`). A plugin carries the contracts it was written against (decision "Contracts come from where the + plugin's code came from"): plugins that change together import each other by relative path within a root + (copied into the importer's script); others take contracts as an npm dependency at a fixed version (slice 10 still + resolves imports by plugin name; the plugin slice changes it). - **Loader D:** one esbuild script per facet (its own files, copied files of other plugins, its npm `dependencies` inlined; host packages, `chord.external` packages and Node builtins external, passed in at evaluation); each facet kind is defined once, `kinds: { : { platform: "node" | "browser", hostPackages } }` (slice 11); evaluated with `vm.Script` under a stable filename per (plugin, kind, digest); no module map: old generations are reclaimed by the diff --git a/packages/chord/docs/next/slices.md b/packages/chord/docs/next/slices.md index 28e094cd5..084d3456e 100644 --- a/packages/chord/docs/next/slices.md +++ b/packages/chord/docs/next/slices.md @@ -181,6 +181,13 @@ deleted (the slice's spec and code are the record). Moved here from `carry-over. - **`rebuild` of tables** (key and column-type changes): `evolveTable` has no edit for them; build one only when a real case needs it. (Slice 6a's second spec review; out of slice 7b's scope, `slice-7b.md` §12.) +### The plugin slice (plugin model) + +- **Cross-plugin imports by relative path and npm** (decision "Contracts come from where the plugin's code came + from"): remove `build.ts`'s by-name lookup, allow relative imports into another plugin of the same root, refuse + paths leaving the root; examples 35-38 and fixtures back to relative imports; drop the six `tsconfig` path aliases; + plugins README; the guide's chapter "Plugins Using Plugins". (Slice 10 built by-name imports.) + ### After slice 11 - **Small examples** (owner, 2026-10-30: the 7c examples are complete but complex): once HTTP, workers, web and TUI From 6ab6c1e0e431a55ddfdd804f35c16073b1031e56 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 09:41:22 +0000 Subject: [PATCH 123/175] docs(chord): plugins import only the app's versioned API --- packages/chord/docs/next/decisions.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 843af1a7c..3b38c7bf3 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2153,6 +2153,18 @@ validation and copies the v2 kernel added around Delta. consumer waits "contract mismatch". Which package carries the app's contracts is point 5. The by-name lookup and the `tsconfig` path aliases go; built in the plugin slice (examples 35-38, fixtures, design §21, the plugins README, the guide's chapter "Plugins Using Plugins"). +- **Plugins import only the app's versioned API; Chord is re-exported in it (owner, 2026-11-01: "makes sense"; + plugin model point 2):** a plugin never imports Chord directly: Chord is not a host package plugins may name, so a + direct import is a build error pointing to the app's API. The app's API module per version (`@acme/app/v3`) + re-exports the Chord surface the app supports (contract and schema builders, `defineFacet`, `defineRegistry`, the + types plugins see) next to its own small part (host kinds, the principal type, `defineFacet` bound to it, core + contracts, registry tokens). One version number, the app's, describes everything a plugin gets from its host. The + process runs one Chord: an older API version still listed is a layer over today's Chord; when Chord breaks its + API it keeps its previous API as a compatibility module for a while, which the app's older version re-exports; + once that is dropped the app drops the version and old plugins fail at build naming the available versions. + Contracts are data and bind through `accepts` whatever Chord runs. One facet imports from one API version only + (checked at build). Only base code imports Chord directly. Bundling is slice 10's host packages: the version is part + of the name (`kinds.hostPackages` lists `@acme/app/v3`, `@acme/app/v2`), each process provides its own modules. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From d0fe213089aef0599aa9c90bf910c8b562e0dcf8 Mon Sep 17 00:00:00 2001 From: "robopipi[bot]" <336283572+robopipi[bot]@users.noreply.github.com> Date: Mon, 2 Nov 2026 10:55:30 +0100 Subject: [PATCH 124/175] fix(coding-agent): apply context edits to branch summary input (#11680) When a partial assistant reply fails and automatic retry succeeds, pi appends a `context_edit` that hides the failed attempt from model context. Branch summarization (`/tree` with summarize) built its input from raw session entries through `getMessageFromEntry`, which ignored `context_edit` entries. As a result, the summarizer received the hidden attempt again. Content replacements were ignored the same way. What the change does: - `prepareBranchEntries` now collects the latest `context_edit` for each target in the branch. - It applies those edits to the entries it converts, using the same `projectContextEntry` logic as the live context and compaction: - an omission (`replacement: null`) drops the entry; - a replacement swaps its content. - `projectContextEntry` is now exported from `session-manager.ts` so both paths share it. - The existing branch-summary behavior is unchanged: tool results are still skipped, and compaction or branch-summary entries still become summary messages. Testing: - Added `applies context edits to branch entries` to `test/branch-summarization.test.ts`. It covers one omitted assistant entry and one replaced assistant entry. The test fails without the change and passes with it. - Ran `branch-summarization`, `branch-summary-extensions`, `agent-session-branching` and the compaction tests: all pass. - `npm run check`: biome and the coding-agent checks pass. `tsc` reports 6 errors in `packages/ai` model-catalog tests, and the same 6 errors occur on unmodified `main`. - End-to-end reproduction with the faux provider and `AgentSession`: a failed attempt, a successful retry, then `navigateTree(target, { summarize: true })`. Before the change, the summary request contained the failed attempt's text. After the change, it contains only the successful retry. Fixes #11602 Requested by @davidbrai Co-authored-by: triagent --- .../core/compaction/branch-summarization.ts | 20 +++++-- .../coding-agent/src/core/session-manager.ts | 3 +- .../test/branch-summarization.test.ts | 52 ++++++++++++++++++- 3 files changed, 69 insertions(+), 6 deletions(-) diff --git a/packages/coding-agent/src/core/compaction/branch-summarization.ts b/packages/coding-agent/src/core/compaction/branch-summarization.ts index 69bd75133..6f8bc0cb3 100644 --- a/packages/coding-agent/src/core/compaction/branch-summarization.ts +++ b/packages/coding-agent/src/core/compaction/branch-summarization.ts @@ -15,7 +15,12 @@ import { createCompactionSummaryMessage, createCustomMessage, } from "../messages.ts"; -import type { ReadonlySessionManager, SessionEntry } from "../session-manager.ts"; +import { + type ContextEditEntry, + projectContextEntry, + type ReadonlySessionManager, + type SessionEntry, +} from "../session-manager.ts"; import { completeSummarization, estimateTokens, getSummarizationFailure } from "./compaction.ts"; import { computeFileLists, @@ -155,14 +160,15 @@ export function collectEntriesForBranchSummary( * Extract AgentMessage from a session entry. * Similar to getMessageFromEntry in compaction.ts but also handles compaction entries. */ -function getMessageFromEntry(entry: SessionEntry): AgentMessage | undefined { +function getMessageFromEntry(entry: SessionEntry, edit?: ContextEditEntry): AgentMessage | undefined { switch (entry.type) { case "message": // Skip tool results - context is in assistant's tool call if (entry.message.role === "toolResult") return undefined; - return entry.message; + return edit ? projectContextEntry(entry, edit)[0] : entry.message; case "custom_message": + if (edit) return projectContextEntry(entry, edit)[0]; return createCustomMessage(entry.customType, entry.content, entry.display, entry.details, entry.timestamp); case "branch_summary": @@ -217,10 +223,16 @@ export function prepareBranchEntries(entries: SessionEntry[], tokenBudget: numbe } } + // Context edits hide or replace earlier entries in model context; honor the latest edit per target. + const edits = new Map(); + for (const entry of entries) { + if (entry.type === "context_edit") edits.set(entry.targetId, entry); + } + // Second pass: walk from newest to oldest, adding messages until token budget for (let i = entries.length - 1; i >= 0; i--) { const entry = entries[i]; - const message = getMessageFromEntry(entry); + const message = getMessageFromEntry(entry, edits.get(entry.id)); if (!message) continue; // Extract file ops from assistant messages (tool calls) diff --git a/packages/coding-agent/src/core/session-manager.ts b/packages/coding-agent/src/core/session-manager.ts index 822aad816..20b6edaa9 100644 --- a/packages/coding-agent/src/core/session-manager.ts +++ b/packages/coding-agent/src/core/session-manager.ts @@ -516,7 +516,8 @@ export function buildContextEntries( * If leafId is provided, walks from that entry to root. * Handles compaction and branch summaries along the path. */ -function projectContextEntry(entry: SessionEntry, edit: ContextEditEntry | undefined): AgentMessage[] { +/** Project one entry into model-visible messages, applying its latest context edit if any. */ +export function projectContextEntry(entry: SessionEntry, edit: ContextEditEntry | undefined): AgentMessage[] { const messages = sessionEntryToContextMessages(entry); if (!edit) return messages; const replacement = edit.replacement; diff --git a/packages/coding-agent/test/branch-summarization.test.ts b/packages/coding-agent/test/branch-summarization.test.ts index ade25dbfd..3164a8773 100644 --- a/packages/coding-agent/test/branch-summarization.test.ts +++ b/packages/coding-agent/test/branch-summarization.test.ts @@ -7,7 +7,7 @@ import { type SimpleStreamOptions, } from "@earendil-works/pi-ai"; import { describe, expect, it } from "vitest"; -import { generateBranchSummary } from "../src/core/compaction/index.ts"; +import { generateBranchSummary, prepareBranchEntries } from "../src/core/compaction/index.ts"; import type { SessionEntry } from "../src/core/session-manager.ts"; const model: Model<"anthropic-messages"> = { @@ -131,4 +131,54 @@ describe("branch summarization", () => { "Branch summarization failed: generation hit the token cap and the summary is incomplete", ); }); + + it("applies context edits to branch entries", () => { + const ts = (n: number) => new Date(n).toISOString(); + const branch: SessionEntry[] = [ + { + type: "message", + id: "u1", + parentId: null, + timestamp: ts(1), + message: { role: "user", content: "Investigate login", timestamp: 1 }, + }, + { + type: "message", + id: "failed", + parentId: "u1", + timestamp: ts(2), + message: fauxAssistantMessage("FAILED_ATTEMPT", { stopReason: "error", errorMessage: "lost" }), + }, + { + type: "context_edit", + id: "e1", + parentId: "failed", + timestamp: ts(3), + targetId: "failed", + replacement: null, + }, + { + type: "message", + id: "ok", + parentId: "e1", + timestamp: ts(4), + message: fauxAssistantMessage("ORIGINAL_TEXT"), + }, + { + type: "context_edit", + id: "e2", + parentId: "ok", + timestamp: ts(5), + targetId: "ok", + replacement: { content: "REPLACED_TEXT" }, + }, + ]; + + const serialized = JSON.stringify(prepareBranchEntries(branch).messages); + + expect(serialized).toContain("Investigate login"); + expect(serialized).not.toContain("FAILED_ATTEMPT"); + expect(serialized).not.toContain("ORIGINAL_TEXT"); + expect(serialized).toContain("REPLACED_TEXT"); + }); }); From ddbe02cd2014bedc9555b7d782da90a2fd5e8abc Mon Sep 17 00:00:00 2001 From: Armin Ronacher Date: Mon, 2 Nov 2026 09:47:27 +0100 Subject: [PATCH 125/175] docs: audit changelogs and add release highlights --- packages/ai/CHANGELOG.md | 8 ++++ packages/coding-agent/CHANGELOG.md | 62 +++++++++++++++++++++++++++++- packages/tui/CHANGELOG.md | 5 +++ 3 files changed, 73 insertions(+), 2 deletions(-) diff --git a/packages/ai/CHANGELOG.md b/packages/ai/CHANGELOG.md index 6aae94ddb..5d197d814 100644 --- a/packages/ai/CHANGELOG.md +++ b/packages/ai/CHANGELOG.md @@ -36,6 +36,8 @@ - Added optional `openBrowserOnConfirm` and `copyCodeOnConfirm` to `device_code` auth events and `OAuthDeviceCodeInfo`. Both default to `true`; set them to `false` to stop clients from offering to open the verification page or copy the code ([#9282](https://github.com/earendil-works/pi/issues/9282), [#9301](https://github.com/earendil-works/pi/pull/9301)) - Added `StreamingJsonParser`, an incremental, tolerant parser for JSON text that arrives in pieces (each piece is read once; snapshots of the partial value never change once returned), and `ToolCallArgumentsStream`, which every provider now uses for streamed tool-call arguments; its `append()` also returns the changes each piece made (`JsonChange`: a value set at a path, or text appended to a string), for writers that keep their own copy ([#9265](https://github.com/earendil-works/pi/issues/9265)) - Added GPT-6.1 Terra to the OpenAI, Azure OpenAI Responses, and OpenAI Codex providers with official pricing ($1 input, $0.05 cached input, $1.25 cache writes, $5 output, long-context tier above 272k), tool search, and mid-conversation effort changes. Unlike GPT-6.1 Sol it supports `off` (reasoning effort `none`); Codex serves it with a 500K context window +- Added `Usage.cost.totalFromProvider` to distinguish provider-reported billed totals from catalog estimates. Missing or `false` means an estimate; combined usage is marked provider-reported only when every part is ([#11178](https://github.com/earendil-works/pi/pull/11178)) +- Added `responseModel` to pi-messages terminal events, preserving the concrete model reported by providers such as Radius when a virtual model is requested ([#11265](https://github.com/earendil-works/pi/pull/11265)) ### Changed @@ -57,6 +59,12 @@ - Fixed GPT-5.4 and GPT-5.5 failing with `Missing namespace for function_call` after switching from another model that called tools. On the OpenAI and OpenCode providers they now declare their tools in the top-level tool list instead of loading them through `additional_tools` or tool search ([#11415](https://github.com/earendil-works/pi/issues/11415)) - Fixed Bedrock requests failing when `strict: "prefer"` tools exceed Claude's strict tool limits, such as more than 16 optional parameters across all strict tools; requests now retry once with preferred tools non-strict while keeping `strict: "require"` tools strict, as the Anthropic provider already does ([#11416](https://github.com/earendil-works/pi/issues/11416)) - Fixed OpenRouter's `~anthropic/claude-*-latest` aliases losing Claude's earlier thinking after each tool call. They used the Chat Completions API, where pi dropped the thinking signature, and OpenRouter then ignored the unsigned thinking. They now use the Anthropic Messages API with adaptive thinking, like the versioned Claude models ([#9534](https://github.com/earendil-works/pi/issues/9534)) +- Fixed OpenRouter cost totals using catalog estimates instead of reported billing for Chat Completions and Anthropic Messages. BYOK totals include both OpenRouter charges and reported upstream costs; component costs remain catalog estimates ([#9980](https://github.com/earendil-works/pi/issues/9980)) +- Fixed OpenCode Zen and OpenCode Go requests using shared endpoints instead of their protocol-specific OpenAI, Anthropic, Google, and System One endpoints ([#11434](https://github.com/earendil-works/pi/pull/11434) by [@vimtor](https://github.com/vimtor)) +- Fixed Zhipu/Z.ai usage-limit errors (code `1308`) being treated as transient HTTP 429 errors, causing automatic retries that cannot succeed until the usage cap resets ([#11537](https://github.com/earendil-works/pi/issues/11537)) +- Fixed LiteLLM's `ContextWindowExceededError` not being recognized as context overflow ([#11492](https://github.com/earendil-works/pi/issues/11492)) +- Fixed Gemini tool schemas sending unsupported `const` keywords. They are now converted to single-value `enum` schemas without rewriting literal data in defaults, examples, or enum values, or losing properties named `__proto__` ([#11308](https://github.com/earendil-works/pi/issues/11308)) +- Fixed the faux provider counting simulated cache-write tokens as both input and cache writes ## [1.4.0] - 2026-10-21 diff --git a/packages/coding-agent/CHANGELOG.md b/packages/coding-agent/CHANGELOG.md index d391a4f56..46a5bbc9f 100644 --- a/packages/coding-agent/CHANGELOG.md +++ b/packages/coding-agent/CHANGELOG.md @@ -2,6 +2,17 @@ ## [Unreleased] +### New Features + +- **Extension update changelogs**: see what changed in installed extension packages on the next fresh launch after an update. Package authors can choose the changelog path, and `collapseChangelog` condenses the notices. See [Show a changelog](docs/packages.md#show-a-changelog). +- **Faster streaming**: large tool calls are parsed incrementally, and code-heavy answers and tool rows reuse highlighting and rendered Markdown instead of repeating the work on every update. See [Keep rendering responsive](docs/tui.md#keep-rendering-responsive). +- **Custom self-update commands**: use `PI_SELF_UPDATE_COMMAND` or `selfUpdateCommand` to update pi through a wrapper or another package manager. See [Custom update command](docs/cli.md#custom-update-command). +- **GPT-6.1 Terra**: available through OpenAI, Azure OpenAI Responses, and OpenAI Codex, with tool search, mid-conversation thinking-level changes, and thinking `off`. Codex offers a 500K context window. See [Models](docs/cli.md#models). + +### Breaking Changes + +- Tool argument validation runs on `@earendil-works/pi-schema`. TypeBox features outside JSON Schema (`Type.Refine()`, `Type.BigInt()`, `Type.Undefined()`, codecs, `Format.Set()` formats) no longer apply to tool arguments; see [Tool parameter schemas and TypeBox](docs/extensions.md#tool-parameter-schemas-and-typebox). Plain JSON Schema tools, including MCP tools, now get the same coercion as TypeBox tools, such as converting numeric strings and wrapping a single value into an array. Validation failures the model sees list pi-schema's messages (`- path: missing`, `- limit: expected number, got string`) instead of TypeBox's (`must have required properties path`, `must be number`). + ### Added - Extension packages can now show their changelog after an update. Pi reads `pi.changelogPath` from the package's `package.json`, or `CHANGELOG.md` in the package root, and on the next fresh launch shows entries newer than the last seen version under "Extension Updates". The `collapseChangelog` setting condenses this to one line per package ([#5958](https://github.com/earendil-works/pi/issues/5958), [#8790](https://github.com/earendil-works/pi/pull/8790)) @@ -9,15 +20,21 @@ - Device-code login (for example GitHub Copilot) now opens the verification page and copies the code when you press Enter. Providers can turn either action off with `openBrowserOnConfirm: false` or `copyCodeOnConfirm: false` ([#9282](https://github.com/earendil-works/pi/issues/9282), [#9301](https://github.com/earendil-works/pi/pull/9301)) - `pi update` can update Pi through a wrapper or another package manager. Set `PI_SELF_UPDATE_COMMAND` (for example `mise use -g npm:{packageName}@{version}`) or the global `selfUpdateCommand` setting; `pi update` runs it when pi.dev reports a newer release and fills `{packageName}`, `{version}`, and `{installSpec}` from that release. See [Custom update command](docs/cli.md#custom-update-command) ([#7756](https://github.com/earendil-works/pi/issues/7756), [#5607](https://github.com/earendil-works/pi/issues/5607)) - Extensions can format the user's configured keybindings with `keyText()` and `keyHint()` from the tool render context and the message and entry renderer options. Importing these functions from `@earendil-works/pi-coding-agent` can resolve the extension's own copy of the package, which does not see pi's keybindings, so hints such as `keyHint("app.tools.expand", "to expand")` rendered without the key. The docs now recommend the injected functions ([#4748](https://github.com/earendil-works/pi/issues/4748)) +- Added persisted session-entry `messageId` and ISO `timestamp` to `MarkdownTransformContext`. They are available when rendering saved entries and undefined for live rendering ([#7952](https://github.com/earendil-works/pi/pull/7952) by [@vipentti](https://github.com/vipentti)) +- Added a `blocked` option to inline `ctx.ui.custom()` components so approval prompts and other custom dialogs can report that pi is waiting for user input. Overlays ignore the option. See [Custom screens and overlays](docs/tui.md#use-custom-screens-and-overlays) ([#11428](https://github.com/earendil-works/pi/issues/11428)) +- Added `Usage.cost.totalFromProvider` to distinguish billed totals from catalog estimates in SDK and extension usage data; combined usage is marked provider-reported only when every part is ([#11178](https://github.com/earendil-works/pi/pull/11178)) +- Added concrete `responseModel` metadata to messages from pi-messages providers such as Radius, including when a virtual model is requested. See [AssistantMessage](docs/message-types.md#assistantmessage) ([#11265](https://github.com/earendil-works/pi/pull/11265)) +- Added GPT-6.1 Terra for OpenAI, Azure OpenAI Responses, and OpenAI Codex, with tool search and mid-conversation thinking-level changes. It supports thinking `off`; Codex offers a 500K context window ### Changed -- Tool argument validation runs on `@earendil-works/pi-schema`. TypeBox features outside JSON Schema (`Type.Refine()`, `Type.BigInt()`, `Type.Undefined()`, codecs, `Format.Set()` formats) no longer apply to tool arguments; see [Tool parameter schemas and TypeBox](docs/extensions.md#tool-parameter-schemas-and-typebox). Validation failures the model sees list pi-schema's messages (`- path: missing`, `- limit: expected number, got string`) instead of TypeBox's (`must have required properties path`, `must be number`); which arguments are accepted and how they are coerced is unchanged for extension and built-in tools. - Extensions keep importing `Type`, `Static` and `TSchema` from `@earendil-works/pi-ai`: Pi maps that import to pi-ai's compat entry plus the TypeBox it bundles, now that pi-ai itself no longer re-exports TypeBox, and its `StringEnum` there stays a TypeBox `TUnsafe`. The built-in tool factories (`createReadTool()` and the others) return `TypeBoxAgentTool`, which keeps `params` typed for TypeBox. - Tightened system prompt - `pi update` for installs from the pi.dev installer no longer passes `--min-release-age=0` to `npm ci`. The release lockfile pins every package, so the flag had no effect, and security scanners that block it stopped the update ([#11262](https://github.com/earendil-works/pi/issues/11262)) - `/model` and `/scoped-models` now show a model's name next to its ID when the name adds words the ID lacks, such as a renamed `models.json` override ([#5916](https://github.com/earendil-works/pi/issues/5916), [#11347](https://github.com/earendil-works/pi/pull/11347)) - MCP OAuth sign-in now uses pi's Client ID Metadata Document by default when the authorization server supports it for public clients, and registers a client dynamically otherwise, like Codex, Claude Code, and opencode. Clients registered before stay in use while their redirect URI matches. The new default `oauth.clientRegistration: "auto"` registers when `clientName` or a non-default `callbackUrl` is set; `"dcr"` always registers. `pi mcp add --oauth-client-registration` sets the mode ([#11426](https://github.com/earendil-works/pi/issues/11426)) +- Changed the default OpenAI model from `gpt-5.5` to `gpt-6.1-sol`, including after ChatGPT sign-in when no model is selected ([#11442](https://github.com/earendil-works/pi/issues/11442)) +- Updated the built-in model catalog to models.dev v2. Realtime and image-only models are no longer offered as chat models; newer OpenAI and Google chat models are included ### Fixed @@ -37,7 +54,7 @@ - Fixed HTML export showing a blank page for sessions with thousands of entries ([#11225](https://github.com/earendil-works/pi/issues/11225), [#11293](https://github.com/earendil-works/pi/pull/11293)) - Fixed HTML export not showing system messages: the system prompt and tools, and later section updates such as after `/reload`. `pi --export` now includes the system prompt and tools too ([#11261](https://github.com/earendil-works/pi/issues/11261), [#3464](https://github.com/earendil-works/pi/issues/3464), [#11290](https://github.com/earendil-works/pi/pull/11290)) - Fixed aborting a bash command on macOS sometimes running the next command of a list such as `long_step; next_step`. pi now stops the process group before killing it ([#11252](https://github.com/earendil-works/pi/issues/11252), [#11289](https://github.com/earendil-works/pi/pull/11289)) -- Fixed streamed tool calls re-running their call renderer on every argument delta; the interactive mode now renders the latest arguments once per frame, so a large streamed `write` no longer re-highlights its whole content per delta ([#9265](https://github.com/earendil-works/pi/issues/9265)) +- Fixed large streamed tool calls taking excessive CPU time and memory because every delta re-parsed the accumulated arguments. Providers and agent proxy streams now parse each delta once, grammar tools no longer copy their entire input per delta, and interactive tool renderers run at most once per frame ([#9265](https://github.com/earendil-works/pi/issues/9265)) - Fixed the login dialog opening sign-in URLs that are not web URLs, and printing control characters from sign-in, device-code, and info URLs, which could end a terminal hyperlink and send escape sequences to the terminal - Fixed `tui.select.pageUp` and `tui.select.pageDown` being ignored in slash command and file autocomplete, `/settings`, and the model, scoped models, login, fork, `/mcp`, and llama.cpp selectors. In fullscreen mode, PageUp/PageDown now move an open list or autocomplete menu instead of scrolling the transcript, and a page key bound to Ctrl+V pages autocomplete instead of pasting an image ([#7629](https://github.com/earendil-works/pi/issues/7629), based on [#7680](https://github.com/earendil-works/pi/pull/7680) by [@muyiyr](https://github.com/muyiyr)) - Fixed tool output and `!` command output showing the parameters of control sequences with a `<`, `=` or `>` prefix as text, for example `35;59;1M` from a mouse report `ESC[<35;59;1M` or `0;276;0c` from a terminal reply `ESC[>0;276;0c`. Text after the keypad-mode sequences `ESC=` and `ESC>` is still kept ([#11377](https://github.com/earendil-works/pi/issues/11377)) @@ -52,6 +69,47 @@ - Fixed `tool_search` and codemode `searchTools()` finding nothing for Chinese, Japanese, Thai, and other non-ASCII queries, because the tokenizer kept only ASCII letters and digits. Words in any script now match, text without spaces between words such as `搜索记录` is split into words, and canonically equivalent text such as `café` and `cafe\u0301` matches ([#10816](https://github.com/earendil-works/pi/issues/10816), [#11037](https://github.com/earendil-works/pi/pull/11037) by [@haoqixu](https://github.com/haoqixu)) - Fixed `pi update` stopping with `git clean -fdx failed with code 1` on Windows when another pi session had a git extension's native module or executable open. Windows cannot delete such files, so the update left the new code with the old dependencies. pi now warns, installs the dependencies anyway, and retries the clean on the next update ([#11597](https://github.com/earendil-works/pi/issues/11597)) - Fixed tool rows and streamed answers highlighting the same code again on every redraw. A running codemode row highlighted its whole script on every progress update and timer tick, a complete `write` row its whole file on every update and expand toggle, a streamed `write` its first 50 lines on every delta, and a streamed answer every earlier code block on every delta. Rows now reuse their last highlighting until the code, theme, or loaded grammars change, and assistant messages keep their rendered Markdown blocks while streaming ([#10971](https://github.com/earendil-works/pi/issues/10971)) +- Fixed resized photos being sent as multi-megabyte PNGs even when JPEG was much smaller, causing large requests to fail or time out. Image resizing now tries the smaller encoding first ([#11551](https://github.com/earendil-works/pi/issues/11551)) +- Fixed `pi update --extensions` reporting success without warning when npm's stale registry metadata left a package at its old version. pi now reports the installed and expected versions ([#11536](https://github.com/earendil-works/pi/issues/11536)) +- Fixed skill validation warnings being labeled `[Skill conflicts]` even when there was no name collision. They are now shown under `[Skill warnings]` ([#11521](https://github.com/earendil-works/pi/issues/11521)) +- Fixed codemode tool descriptions including entire union and multi-line output types in every request. Object unions now summarize their fields and discriminators; other multi-line types point to `describeTool()` for the full declaration ([#11501](https://github.com/earendil-works/pi/issues/11501)) +- Fixed the model-scope startup notice being left in the shell after exiting fullscreen mode. It now appears inside pi's fullscreen header ([#11508](https://github.com/earendil-works/pi/issues/11508)) +- Fixed startup from a deleted working directory crashing with an uncaught `uv_cwd` stack trace. pi now asks the user to change to an existing directory ([#11509](https://github.com/earendil-works/pi/issues/11509)) +- Fixed failed session writes leaving entries in memory that were absent from disk, so later entries pointed to missing parents and resumed sessions lost earlier history and tools ([#11460](https://github.com/earendil-works/pi/issues/11460)) +- Fixed `/tree` branch summaries ignoring the session's thinking level, which could make reasoning models reject summary requests with an unsupported `none` effort ([#11444](https://github.com/earendil-works/pi/issues/11444)) +- Fixed hidden thinking with an empty hidden-thinking label still adding blank rows before answers and tool calls ([#11402](https://github.com/earendil-works/pi/issues/11402)) +- Fixed package installation and removal printing npm audit and funding advice, including `npm audit fix` instructions that could install a second copy of pi into the package directory ([#11409](https://github.com/earendil-works/pi/issues/11409)) +- Fixed replacing an extension editor while a dialog was open hiding the dialog and leaving it unanswerable. The new editor now appears after the dialog closes ([#11431](https://github.com/earendil-works/pi/issues/11431)) +- Fixed `grep` context lines being offset from ripgrep's match line numbers when a file contains a lone carriage return ([#11398](https://github.com/earendil-works/pi/issues/11398)) +- Fixed `--model /` failing for provider IDs containing slashes, such as URL-derived llama.cpp provider IDs ([#11394](https://github.com/earendil-works/pi/issues/11394)) +- Fixed slow session forking on filesystems where opening files is expensive by writing the fork through one file handle ([#11253](https://github.com/earendil-works/pi/issues/11253)) +- Fixed `pi.exec()` corrupting UTF-8 characters split across stdout or stderr chunks ([#11312](https://github.com/earendil-works/pi/issues/11312)) +- Fixed `pi.exec()` returning empty stderr when a command could not be spawned, such as a missing executable. It now preserves the spawn error message ([#11257](https://github.com/earendil-works/pi/pull/11257) by [@muyiyr](https://github.com/muyiyr)) +- Fixed a configured package failing to install crashing startup. pi now skips the package, reports an extension warning, and retries on the next start or `/reload`; explicit install and update commands still fail with a nonzero exit ([#11304](https://github.com/earendil-works/pi/pull/11304) by [@Lincoln504](https://github.com/Lincoln504)) +- Fixed the todo extension example crashing the TUI when a tool call failed argument validation ([#8057](https://github.com/earendil-works/pi/pull/8057) by [@cyzlmh](https://github.com/cyzlmh)) +- Updated `undici` to 8.12.3 and the Gondolin extension example to 0.14.0 for security advisories +- Fixed OpenRouter cost totals using catalog estimates instead of reported billing for Chat Completions and Anthropic Messages. BYOK totals include both OpenRouter charges and reported upstream costs; component costs remain catalog estimates ([#9980](https://github.com/earendil-works/pi/issues/9980)) +- Fixed OpenCode Zen and OpenCode Go requests using shared endpoints instead of their protocol-specific OpenAI, Anthropic, Google, and System One endpoints ([#11434](https://github.com/earendil-works/pi/pull/11434) by [@vimtor](https://github.com/vimtor)) +- Fixed Zhipu/Z.ai usage-limit errors (code `1308`) being treated as transient HTTP 429 errors, causing automatic retries that cannot succeed until the usage cap resets ([#11537](https://github.com/earendil-works/pi/issues/11537)) +- Fixed LiteLLM's `ContextWindowExceededError` not being recognized as context overflow ([#11492](https://github.com/earendil-works/pi/issues/11492)) +- Fixed Gemini tool schemas sending unsupported `const` keywords. They are now converted to single-value `enum` schemas without rewriting literal data in defaults, examples, or enum values, or losing properties named `__proto__` ([#11308](https://github.com/earendil-works/pi/issues/11308)) +- Fixed the faux provider counting simulated cache-write tokens as both input and cache writes +- Fixed Orca terminals falling back to text for inline images. Orca is now recognized as supporting true color and hyperlinks, and Kitty images when it advertises `ORCA_IMAGE_PROTOCOL=kitty` ([#9329](https://github.com/earendil-works/pi/pull/9329) by [@XRX193](https://github.com/XRX193)) +- Fixed bare Markdown URLs swallowing following CJK punctuation and text; explicit Markdown links and angle autolinks are unchanged ([#10934](https://github.com/earendil-works/pi/pull/10934) by [@haoqixu](https://github.com/haoqixu)) +- Fixed Kitty-protocol F1–F12 shortcuts, including modified function keys, not being recognized, and F13+ and other functional keys inserting invisible private-use characters into the editor ([#11419](https://github.com/earendil-works/pi/issues/11419), [#11345](https://github.com/earendil-works/pi/issues/11345)) +- Fixed Kitty images inside padded or colored boxes disappearing when their first row scrolled above the fullscreen viewport. Partially visible images are now cropped correctly, including beside other text +- Fixed deeply nested Markdown blockquotes producing exponentially many lines, hanging or crashing rendering once their borders exhausted the available width ([#11272](https://github.com/earendil-works/pi/issues/11272)) +- Fixed Claude sessions failing with `role 'system' must precede an 'assistant' message` when a system update was followed by an empty or reasoning-only assistant reply ([#11091](https://github.com/earendil-works/pi/issues/11091)) +- Fixed Claude mid-conversation thinking-level changes failing with `Invalid effort level` on accounts whose organization caps effort below `high` ([#10063](https://github.com/earendil-works/pi/issues/10063)) +- Fixed Kimi K3.1 Pro models using Kimi K2 request settings on Kimi Coding, Moonshot, and OpenCode Go. They now send the correct reasoning effort, preserve earlier thinking, accept tools added mid-conversation, and use K3-family pricing +- Fixed OpenCode DeepSeek V4.1 Flash requests failing when long cache retention sent the unsupported `prompt_cache_retention` parameter +- Fixed GPT-5.4 and GPT-5.5 failing with `Missing namespace for function_call` after switching from a model that had called tools ([#11415](https://github.com/earendil-works/pi/issues/11415)) +- Fixed Bedrock Claude requests failing when preferred strict tools exceed the provider's strict-tool limits. Requests now retry once with preferred tools non-strict while keeping required strict tools strict ([#11416](https://github.com/earendil-works/pi/issues/11416)) +- Fixed OpenRouter's `~anthropic/claude-*-latest` aliases losing earlier thinking after tool calls by using the Anthropic Messages API ([#9534](https://github.com/earendil-works/pi/issues/9534)) +- Fixed terminal titles containing control characters being able to inject terminal escape sequences, and pi-messages streams accepting invalid content-block indices +- Fixed fullscreen double-click selection splitting paths and tokens at `~`, `.`, `\`, and `+`; word characters now match iTerm2's defaults ([#10980](https://github.com/earendil-works/pi/issues/10980)) +- Fixed display math wider than the content area being split apart by line wrapping; it now falls back to raw LaTeX ([#11483](https://github.com/earendil-works/pi/issues/11483)) +- Fixed styled words ending exactly at the wrap width adding a leading space on the next line, and strikethrough leaking into wrapped-line padding ([#10925](https://github.com/earendil-works/pi/issues/10925)) ### Removed diff --git a/packages/tui/CHANGELOG.md b/packages/tui/CHANGELOG.md index 1b13cf0a7..04fdf0893 100644 --- a/packages/tui/CHANGELOG.md +++ b/packages/tui/CHANGELOG.md @@ -20,6 +20,11 @@ - Fixed `wrapTextWithAnsi()` starting the next line with a space when a styled word ended exactly at the wrap width, and strikethrough continuing into the right padding of wrapped lines ([#10925](https://github.com/earendil-works/pi/issues/10925)) - Fixed `Markdown` styling, highlighting, and wrapping every block on each render. It now reuses the top-level blocks of its last render, so a streaming message renders only its changed last block, and a resize reuses highlighted code. Reused blocks are held weakly and dropped by `invalidate()`, which a theme change must call because `MarkdownTheme.highlightCode` results are reused until then ([#10971](https://github.com/earendil-works/pi/issues/10971)) - Fixed `Text.setText()` with unchanged text dropping the wrapped lines, so renderers that set the same text on every update wrapped it again ([#10971](https://github.com/earendil-works/pi/issues/10971)) +- Fixed Orca terminals falling back to text for inline images. Orca is now recognized as supporting true color and hyperlinks, and Kitty images when it advertises `ORCA_IMAGE_PROTOCOL=kitty` ([#9329](https://github.com/earendil-works/pi/pull/9329) by [@XRX193](https://github.com/XRX193)) +- Fixed bare Markdown URLs swallowing following CJK punctuation and text; explicit Markdown links and angle autolinks are unchanged ([#10934](https://github.com/earendil-works/pi/pull/10934) by [@haoqixu](https://github.com/haoqixu)) +- Fixed Kitty-protocol F1–F12 shortcuts, including modified function keys, not being recognized, and F13+ and other functional keys inserting invisible private-use characters into the editor ([#11419](https://github.com/earendil-works/pi/issues/11419), [#11345](https://github.com/earendil-works/pi/issues/11345)) +- Fixed Kitty images inside padded or colored boxes disappearing when their first row scrolled above the fullscreen viewport. Partially visible images are now cropped correctly, including beside other text +- Fixed deeply nested Markdown blockquotes producing exponentially many lines, hanging or crashing rendering once their borders exhausted the available width ([#11272](https://github.com/earendil-works/pi/issues/11272)) ## [1.4.0] - 2026-10-21 From ee7f61f92c89da29b2df05b21b2bb59fee855559 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 10:19:47 +0000 Subject: [PATCH 126/175] docs(chord): the plugin model's points 3, 4, 5 and 8, imports and builds --- packages/chord/docs/next/decisions.md | 40 +++++++++++++++++++++++++++ packages/chord/docs/next/slices.md | 5 ++++ 2 files changed, 45 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 3b38c7bf3..594900030 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2165,6 +2165,46 @@ validation and copies the v2 kernel added around Delta. Contracts are data and bind through `accepts` whatever Chord runs. One facet imports from one API version only (checked at build). Only base code imports Chord directly. Bundling is slice 10's host packages: the version is part of the name (`kinds.hostPackages` lists `@acme/app/v3`, `@acme/app/v2`), each process provides its own modules. +- **The plugin model, points 3, 4, 5 and 8, imports and builds (owner, 2026-11-01, by voice: "yeah that checks out"; "lock the decisions on the + points which was discussed"; primer `docs/next/plugin-primer/`):** + (3) **Split by host kind, for the app and for plugins alike.** The app's API per version has a shared module + (`@acme/app/v3`: Chord's re-exported surface, host kinds, the principal type, `defineFacet` bound to it, contracts + and registries that cross hosts) and per-kind modules (`/v3/web`, `/v3/tui`, `/v3/server`, `/v3/worker` where + needed: UI components, registries and `local` contracts that live on that kind only). A plugin's contract files + mirror it: `contract.ts` (shared) and `contract.server.ts`, `contract.worker.ts`, `contract.web.tsx`, + `contract.tui.ts` as needed. A contract file mentions native things (React components, functions) only through + `import type`; at runtime it imports only the shared API and its plugin's data-only files (a hard rule). The build + checks every import by the file it resolves to: a per-kind file (`.web.` etc.) or per-kind API module only for that + kind, and a file importing a package the kind does not provide (React in a worker) refused. Shared UI between + plugins: a plain component through an imported file (inlined into the importer's script, frozen at its build), or + a registry when one live instance is needed (React contexts, a feature others embed). + (4) **Host kinds on registry tokens** are covered by placement: a token used on one kind lives in that kind's file, + so other kinds cannot import it; an explicit `hosts` marker only if a shared token must be limited later. + Registries version by hand: readers and writers work across versions through the token's version and the defining + plugin's `accepts`; no fingerprint for native values (decided earlier, unchanged). + (5) **Third parties get the app's contracts through its versioned API:** each built-in plugin keeps its contract + files; the API re-exports the ones the app promises (`export { Notes } from "../plugins/notes/contract.ts"`), so a + plugin built against `/v2` keeps v2's contracts; the others stay private. No per-area subpackages. + (Imports) **Plainly, with no resolution magic:** within one repository (a root, or a repository of several + plugins), relative paths to the files (`../notes/contract.web.tsx`); across repositories, the package name with its + exports (`@bob/notes`, `@bob/notes/web`), pinned in `dependencies` by an npm version or a git ref + (`github:bob/notes#`; npm's git dependencies, no registry publishing needed; a plugin in a subfolder of a + multi-plugin repository is cloned and linked by the plugins component, which npm cannot do). Rejected: one + specifier resolving to a sibling or an npm package depending on `dependencies` ("kinda magic"). Plugins are + installed from git by cloning a branch, tag or commit into a plugins root (as the coding agent's git packages). + (8, base code) **Base code shrinks to the API, bootstrap and a start-data facet:** the vocabulary (the versioned API modules, + host packages, never reloaded), bootstrap (creating hosts, opening the store, components such as `createHttp` and + `createWorkers`, which take settings and process values and return facets, their natives opened by Chord, passed as + the same object at every reload) and one small facet serving process values (start data, the opened store) as core + objects. Every implementation, the harness and the providers of core registries included, is a plugin in the app's + own root, loaded, reloaded and failing per plugin like any other; the app may mark its root required, so a core + plugin failing at startup is fatal. Built-in plugins follow the same rules as any plugin. + (Build) **No build hooks from plugins:** the server builds every plugin with one esbuild call from TypeScript source + (no `dist` needed; installs run with scripts off); what the build handles (TS, TSX, JS, JSON, CSS, assets and WASM + by URL) is the plugin's toolbox, generated files are committed; a CSS preprocessor or similar, if ever needed, is + enabled by the app per host kind, never by a plugin (a stranger's web plugin must not run code on the server). + To do: a skill and documentation for agents describing a plugin's layout, its contract files, imports and how the + host builds it. Open from the primer: (6) `testPlugin` with stub facets, (7) single-file plugins. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a diff --git a/packages/chord/docs/next/slices.md b/packages/chord/docs/next/slices.md index 084d3456e..c4619becc 100644 --- a/packages/chord/docs/next/slices.md +++ b/packages/chord/docs/next/slices.md @@ -187,6 +187,11 @@ deleted (the slice's spec and code are the record). Moved here from `carry-over. from"): remove `build.ts`'s by-name lookup, allow relative imports into another plugin of the same root, refuse paths leaving the root; examples 35-38 and fixtures back to relative imports; drop the six `tsconfig` path aliases; plugins README; the guide's chapter "Plugins Using Plugins". (Slice 10 built by-name imports.) +- **The plugin model's other points** (decision "The plugin model, points 3, 4, 5 and 8, imports and builds"): per-kind + API modules and plugin contract files with the build's per-kind import check; package-name imports with npm or git + pins (subfolders of multi-plugin repositories cloned and linked); installing plugins from git; base code reduced to + the API, bootstrap and a start-data facet, the harness and core registry providers as plugins, a required root; a + skill and documentation for agents on a plugin's layout and build. ### After slice 11 From 95e2015f3bbca7bace8311b0ddf743367382b9b1 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 10:22:50 +0000 Subject: [PATCH 127/175] docs(chord): single-file plugins --- packages/chord/docs/next/decisions.md | 7 +++++++ packages/chord/docs/next/slices.md | 2 ++ 2 files changed, 9 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 594900030..b972b1044 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2205,6 +2205,13 @@ validation and copies the v2 kernel added around Delta. enabled by the app per host kind, never by a plugin (a stranger's web plugin must not run code on the server). To do: a skill and documentation for agents describing a plugin's layout, its contract files, imports and how the host builds it. Open from the primer: (6) `testPlugin` with stub facets, (7) single-file plugins. +- **Single-file plugins (owner, 2026-11-01: "agreed"; plugin model point 7):** a root may hold plain files named + `..ts(x)` (`hello.server.ts`, `hello.web.tsx`): files with the same first name are one plugin named + after it, one facet per file. The name must be a valid unscoped name; a collision with a directory plugin of the + same root is reported like any duplicate. No `package.json`, so no npm dependencies; it imports only the app's API + (host packages), no other files and no contract files for others; it may serve objects, define contracts for + itself inline and contribute to registries. When it needs more, it becomes a directory with the same name, which a + reload treats as the same plugin. The watcher's `set` mark covers files appearing and disappearing at a root's top. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a diff --git a/packages/chord/docs/next/slices.md b/packages/chord/docs/next/slices.md index c4619becc..9b9c357c1 100644 --- a/packages/chord/docs/next/slices.md +++ b/packages/chord/docs/next/slices.md @@ -192,6 +192,8 @@ deleted (the slice's spec and code are the record). Moved here from `carry-over. pins (subfolders of multi-plugin repositories cloned and linked); installing plugins from git; base code reduced to the API, bootstrap and a start-data facet, the harness and core registry providers as plugins, a required root; a skill and documentation for agents on a plugin's layout and build. +- **Single-file plugins** (decision "Single-file plugins"): discovery of `..ts(x)` files, grouped by + name; their import rule; the watcher's `set` for them. ### After slice 11 From 7edaa131dd2c1d7ebcedb8ca76da7e38ce719c74 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 10:26:25 +0000 Subject: [PATCH 128/175] docs(chord): testing a plugin alone --- packages/chord/docs/next/decisions.md | 11 +++++++++++ packages/chord/docs/next/slices.md | 2 ++ 2 files changed, 13 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index b972b1044..ea37ad100 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2212,6 +2212,17 @@ validation and copies the v2 kernel added around Delta. (host packages), no other files and no contract files for others; it may serve objects, define contracts for itself inline and contribute to registries. When it needs more, it becomes a directory with the same name, which a reload treats as the same plugin. The watcher's `set` mark covers files appearing and disappearing at a root's top. +- **Testing a plugin alone (owner, 2026-11-01: "let's do that"; plugin model point 6):** `testPlugin(dir, { kind, + modules, stubs, start?, principal? }, ctx)` builds the plugin as production does and loads it on an in-memory host + of that kind with an in-memory store. A plugin depends on four things, each answered without a mocking framework: + the app's API and Chord are the real modules (definitions only); other plugins' and core objects are reached + through contracts, so the test passes stubs, ordinary facets serving those contracts with fake data; registries + the plugin contributes to are served by the helper, which collects the contributions for the test to read (a + registry it reads gets fake entries from a stub); process values come as fake start data. The result is a live host: + the test gets the plugin's objects, calls them, watches their state, and acts as any principal. A contract no stub + serves leaves its requests waiting `not_served`; the helper lists those names, so the test says what to stub. + Several hosts at once (a server plugin calling a worker) is a later, larger helper over memory links; the app kit + may ship stubs for its core contracts. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a diff --git a/packages/chord/docs/next/slices.md b/packages/chord/docs/next/slices.md index 9b9c357c1..3c300ee62 100644 --- a/packages/chord/docs/next/slices.md +++ b/packages/chord/docs/next/slices.md @@ -194,6 +194,8 @@ deleted (the slice's spec and code are the record). Moved here from `carry-over. skill and documentation for agents on a plugin's layout and build. - **Single-file plugins** (decision "Single-file plugins"): discovery of `..ts(x)` files, grouped by name; their import rule; the watcher's `set` for them. +- **`testPlugin`** (decision "Testing a plugin alone"): stub facets, collected contributions, the list of contracts + nobody serves. ### After slice 11 From 2dd44fb1c8078267c9507d95d257671dfb7e990d Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 10:33:06 +0000 Subject: [PATCH 129/175] docs(chord): app kit shape --- packages/chord/docs/next/decisions.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index ea37ad100..fa7a39c94 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2223,6 +2223,15 @@ validation and copies the v2 kernel added around Delta. serves leaves its requests waiting `not_served`; the helper lists those names, so the test says what to stub. Several hosts at once (a server plugin calling a worker) is a later, larger helper over memory links; the app kit may ship stubs for its core contracts. +- **The app kit, point 1: its shape (owner, 2026-11-01, by voice: "yeah"; app kit design pass):** the app's own + layout is fixed: `src/` with the API versions (`src/api/v3/`, per host kind as decided for the plugin model) and the + built-in plugins root (`plugins/`). The data directory is deployment configuration, outside the app: a `startApp` + option set by the app's own start script (no environment variables: owner, "no environment variables"), default a + dot directory named after the app in the home directory; it holds the store, the user's plugins root, logs and the + workers' data. The server is fixed (one per app, clients connect to it). Workers are a list of worker kinds the app + defines, each a host kind with the app's own params (sharding is the app's: a project, a tenant, a document; the kit + knows no sessions) and how to start it (entry, start data per address, principal, idle time: the workers + component's options); zero kinds is a server with clients only (a chat app). - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 87f69dbf3cef9a22f1b4ad82780edf8028aad3f3 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 10:36:42 +0000 Subject: [PATCH 130/175] docs(chord): app kit principals and users --- packages/chord/docs/next/decisions.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index fa7a39c94..e513dce86 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2232,6 +2232,16 @@ validation and copies the v2 kernel added around Delta. defines, each a host kind with the app's own params (sharding is the app's: a project, a tenant, a document; the kit knows no sessions) and how to start it (entry, start data per address, principal, idle time: the workers component's options); zero kinds is a server with clients only (a chat app). +- **The app kit, point 2: principals and users (owner, 2026-11-01, by voice: "yeah I guess. just make it nice and + concise and easy to use and flexible"; "too opinionated might be a problem"):** three things kept apart. (1) The + principal is tiny (a kind and an id: it travels on every call and is part of objects' identity) and the app defines + its type; the kit requires only the variants it needs itself (the server and its worker kinds). (2) User data + (avatar, bio, availability) is ordinary app data in objects keyed by the user's id, never in the principal. (3) + Authentication turns a connection into a principal. The kit ships optional plugins: a users plugin storing accounts + with a profile schema the app supplies (a Slack-like profile or just a name), and authenticator plugins (device + pairing, invite links, an ephemeral one for an IRC-like "pick a name and you're in": a fresh id with the name, no + stored account, gone with the connection); an app picks, replaces or writes its own. Rejected: a fixed principal type + with roles. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 7e7c98a0f505075f67291d4425cf41643678f278 Mon Sep 17 00:00:00 2001 From: Armin Ronacher Date: Mon, 2 Nov 2026 11:27:14 +0100 Subject: [PATCH 131/175] docs(coding-agent): note branch summary context edits --- packages/coding-agent/CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/packages/coding-agent/CHANGELOG.md b/packages/coding-agent/CHANGELOG.md index 46a5bbc9f..15c302597 100644 --- a/packages/coding-agent/CHANGELOG.md +++ b/packages/coding-agent/CHANGELOG.md @@ -110,6 +110,7 @@ - Fixed fullscreen double-click selection splitting paths and tokens at `~`, `.`, `\`, and `+`; word characters now match iTerm2's defaults ([#10980](https://github.com/earendil-works/pi/issues/10980)) - Fixed display math wider than the content area being split apart by line wrapping; it now falls back to raw LaTeX ([#11483](https://github.com/earendil-works/pi/issues/11483)) - Fixed styled words ending exactly at the wrap width adding a leading space on the next line, and strikethrough leaking into wrapped-line padding ([#10925](https://github.com/earendil-works/pi/issues/10925)) +- Fixed `/tree` branch summaries including failed assistant replies and original content that context edits had hidden or replaced. Summaries now honor the same context edits as live requests and compaction ([#11602](https://github.com/earendil-works/pi/issues/11602)) ### Removed From 513f6f1ce8fc982b36ae0e9e2ef4c7379cd78fe3 Mon Sep 17 00:00:00 2001 From: Armin Ronacher Date: Mon, 2 Nov 2026 11:30:55 +0100 Subject: [PATCH 132/175] fix: keep Chord example dependencies out of releases --- .../37-plugin-imports/deps/example-greeting/package.json | 1 + .../examples/37-plugin-imports/deps/example-native/package.json | 1 + 2 files changed, 2 insertions(+) diff --git a/packages/chord/test/next/examples/37-plugin-imports/deps/example-greeting/package.json b/packages/chord/test/next/examples/37-plugin-imports/deps/example-greeting/package.json index ae18d3b71..40d2288d8 100644 --- a/packages/chord/test/next/examples/37-plugin-imports/deps/example-greeting/package.json +++ b/packages/chord/test/next/examples/37-plugin-imports/deps/example-greeting/package.json @@ -1,6 +1,7 @@ { "name": "example-greeting", "version": "1.0.0", + "private": true, "type": "module", "main": "index.js" } diff --git a/packages/chord/test/next/examples/37-plugin-imports/deps/example-native/package.json b/packages/chord/test/next/examples/37-plugin-imports/deps/example-native/package.json index c05122b96..0e7a2555f 100644 --- a/packages/chord/test/next/examples/37-plugin-imports/deps/example-native/package.json +++ b/packages/chord/test/next/examples/37-plugin-imports/deps/example-native/package.json @@ -1,6 +1,7 @@ { "name": "example-native", "version": "1.0.0", + "private": true, "type": "commonjs", "main": "index.js" } From f9a26365568127461c47a07a37dcb42c140a3ac5 Mon Sep 17 00:00:00 2001 From: Armin Ronacher Date: Mon, 2 Nov 2026 11:35:40 +0100 Subject: [PATCH 133/175] Release v1.5.0 --- nix/model-catalog.json | 2 +- package-lock.json | 72 +++++++++---------- packages/agent/CHANGELOG.md | 2 +- packages/agent/package.json | 6 +- packages/ai/CHANGELOG.md | 2 +- packages/ai/package.json | 6 +- packages/ai/src/utils/version.ts | 2 +- packages/chord/CHANGELOG.md | 2 +- packages/chord/package.json | 6 +- packages/codemode/CHANGELOG.md | 2 +- packages/codemode/package.json | 2 +- packages/coding-agent/CHANGELOG.md | 2 +- .../package-lock.json | 4 +- .../custom-provider-anthropic/package.json | 2 +- .../custom-provider-gitlab-duo/package.json | 2 +- .../extensions/gondolin/package-lock.json | 4 +- .../examples/extensions/gondolin/package.json | 2 +- .../extensions/with-deps/package-lock.json | 4 +- .../extensions/with-deps/package.json | 2 +- .../install-lock/package-lock.json | 56 +++++++-------- .../coding-agent/install-lock/package.json | 4 +- packages/coding-agent/package.json | 12 ++-- packages/durable/CHANGELOG.md | 2 +- packages/durable/package.json | 12 ++-- packages/env/CHANGELOG.md | 2 +- packages/env/package.json | 6 +- packages/evals/package.json | 6 +- packages/mcp/CHANGELOG.md | 2 +- packages/mcp/package.json | 2 +- packages/schema/CHANGELOG.md | 2 +- packages/schema/package.json | 2 +- packages/telemetry/CHANGELOG.md | 2 +- packages/telemetry/package.json | 2 +- packages/tui/CHANGELOG.md | 2 +- packages/tui/package.json | 2 +- 35 files changed, 121 insertions(+), 121 deletions(-) diff --git a/nix/model-catalog.json b/nix/model-catalog.json index 3c31c2cca..3b4641ee5 100644 --- a/nix/model-catalog.json +++ b/nix/model-catalog.json @@ -1,3 +1,3 @@ { - "revision": "sha256-7fe1b41feaf43a7aad4bc832e6b306c7ced56e6d41bbc724b74e91eeacff6edf" + "revision": "sha256-f4dbabe79cda4b38aee4a761b23fd5eaf526ac2e28fed3c9ee86e37ca0e5d71e" } diff --git a/package-lock.json b/package-lock.json index 8abc3354d..a0749b93b 100644 --- a/package-lock.json +++ b/package-lock.json @@ -5654,11 +5654,11 @@ }, "packages/agent": { "name": "@earendil-works/pi-agent-core", - "version": "1.4.0", + "version": "1.5.0", "license": "MIT", "dependencies": { - "@earendil-works/pi-ai": "1.4.0", - "@earendil-works/pi-schema": "1.4.0" + "@earendil-works/pi-ai": "1.5.0", + "@earendil-works/pi-schema": "1.5.0" }, "devDependencies": { "@types/node": "22.19.19", @@ -5677,13 +5677,13 @@ }, "packages/ai": { "name": "@earendil-works/pi-ai", - "version": "1.4.0", + "version": "1.5.0", "license": "MIT", "dependencies": { "@anthropic-ai/sdk": "0.129.0", "@aws-sdk/client-bedrock-runtime": "3.1127.0", - "@earendil-works/pi-schema": "1.4.0", - "@earendil-works/pi-telemetry": "1.4.0", + "@earendil-works/pi-schema": "1.5.0", + "@earendil-works/pi-telemetry": "1.5.0", "@google/genai": "2.21.0", "@smithy/node-http-handler": "4.12.1", "http-proxy-agent": "9.1.0", @@ -5766,11 +5766,11 @@ }, "packages/chord": { "name": "@earendil-works/chord", - "version": "1.4.0", + "version": "1.5.0", "license": "MIT", "dependencies": { - "@earendil-works/pi-schema": "1.4.0", - "@earendil-works/pi-telemetry": "1.4.0" + "@earendil-works/pi-schema": "1.5.0", + "@earendil-works/pi-telemetry": "1.5.0" }, "devDependencies": { "@types/ws": "8.18.2", @@ -5795,7 +5795,7 @@ }, "packages/codemode": { "name": "@earendil-works/pi-codemode", - "version": "1.4.0", + "version": "1.5.0", "license": "MIT", "dependencies": { "quickjs-wasi": "3.6.2" @@ -5809,14 +5809,14 @@ }, "packages/coding-agent": { "name": "@earendil-works/pi-coding-agent", - "version": "1.4.0", + "version": "1.5.0", "license": "MIT", "dependencies": { - "@earendil-works/pi-agent-core": "1.4.0", - "@earendil-works/pi-ai": "1.4.0", - "@earendil-works/pi-codemode": "1.4.0", - "@earendil-works/pi-mcp": "1.4.0", - "@earendil-works/pi-tui": "1.4.0", + "@earendil-works/pi-agent-core": "1.5.0", + "@earendil-works/pi-ai": "1.5.0", + "@earendil-works/pi-codemode": "1.5.0", + "@earendil-works/pi-mcp": "1.5.0", + "@earendil-works/pi-tui": "1.5.0", "@silvia-odwyer/photon-node": "0.3.4", "brace-expansion": "5.0.12", "chalk": "6.0.0", @@ -5853,25 +5853,25 @@ }, "packages/coding-agent/examples/extensions/custom-provider-anthropic": { "name": "pi-extension-custom-provider-anthropic", - "version": "1.4.0", + "version": "1.5.0", "dependencies": { "@anthropic-ai/sdk": "0.52.0" } }, "packages/coding-agent/examples/extensions/custom-provider-gitlab-duo": { "name": "pi-extension-custom-provider-gitlab-duo", - "version": "1.4.0" + "version": "1.5.0" }, "packages/coding-agent/examples/extensions/gondolin": { "name": "pi-extension-gondolin", - "version": "1.4.0", + "version": "1.5.0", "dependencies": { "@earendil-works/gondolin": "0.14.0" } }, "packages/coding-agent/examples/extensions/with-deps": { "name": "pi-extension-with-deps", - "version": "1.4.0", + "version": "1.5.0", "dependencies": { "ms": "2.1.3" }, @@ -5925,14 +5925,14 @@ }, "packages/durable": { "name": "@earendil-works/pi-durable", - "version": "1.4.0", + "version": "1.5.0", "license": "MIT", "dependencies": { - "@earendil-works/chord": "1.4.0", - "@earendil-works/pi-ai": "1.4.0", - "@earendil-works/pi-codemode": "1.4.0", - "@earendil-works/pi-schema": "1.4.0", - "@earendil-works/pi-telemetry": "1.4.0", + "@earendil-works/chord": "1.5.0", + "@earendil-works/pi-ai": "1.5.0", + "@earendil-works/pi-codemode": "1.5.0", + "@earendil-works/pi-schema": "1.5.0", + "@earendil-works/pi-telemetry": "1.5.0", "@silvia-odwyer/photon": "0.3.3", "diff": "8.0.4" }, @@ -5947,11 +5947,11 @@ }, "packages/env": { "name": "@earendil-works/pi-env", - "version": "1.4.0", + "version": "1.5.0", "license": "MIT", "dependencies": { - "@earendil-works/chord": "1.4.0", - "@earendil-works/pi-durable": "1.4.0" + "@earendil-works/chord": "1.5.0", + "@earendil-works/pi-durable": "1.5.0" }, "devDependencies": { "vitest": "4.1.12" @@ -5962,10 +5962,10 @@ }, "packages/evals": { "name": "@earendil-works/pi-evals", - "version": "1.4.0", + "version": "1.5.0", "devDependencies": { - "@earendil-works/pi-ai": "1.4.0", - "@earendil-works/pi-coding-agent": "1.4.0", + "@earendil-works/pi-ai": "1.5.0", + "@earendil-works/pi-coding-agent": "1.5.0", "@types/node": "22.19.19", "@vitest-evals/core": "0.15.0", "vitest": "4.1.12", @@ -5974,7 +5974,7 @@ }, "packages/mcp": { "name": "@earendil-works/pi-mcp", - "version": "1.4.0", + "version": "1.5.0", "license": "MIT", "dependencies": { "cross-spawn": "7.0.6" @@ -5989,7 +5989,7 @@ }, "packages/schema": { "name": "@earendil-works/pi-schema", - "version": "1.4.0", + "version": "1.5.0", "license": "MIT", "devDependencies": { "@cfworker/json-schema": "4.1.1", @@ -6007,7 +6007,7 @@ }, "packages/telemetry": { "name": "@earendil-works/pi-telemetry", - "version": "1.4.0", + "version": "1.5.0", "license": "MIT", "devDependencies": { "@types/node": "22.19.19", @@ -6019,7 +6019,7 @@ }, "packages/tui": { "name": "@earendil-works/pi-tui", - "version": "1.4.0", + "version": "1.5.0", "license": "MIT", "dependencies": { "get-east-asian-width": "1.6.0", diff --git a/packages/agent/CHANGELOG.md b/packages/agent/CHANGELOG.md index 96b4b9428..a481d47f0 100644 --- a/packages/agent/CHANGELOG.md +++ b/packages/agent/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -## [Unreleased] +## [1.5.0] - 2026-11-02 ### Breaking Changes diff --git a/packages/agent/package.json b/packages/agent/package.json index 5d7091e4a..e00a8e1e7 100644 --- a/packages/agent/package.json +++ b/packages/agent/package.json @@ -1,6 +1,6 @@ { "name": "@earendil-works/pi-agent-core", - "version": "1.4.0", + "version": "1.5.0", "description": "General-purpose agent with transport abstraction, state management, and attachment support", "type": "module", "main": "./dist/index.js", @@ -23,8 +23,8 @@ "prepublishOnly": "npm run build" }, "dependencies": { - "@earendil-works/pi-ai": "1.4.0", - "@earendil-works/pi-schema": "1.4.0" + "@earendil-works/pi-ai": "1.5.0", + "@earendil-works/pi-schema": "1.5.0" }, "keywords": [ "ai", diff --git a/packages/ai/CHANGELOG.md b/packages/ai/CHANGELOG.md index 5d197d814..cc4f0d9ac 100644 --- a/packages/ai/CHANGELOG.md +++ b/packages/ai/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -## [Unreleased] +## [1.5.0] - 2026-11-02 ### Breaking Changes diff --git a/packages/ai/package.json b/packages/ai/package.json index 8de0434f8..57e8f262c 100644 --- a/packages/ai/package.json +++ b/packages/ai/package.json @@ -1,6 +1,6 @@ { "name": "@earendil-works/pi-ai", - "version": "1.4.0", + "version": "1.5.0", "description": "Unified LLM API with automatic model discovery and provider configuration", "type": "module", "main": "./dist/index.js", @@ -69,8 +69,8 @@ "dependencies": { "@anthropic-ai/sdk": "0.129.0", "@aws-sdk/client-bedrock-runtime": "3.1127.0", - "@earendil-works/pi-schema": "1.4.0", - "@earendil-works/pi-telemetry": "1.4.0", + "@earendil-works/pi-schema": "1.5.0", + "@earendil-works/pi-telemetry": "1.5.0", "@google/genai": "2.21.0", "@smithy/node-http-handler": "4.12.1", "http-proxy-agent": "9.1.0", diff --git a/packages/ai/src/utils/version.ts b/packages/ai/src/utils/version.ts index 18fe5a582..f1e6f3211 100644 --- a/packages/ai/src/utils/version.ts +++ b/packages/ai/src/utils/version.ts @@ -1,2 +1,2 @@ // The @earendil-works/pi-ai package version. scripts/sync-versions.js keeps it in sync with package.json. -export const PI_AI_VERSION = "1.4.0"; +export const PI_AI_VERSION = "1.5.0"; diff --git a/packages/chord/CHANGELOG.md b/packages/chord/CHANGELOG.md index 1a1382eb7..b4fd5fdfd 100644 --- a/packages/chord/CHANGELOG.md +++ b/packages/chord/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -## [Unreleased] +## [1.5.0] - 2026-11-02 ### Breaking Changes diff --git a/packages/chord/package.json b/packages/chord/package.json index edced8cd5..6e8e12a4d 100644 --- a/packages/chord/package.json +++ b/packages/chord/package.json @@ -1,6 +1,6 @@ { "name": "@earendil-works/chord", - "version": "1.4.0", + "version": "1.5.0", "description": "Typed host runtime for plugins, contracts, replicated state, and RPC over JSON links", "type": "module", "main": "./dist/index.js", @@ -90,8 +90,8 @@ "node": ">=22.19.0" }, "dependencies": { - "@earendil-works/pi-schema": "1.4.0", - "@earendil-works/pi-telemetry": "1.4.0" + "@earendil-works/pi-schema": "1.5.0", + "@earendil-works/pi-telemetry": "1.5.0" }, "peerDependencies": { "esbuild": "0.28.2", diff --git a/packages/codemode/CHANGELOG.md b/packages/codemode/CHANGELOG.md index a4c2fd624..446a26034 100644 --- a/packages/codemode/CHANGELOG.md +++ b/packages/codemode/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -## [Unreleased] +## [1.5.0] - 2026-11-02 ### Changed diff --git a/packages/codemode/package.json b/packages/codemode/package.json index 9fed9d017..006549146 100644 --- a/packages/codemode/package.json +++ b/packages/codemode/package.json @@ -1,6 +1,6 @@ { "name": "@earendil-works/pi-codemode", - "version": "1.4.0", + "version": "1.5.0", "description": "Sandboxed JavaScript execution where the only capability is calling injected tools", "type": "module", "main": "./dist/index.js", diff --git a/packages/coding-agent/CHANGELOG.md b/packages/coding-agent/CHANGELOG.md index 15c302597..b3be0124e 100644 --- a/packages/coding-agent/CHANGELOG.md +++ b/packages/coding-agent/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -## [Unreleased] +## [1.5.0] - 2026-11-02 ### New Features diff --git a/packages/coding-agent/examples/extensions/custom-provider-anthropic/package-lock.json b/packages/coding-agent/examples/extensions/custom-provider-anthropic/package-lock.json index b7d2ebab0..c6ed32b21 100644 --- a/packages/coding-agent/examples/extensions/custom-provider-anthropic/package-lock.json +++ b/packages/coding-agent/examples/extensions/custom-provider-anthropic/package-lock.json @@ -1,12 +1,12 @@ { "name": "pi-extension-custom-provider", - "version": "1.4.0", + "version": "1.5.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "pi-extension-custom-provider", - "version": "1.4.0", + "version": "1.5.0", "dependencies": { "@anthropic-ai/sdk": "^0.52.0" } diff --git a/packages/coding-agent/examples/extensions/custom-provider-anthropic/package.json b/packages/coding-agent/examples/extensions/custom-provider-anthropic/package.json index b22368863..b4900f974 100644 --- a/packages/coding-agent/examples/extensions/custom-provider-anthropic/package.json +++ b/packages/coding-agent/examples/extensions/custom-provider-anthropic/package.json @@ -1,7 +1,7 @@ { "name": "pi-extension-custom-provider-anthropic", "private": true, - "version": "1.4.0", + "version": "1.5.0", "type": "module", "scripts": { "clean": "echo 'nothing to clean'", diff --git a/packages/coding-agent/examples/extensions/custom-provider-gitlab-duo/package.json b/packages/coding-agent/examples/extensions/custom-provider-gitlab-duo/package.json index dbdc63c72..54e070434 100644 --- a/packages/coding-agent/examples/extensions/custom-provider-gitlab-duo/package.json +++ b/packages/coding-agent/examples/extensions/custom-provider-gitlab-duo/package.json @@ -1,7 +1,7 @@ { "name": "pi-extension-custom-provider-gitlab-duo", "private": true, - "version": "1.4.0", + "version": "1.5.0", "type": "module", "scripts": { "clean": "echo 'nothing to clean'", diff --git a/packages/coding-agent/examples/extensions/gondolin/package-lock.json b/packages/coding-agent/examples/extensions/gondolin/package-lock.json index 99a06ff69..0887734dd 100644 --- a/packages/coding-agent/examples/extensions/gondolin/package-lock.json +++ b/packages/coding-agent/examples/extensions/gondolin/package-lock.json @@ -1,12 +1,12 @@ { "name": "pi-extension-gondolin", - "version": "1.4.0", + "version": "1.5.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "pi-extension-gondolin", - "version": "1.4.0", + "version": "1.5.0", "dependencies": { "@earendil-works/gondolin": "0.14.0" } diff --git a/packages/coding-agent/examples/extensions/gondolin/package.json b/packages/coding-agent/examples/extensions/gondolin/package.json index d19d7190f..762d1d230 100644 --- a/packages/coding-agent/examples/extensions/gondolin/package.json +++ b/packages/coding-agent/examples/extensions/gondolin/package.json @@ -1,7 +1,7 @@ { "name": "pi-extension-gondolin", "private": true, - "version": "1.4.0", + "version": "1.5.0", "type": "module", "scripts": { "clean": "echo 'nothing to clean'", diff --git a/packages/coding-agent/examples/extensions/with-deps/package-lock.json b/packages/coding-agent/examples/extensions/with-deps/package-lock.json index 90c4e6339..2d84832c4 100644 --- a/packages/coding-agent/examples/extensions/with-deps/package-lock.json +++ b/packages/coding-agent/examples/extensions/with-deps/package-lock.json @@ -1,12 +1,12 @@ { "name": "pi-extension-with-deps", - "version": "1.4.0", + "version": "1.5.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "pi-extension-with-deps", - "version": "1.4.0", + "version": "1.5.0", "dependencies": { "ms": "^2.1.3" }, diff --git a/packages/coding-agent/examples/extensions/with-deps/package.json b/packages/coding-agent/examples/extensions/with-deps/package.json index f22d436c7..c71ad77f2 100644 --- a/packages/coding-agent/examples/extensions/with-deps/package.json +++ b/packages/coding-agent/examples/extensions/with-deps/package.json @@ -1,7 +1,7 @@ { "name": "pi-extension-with-deps", "private": true, - "version": "1.4.0", + "version": "1.5.0", "type": "module", "scripts": { "clean": "echo 'nothing to clean'", diff --git a/packages/coding-agent/install-lock/package-lock.json b/packages/coding-agent/install-lock/package-lock.json index 521c82fef..e37a4fb76 100644 --- a/packages/coding-agent/install-lock/package-lock.json +++ b/packages/coding-agent/install-lock/package-lock.json @@ -1,14 +1,14 @@ { "name": "@earendil-works/pi-coding-agent-install", - "version": "1.4.0", + "version": "1.5.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@earendil-works/pi-coding-agent-install", - "version": "1.4.0", + "version": "1.5.0", "dependencies": { - "@earendil-works/pi-coding-agent": "1.4.0" + "@earendil-works/pi-coding-agent": "1.5.0" }, "engines": { "node": ">=22.19.0" @@ -386,26 +386,26 @@ } }, "node_modules/@earendil-works/pi-agent-core": { - "version": "1.4.0", - "resolved": "https://registry.npmjs.org/@earendil-works/pi-agent-core/-/pi-agent-core-1.4.0.tgz", + "version": "1.5.0", + "resolved": "https://registry.npmjs.org/@earendil-works/pi-agent-core/-/pi-agent-core-1.5.0.tgz", "license": "MIT", "dependencies": { - "@earendil-works/pi-ai": "1.4.0", - "@earendil-works/pi-schema": "1.4.0" + "@earendil-works/pi-ai": "1.5.0", + "@earendil-works/pi-schema": "1.5.0" }, "engines": { "node": ">=22.19.0" } }, "node_modules/@earendil-works/pi-ai": { - "version": "1.4.0", - "resolved": "https://registry.npmjs.org/@earendil-works/pi-ai/-/pi-ai-1.4.0.tgz", + "version": "1.5.0", + "resolved": "https://registry.npmjs.org/@earendil-works/pi-ai/-/pi-ai-1.5.0.tgz", "license": "MIT", "dependencies": { "@anthropic-ai/sdk": "0.129.0", "@aws-sdk/client-bedrock-runtime": "3.1127.0", - "@earendil-works/pi-schema": "1.4.0", - "@earendil-works/pi-telemetry": "1.4.0", + "@earendil-works/pi-schema": "1.5.0", + "@earendil-works/pi-telemetry": "1.5.0", "@google/genai": "2.21.0", "@smithy/node-http-handler": "4.12.1", "http-proxy-agent": "9.1.0", @@ -481,8 +481,8 @@ } }, "node_modules/@earendil-works/pi-codemode": { - "version": "1.4.0", - "resolved": "https://registry.npmjs.org/@earendil-works/pi-codemode/-/pi-codemode-1.4.0.tgz", + "version": "1.5.0", + "resolved": "https://registry.npmjs.org/@earendil-works/pi-codemode/-/pi-codemode-1.5.0.tgz", "license": "MIT", "dependencies": { "quickjs-wasi": "3.6.2" @@ -492,15 +492,15 @@ } }, "node_modules/@earendil-works/pi-coding-agent": { - "version": "1.4.0", - "resolved": "https://registry.npmjs.org/@earendil-works/pi-coding-agent/-/pi-coding-agent-1.4.0.tgz", + "version": "1.5.0", + "resolved": "https://registry.npmjs.org/@earendil-works/pi-coding-agent/-/pi-coding-agent-1.5.0.tgz", "license": "MIT", "dependencies": { - "@earendil-works/pi-agent-core": "1.4.0", - "@earendil-works/pi-ai": "1.4.0", - "@earendil-works/pi-codemode": "1.4.0", - "@earendil-works/pi-mcp": "1.4.0", - "@earendil-works/pi-tui": "1.4.0", + "@earendil-works/pi-agent-core": "1.5.0", + "@earendil-works/pi-ai": "1.5.0", + "@earendil-works/pi-codemode": "1.5.0", + "@earendil-works/pi-mcp": "1.5.0", + "@earendil-works/pi-tui": "1.5.0", "@silvia-odwyer/photon-node": "0.3.4", "brace-expansion": "5.0.12", "chalk": "6.0.0", @@ -527,8 +527,8 @@ } }, "node_modules/@earendil-works/pi-mcp": { - "version": "1.4.0", - "resolved": "https://registry.npmjs.org/@earendil-works/pi-mcp/-/pi-mcp-1.4.0.tgz", + "version": "1.5.0", + "resolved": "https://registry.npmjs.org/@earendil-works/pi-mcp/-/pi-mcp-1.5.0.tgz", "license": "MIT", "dependencies": { "cross-spawn": "7.0.6" @@ -538,24 +538,24 @@ } }, "node_modules/@earendil-works/pi-schema": { - "version": "1.4.0", - "resolved": "https://registry.npmjs.org/@earendil-works/pi-schema/-/pi-schema-1.4.0.tgz", + "version": "1.5.0", + "resolved": "https://registry.npmjs.org/@earendil-works/pi-schema/-/pi-schema-1.5.0.tgz", "license": "MIT", "engines": { "node": ">=22.19.0" } }, "node_modules/@earendil-works/pi-telemetry": { - "version": "1.4.0", - "resolved": "https://registry.npmjs.org/@earendil-works/pi-telemetry/-/pi-telemetry-1.4.0.tgz", + "version": "1.5.0", + "resolved": "https://registry.npmjs.org/@earendil-works/pi-telemetry/-/pi-telemetry-1.5.0.tgz", "license": "MIT", "engines": { "node": ">=22.19.0" } }, "node_modules/@earendil-works/pi-tui": { - "version": "1.4.0", - "resolved": "https://registry.npmjs.org/@earendil-works/pi-tui/-/pi-tui-1.4.0.tgz", + "version": "1.5.0", + "resolved": "https://registry.npmjs.org/@earendil-works/pi-tui/-/pi-tui-1.5.0.tgz", "license": "MIT", "dependencies": { "get-east-asian-width": "1.6.0", diff --git a/packages/coding-agent/install-lock/package.json b/packages/coding-agent/install-lock/package.json index 004cf3c90..a8746dfab 100644 --- a/packages/coding-agent/install-lock/package.json +++ b/packages/coding-agent/install-lock/package.json @@ -1,10 +1,10 @@ { "name": "@earendil-works/pi-coding-agent-install", - "version": "1.4.0", + "version": "1.5.0", "private": true, "description": "Lockfile root used by the Pi installer and updater.", "dependencies": { - "@earendil-works/pi-coding-agent": "1.4.0" + "@earendil-works/pi-coding-agent": "1.5.0" }, "overrides": { "protobufjs": "7.6.6" diff --git a/packages/coding-agent/package.json b/packages/coding-agent/package.json index f8e6d41cc..705be3d3d 100644 --- a/packages/coding-agent/package.json +++ b/packages/coding-agent/package.json @@ -1,6 +1,6 @@ { "name": "@earendil-works/pi-coding-agent", - "version": "1.4.0", + "version": "1.5.0", "description": "Coding agent CLI with read, bash, edit, write tools and session management", "type": "module", "piConfig": { @@ -41,11 +41,11 @@ "prepublishOnly": "npm run clean && npm run build" }, "dependencies": { - "@earendil-works/pi-agent-core": "1.4.0", - "@earendil-works/pi-ai": "1.4.0", - "@earendil-works/pi-codemode": "1.4.0", - "@earendil-works/pi-mcp": "1.4.0", - "@earendil-works/pi-tui": "1.4.0", + "@earendil-works/pi-agent-core": "1.5.0", + "@earendil-works/pi-ai": "1.5.0", + "@earendil-works/pi-codemode": "1.5.0", + "@earendil-works/pi-mcp": "1.5.0", + "@earendil-works/pi-tui": "1.5.0", "@silvia-odwyer/photon-node": "0.3.4", "brace-expansion": "5.0.12", "chalk": "6.0.0", diff --git a/packages/durable/CHANGELOG.md b/packages/durable/CHANGELOG.md index 6e9dcdb21..f52841059 100644 --- a/packages/durable/CHANGELOG.md +++ b/packages/durable/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -## [Unreleased] +## [1.5.0] - 2026-11-02 ### Breaking Changes diff --git a/packages/durable/package.json b/packages/durable/package.json index 495197c59..c32091c32 100644 --- a/packages/durable/package.json +++ b/packages/durable/package.json @@ -1,6 +1,6 @@ { "name": "@earendil-works/pi-durable", - "version": "1.4.0", + "version": "1.5.0", "description": "Durable conversation, task, and document runtime for Pi", "type": "module", "main": "./dist/index.js", @@ -158,11 +158,11 @@ "node": ">=22.19.0" }, "dependencies": { - "@earendil-works/chord": "1.4.0", - "@earendil-works/pi-ai": "1.4.0", - "@earendil-works/pi-codemode": "1.4.0", - "@earendil-works/pi-schema": "1.4.0", - "@earendil-works/pi-telemetry": "1.4.0", + "@earendil-works/chord": "1.5.0", + "@earendil-works/pi-ai": "1.5.0", + "@earendil-works/pi-codemode": "1.5.0", + "@earendil-works/pi-schema": "1.5.0", + "@earendil-works/pi-telemetry": "1.5.0", "@silvia-odwyer/photon": "0.3.3", "diff": "8.0.4" }, diff --git a/packages/env/CHANGELOG.md b/packages/env/CHANGELOG.md index 96bd4e91e..aada8be3d 100644 --- a/packages/env/CHANGELOG.md +++ b/packages/env/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -## [Unreleased] +## [1.5.0] - 2026-11-02 ### Breaking Changes diff --git a/packages/env/package.json b/packages/env/package.json index 2bb941ce0..f78d49c20 100644 --- a/packages/env/package.json +++ b/packages/env/package.json @@ -1,6 +1,6 @@ { "name": "@earendil-works/pi-env", - "version": "1.4.0", + "version": "1.5.0", "description": "Remote execution environments for Pi Durable: a small daemon deployed over SSH and its ExecutionEnv client", "type": "module", "main": "./dist/index.js", @@ -45,8 +45,8 @@ "node": ">=22.19.0" }, "dependencies": { - "@earendil-works/chord": "1.4.0", - "@earendil-works/pi-durable": "1.4.0" + "@earendil-works/chord": "1.5.0", + "@earendil-works/pi-durable": "1.5.0" }, "devDependencies": { "vitest": "4.1.12" diff --git a/packages/evals/package.json b/packages/evals/package.json index fa4400219..f740c3368 100644 --- a/packages/evals/package.json +++ b/packages/evals/package.json @@ -1,6 +1,6 @@ { "name": "@earendil-works/pi-evals", - "version": "1.4.0", + "version": "1.5.0", "private": true, "type": "module", "scripts": { @@ -12,8 +12,8 @@ "test": "vitest run --config vitest.test.config.ts" }, "devDependencies": { - "@earendil-works/pi-ai": "1.4.0", - "@earendil-works/pi-coding-agent": "1.4.0", + "@earendil-works/pi-ai": "1.5.0", + "@earendil-works/pi-coding-agent": "1.5.0", "@types/node": "22.19.19", "@vitest-evals/core": "0.15.0", "vitest": "4.1.12", diff --git a/packages/mcp/CHANGELOG.md b/packages/mcp/CHANGELOG.md index 92e01bf9d..566400d62 100644 --- a/packages/mcp/CHANGELOG.md +++ b/packages/mcp/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -## [Unreleased] +## [1.5.0] - 2026-11-02 ### Changed diff --git a/packages/mcp/package.json b/packages/mcp/package.json index a884d2b0f..8a9bac663 100644 --- a/packages/mcp/package.json +++ b/packages/mcp/package.json @@ -1,6 +1,6 @@ { "name": "@earendil-works/pi-mcp", - "version": "1.4.0", + "version": "1.5.0", "description": "Standalone Model Context Protocol client for pi and other applications", "type": "module", "main": "./dist/index.js", diff --git a/packages/schema/CHANGELOG.md b/packages/schema/CHANGELOG.md index 404e93c57..57ce17ba7 100644 --- a/packages/schema/CHANGELOG.md +++ b/packages/schema/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -## [Unreleased] +## [1.5.0] - 2026-11-02 ### Added diff --git a/packages/schema/package.json b/packages/schema/package.json index 3fdb1b14f..4e176a57c 100644 --- a/packages/schema/package.json +++ b/packages/schema/package.json @@ -1,6 +1,6 @@ { "name": "@earendil-works/pi-schema", - "version": "1.4.0", + "version": "1.5.0", "description": "JSON Schema validation without eval: drafts 2020-12 and 7, compiled into plain closures", "type": "module", "main": "./dist/index.js", diff --git a/packages/telemetry/CHANGELOG.md b/packages/telemetry/CHANGELOG.md index 52617bb21..d673bc233 100644 --- a/packages/telemetry/CHANGELOG.md +++ b/packages/telemetry/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -## [Unreleased] +## [1.5.0] - 2026-11-02 ### Fixed diff --git a/packages/telemetry/package.json b/packages/telemetry/package.json index eca0a94e7..eeb30b234 100644 --- a/packages/telemetry/package.json +++ b/packages/telemetry/package.json @@ -1,6 +1,6 @@ { "name": "@earendil-works/pi-telemetry", - "version": "1.4.0", + "version": "1.5.0", "description": "Vendor-neutral telemetry contracts and typed schema utilities for pi", "type": "module", "main": "./dist/index.js", diff --git a/packages/tui/CHANGELOG.md b/packages/tui/CHANGELOG.md index 04fdf0893..cc7e85378 100644 --- a/packages/tui/CHANGELOG.md +++ b/packages/tui/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -## [Unreleased] +## [1.5.0] - 2026-11-02 ### Added diff --git a/packages/tui/package.json b/packages/tui/package.json index d5ce578be..74a49f0d3 100644 --- a/packages/tui/package.json +++ b/packages/tui/package.json @@ -1,6 +1,6 @@ { "name": "@earendil-works/pi-tui", - "version": "1.4.0", + "version": "1.5.0", "description": "Terminal User Interface library with differential rendering for efficient text-based applications", "type": "module", "main": "dist/index.js", From db6c08445593c4368460f42dc0b70656ad5c179e Mon Sep 17 00:00:00 2001 From: Armin Ronacher Date: Mon, 2 Nov 2026 11:35:40 +0100 Subject: [PATCH 134/175] Add [Unreleased] section for next cycle --- packages/agent/CHANGELOG.md | 2 ++ packages/ai/CHANGELOG.md | 2 ++ packages/chord/CHANGELOG.md | 2 ++ packages/codemode/CHANGELOG.md | 2 ++ packages/coding-agent/CHANGELOG.md | 2 ++ packages/durable/CHANGELOG.md | 2 ++ packages/env/CHANGELOG.md | 2 ++ packages/mcp/CHANGELOG.md | 2 ++ packages/schema/CHANGELOG.md | 2 ++ packages/telemetry/CHANGELOG.md | 2 ++ packages/tui/CHANGELOG.md | 2 ++ 11 files changed, 22 insertions(+) diff --git a/packages/agent/CHANGELOG.md b/packages/agent/CHANGELOG.md index a481d47f0..9e80e2abf 100644 --- a/packages/agent/CHANGELOG.md +++ b/packages/agent/CHANGELOG.md @@ -1,5 +1,7 @@ # Changelog +## [Unreleased] + ## [1.5.0] - 2026-11-02 ### Breaking Changes diff --git a/packages/ai/CHANGELOG.md b/packages/ai/CHANGELOG.md index cc4f0d9ac..67df897b0 100644 --- a/packages/ai/CHANGELOG.md +++ b/packages/ai/CHANGELOG.md @@ -1,5 +1,7 @@ # Changelog +## [Unreleased] + ## [1.5.0] - 2026-11-02 ### Breaking Changes diff --git a/packages/chord/CHANGELOG.md b/packages/chord/CHANGELOG.md index b4fd5fdfd..ed7af6305 100644 --- a/packages/chord/CHANGELOG.md +++ b/packages/chord/CHANGELOG.md @@ -1,5 +1,7 @@ # Changelog +## [Unreleased] + ## [1.5.0] - 2026-11-02 ### Breaking Changes diff --git a/packages/codemode/CHANGELOG.md b/packages/codemode/CHANGELOG.md index 446a26034..6c6de2736 100644 --- a/packages/codemode/CHANGELOG.md +++ b/packages/codemode/CHANGELOG.md @@ -1,5 +1,7 @@ # Changelog +## [Unreleased] + ## [1.5.0] - 2026-11-02 ### Changed diff --git a/packages/coding-agent/CHANGELOG.md b/packages/coding-agent/CHANGELOG.md index b3be0124e..36af238b5 100644 --- a/packages/coding-agent/CHANGELOG.md +++ b/packages/coding-agent/CHANGELOG.md @@ -1,5 +1,7 @@ # Changelog +## [Unreleased] + ## [1.5.0] - 2026-11-02 ### New Features diff --git a/packages/durable/CHANGELOG.md b/packages/durable/CHANGELOG.md index f52841059..14750ec5a 100644 --- a/packages/durable/CHANGELOG.md +++ b/packages/durable/CHANGELOG.md @@ -1,5 +1,7 @@ # Changelog +## [Unreleased] + ## [1.5.0] - 2026-11-02 ### Breaking Changes diff --git a/packages/env/CHANGELOG.md b/packages/env/CHANGELOG.md index aada8be3d..6c101bdd4 100644 --- a/packages/env/CHANGELOG.md +++ b/packages/env/CHANGELOG.md @@ -1,5 +1,7 @@ # Changelog +## [Unreleased] + ## [1.5.0] - 2026-11-02 ### Breaking Changes diff --git a/packages/mcp/CHANGELOG.md b/packages/mcp/CHANGELOG.md index 566400d62..a0e3f2cc8 100644 --- a/packages/mcp/CHANGELOG.md +++ b/packages/mcp/CHANGELOG.md @@ -1,5 +1,7 @@ # Changelog +## [Unreleased] + ## [1.5.0] - 2026-11-02 ### Changed diff --git a/packages/schema/CHANGELOG.md b/packages/schema/CHANGELOG.md index 57ce17ba7..1fb3218b0 100644 --- a/packages/schema/CHANGELOG.md +++ b/packages/schema/CHANGELOG.md @@ -1,5 +1,7 @@ # Changelog +## [Unreleased] + ## [1.5.0] - 2026-11-02 ### Added diff --git a/packages/telemetry/CHANGELOG.md b/packages/telemetry/CHANGELOG.md index d673bc233..a6dd650a8 100644 --- a/packages/telemetry/CHANGELOG.md +++ b/packages/telemetry/CHANGELOG.md @@ -1,5 +1,7 @@ # Changelog +## [Unreleased] + ## [1.5.0] - 2026-11-02 ### Fixed diff --git a/packages/tui/CHANGELOG.md b/packages/tui/CHANGELOG.md index cc7e85378..5c7b22acf 100644 --- a/packages/tui/CHANGELOG.md +++ b/packages/tui/CHANGELOG.md @@ -1,5 +1,7 @@ # Changelog +## [Unreleased] + ## [1.5.0] - 2026-11-02 ### Added From 09f2dac685ca67a33a45e8b5068db0805dfe85f6 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 10:40:04 +0000 Subject: [PATCH 135/175] docs(chord): app kit authentication --- packages/chord/docs/next/decisions.md | 9 +++++++++ packages/chord/docs/next/slices.md | 5 +++++ 2 files changed, 14 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index e513dce86..7ccf9cefa 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2242,6 +2242,15 @@ validation and copies the v2 kernel added around Delta. pairing, invite links, an ephemeral one for an IRC-like "pick a name and you're in": a fresh id with the name, no stored account, gone with the connection); an app picks, replaces or writes its own. Rejected: a fixed principal type with roles. +- **The app kit, point 3: authentication (owner, 2026-11-01, by voice: "yeah, I guess something like that"):** (a) + no owner by loopback alone (any local process or OS user could claim it): the start script prints a link with a + one-time token (as Jupyter), whose visit sets a signed `HttpOnly` owner cookie for that browser; drive-by requests + from pages in that browser stay blocked by the security baseline (foreign requests, also other localhost ports, + never authenticated; Host and Origin checks). (b) Deny by default, in the HTTP component itself (a change to slice + 10's routes and sockets): a request without a principal is refused before plugin code unless its route or socket is + marked public (a landing page, a webhook, the ephemeral name login, which hands out a principal). Authenticators stay + the `Authenticators` registry, filled by the authenticator plugins the app chose; a session helper issues and revokes + sessions so authenticators do not each reinvent cookies and logout. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a diff --git a/packages/chord/docs/next/slices.md b/packages/chord/docs/next/slices.md index 3c300ee62..766d32211 100644 --- a/packages/chord/docs/next/slices.md +++ b/packages/chord/docs/next/slices.md @@ -181,6 +181,11 @@ deleted (the slice's spec and code are the record). Moved here from `carry-over. - **`rebuild` of tables** (key and column-type changes): `evolveTable` has no edit for them; build one only when a real case needs it. (Slice 6a's second spec review; out of slice 7b's scope, `slice-7b.md` §12.) +### The app kit + +- **Deny by default in the HTTP component** (decision "The app kit, point 3: authentication"): routes and sockets + refuse requests without a principal unless marked public; the owner link with a one-time token; a session helper. + ### The plugin slice (plugin model) - **Cross-plugin imports by relative path and npm** (decision "Contracts come from where the plugin's code came From e8c5ac945fb00ec1e37899e545fbdcc0d985b8b5 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 10:42:09 +0000 Subject: [PATCH 136/175] docs(chord): app kit command-line clients --- packages/chord/docs/next/decisions.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 7ccf9cefa..da0af569b 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2251,6 +2251,11 @@ validation and copies the v2 kernel added around Delta. marked public (a landing page, a webhook, the ephemeral name login, which hands out a principal). Authenticators stay the `Authenticators` registry, filled by the authenticator plugins the app chose; a session helper issues and revokes sessions so authenticators do not each reinvent cookies and logout. + (c) Command-line clients (owner: "that sounds good"; the TUI client too) connect over the same WebSocket with a + token in a header (no Origin, no Fetch Metadata: not a browser request; the token decides): on the same machine an + owner token file in the data directory readable only by the OS user running the app; from another machine a device + pairing flow (the tool shows a code and a link, the owner approves in a browser, the tool stores the token), the + token a session like any other, listed and revocable. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From dbfb5bc105d63e11311779b2f3e8a4bf79a6856f Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 10:43:09 +0000 Subject: [PATCH 137/175] docs(chord): app kit storage --- packages/chord/docs/next/decisions.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index da0af569b..916af9f51 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2256,6 +2256,11 @@ validation and copies the v2 kernel added around Delta. owner token file in the data directory readable only by the OS user running the app; from another machine a device pairing flow (the tool shows a code and a link, the owner approves in a browser, the tool stores the token), the token a session like any other, listed and revocable. +- **The app kit, point 4: storage (owner, 2026-11-01, by voice: "yeah, I suppose that sounds right"):** in the data + directory the server has one store; each worker has its own directory under its worker kind, named by its address, + with its store in it; the kit opens and closes them. Plugins never use paths of the data directory: their state is + in their objects' documents and tables; a plugin that needs raw files gets a subdirectory of its own, owned by one of + its objects. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 3cec8e2b66fa8408c0cb9cbb6050df2574eacb86 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 10:50:22 +0000 Subject: [PATCH 138/175] docs(chord): app kit plugins, files and export --- packages/chord/docs/next/decisions.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 916af9f51..5494adbd9 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2261,6 +2261,17 @@ validation and copies the v2 kernel added around Delta. with its store in it; the kit opens and closes them. Plugins never use paths of the data directory: their state is in their objects' documents and tables; a plugin that needs raw files gets a subdirectory of its own, owned by one of its objects. +- **The app kit, point 5: plugins, files and export (owner, 2026-11-01: "okay", "yeah"):** two plugin roots: the + app's built-in `plugins/` (required: a plugin failing at startup stops the app with its error) and the user's in the + data directory (failures reported). Installs from git or npm into the user's root. One "apply changes" call builds the + dirty plugins and rolls them out to the server, then every running worker; the app exposes it as a button, a + command or an agent tool; changes are never applied by themselves. **Files:** the kit ships a files plugin, one owner + per address (`self.get(Files, { scope })`: `put`, chunked transfers for large files (design §23), a watchable + index, caps), its files in that worker's directory; apps build on it (p3's artifacts: versions, previews, sharing; + design §26 #17). Raw directories only where a real path is needed (a git checkout, a library writing its own + database), each owned by one object, under the worker's directory. **Export and import per address:** the store's + archive plus the worker's directory as one archive; importing it under another address copies or moves it. + Refines point 4 (storage): plugins use the files plugin instead of raw subdirectories of their own. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From cdd50052f61d735ca6cd83b87f7173204dc86fba Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 10:55:18 +0000 Subject: [PATCH 139/175] docs(chord): slice 11 base bundle and plugin scripts --- packages/chord/docs/next/decisions.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 5494adbd9..853b2ab5b 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2272,6 +2272,13 @@ validation and copies the v2 kernel added around Delta. database), each owned by one object, under the worker's directory. **Export and import per address:** the store's archive plus the worker's directory as one archive; importing it under another address copies or moves it. Refines point 4 (storage): plugins use the files plugin instead of raw subdirectories of their own. +- **Slice 11, P3: the base bundle and plugin scripts (owner, 2026-11-01, by voice: "yep, that sounds good"):** the + server builds one base bundle per client kind from an entry in the app's source (the app's web start code, its API + with the kind's module, Chord, React), content-hashed and cached by browsers forever; a change of base code reloads + the page (as a restart of the server). Plugin scripts: at page load the server serves the current set of the kind's + plugins as one combined file named by a hash of the set (one request, cacheable); while a page is open, a rollout's + changed plugins are fetched one script each. A fresh page makes two requests, an update downloads only what changed. + Rejected: a prebuilt base bundle shipped with the app (could disagree with the plugins' build). - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 7f7ec86806764ef1d77d3509696060f57391d980 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 10:56:16 +0000 Subject: [PATCH 140/175] docs(chord): slice 11 applies plugin scripts in place --- packages/chord/docs/next/decisions.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 853b2ab5b..440329fee 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2279,6 +2279,11 @@ validation and copies the v2 kernel added around Delta. plugins as one combined file named by a hash of the set (one request, cacheable); while a page is open, a rollout's changed plugins are fetched one script each. A fresh page makes two requests, an update downloads only what changed. Rejected: a prebuilt base bundle shipped with the app (could disagree with the plugins' build). +- **Slice 11, P4: new plugin scripts apply in place (owner, 2026-11-01: "yep, agree"):** a page's host reloads the + changed plugins per plugin (`host.reload`, as on the server): objects swap or restart, the UI re-renders from its + registries, keeping what is on screen (scroll, a half-typed message). A change of base code still reloads the whole + page. Rejected: reloading the page for every plugin change (p2's behaviour, which lost the screen on every agent + edit). - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 9942d5bbcfeec633da80f69c57747a3cf48da9a5 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 10:57:16 +0000 Subject: [PATCH 141/175] docs(chord): slice 11 pages learn of new scripts through Chord --- packages/chord/docs/next/decisions.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 440329fee..07af2737d 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2284,6 +2284,11 @@ validation and copies the v2 kernel added around Delta. registries, keeping what is on screen (scroll, a half-typed message). A change of base code still reloads the whole page. Rejected: reloading the page for every plugin change (p2's behaviour, which lost the screen on every agent edit). +- **Slice 11, P5: pages learn of new scripts through Chord (owner, 2026-11-01: "agree"):** the web component serves a + small object whose state is the current set of the kind's plugin scripts (names, digests, URLs); every page holds + it; a rollout publishes the new set without waiting for pages; each page gets one frame and fetches what changed; + after a reconnect the hold's snapshot brings the current set, so a page catches up on anything it missed. Rejected: + polling a manifest (p2: every 5 s while visible). - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 2f153034b45328592bcb35d0d38b4344ad6e468d Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 10:58:40 +0000 Subject: [PATCH 142/175] docs(chord): slice 11 connection loop --- packages/chord/docs/next/decisions.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 07af2737d..962e5cbd1 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2289,6 +2289,11 @@ validation and copies the v2 kernel added around Delta. it; a rollout publishes the new set without waiting for pages; each page gets one frame and fetches what changed; after a reconnect the hold's snapshot brings the current set, so a page catches up on anything it missed. Rejected: polling a manifest (p2: every 5 s while visible). +- **Slice 11, P6: the connection loop (owner, 2026-11-01: "agree"):** p2's `keepConnected`: after a drop, reconnect + with a growing pause (250 ms to 5 s); try at once when the tab becomes visible, the browser comes back online or the + network changes; replace a socket open but silent; stop when the server reports an outdated protocol (the page + reloads) or the app's `canRetry` says no (a guest's share ended). References keep their last values and wait while + down, so the UI shows a banner from the connection state. A network reconnect, not a crash restart. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 20f600e58d416dd8f0b9ee07238d11b7890b2c06 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 10:59:40 +0000 Subject: [PATCH 143/175] docs(chord): slice 11 React bindings --- packages/chord/docs/next/decisions.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 962e5cbd1..6b74038e7 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2294,6 +2294,11 @@ validation and copies the v2 kernel added around Delta. network changes; replace a socket open but silent; stop when the server reports an outdated protocol (the page reloads) or the app's `canRetry` says no (a guest's share ended). References keep their last values and wait while down, so the UI shows a banner from the connection state. A network reconnect, not a crash restart. +- **Slice 11, P7: React bindings (owner, 2026-11-01: "we definitely want something like that"; p2's `useObject`, + `useReadable`, `useRegistry` in `runtime/web.tsx`):** an optional subpath (`chord/next/react`): `HostProvider`, + `useHold(Type, params)`, `useWatch(watchable)`, `useRegistry(R)`, `useRefState(ref)`; releasing holds on unmount, + and a component unmounted by a reload never counts as leaving (design §21). No components, no styling; non-React apps + never load it. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From e8504dfab5fb3aa023322a7e89b43c0eb49e6120 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 11:00:43 +0000 Subject: [PATCH 144/175] docs(chord): slice 11 terminal client always sandboxed --- packages/chord/docs/next/decisions.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 6b74038e7..108821ff2 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2299,6 +2299,11 @@ validation and copies the v2 kernel added around Delta. `useHold(Type, params)`, `useWatch(watchable)`, `useRegistry(R)`, `useRefState(ref)`; releasing holds on unmount, and a component unmounted by a reload never counts as leaving (design §21). No components, no styling; non-React apps never load it. +- **Slice 11, P8: the terminal client always runs sandboxed (owner, 2026-11-01: "the terminal should always use the + sandbox approach"):** `chord/next/tui` is a Node client host that connects with a token (decision "The app kit, point + 3", (c)), downloads the base bundle and the TUI plugin scripts from the server and runs the app's TUI entry, always + inside p2's sandbox: a permission-dropped worker thread with a bridge terminal, the trusted launcher outside it + holding the credentials, the terminal and local preferences. No unsandboxed mode, also for the owner's own server. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 752019da075163d72eb447cd9fe9b27d1fd072c7 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 11:03:04 +0000 Subject: [PATCH 145/175] docs(chord): slice 11 terminal component scope --- packages/chord/docs/next/decisions.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 108821ff2..8eb76029e 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2304,6 +2304,11 @@ validation and copies the v2 kernel added around Delta. 3", (c)), downloads the base bundle and the TUI plugin scripts from the server and runs the app's TUI entry, always inside p2's sandbox: a permission-dropped worker thread with a bridge terminal, the trusted launcher outside it holding the credentials, the terminal and local preferences. No unsandboxed mode, also for the owner's own server. +- **Slice 11, P9: a terminal component scope, library-agnostic (owner, 2026-11-01: "yeah, sure"; p2's `TuiScope` + in `src/runtime/tui.ts`):** the counterpart of the React bindings: a scope tied to a component's life with `hold`, + `watch` and `registry`, releasing everything on dispose, calling one `changed` callback the app wires to its redraw + (one line with pi-tui; any library works). p2's retry with backoff for gone references is not carried over (v2 + references wait and rebind by themselves). No pi-tui dependency. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From ebce9debd4206e5a0c176ca6bed28fcb40d0edc1 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 11:13:08 +0000 Subject: [PATCH 146/175] docs(chord): fold the plugin model, app kit and slice 11 decisions into the design; slice 12 spec --- packages/chord/docs/next/design.md | 72 +++++++++++++++ packages/chord/docs/next/slice-12.md | 131 +++++++++++++++++++++++++++ packages/chord/docs/next/slices.md | 8 +- 3 files changed, 208 insertions(+), 3 deletions(-) create mode 100644 packages/chord/docs/next/slice-12.md diff --git a/packages/chord/docs/next/design.md b/packages/chord/docs/next/design.md index c11cb89e4..413674592 100644 --- a/packages/chord/docs/next/design.md +++ b/packages/chord/docs/next/design.md @@ -1980,6 +1980,39 @@ type ReloadReport = { - Project plugins, trust, snapshots, scopes and npm/git package installs: `apps/p2/docs/plugin-loading.md` (still the reference for those; not in 7c). +### The plugin model (2026-11-01; slice 12) + +Decisions "Contracts come from where the plugin's code came from", "Plugins import only the app's versioned API", +"The plugin model, points 3, 4, 5 and 8, imports and builds", "Single-file plugins", "Testing a plugin alone". The +rule underneath: a plugin carries what it was written against, never the running app's newest file. + +- **What a plugin may import:** its own files; the app's versioned API (`@acme/app/v3`, never Chord directly: Chord is + not a host package plugins may name; the API re-exports the Chord surface the app supports); its npm `dependencies` + (bundled at the installed version); other plugins' contract files: by relative path within one repository + (`../notes/contract.ts`, the same commit holds both sides), by package name across repositories (`@bob/notes`, + `@bob/notes/web`, pinned by an npm version or a git ref `github:bob/notes#`; a subfolder of a multi-plugin + repository is cloned and linked by the plugins component). One facet imports one API version only. +- **Versions:** the app lists the API versions it provides; an unlisted one is a build error naming the available ones. + The process runs one Chord: an older API version is a layer over today's, re-exporting Chord's compatibility module + for its previous API while Chord keeps one; contracts bind through `accepts` whatever Chord runs. +- **Split by host kind:** the API has a shared module and per-kind modules (`/v3/web`, `/v3/tui`, `/v3/server`, + `/v3/worker`); a plugin's contract files mirror it (`contract.ts`, `contract.web.tsx`, `contract.tui.ts`, ...). A + contract file mentions native things only through `import type`. The build checks every import by the file it + resolves to: a per-kind file or module only for its kind, a package the kind does not provide refused. A token or + `local` contract used on one kind lives in that kind's file. Shared UI between plugins: an imported file (inlined, + frozen at the importer's build) or a registry entry (one live instance). +- **Base code** is the API modules, bootstrap (hosts, store, components such as `createHttp` and `createWorkers`, which + take settings and process values and return facets) and one start-data facet serving process values as core + objects. Every implementation, the harness and core registry providers included, is a plugin; the app's own root is + required (a failure at startup stops the app). +- **Single-file plugins:** `..ts(x)` at a root's top, grouped by name, no `package.json`, importing only the + API; a directory of the same name replaces it as the same plugin. +- **Installs** from git (a branch, tag or commit cloned into a root) or npm. **Builds** are one esbuild call per facet + from source (TS, TSX, JS, JSON, CSS, assets and WASM by URL); plugins bring no build hooks. +- **`testPlugin(dir, { kind, modules, stubs, start?, principal? }, ctx)`:** the plugin built as in production on an + in-memory host and store, its dependencies served by stub facets, contributions to registries collected for the + test, the contracts nobody serves listed. + - **UI reload vs. leaving:** "leaving" is never derived from an unmount, only from explicit navigation or presence (a dropped reference); a component unmounted by a reload has not left. @@ -2236,6 +2269,45 @@ fails closed when stale; a start waits for a draining old holder without polling complete sets (`createPluginRuntime`, no esbuild); `workers.step(name)` ships a rollout's set to every worker (D12: `{ host, report } | { host, stopped: true } | { host, error }` per worker). +### The app kit (`chord/next/app`; slice 13) + +Decisions "The app kit, point 1" to "point 5". `startApp(options)` on the server, `startWorker(options)` in a worker +entry. Opinionated and small: an app is its API, its built-in plugins and a start script. + +- **Shape:** the app's layout is fixed: `src/api/v/` (shared and per-kind modules) and the built-in plugins root + `plugins/`. The data directory is a `startApp` option (no environment variables), default a dot directory named after + the app in the home directory. One server; worker kinds are a list the app defines, each a host kind with its own + params (sharding is the app's) and how it starts (entry, start data, principal, idle time); zero kinds is a server + with clients only. +- **Principals and users:** the principal is tiny (a kind and an id) and the app's type; the kit needs only its server + and worker variants. User data lives in objects keyed by the user's id. Optional kit plugins: users (accounts with an + app-supplied profile schema), authenticators (device pairing, invite links, an ephemeral pick-a-name login). +- **Authentication:** no owner by loopback alone: the start script prints a link with a one-time token that sets a + signed owner cookie; deny by default (the HTTP component refuses requests without a principal unless the route or + socket is public); a session helper issues and revokes sessions; command-line clients send a token in a header (an + owner token file readable only by the app's OS user; a device pairing flow from other machines). +- **Storage:** one store for the server, a directory per worker under its kind and address with its store; plugins + never use data-directory paths. A files plugin per address (`put`, chunked transfers, a watchable index, caps) is + the default file storage; raw directories only where a real path is needed, owned by one object, under the worker's + directory. Export and import per address: the store's archive plus the worker's directory. +- **Plugins:** the app's root (required) and the user's root in the data directory; installs from git or npm; one + "apply changes" call builds and rolls out to the server, then every running worker, exposed by the app as it likes. + +### Web and TUI (slice 11) + +Decision "Slice 11's points" and "Slice 11, P3" to "P9". + +- `chord/next/web`: serving the surface, a page's client host with its connection loop (p2's `keepConnected`), loading + and applying the plugins' web facets. A base bundle per client kind built on the server from the app's entry + (content-hashed); at page load the kind's current plugin set as one combined file named by its hash, later changed + plugins one script each, each a script that registers itself (no eval); new scripts apply in place (`host.reload` in + the page); pages learn of them through an object the web component serves (the current set); a base code change + reloads the page. +- `chord/next/react`: `HostProvider`, `useHold`, `useWatch`, `useRegistry`, `useRefState`. +- `chord/next/tui`: a Node client connecting with a token, always running the downloaded code in a sandbox (a + permission-dropped worker thread, a bridge terminal, the trusted launcher outside); a component scope (`hold`, + `watch`, `registry`, released on dispose, one `changed` callback for the app's redraw), no pi-tui dependency. + ### Channels WebSocket byte channels with acknowledged windows, separate from object links, wait for a second user after the diff --git a/packages/chord/docs/next/slice-12.md b/packages/chord/docs/next/slice-12.md new file mode 100644 index 000000000..4bfd790e7 --- /dev/null +++ b/packages/chord/docs/next/slice-12.md @@ -0,0 +1,131 @@ +# Slice 12: the plugin model (spec) + +Status: **approved** (owner, 2026-11-01: the plugin model's decisions, then "go ahead, just do all the things"; this +spec written by the redesign session from them, no separate spec review). Builds on 7c's plugins component +(`src/next-node/plugins/`) and slice 10's kinds and `createPluginRuntime`. + +Read in full before writing code: this spec; `decisions.md` entries of 2026-11-01 "Contracts come from where the +plugin's code came from", "Plugins import only the app's versioned API", "The plugin model, points 3, 4, 5 and 8, +imports and builds", "Single-file plugins", "Testing a plugin alone"; `design.md` §21 (all of it, "The plugin model" +included) and §16-§17; `slices.md` (carry-over "Slice 12"); `slice-7c.md`, `slice-10.md` §3 "Plugins in workers"; +`src/next-node/plugins/README.md` and its sources; the primer `docs/next/plugin-primer/book.md`. + +## 1. The app's API (`createPlugins` option `api`) + +```ts +const plugins = createPlugins({ + roots: [{ path: "./plugins", required: true }, { path: join(dataDir, "plugins") }], + api: { name: "@acme/app", versions: ["v3", "v2"] }, // newest first + kinds: { + server: { platform: "node", hostPackages: ["@acme/app/v3", "@acme/app/v3/server", "@acme/app/v2"] }, + worker: { platform: "node", hostPackages: ["@acme/app/v3", "@acme/app/v3/worker", "@acme/app/v2"] }, + web: { platform: "browser", hostPackages: ["@acme/app/v3", "@acme/app/v3/web", "@acme/app/v2", "react", "react/jsx-runtime"] }, + }, + onError, +}, ctx); +``` + +- `api` names the app's API package and the versions it provides. A plugin import of `/[/]`: + a version not listed is a build error naming the listed ones ("@acme/app/v1 is not available: this app provides v3, + v2"); a listed one that the kind's `hostPackages` lacks (`@acme/app/v3/web` from a worker facet) is a build error + naming the kind. One facet's imports of the API all use one version (a mix is a build error naming both). +- Chord itself (`@earendil-works/chord/next` and its subpaths other than `/plugins`, which stays as in 7c for + `pluginFile`) is not importable by plugins: a build error pointing to the app's API. Apps that do not set `api` keep + 7c's behaviour (host packages as listed), so Chord's own tests and examples keep importing Chord directly where they + model base code. +- `roots` entries may be strings (as today) or `{ path, required? }`. A plugin of a required root whose build, + evaluation or setup fails at `plugins.load` makes `load` reject (`PluginError` naming it), after the host's own load + settled; failures in other roots are reported as today. At a reload, a required plugin failing keeps its old code as + any plugin (reported). + +## 2. Imports between plugins (replaces slice 10's by-name lookup) + +- **Relative paths within a root:** an import from one plugin into another plugin's directory under the same root + (`../notes/contract.ts`) is inlined (listed in `copied`), the copied file under 7c's rule (only host packages and + files of its own plugin). A relative path leaving the root (base code, another root) is a build error naming the file + and pointing to the API or a dependency. A git checkout of several plugins cloned into a root is "within a root". +- **Package names across repositories:** an import of a package in the plugin's `dependencies` resolves as npm does + through that package's `exports` (`@bob/notes` → its `contract.ts`, `@bob/notes/web` → `contract.web.tsx`), inlined + at the installed version. A dependency may be an npm version or a git URL with a ref (`github:bob/notes#`, + `git+https://...#`); `installDependencies` installs it with npm as today. A package from a subfolder of a git + repository (which npm cannot install) is declared as `"chord": { "plugins": { "@bob/notes": "github:bob/plugins#:notes" } }` + and installed by `installDependencies` itself: a shallow clone at the ref into the plugin's `node_modules/.chord-git/` + and a link `node_modules/@bob/notes` to the subfolder. +- The by-name lookup (`build.ts` `#byName`), its error texts and the six `tsconfig.json` path aliases go; fixtures and + examples 35-38 import relatively again. + +## 3. Per-kind files and the import check + +- A file whose name has a kind segment before its extension (`contract.web.tsx`, `ui.tui.ts`, any `*..` with + `` one of `kinds`) is a file of that kind: importing it (directly or through other files) from a facet of + another kind is a build error naming the chain. Facet files keep their names (`web.tsx`); contract files are + `contract.ts` (shared) and `contract..ts(x)`. +- The check runs in the existing esbuild `onResolve` path, on the resolved file, for relative imports, dependencies and + copied files alike. +- Documented, not checked: a contract file mentions native things only through `import type` (TypeScript erases it; + a runtime import of a package the kind lacks already fails as above). + +## 4. Single-file plugins + +- Discovery also lists files at a root's top named `..(ts|tsx|js|mjs)`; files of one `` form one + plugin named `` (an unscoped valid npm name, else reported), one facet per file, `dir` the root, no + `package.json` (no dependencies, no `chord.external`). A directory plugin of the same name in the same root is a + reported duplicate (the directory wins). +- Its build allows imports of host packages only (the app's API); anything else is a build error saying to make it a + directory. `pluginFile` refuses (it has no directory of its own). +- The watcher: such a file appearing or disappearing at a root's top sets `set`; a change marks the plugin dirty. + +## 5. Installing plugins + +`installPlugin(source, root, ctx)`: `source` is `{ git: url, ref }` (a shallow clone of the ref into +`/`; a repository whose top has a `package.json` is one plugin, otherwise its subdirectories +are plugins) or `{ npm: name, version }` (`npm pack` and extract into `/`), then `installDependencies` for +each new plugin; marks `set`. Returns the installed plugin names. Removal is deleting the directory (`set`). No +updating in place: install another ref (a new directory) or delete and install. + +## 6. `testPlugin` + +```ts +const t = await testPlugin("./plugins/stats", { + kind: "worker", + api: { name: "@acme/app", versions: ["v3"] }, + modules: { "@acme/app/v3": apiV3, "@acme/app/v3/worker": apiV3Worker }, // the real API modules + stubs: [{ name: "@acme/notes", facet: notesStub }], // facets serving what it uses + principal: { kind: "user", id: "u1" }, // the host's principal (optional) +}, ctx); +const stats = await t.host.get(Stats, { project: "p1" }, ctx); +t.contributions(Panels); // what the plugin contributed to a registry nobody serves (the helper serves it) +t.unserved(); // contract names requests wait for unbound `not_served` +await t.close(ctx); +``` + +- Built with the same build as production (`buildFacet` with the kind's options), evaluated with `modules`, loaded on a + host with `address`/`principal` as given, an in-memory store (`:memory:` SQLite); stubs loaded beside it. +- The helper serves, as a collecting registry, every registry the plugin contributes to that no stub or the plugin + itself serves (`t.contributions(R)`: the current entries). +- `t.unserved()` lists the names of contracts whose requests wait unbound `not_served` on the host (the unbound + index), so a test knows what to stub. `t.root(fn, ctx, { principal })` runs as another principal. +- Exported from `chord/next/plugins` (`testing.ts`); no test-only modes elsewhere. + +## 7. Docs for authors and agents + +- `docs/next/plugin-authoring.md`: a plugin's layout (directory and single-file), its facet and contract files per + kind, what it may import and how each import resolves, versions, how the server builds it (esbuild, what the build + handles, no build hooks), installing from git or npm, testing with `testPlugin`; written for agents (short, exact, + with one complete example plugin). +- Update `src/next-node/plugins/README.md`, the guide `docs/next/guide-plugins-hosts/book.md` (chapters "Plugins Using + Plugins" and "Base Code and Plugins"; rebuild `index.html`), design §21's slice-10 note. + +## 8. Tests, examples + +- One test file per source file; build tests for every refusal and allowance of §1-§4 (an unlisted version, a kind + module from another kind, mixed versions, a Chord import, a relative path leaving the root, a per-kind file through + a chain, a package from `dependencies` through `exports`, a git dependency with a ref (a local bare repository in a + temp directory, no network), a subfolder dependency); discovery of single-file plugins; required roots at load; + `installPlugin` from a local git repository and a local npm tarball; `testPlugin` (stubs, collected contributions, + `unserved`, a principal). +- Examples: `49-plugin-api` (an app API with two versions and per-kind modules, a plugin per version, the refusals), + `50-plugin-imports` (relative within a root, a git dependency pinned to a commit, per-kind contract files), + `51-single-file-plugin`, `52-test-plugin` (a plugin tested with a stub). Examples 35-38 updated. +- No hot path changes; no benchmark beyond a cold build of the 20-plugin set (as 7c's) to show the checks cost nothing + measurable. diff --git a/packages/chord/docs/next/slices.md b/packages/chord/docs/next/slices.md index 766d32211..edd891029 100644 --- a/packages/chord/docs/next/slices.md +++ b/packages/chord/docs/next/slices.md @@ -45,7 +45,7 @@ slice spec's section citations are for orientation, not a reading limit (owner, Order (owner, 2026-10-22: routines and jobs before the store and reload; 2026-10-27: slice 8 before slice 7, whose reload has to reload real registries; 2026-10-29: slice 7 split into 7a, 7b, 7c; the app kit as slices 10 and 11; -telemetry last, since it instruments the app kit too). +telemetry last, since it instruments the app kit too; 2026-11-01: the plugin model (12) and the app kit (13) before web and TUI (11), which builds into the kit). | # | Slice | In short | | --- | --- | --- | @@ -63,6 +63,8 @@ telemetry last, since it instruments the app kit too). | 7b | **Reload in the kernel** | `host.reload`: staging, commit, rollback; swaps, restarts, removal and re-serving; holds and `accepts` across a reload; jobs and registries across a reload; per-name wakes (`slice-7b/` brief) | | 7c | **Plugins** | `chord/next/plugins` (discovery, `buildFacet`, `evaluateFacet`, `pluginFile`, `installDependencies`, the watcher, `rollout`, `mapStack`; no dev/prod split; the kernel's `load`/`reload` per plugin; `slice-7c.md`) | | 10 | **HTTP and workers** | `chord/next/http` (server, `Http` registry, responses as descriptions, WebSocket entry) and `chord/next/workers` (a worker per address, IPC, idle stop) | +| 12 | **The plugin model** | imports (relative within a repository, package names pinned across), versioned app API with Chord re-exported, per-kind API modules and contract files with the build's import check, single-file plugins, git installs, `testPlugin`, base code as API, bootstrap and a start-data facet (`slice-12.md`) | +| 13 | **The app kit** | `chord/next/app`: `startApp`, `startWorker`, layout, worker kinds, owner link, deny by default, sessions, CLI tokens, storage, the files plugin, export per address, plugin roots and "apply changes" (`slice-13.md`) | | 11 | **Web and TUI** | `chord/next/web` (page host, React hooks) and `chord/next/tui` (launcher, sandbox), from p2's code | | 9 | **Telemetry** | spans, events, metrics across hosts | @@ -181,12 +183,12 @@ deleted (the slice's spec and code are the record). Moved here from `carry-over. - **`rebuild` of tables** (key and column-type changes): `evolveTable` has no edit for them; build one only when a real case needs it. (Slice 6a's second spec review; out of slice 7b's scope, `slice-7b.md` §12.) -### The app kit +### Slice 13: the app kit - **Deny by default in the HTTP component** (decision "The app kit, point 3: authentication"): routes and sockets refuse requests without a principal unless marked public; the owner link with a one-time token; a session helper. -### The plugin slice (plugin model) +### Slice 12: the plugin model - **Cross-plugin imports by relative path and npm** (decision "Contracts come from where the plugin's code came from"): remove `build.ts`'s by-name lookup, allow relative imports into another plugin of the same root, refuse From 8d81a0f49db021e8f183a7a953d862ccb8681f12 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 11:14:37 +0000 Subject: [PATCH 147/175] docs(chord): slice 13 spec, the app kit --- packages/chord/docs/next/slice-13.md | 120 +++++++++++++++++++++++++++ 1 file changed, 120 insertions(+) create mode 100644 packages/chord/docs/next/slice-13.md diff --git a/packages/chord/docs/next/slice-13.md b/packages/chord/docs/next/slice-13.md new file mode 100644 index 000000000..f5b8fb029 --- /dev/null +++ b/packages/chord/docs/next/slice-13.md @@ -0,0 +1,120 @@ +# Slice 13: the app kit (spec) + +Status: **approved** (owner, 2026-11-01: decisions "The app kit, point 1" to "point 5", then "go ahead, just do all +the things"). Builds on slices 10 (`http`, `workers`) and 12 (the plugin model). `chord/next/app` in +`src/next-node/app/`: an opinionated layer so an app is its API, its built-in plugins and a start script. + +Read in full before writing code: this spec; `decisions.md` entries of 2026-10-31 and 2026-11-01 (the app kit, the +plugin model, "The app kit is opinionated", "Settings are arguments, extensions are registries", "The HTTP +component's security baseline"); `design.md` §3, §4, §10, §21, §23 (all of it, "The app kit" included); `slices.md` +(carry-over "Slice 13"); `slice-10.md`, `slice-12.md`; the READMEs of `next-node/http`, `workers`, `plugins`; p2's +`src/runtime/server.ts`, `src/hosts/server.ts`, `src/hosts/worker.ts`, `src/runtime/loopback-cookie.ts`, +`src/hosts/principals.ts`, `plugins/devices/`, `plugins/sharing/`, `plugins/files/` for what exists. + +## 1. `startApp` and `startWorker` + +```ts +// start.ts (the app's start script) +const app = await startApp

    ({ + name: "acme", // the default data directory: ~/.acme + dataDir: process.argv[2], // optional; no environment variables + listen: { host: "127.0.0.1", port: 8788 }, + api: { name: "@acme/app", versions: ["v3"], modules: { "@acme/app/v3": apiV3, "@acme/app/v3/server": apiV3Server } }, + kinds: { server: { platform: "node" }, worker: { platform: "node" }, web: { platform: "browser", packages: { react, "react/jsx-runtime": jsx } } }, + principal: { server: { kind: "server" }, worker: (address) => ({ kind: "worker", key: address.key }) }, + workers: [{ + kind: Project, // a host kind the app defines, any params + entry: "./worker.ts", + start: (address, ctx) => ({ id: address.params.id }), + authorize: (principal, address) => mayReach(principal, address), + idleMs: () => 5 * 60_000, + }], + onError: (error) => log(error), +}, ctx); +// app.host, app.http, app.workers, app.plugins, app.applyChanges(ctx), app.exportAddress(address, ctx), +// app.importAddress(archive, address, ctx), app.ownerLink(), app.close(ctx) +``` + +```ts +// worker.ts +await startWorker

    ({ api, kinds, onError }); // reads the start message, opens the worker's store, loads plugins +``` + +- `startApp` creates the data directory layout (§4), opens the server store, creates the host (address `Server()`, + the server principal), `createHttp` (settings as arguments), `createWorkers` per worker kind (`kind` named after the + host kind; the worker's facet kind is `worker` unless the entry says otherwise), `createPlugins` with the roots of §5 + and the app's API (slice 12), loads the server's plugins with the kit's own facets (§2, §3, §4) as core, and resolves + once loaded. `close` closes in the right order (http, workers, host, store). +- `startWorker` is `workerMain` plus: the worker's directory and store under its kind and address, the host with the + principal from the start message, `connect` first, plugins loaded with the kit's worker facets, `busy` from the app + if given. +- The start-data facet: the kit serves `AppInfo` (data directory, app name, version) on the server and `WorkerInfo` + (address, start data) on workers as core objects plugins `get`. +- Zero worker kinds: no workers component. + +## 2. Principals and sessions + +- The app's principal type `P`; the kit needs only `principal.server` and `principal.worker(address)`. The kit adds + one variant for its owner: `{ kind: "owner" }` must be assignable to `P` (a type check). +- **Sessions** (`app/sessions.ts`, a kit object on the server, persisted in the server store): `issue(principal, + { label, kind: "browser" | "device" }, ctx)` returns a token; tokens stored hashed; `revoke(id)`; `list` (watchable: + label, kind, created, last used). The kit's authenticator (contributed to `Authenticators`) reads the session cookie + (signed, `HttpOnly`, `SameSite=Strict`) or an `Authorization: Bearer` header, and gives its principal. +- **Owner link:** at start the kit issues a one-time token (random, single use, valid 10 minutes, replaced at each + start) and `app.ownerLink()` returns `http://127.0.0.1:8788/_chord/owner#`; the start script prints it. The + route `GET /_chord/owner` serves a page that posts the token (from the fragment, never sent in a URL to logs) to + `POST /_chord/owner`, which consumes it and sets a browser session cookie for the owner. No owner by loopback alone. +- **Owner token file:** `/owner-token` (mode 0600, created at first start, a device session of the owner) for + command-line clients on the same machine. +- **Device pairing** (for remote command-line and TUI clients): `POST /_chord/pair` (public) returns `{ code, poll }`; + the owner approves the code in a browser (`GET /_chord/pair/`, owner only, a confirm page); the client polls + and receives a device token. Codes expire after 10 minutes. +- **Deny by default** (an HTTP component change): a route or socket without a principal is refused (401) before plugin + code, unless its contribution marks it `public: true` (routes: `{ handler, public: true }`; sockets likewise). The + `/chord` links endpoint requires a principal. Foreign requests stay anonymous as before, so they reach only public + routes. Slice 10's examples and tests mark their anonymous routes public. + +## 3. Optional kit plugins + +Shipped as plugins (in `packages/chord/plugins/` or exported facets the app adds to its root; choose one and say +why): `users` (accounts keyed by id, a profile schema the app passes in, `self.get(Users, {})` with `get`, `update`, +`list`), `invite` (invite links issuing a session for a principal the owner chooses), `ephemeral-login` (a public +route taking a name, issuing a session for `{ kind: "user", id: , name }` that ends with the connection; the +app's `P` must allow it). Each contributes to `Authenticators` or issues sessions through §2. + +## 4. Storage, files, export + +- Layout: `/server/store.sqlite`, `/workers///store.sqlite` plus that + worker's files, `/plugins/` (the user's root), `/logs/`, `/owner-token`. +- **Files** (a kit facet on workers, and on the server for server-scoped files): `Files` (`local`, params `{}`, one per + host) with `put({ name, mime, bytes })` for small files, `begin`/`append`/`finish` for large ones (design §23 chunked + transfers), `read({ id, offset, length })`, `remove`, `list` (watchable index: id, name, mime, size, created), caps + per host (an option). Stored under the host's directory (`files/`), the index in the store. +- **Export / import per address:** `app.exportAddress(address, ctx)` stops the worker (as `workers.stop`), writes one + archive: the store's export archive (slice 6a) plus the worker's directory (files), and returns a readable stream or + writes to a path; `importAddress(archive, address, ctx)` refuses an existing address, unpacks, and the next hold + starts the worker. Format: a tar of `store.archive` and `files/`, with a small manifest (app name, API version, + kit format version). + +## 5. Plugins + +- Roots: the app's `plugins/` (required, slice 12) and `/plugins/`. Installs (`app.plugins.install(source, + ctx)`, slice 12's `installPlugin`) go to the user's root. +- `app.applyChanges(ctx)`: build the dirty plugins and roll out (server step, then the workers step); returns the + rollout result; `app.plugins.state` for the app's button. Never applied by itself. + +## 6. Not in slice 13 + +Web and TUI clients (slice 11, which builds on this kit: `startApp` gains the web surface there); telemetry (9); +project plugins (§26 #14). + +## 7. Tests, examples, docs + +- One test file per source file: start and close order; a server without workers; two worker kinds; the owner link + (single use, expiry, the fragment), sessions (issue, revoke, list, hashed storage), the token file mode, device + pairing, deny by default (a route without principal 401, a public route served, `/chord` refused anonymous), Files + (small and chunked, caps, list, remove), export and import (a worker with documents, tables and files round-trips; + importing over an existing address refused), `applyChanges`, required root failure stopping `startApp`. +- Examples: `53-app` (a small app: start script, API, two plugins, a worker kind, the owner link), `54-app-sessions` + (pairing a command-line client), `55-app-export` (export and import a worker's address). +- `src/next-node/app/README.md`; the guide "Hosts and Plugins" gets a closing chapter on the app kit. From 7b991fd96aee530b2ecfb5327f6c82d197f36395 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 11:15:21 +0000 Subject: [PATCH 148/175] docs(chord): slice 11 spec, web and TUI on the app kit --- packages/chord/docs/next/slice-11.md | 150 ++++++++++++++------------- 1 file changed, 78 insertions(+), 72 deletions(-) diff --git a/packages/chord/docs/next/slice-11.md b/packages/chord/docs/next/slice-11.md index c7b752596..e13899000 100644 --- a/packages/chord/docs/next/slice-11.md +++ b/packages/chord/docs/next/slice-11.md @@ -1,93 +1,99 @@ # Slice 11: web and TUI surfaces (spec) -Status: **draft** (owner, 2026-10-30: "start on slice 11"; "this one needs to be generic, so apps can do whatever they -want: css, tui components, themes"). Points marked **Decision needed** go to the owner one at a time. +Status: **approved** (owner, 2026-10-30/11-01: P1-P9 in the decisions "Slice 11's points" and "Slice 11, P3" to "P9", +then "go ahead, just do all the things"). Built after slices 12 and 13: the surfaces are generic plumbing that the app +kit wires in (`startApp` serves the web surface; a TUI client entry). The app keeps everything visible: HTML shell, +CSS, components, themes, navigation, layout, keys. -Slice 11 builds the generic plumbing of the app kit's two client surfaces (decision "Slice 7b's open points" (6)): -`chord/next/web` (a browser page as a Chord client host that loads the plugins' web facets the server built) and -`chord/next/tui` (a terminal client that does the same for TUI facets). The app keeps everything visible: its HTML -shell, CSS, components, themes, navigation, layout, keys. Built on slice 10 (`http` serves, the plugins component -builds and ships) and 7c (scripts as plain data, `createPluginRuntime`). Rule "The app kit is malleable by plugins" -applies. +Read in full before writing code: this spec; `decisions.md` (slice 11's entries, the plugin model, the app kit); +`design.md` §3 (clients), §21, §23 (all, "Web and TUI" included); `slice-10.md`, `slice-12.md`, `slice-13.md`; p2's +`src/hosts/web.ts`, `src/hosts/web/main.tsx`, `src/web/reconnect.ts`, `src/runtime/web.tsx`, `src/hosts/bundles.ts`, +`src/hosts/tui.ts`, `src/hosts/tui-client.ts`, `src/hosts/tui-sandbox.ts`, `src/hosts/tui-bootstrap.mjs`, +`src/hosts/tui-worker-runtime.ts`, `src/runtime/tui.ts` (`TuiScope`). -Read in full before writing code: this spec; `decisions.md` ("Slice 7c's points", "Slice 10's points"); `design.md` -§3 (clients), §21, §23; `slice-10.md`; p2's `src/hosts/web.ts`, `src/hosts/web/main.tsx`, `src/web/reconnect.ts`, -`src/runtime/web.tsx`, `src/hosts/bundles.ts`, `src/hosts/tui.ts`, `src/hosts/tui-client.ts`, `src/hosts/tui-sandbox.ts`, -`src/hosts/tui-bootstrap.mjs`, `src/hosts/tui-worker-runtime.ts`. - -## 1. What p2 has, and what is generic - -| p2 today | Generic (slice 11) | The app's | -| --- | --- | --- | -| one esbuild bundle per surface: runtime, React, every plugin's web facet, CSS; the page reloads whole when its version changes | a base bundle per surface (host packages and the surface runtime) plus each plugin's facet script (7c); new scripts applied in place | which host packages a surface has | -| the page: HTML with a CSP nonce, a theme script, `#root`, the bundle | serving the base bundle, scripts, stylesheets and assets through `http`; a page template hook | the HTML shell, CSP, theme, manifest and icons, service worker | -| `startWeb`: a client host, `keepConnected` (reconnect with backoff, wake on visibility and network change, stale-socket checks), protocol mismatch reloads the page | `startPage`: the client host, the connection loop, facet loading and reloading | `canRetry` policy (an ended share), connection UI | -| `runtime/web.tsx`: `HostContext`, `useObject`, `useReadable`, `useRegistry`, slots, sheets, navigation, owner state | optional React bindings: provider, hold, watch, registry, reference state | slots, sheets, navigation, everything else | -| TUI: a trusted launcher (auth, terminal, local preferences, actions), a permission-dropped worker thread running the downloaded bundle, a bridge terminal, a link through the launcher | a Node client surface: connect, download the base bundle and TUI scripts, evaluate, reload in place; the sandbox as an option | the TUI's components (pi-tui or other), commands, keys, themes, launcher actions | - -## 2. `chord/next/web` +## 1. `chord/next/web` (server side) ```ts -// server: serve the page surface (routes contributed to `Http`, so they reload like any contribution) +// inside startApp (slice 13), or by hand: const web = webSurface(server, { - kind: "web", - plugins, // the plugins component: the web kind's scripts - hostPackages: { "@earendil-works/chord/next": "chord-next", react: "react", "react/jsx-runtime": "react-jsx", "@app/web": "app-web" }, - base: "./src/web/base.ts", // the surface runtime's entry: built once into the base bundle - page: (request, ctx, assets) => appShell(assets), // the app's HTML; `assets` lists the base script and stylesheets + kind: "web", plugins, + entry: "./src/web/main.tsx", // the app's web start code: the base bundle's entry + page: (request, assets) => shell(request.nonce, assets), // the app's HTML; assets: base script, plugins file, styles }, ctx); +// its facet (routes to Http, the WebScripts object) goes into the server's core facets +``` -// browser: src/web/base.ts (in the base bundle) -startPage({ - principal: { kind: "server" }, +- **Base bundle** per client kind: built on the server by esbuild from `entry`, with the kind's host packages inlined + once (the app's API modules for the kind, Chord, React), content-hashed (`/_chord/web/base..js`), cached + forever (`Cache-Control: immutable`, overriding the baseline's `no-store` for these routes only); rebuilt when base + code changes (the server restarts anyway: built at start). +- **Plugin scripts:** each web facet served as a script that registers itself (`chordFacet("", "", + function (require, module, exports) { … })`, P2), at `/_chord/web/plugin/..js`, immutable; the + current set as one combined file `/_chord/web/plugins..js` (P3); stylesheets and assets of the build + likewise, under `publicPath`. +- **`WebScripts`** (a `remote` object the surface serves, params `{}`, any principal that may load the app): its state + is the current set `{ plugins: [{ name, digest, url, css? }], base }`; a rollout that ships the web kind updates it + (the plugins component's `scripts("web")` changes; the surface follows them) without waiting for pages (P5). +- Routes and the page are `Http` contributions of the surface's facet; the page route is `page: true` (navigation from + another site allowed) and public or not as the app says. + +## 2. `chord/next/web` (browser side) + +```ts +// src/web/main.tsx (the app's entry, in the base bundle) +startPage

    ({ connect: () => new WebSocket(`${location.origin.replace(/^http/, "ws")}/chord`), - hostPackages: { "@earendil-works/chord/next": chordNext, react: React, "react/jsx-runtime": jsx, "@app/web": appWeb }, - core: [{ name: "@app/web", facet: webCore }], - canRetry: async () => !(await shareEnded()), - onConnected: (connected) => setBanner(!connected), + modules: { "@acme/app/v3": api, "@acme/app/v3/web": apiWeb, react: React, "react/jsx-runtime": jsx }, + core: [{ name: "@acme/web", facet: webCore }], + canRetry: async () => true, + onConnection: (state) => setBanner(state), + render: (host) => createRoot(document.getElementById("root")!).render(), }); ``` -> **Decision needed P1: scope.** What `chord/next/web` ships: serving the surface (base bundle, scripts, stylesheets, -> assets, the page through an app template), the browser's client host with its connection loop, loading the plugins' -> web facets and applying new ones. Not shipped: HTML, CSS, components, themes, navigation, slots. React bindings -> separately (P7). +- A client host (no address); the connection loop (P6, p2's `keepConnected`: reconnect 250 ms to 5 s, wake on + visibility, `online` and network change, replace a silent socket, stop on an outdated protocol (reload the page) or + when `canRetry` says no); `onConnection` reports `connecting | connected | waiting`. +- Loads the combined plugins file, evaluates each registered facet with `modules` (the `require` they get answers only + host packages and `@earendil-works/chord/next/plugins`'s browser half if any), loads them with `core` in one + `host.load`, then holds `WebScripts`: on a new set it fetches the changed scripts (` + From 0ec20e157b1f76ffcd7b12c216a823c08aa8c48e Mon Sep 17 00:00:00 2001 From: m3 Date: Mon, 2 Nov 2026 23:12:43 +0800 Subject: [PATCH 155/175] feat(ai): add Microsoft-Decision-1 classifier to GitHub Copilot (#11679) Closes #11678 Changes: - `catalog/providers/github-copilot.ts`: add `microsoft-decision-1` as a `typesafe-system-one` classifier (text input, 32768 context, $0.042/M input, Copilot's static headers). - `src/providers/github-copilot.ts`: register the classifier and wrap `typesafeSystemOneApi()` to add `/v1` to the base URL, since Copilot serves `/v1/systemone` and the OAuth login replaces the base URL with the account endpoint. - README and `docs/models.md`: add the model to the classifier tables. Tests: - `test/classifier-models.test.ts`: with an OAuth credential, the request goes to `https://api.enterprise.githubcopilot.com/v1/systemone` with the Copilot headers, and the model stays available although `availableModelIds` lists only chat models. With an API key, it goes to `https://api.individual.githubcopilot.com/v1/systemone`. Both tests fail without the wrapper. - Ran against `api.enterprise.githubcopilot.com` with a Copilot login: bool, choice, and score questions return answers and usage. Co-authored-by: Hongwei Ma <11957602+Marvae@users.noreply.github.com> --- packages/ai/README.md | 3 +- .../ai/catalog/providers/github-copilot.ts | 19 ++++++- packages/ai/src/providers/github-copilot.ts | 24 ++++++-- packages/ai/test/classifier-models.test.ts | 56 +++++++++++++++++++ packages/coding-agent/docs/models.md | 3 +- 5 files changed, 98 insertions(+), 7 deletions(-) diff --git a/packages/ai/README.md b/packages/ai/README.md index 7af343eab..73f831905 100644 --- a/packages/ai/README.md +++ b/packages/ai/README.md @@ -891,7 +891,7 @@ console.log(model.output); // ['image'] or ['image', 'text'] ## Classification -Classifier models consume structured JSON state, and optionally images, and answer one or more typed questions. They do not use chat or image-generation APIs. TypeSafe's Jev model, Cloudflare's Clef models, and OpenAI's GPT-6 Luna are available from these built-in providers: +Classifier models consume structured JSON state, and optionally images, and answer one or more typed questions. They do not use chat or image-generation APIs. TypeSafe's Jev model, Cloudflare's Clef models, OpenAI's GPT-6 Luna, and Microsoft-Decision-1 are available from these built-in providers: | Provider | Model IDs | Auth | | --- | --- | --- | @@ -901,6 +901,7 @@ Classifier models consume structured JSON state, and optionally images, and answ | `vercel-ai-gateway` | `typesafe-ai/jev` | `AI_GATEWAY_API_KEY` | | `opencode` | `jev-1.13`, `jev-1.13-free` | `OPENCODE_API_KEY` | | `openai` | `gpt-6-luna` | `OPENAI_API_KEY` | +| `github-copilot` | `microsoft-decision-1` | `COPILOT_GITHUB_TOKEN` or GitHub Copilot OAuth | ```typescript import { builtinModels } from '@earendil-works/pi-ai/providers/all'; diff --git a/packages/ai/catalog/providers/github-copilot.ts b/packages/ai/catalog/providers/github-copilot.ts index 2bea3586b..b91a03213 100644 --- a/packages/ai/catalog/providers/github-copilot.ts +++ b/packages/ai/catalog/providers/github-copilot.ts @@ -1,4 +1,4 @@ -import type { Api, Model, ThinkingLevelMap } from "../../src/types.ts"; +import type { Api, ClassifierModel, Model, ThinkingLevelMap } from "../../src/types.ts"; import { addMissing, type ChatModel, defineProvider, mergeCompat, mergeThinking, modelsDevModels } from "../define.ts"; import { claude55ThinkingLevels, supportsClaudeMidConvoSystemMessages } from "../families/anthropic.ts"; import { @@ -67,6 +67,22 @@ const GPT_6_LUNA: Model<"openai-responses"> = { headers: { ...STATIC_HEADERS }, }; +// The authenticated Copilot catalog lists this decision model on /v1/systemone (checked 2026-11-02) without +// a context window or price, and GitHub's Copilot pricing page does not list it. These are Azure's values, +// which OpenRouter and Vercel AI Gateway also list. +const MICROSOFT_DECISION_1: ClassifierModel<"typesafe-system-one"> = { + type: "classifier", + id: "microsoft-decision-1", + name: "Microsoft-Decision-1", + api: "typesafe-system-one", + provider: "github-copilot", + baseUrl: BASE_URL, + input: ["text"], + cost: { input: 0.042, output: 0, cacheRead: 0, cacheWrite: 0 }, + contextWindow: 32768, + headers: { ...STATIC_HEADERS }, +}; + export default defineProvider({ id: "github-copilot", models: (ctx) => @@ -94,6 +110,7 @@ export default defineProvider({ }), [GPT_6_LUNA], ).map((model) => (EXTENDED_CONTEXT_MODEL_IDS.has(model.id) ? { ...model, contextWindow: 1000000 } : model)), + classifiers: () => [MICROSOFT_DECISION_1], compat: { "openai-completions": { supportsStrictMode: true } }, finalize: (model) => { if (model.id.startsWith("gpt-5")) mergeThinking(model, { minimal: "low" }); diff --git a/packages/ai/src/providers/github-copilot.ts b/packages/ai/src/providers/github-copilot.ts index eab784d88..282bc1da1 100644 --- a/packages/ai/src/providers/github-copilot.ts +++ b/packages/ai/src/providers/github-copilot.ts @@ -1,13 +1,28 @@ import { anthropicMessagesApi } from "../api/anthropic-messages.lazy.ts"; import { openAICompletionsApi } from "../api/openai-completions.lazy.ts"; import { openAIResponsesApi } from "../api/openai-responses.lazy.ts"; +import { typesafeSystemOneApi } from "../api/typesafe-system-one.lazy.ts"; import { envApiKeyAuth, lazyOAuth } from "../auth/helpers.ts"; import { loadGitHubCopilotOAuth } from "../auth/oauth/load.ts"; import { createProvider, type Provider } from "../models.ts"; -import { GITHUB_COPILOT_MODELS } from "./github-copilot.models.ts"; +import type { ProviderClassifier } from "../types.ts"; +import { GITHUB_COPILOT_CLASSIFIER_MODELS, GITHUB_COPILOT_MODELS } from "./github-copilot.models.ts"; -export function githubCopilotProvider(): Provider<"anthropic-messages" | "openai-completions" | "openai-responses"> { - return createProvider({ +type GitHubCopilotApi = "anthropic-messages" | "openai-completions" | "openai-responses"; + +/** + * Copilot serves System One at `/v1/systemone`, while its chat APIs have no version prefix. + * The login replaces the catalog base URL with the account's endpoint, so the prefix is added per request. + */ +function copilotSystemOne(classifier: ProviderClassifier): ProviderClassifier { + return { + classify: (model, context, options) => + classifier.classify({ ...model, baseUrl: `${model.baseUrl.replace(/\/+$/u, "")}/v1` }, context, options), + }; +} + +export function githubCopilotProvider(): Provider { + return createProvider({ id: "github-copilot", name: "GitHub Copilot", baseUrl: "https://api.individual.githubcopilot.com", @@ -15,7 +30,7 @@ export function githubCopilotProvider(): Provider<"anthropic-messages" | "openai apiKey: envApiKeyAuth("GitHub Copilot token", ["COPILOT_GITHUB_TOKEN"]), oauth: lazyOAuth({ name: "GitHub Copilot", isSubscription: true, load: loadGitHubCopilotOAuth }), }, - models: Object.values(GITHUB_COPILOT_MODELS), + models: [...Object.values(GITHUB_COPILOT_MODELS), ...Object.values(GITHUB_COPILOT_CLASSIFIER_MODELS)], filterModels: (models, credential) => { if (credential?.type !== "oauth") return models; const availableModelIds = credential.availableModelIds; @@ -30,5 +45,6 @@ export function githubCopilotProvider(): Provider<"anthropic-messages" | "openai "openai-completions": openAICompletionsApi(), "openai-responses": openAIResponsesApi(), }, + classifiers: { "typesafe-system-one": copilotSystemOne(typesafeSystemOneApi()) }, }); } diff --git a/packages/ai/test/classifier-models.test.ts b/packages/ai/test/classifier-models.test.ts index cae1df4a5..9f26b199c 100644 --- a/packages/ai/test/classifier-models.test.ts +++ b/packages/ai/test/classifier-models.test.ts @@ -245,6 +245,62 @@ describe("Models with classifier models", () => { expect((await withOAuth.getAvailable("openai")).some((model) => model.id === "gpt-6-luna")).toBe(true); }); + it("routes GitHub Copilot Microsoft-Decision-1 to the account's /v1/systemone endpoint", async () => { + const credentials = new InMemoryCredentialStore(); + await credentials.modify("github-copilot", async () => ({ + type: "oauth", + access: "tid=test;exp=9999999999;proxy-ep=proxy.enterprise.githubcopilot.com;", + refresh: "refresh", + expires: Date.now() + 3_600_000, + // The login records only picker models; classifiers stay listed. + availableModelIds: ["gpt-6-luna"], + })); + const models = builtinModels({ credentials }); + const decision = models.getModelOfType("classifier", "github-copilot", "microsoft-decision-1"); + if (!decision) throw new Error("missing GitHub Copilot decision model"); + expect(decision).toMatchObject({ + api: "typesafe-system-one", + baseUrl: "https://api.individual.githubcopilot.com", + input: ["text"], + contextWindow: 32768, + }); + expect(models.getModel("github-copilot", "microsoft-decision-1")).toBeUndefined(); + expect(await models.getAvailableOfType("classifier", "github-copilot")).toEqual([decision]); + + const requests: Array<{ url: string; headers: Headers }> = []; + const result = await models.classify(decision, context, { + fetch: async (input, init) => { + requests.push({ url: String(input), headers: new Headers(init?.headers) }); + return Response.json({ answers: { approved: { type: "noul", noul: 0.8 } } }); + }, + }); + + expect(requests.map((request) => request.url)).toEqual(["https://api.enterprise.githubcopilot.com/v1/systemone"]); + expect(requests[0].headers.get("authorization")).toBe( + "Bearer tid=test;exp=9999999999;proxy-ep=proxy.enterprise.githubcopilot.com;", + ); + // Copilot rejects requests without an editor version. + expect(requests[0].headers.get("editor-version")).toBeTruthy(); + expect(requests[0].headers.get("copilot-integration-id")).toBe("vscode-chat"); + expect(result.stopReason).toBe("stop"); + expect(result.answers.approved).toEqual({ type: "bool", probability: 0.8 }); + }); + + it("adds the System One version prefix to the GitHub Copilot catalog base URL", async () => { + const models = builtinModels(); + const decision = models.getModelOfType("classifier", "github-copilot", "microsoft-decision-1"); + if (!decision) throw new Error("missing GitHub Copilot decision model"); + const urls: string[] = []; + await models.classify(decision, context, { + apiKey: "secret", + fetch: async (input) => { + urls.push(String(input)); + return Response.json({ answers: { approved: { type: "noul", noul: 0.8 } } }); + }, + }); + expect(urls).toEqual(["https://api.individual.githubcopilot.com/v1/systemone"]); + }); + it("routes OpenRouter classifier models through the System One API", () => { const models = builtinModels(); for (const model of getBuiltinClassifierModels("openrouter")) { diff --git a/packages/coding-agent/docs/models.md b/packages/coding-agent/docs/models.md index e076ac139..803458a8b 100644 --- a/packages/coding-agent/docs/models.md +++ b/packages/coding-agent/docs/models.md @@ -141,7 +141,7 @@ Compatibility settings should describe verified differences in the endpoint's re ## Use classifier models -Classifier models do not chat. They answer typed questions about JSON state: pick one of several choices, answer yes or no, or give a score, each with probabilities. Pi includes TypeSafe's Jev model from these providers, Cloudflare's Clef, Clef Flash, and Clef Omni models from Workers AI and OpenRouter, and OpenAI's GPT-6 Luna through the [Decisions API](https://developers.openai.com/api/docs/guides/decisions): +Classifier models do not chat. They answer typed questions about JSON state: pick one of several choices, answer yes or no, or give a score, each with probabilities. Pi includes TypeSafe's Jev model from these providers, Cloudflare's Clef, Clef Flash, and Clef Omni models from Workers AI and OpenRouter, OpenAI's GPT-6 Luna through the [Decisions API](https://developers.openai.com/api/docs/guides/decisions), and Microsoft-Decision-1 through GitHub Copilot: | Provider | Model IDs | Authentication | |---|---|---| @@ -151,6 +151,7 @@ Classifier models do not chat. They answer typed questions about JSON state: pic | `vercel-ai-gateway` | `typesafe-ai/jev` | `AI_GATEWAY_API_KEY` | | `opencode` | `jev-1.13`, `jev-1.13-free` | `OPENCODE_API_KEY` | | `openai` | `gpt-6-luna` | `OPENAI_API_KEY` | +| `github-copilot` | `microsoft-decision-1` | `COPILOT_GITHUB_TOKEN` or `/login` | Chat models on a [llama.cpp router](llama-cpp.md#classification) are also listed as classifier models. From 072e413bf3f8ee0cf8d9f2862fd1dbfc5d337afa Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 15:13:36 +0000 Subject: [PATCH 156/175] feat(chord): Chord v2 slice 13, the app kit chord/next/app: startApp and startWorker with the fixed app layout, worker kinds, AppInfo and WorkerInfo; sessions (hashed tokens, rolling cookies, bearer tokens), the one-time owner link, the owner token file, device pairing; Files per host with ordered chunked uploads; export and import per address; plugin roots, installs and applyChanges; optional users, invite and ephemeral login facets. HTTP: deny by default (public routes and sockets), remoteAddress, readSignedCookie. Examples 53-55. --- packages/chord/CHANGELOG.md | 4 + .../docs/next/guide-plugins-hosts/book.md | 101 +++- .../docs/next/guide-plugins-hosts/index.html | 100 +++- packages/chord/docs/next/slices.md | 14 +- packages/chord/src/next-node/app/README.md | 198 +++++++ packages/chord/src/next-node/app/app.ts | 488 ++++++++++++++++++ packages/chord/src/next-node/app/archive.ts | 194 +++++++ packages/chord/src/next-node/app/ephemeral.ts | 43 ++ packages/chord/src/next-node/app/files.ts | 196 +++++++ packages/chord/src/next-node/app/index.ts | 24 + packages/chord/src/next-node/app/info.ts | 52 ++ packages/chord/src/next-node/app/invite.ts | 72 +++ packages/chord/src/next-node/app/layout.ts | 105 ++++ packages/chord/src/next-node/app/owner.ts | 123 +++++ packages/chord/src/next-node/app/pairing.ts | 205 ++++++++ packages/chord/src/next-node/app/secrets.ts | 75 +++ packages/chord/src/next-node/app/sessions.ts | 267 ++++++++++ packages/chord/src/next-node/app/tar.ts | 181 +++++++ packages/chord/src/next-node/app/uploads.ts | 172 ++++++ packages/chord/src/next-node/app/users.ts | 68 +++ packages/chord/src/next-node/app/worker.ts | 72 +++ packages/chord/src/next-node/http/README.md | 21 +- packages/chord/src/next-node/http/cookies.ts | 23 +- packages/chord/src/next-node/http/http.ts | 14 +- packages/chord/src/next-node/http/index.ts | 2 +- packages/chord/src/next-node/http/request.ts | 8 + packages/chord/src/next-node/http/router.ts | 24 +- packages/chord/src/next-node/http/upgrades.ts | 7 +- packages/chord/test/next-node/app/app.test.ts | 308 +++++++++++ .../chord/test/next-node/app/archive.test.ts | 142 +++++ .../test/next-node/app/ephemeral.test.ts | 55 ++ .../chord/test/next-node/app/files.test.ts | 190 +++++++ .../chord/test/next-node/app/info.test.ts | 56 ++ .../chord/test/next-node/app/invite.test.ts | 55 ++ .../chord/test/next-node/app/layout.test.ts | 47 ++ .../chord/test/next-node/app/owner.test.ts | 112 ++++ .../chord/test/next-node/app/pairing.test.ts | 148 ++++++ .../chord/test/next-node/app/secrets.test.ts | 65 +++ .../chord/test/next-node/app/sessions.test.ts | 227 ++++++++ packages/chord/test/next-node/app/support.ts | 187 +++++++ packages/chord/test/next-node/app/tar.test.ts | 107 ++++ .../chord/test/next-node/app/uploads.test.ts | 28 + .../chord/test/next-node/app/users.test.ts | 42 ++ .../chord/test/next-node/app/worker.test.ts | 40 ++ .../test/next-node/fixtures/app/worker.ts | 13 + .../chord/test/next-node/http/file.test.ts | 7 +- .../chord/test/next-node/http/http.test.ts | 283 +++++++--- .../chord/test/next-node/http/node.test.ts | 34 +- .../chord/test/next-node/http/request.test.ts | 29 +- .../test/next-node/http/response.test.ts | 43 +- .../chord/test/next-node/http/router.test.ts | 13 +- .../test/next-node/http/upgrades.test.ts | 23 +- .../examples/40-http/plugins/notes/server.ts | 19 +- .../44-http-wraps/plugins/notes/server.ts | 4 +- .../46-allowed-hosts/plugins/tunnel/server.ts | 7 +- .../plugins/notes/server.ts | 5 +- .../next/examples/48-http-baseline/main.ts | 14 +- .../48-http-baseline/plugins/notes/server.ts | 6 +- .../next/examples/53-app/app/principals.ts | 8 + .../chord/test/next/examples/53-app/app/v1.ts | 31 ++ .../next/examples/53-app/app/v1/server.ts | 5 + .../chord/test/next/examples/53-app/client.ts | 44 ++ .../chord/test/next/examples/53-app/main.ts | 56 ++ .../examples/53-app/plugins/home/package.json | 5 + .../examples/53-app/plugins/home/server.ts | 27 + .../examples/53-app/plugins/notes/contract.ts | 22 + .../53-app/plugins/notes/package.json | 5 + .../examples/53-app/plugins/notes/server.ts | 23 + .../examples/53-app/plugins/notes/worker.ts | 49 ++ .../chord/test/next/examples/53-app/start.ts | 50 ++ .../chord/test/next/examples/53-app/worker.ts | 7 + .../next/examples/54-app-sessions/main.ts | 62 +++ .../test/next/examples/55-app-export/main.ts | 66 +++ tsconfig.json | 2 + 74 files changed, 5437 insertions(+), 187 deletions(-) create mode 100644 packages/chord/src/next-node/app/README.md create mode 100644 packages/chord/src/next-node/app/app.ts create mode 100644 packages/chord/src/next-node/app/archive.ts create mode 100644 packages/chord/src/next-node/app/ephemeral.ts create mode 100644 packages/chord/src/next-node/app/files.ts create mode 100644 packages/chord/src/next-node/app/index.ts create mode 100644 packages/chord/src/next-node/app/info.ts create mode 100644 packages/chord/src/next-node/app/invite.ts create mode 100644 packages/chord/src/next-node/app/layout.ts create mode 100644 packages/chord/src/next-node/app/owner.ts create mode 100644 packages/chord/src/next-node/app/pairing.ts create mode 100644 packages/chord/src/next-node/app/secrets.ts create mode 100644 packages/chord/src/next-node/app/sessions.ts create mode 100644 packages/chord/src/next-node/app/tar.ts create mode 100644 packages/chord/src/next-node/app/uploads.ts create mode 100644 packages/chord/src/next-node/app/users.ts create mode 100644 packages/chord/src/next-node/app/worker.ts create mode 100644 packages/chord/test/next-node/app/app.test.ts create mode 100644 packages/chord/test/next-node/app/archive.test.ts create mode 100644 packages/chord/test/next-node/app/ephemeral.test.ts create mode 100644 packages/chord/test/next-node/app/files.test.ts create mode 100644 packages/chord/test/next-node/app/info.test.ts create mode 100644 packages/chord/test/next-node/app/invite.test.ts create mode 100644 packages/chord/test/next-node/app/layout.test.ts create mode 100644 packages/chord/test/next-node/app/owner.test.ts create mode 100644 packages/chord/test/next-node/app/pairing.test.ts create mode 100644 packages/chord/test/next-node/app/secrets.test.ts create mode 100644 packages/chord/test/next-node/app/sessions.test.ts create mode 100644 packages/chord/test/next-node/app/support.ts create mode 100644 packages/chord/test/next-node/app/tar.test.ts create mode 100644 packages/chord/test/next-node/app/uploads.test.ts create mode 100644 packages/chord/test/next-node/app/users.test.ts create mode 100644 packages/chord/test/next-node/app/worker.test.ts create mode 100644 packages/chord/test/next-node/fixtures/app/worker.ts create mode 100644 packages/chord/test/next/examples/53-app/app/principals.ts create mode 100644 packages/chord/test/next/examples/53-app/app/v1.ts create mode 100644 packages/chord/test/next/examples/53-app/app/v1/server.ts create mode 100644 packages/chord/test/next/examples/53-app/client.ts create mode 100644 packages/chord/test/next/examples/53-app/main.ts create mode 100644 packages/chord/test/next/examples/53-app/plugins/home/package.json create mode 100644 packages/chord/test/next/examples/53-app/plugins/home/server.ts create mode 100644 packages/chord/test/next/examples/53-app/plugins/notes/contract.ts create mode 100644 packages/chord/test/next/examples/53-app/plugins/notes/package.json create mode 100644 packages/chord/test/next/examples/53-app/plugins/notes/server.ts create mode 100644 packages/chord/test/next/examples/53-app/plugins/notes/worker.ts create mode 100644 packages/chord/test/next/examples/53-app/start.ts create mode 100644 packages/chord/test/next/examples/53-app/worker.ts create mode 100644 packages/chord/test/next/examples/54-app-sessions/main.ts create mode 100644 packages/chord/test/next/examples/55-app-export/main.ts diff --git a/packages/chord/CHANGELOG.md b/packages/chord/CHANGELOG.md index 39a18568b..fb3520709 100644 --- a/packages/chord/CHANGELOG.md +++ b/packages/chord/CHANGELOG.md @@ -5,9 +5,13 @@ ### Added - Chord v2's plugins component gains the plugin model (slice 12): `createPlugins({ api: { name, versions } })` checks every import of the app's versioned API (`@acme/app/v3`, per-kind modules such as `@acme/app/v3/web`): an unlisted version, another kind's module, two versions in one facet and a direct import of Chord are build errors naming the file and line. Roots may be `{ path, required }`: a required root's plugin failing at `load` rejects it with a `PluginError` after the load settled. Files named `..` (`contract.web.tsx`) are of that kind, refused in facets of another kind with the chain of imports named. Single-file plugins (`hello.server.ts` at a root's top, grouped by name, host packages only). `installPlugin({ git, ref } | { npm, version }, root, ctx)` installs plugins from git (a shallow clone of a branch, tag or commit; a repository of several plugins too) or npm; a package lists the plugin folders it provides in its `package.json` (`"chord": { "plugins": ["./notes", "plugins/*", "!plugins/draft"] }`), used by discovery and `installPlugin` (without it: one plugin when the package's top has facet files, else its subfolders with a `package.json`). `testPlugin(dir, { kind, modules, stubs, principal }, ctx)` builds a plugin as production does and loads it on an in-memory host beside stub facets, collecting its contributions to registries nobody serves and listing the contracts it waits for. Agents' guide: `docs/next/plugin-authoring.md`. +- Chord v2 gains the app kit (slice 13, `src/next-node/app`, to be exported as `@earendil-works/chord/next/app`): `startApp

    ({ name, dataDir?, root?, listen?, api, kinds, principal, workers?, files?, install?, onError }, ctx)` lays out the data directory (`server/store.sqlite`, `workers//

    /`, `plugins/`, `logs/`, `owner-token`; one server per data directory), creates the server's store and host, HTTP, one workers component per worker kind and the plugins component over the app's required root and the user's root, and loads the server's plugins with the kit's objects; `startWorker({ api, kinds, busy?, onError })` is a worker's entry (its directory and store, the link to the server first, its plugins). The start-data objects `AppInfo` and `WorkerInfo` serve process values to plugins. Sessions (`Sessions`: `issue`, `verify`, `revoke`, `list`; tokens kept as SHA-256 hashes, compared in constant time; persisted ones end after 400 days unused and their browser cookie is renewed daily while used, ephemeral ones live in memory, at most `limits.ephemeralSessions`) authenticate a signed `HttpOnly`, `SameSite=Strict` cookie or `Authorization: Bearer`; a one-time owner link per start (`app.ownerLink()`, the token in the URL's fragment, single use, 10 minutes), an owner token file (mode 0600, written atomically) and device pairing (`POST /_chord/pair`, owner approval, a poll returning the token once; bounded in all and per client address) sign the owner in. The data directory, its stores and secret files are made private (0700, 0600) at every start; the home directory, a file system's root and another user's directory are refused. `Files` per host: small and chunked writes at a stored offset, ranged reads, a watchable index, caps, at most `limits.uploads` uploads under way. `app.exportAddress(address, path, ctx)` and `app.importAddress(archive, address, ctx)` move a worker's store and files as a tar, holding the worker's process lock meanwhile (refusing an existing address, another app's archive, every path leaving the target, and archives past `limits.importBytes` or `limits.importEntries`). `app.applyChanges(ctx)` and `app.plugins.install(source, ctx)`. Optional plugins as exported facets: `defineUsers(profile)`, `inviteFacet()`, `ephemeralLoginFacet()`. +- Chord v2's `HttpRequest` gains `remoteAddress`, the peer's address as the listener knows it (for limits per client). +- Chord v2's HTTP component gains `readSignedCookie(request, name, secret)`: `readCookie`'s value with its signed expiry, for renewing a cookie before it expires. ### Changed +- Chord v2's HTTP component denies by default (slice 13): a request or plugin socket without a principal gets 401 before plugin code runs unless its route or socket is marked `public: true` (`{ handler, public: true }`; a `crossSite` route is public), and the `/chord` links endpoint refuses requests without a principal. - Chord v2 plugins import each other by relative path within a root (copied into the importer's script) and across repositories by package name through the package's `exports`, pinned in `dependencies`; slice 10's imports by plugin name (`@app/notes/contract`) are removed, and a relative path leaving the root is a build error. ## [1.5.0] - 2026-11-02 diff --git a/packages/chord/docs/next/guide-plugins-hosts/book.md b/packages/chord/docs/next/guide-plugins-hosts/book.md index 747cc4e32..ad2d511f2 100644 --- a/packages/chord/docs/next/guide-plugins-hosts/book.md +++ b/packages/chord/docs/next/guide-plugins-hosts/book.md @@ -1,7 +1,7 @@ --- title: Hosts and Plugins: a tutorial for Chord v2 cover: Hosts / and Plugins -kicker: A guide · Chord v2 · 2026-10-31 +kicker: A guide · Chord v2 · 2026-11-02 subtitle: Build a small notes app step by step, from one host to plugins on a server and on workers, edited while it runs stats: - 3 | kinds of host in an app: server, worker, client @@ -10,10 +10,10 @@ stats: - 1 | rule: plugins talk only through object types and registries note: | A tutorial for people building apps on Chord v2, written by the redesign session from `design.md`, `slice-7c.md`, - `slice-10.md`, the decision log and the code in `packages/chord/src/next` and `src/next-node`. API names are the - current ones; where slice 10's fix round or the app kit (`chord/next/app`, being designed) will change what app code - writes, a note says so. -colophon: Hosts and Plugins · second edition · 2026-11-01 · Chord v2 slices 1-10 and 12 + `slice-10.md`, `slice-12.md`, `slice-13.md`, the decision log and the code in `packages/chord/src/next` and + `src/next-node`. API names are the current ones. The early chapters wire hosts by hand to show the parts; the last + chapter shows the app kit (`chord/next/app`), which does that wiring for an app. +colophon: Hosts and Plugins · third edition · 2026-11-02 · Chord v2 slices 1-10, 12 and 13 --- # Before we begin {.preface} @@ -265,10 +265,10 @@ tab.connect(webSocketLink(new WebSocket("/chord")), { principal: { kind: "server `P` is your app's type of **principal**, who is calling: an owner, a guest, the server, a worker. Links stamp it; a `remote` object is made per principal, so a guest can get a read-only variant of the same notes. -:::note | Coming with the app kit -Starting workers, routing a host kind to them, the WebSocket entry and authentication are slice 10's components today -(`createWorkers`, `chord.http`), wired by the app. The opinionated app kit (`chord/next/app`, being designed) will do -this wiring for you: `startApp(...)` and `startWorker()`. +:::note | The app kit does this wiring +Starting workers, routing a host kind to them, the WebSocket entry and authentication are slice 10's components +(`createWorkers`, `chord.http`). An app built with the app kit (`chord/next/app`, the last chapter) does not wire them: +`startApp(...)` on the server and `startWorker(...)` in a worker's entry do. ::: # Client Hosts {#clients lede="Browser tabs and terminal UIs are hosts too: what they can do, and what they cannot."} @@ -633,6 +633,84 @@ same name, the first root wins, and the other is reported. as production does, loads it on an in-memory host beside stub facets serving what it uses, collects what it contributes to registries nobody serves, and lists the contracts it waits for (`docs/next/plugin-authoring.md`). +# The App Kit {#kit lede="startApp and startWorker: the hosts, stores, sign-in, files and plugin roots every app needs, so an app is its API, its plugins and a start script."} + +The chapters so far wired an app by hand: a store, a host, `createHttp`, `createWorkers`, `createPlugins`, an +authenticator, a worker entry connecting to its parent. Every app needs the same wiring, and most of it has one right +answer. The **app kit** (`chord/next/app`) is that answer, opinionated on purpose: one data directory layout, one way +to sign in, one place for files. + +## A whole app + +The notes app of example 53 is four files of base code and two plugins: + +```ts +// start.ts +const app = await startApp({ + name: "notes", // the data directory: ~/.notes unless dataDir says otherwise + root: "./plugins", // the app's own plugins: a required root + api: { name: "@notes/app", versions: ["v1"], modules: { "@notes/app/v1": v1, "@notes/app/v1/server": v1server } }, + kinds: { server: { platform: "node" }, worker: { platform: "node" } }, + principal: { server: { kind: "server" }, worker: (address) => ({ kind: "worker", project: address.key }) }, + workers: [{ + kind: Project, // one worker per project + entry: "./worker.ts", + start: (address) => ({ project: address.key }), + authorize: (principal) => principal.kind === "owner" || principal.kind === "server", + }], + onError: (error) => log(error), +}, ctx); +console.log(`Open ${app.ownerLink()}`); + +// worker.ts +startWorker({ api: API, kinds: KINDS, onError: (error) => log(error) }); +``` + +`startApp` creates the data directory (`server/store.sqlite`, `workers//
    /`, `plugins/`, `logs/`, +`owner-token`), opens the server's store, creates the server host, HTTP, a workers component per worker kind and the +plugins component over two roots: the app's `plugins/` and the user's `/plugins/`. Then it loads the server's +plugins with the kit's own facets beside them and resolves. `startWorker` does the worker's half: the project's +directory and store, the host at the project's address, the link to the server first, the worker's plugins. + +## What plugins get from the kit + +Plugins still import only the app's API; the API re-exports the kit's objects it offers: + +- `AppInfo` on the server (data directory, app name, API version) and `WorkerInfo` on workers (the address and the + start data `start(address)` made): the **start-data objects**. Process values reach plugins only through them. +- `Files`, one per host: `put` for small files, `begin`, `append` and `finish` for large ones, ranged `read`, `remove` + and a watchable `list`, stored in the host's directory with caps per host. +- `Sessions` on the server: `issue`, `revoke`, `list`. + +This finishes what the previous chapter started. Base code is the API, the start script and the worker's entry. The +providers of the app's own registries live in its required root: example 53's API declares a `Sections` registry, and +its home plugin serves it and reads `AppInfo`, so it reloads like any other plugin. + +## Who may come in + +HTTP **denies by default**: a request without a principal gets 401 before any plugin code runs, unless the route says +`public: true`. Someone gets a principal through a **session**: a random token, kept in the store only as a hash, sent +as a signed cookie by a browser or as `Authorization: Bearer` by a command-line client. + +The first session is the owner's. At each start the kit makes a one-time **owner link**, +`http://127.0.0.1:8788/_chord/owner#`, valid ten minutes; the start script prints it. The token is in the +fragment, which browsers never send, so it never reaches a log: the page posts it, the kit consumes it and sets the +owner's cookie. A command-line client on the same machine reads `/owner-token` (mode 0600). One on another +machine **pairs**: it asks for a code, the owner approves the code in a signed-in browser, and the client's next poll +returns its token (example 54). + +Users other than the owner come from optional plugins the app adds: `defineUsers(profile)` for accounts, +`inviteFacet()` for invite links, `ephemeralLoginFacet()` for "pick a name and you're in". + +## Applying changes, moving projects + +The kit never reloads by itself. An edit or an install marks plugins dirty; `app.plugins.state` drives the app's +"apply changes" button, and `app.applyChanges(ctx)` builds and rolls out to the server, then the workers. + +A project's data is its directory. `app.exportAddress(Project({ id }), path, ctx)` stops its worker and writes one +archive (a manifest, the store's export, the files); `app.importAddress(path, Project({ id: "copy" }), ctx)` makes a new +project from it and refuses one that already has data (example 55). + # Reference {.appendix} | Call | What it does | @@ -647,3 +725,8 @@ same name, the first root wins, and the other is reported. | `plugins.load(host, { kind, core, modules }, ctx)`, `build(names)`, `rollout(steps, built)` | load, build, roll out | | `plugins.installPlugin(source, root, ctx)`, `testPlugin(dir, options, ctx)` | install a plugin, test one alone | | `hostKind(name, params?)`, `Kind(params)` | host kinds and addresses | +| `startApp

    ({ name, dataDir?, root?, listen?, api, kinds, principal, workers?, files?, onError }, ctx)` | an app's server, with the kit | +| `startWorker({ api, kinds, busy?, onError })` | a worker's entry, with the kit | +| `app.ownerLink()`, `app.applyChanges(ctx)`, `app.plugins.install(source, ctx)` | sign in, apply changes, install | +| `app.exportAddress(address, path, ctx)`, `app.importAddress(path, address, ctx)` | move a worker's data | +| `AppInfo`, `WorkerInfo`, `Files`, `Sessions` | the kit's objects plugins get | diff --git a/packages/chord/docs/next/guide-plugins-hosts/index.html b/packages/chord/docs/next/guide-plugins-hosts/index.html index 9b30188ea..beac6027f 100644 --- a/packages/chord/docs/next/guide-plugins-hosts/index.html +++ b/packages/chord/docs/next/guide-plugins-hosts/index.html @@ -109,14 +109,14 @@ table{max-width:100%}

    -
    A guide · Chord v2 · 2026-10-31
    +
    A guide · Chord v2 · 2026-11-02

    Hosts
    and Plugins

    Build a small notes app step by step, from one host to plugins on a server and on workers, edited while it runs

    3kinds of host in an app: server, worker, client
    1file per host kind in a plugin
    0restarts to apply a plugin edit
    1rule: plugins talk only through object types and registries

    A tutorial for people building apps on Chord v2, written by the redesign session from design.md, slice-7c.md, -slice-10.md, the decision log and the code in packages/chord/src/next and src/next-node. API names are the -current ones; where slice 10's fix round or the app kit (chord/next/app, being designed) will change what app code -writes, a note says so.

    +slice-10.md, slice-12.md, slice-13.md, the decision log and the code in packages/chord/src/next and +src/next-node. API names are the current ones. The early chapters wire hosts by hand to show the parts; the last +chapter shows the app kit (chord/next/app), which does that wiring for an app.

    @@ -306,9 +307,9 @@ worker.connect(parentLink, { principal: { kind: connect(webSocketLink(new WebSocket("/chord")), { principal: { kind: "server" } }, ctx);

    P is your app's type of principal, who is calling: an owner, a guest, the server, a worker. Links stamp it; a remote object is made per principal, so a guest can get a read-only variant of the same notes.

    -
    Coming with the app kit

    Starting workers, routing a host kind to them, the WebSocket entry and authentication are slice 10's components today -(createWorkers, chord.http), wired by the app. The opinionated app kit (chord/next/app, being designed) will do -this wiring for you: startApp(...) and startWorker().

    +
    The app kit does this wiring

    Starting workers, routing a host kind to them, the WebSocket entry and authentication are slice 10's components +(createWorkers, chord.http). An app built with the app kit (chord/next/app, the last chapter) does not wire them: +startApp(...) on the server and startWorker(...) in a worker's entry do.

    @@ -593,6 +594,69 @@ contributes to registries nobody serves, and lists the contracts it waits for (< +
    X

    The App Kit

    startApp and startWorker: the hosts, stores, sign-in, files and plugin roots every app needs, so an app is its API, its plugins and a start script.

    +

    The chapters so far wired an app by hand: a store, a host, createHttp, createWorkers, createPlugins, an +authenticator, a worker entry connecting to its parent. Every app needs the same wiring, and most of it has one right +answer. The app kit (chord/next/app) is that answer, opinionated on purpose: one data directory layout, one way +to sign in, one place for files.

    +

    A whole app

    +

    The notes app of example 53 is four files of base code and two plugins:

    +
    // start.ts
+const app = await startApp<NotesPrincipal>({
+	name: "notes",                                  // the data directory: ~/.notes unless dataDir says otherwise
+	root: "./plugins",                              // the app's own plugins: a required root
+	api: { name: "@notes/app", versions: ["v1"], modules: { "@notes/app/v1": v1, "@notes/app/v1/server": v1server } },
+	kinds: { server: { platform: "node" }, worker: { platform: "node" } },
+	principal: { server: { kind: "server" }, worker: (address) => ({ kind: "worker", project: address.key }) },
+	workers: [{
+		kind: Project,                              // one worker per project
+		entry: "./worker.ts",
+		start: (address) => ({ project: address.key }),
+		authorize: (principal) => principal.kind === "owner" || principal.kind === "server",
+	}],
+	onError: (error) => log(error),
+}, ctx);
+console.log(`Open ${app.ownerLink()}`);
+
+// worker.ts
+startWorker({ api: API, kinds: KINDS, onError: (error) => log(error) });
    +

    startApp creates the data directory (server/store.sqlite, workers/<kind>/<address>/, plugins/, logs/, +owner-token), opens the server's store, creates the server host, HTTP, a workers component per worker kind and the +plugins component over two roots: the app's plugins/ and the user's <dataDir>/plugins/. Then it loads the server's +plugins with the kit's own facets beside them and resolves. startWorker does the worker's half: the project's +directory and store, the host at the project's address, the link to the server first, the worker's plugins.

    +

    What plugins get from the kit

    +

    Plugins still import only the app's API; the API re-exports the kit's objects it offers:

    +
      +
    • AppInfo on the server (data directory, app name, API version) and WorkerInfo on workers (the address and the +start data start(address) made): the start-data objects. Process values reach plugins only through them.
      • +
    • Files, one per host: put for small files, begin, append and finish for large ones, ranged read, remove +and a watchable list, stored in the host's directory with caps per host.
      • +
    • Sessions on the server: issue, revoke, list.
      • +
    +

    This finishes what the previous chapter started. Base code is the API, the start script and the worker's entry. The +providers of the app's own registries live in its required root: example 53's API declares a Sections registry, and +its home plugin serves it and reads AppInfo, so it reloads like any other plugin.

    +

    Who may come in

    +

    HTTP denies by default: a request without a principal gets 401 before any plugin code runs, unless the route says +public: true. Someone gets a principal through a session: a random token, kept in the store only as a hash, sent +as a signed cookie by a browser or as Authorization: Bearer by a command-line client.

    +

    The first session is the owner's. At each start the kit makes a one-time owner link, +http://127.0.0.1:8788/_chord/owner#<token>, valid ten minutes; the start script prints it. The token is in the +fragment, which browsers never send, so it never reaches a log: the page posts it, the kit consumes it and sets the +owner's cookie. A command-line client on the same machine reads <dataDir>/owner-token (mode 0600). One on another +machine pairs: it asks for a code, the owner approves the code in a signed-in browser, and the client's next poll +returns its token (example 54).

    +

    Users other than the owner come from optional plugins the app adds: defineUsers(profile) for accounts, +inviteFacet() for invite links, ephemeralLoginFacet() for "pick a name and you're in".

    +

    Applying changes, moving projects

    +

    The kit never reloads by itself. An edit or an install marks plugins dirty; app.plugins.state drives the app's +"apply changes" button, and app.applyChanges(ctx) builds and rolls out to the server, then the workers.

    +

    A project's data is its directory. app.exportAddress(Project({ id }), path, ctx) stops its worker and writes one +archive (a manifest, the store's export, the files); app.importAddress(path, Project({ id: "copy" }), ctx) makes a new +project from it and refuses one that already has data (example 55).

    +
    +
    A

    Reference

    @@ -641,10 +705,30 @@ contributes to registries nobody serves, and lists the contracts it waits for (< + + + + + + + + + + + + + + + + + + + +
    hostKind(name, params?), Kind(params) host kinds and addresses
    startApp<P>({ name, dataDir?, root?, listen?, api, kinds, principal, workers?, files?, onError }, ctx)an app's server, with the kit
    startWorker({ api, kinds, busy?, onError })a worker's entry, with the kit
    app.ownerLink(), app.applyChanges(ctx), app.plugins.install(source, ctx)sign in, apply changes, install
    app.exportAddress(address, path, ctx), app.importAddress(path, address, ctx)move a worker's data
    AppInfo, WorkerInfo, Files, Sessionsthe kit's objects plugins get
    -

    Hosts and Plugins · second edition · 2026-11-01 · Chord v2 slices 1-10 and 12

    +

    Hosts and Plugins · third edition · 2026-11-02 · Chord v2 slices 1-10, 12 and 13

    ", + ].join("\n"); +} + +/** + * The owner token file at `path`: kept when its token still authenticates as the owner (its session exists, its secret + * matches, it has not expired; its mode set to 0600 again), else a new device session of the owner written there (mode + * 0600, in one step). Its token, for the start script. + */ +export async function ownerTokenFile

    ( + path: string, + host: Host

    , + context: Context, +): Promise { + const existing = await readSecret(path); + return host.root(async (ctx) => { + const sessions = await ctx.get(Sessions, {}); + if (existing !== undefined && isOwner(await sessions.verify(existing, ctx))) { + await chmod(path, 0o600); + return existing; + } + const { token } = await sessions.issue({ principal: OWNER, label: "owner token file", kind: "device" }, ctx); + await writeSecret(path, token, existing === undefined ? "create" : "replace"); + return token; + }, context); +} + +/** Whether `principal` (as `Sessions.verify` gives it) is the owner. */ +function isOwner(principal: unknown): boolean { + return typeof principal === "object" && principal !== null && (principal as { kind?: unknown }).kind === "owner"; +} diff --git a/packages/chord/src/next-node/app/pairing.ts b/packages/chord/src/next-node/app/pairing.ts new file mode 100644 index 000000000..8741aa5cd --- /dev/null +++ b/packages/chord/src/next-node/app/pairing.ts @@ -0,0 +1,205 @@ +// Device pairing (slice 13 §2, decision "The app kit, point 3" (c)): a command-line or terminal client on another +// machine asks for a code (`POST /_chord/pair`, public) and polls with a secret it got with it; the owner opens +// `/_chord/pair/` in a signed-in browser and approves; the next poll returns a device session's token, once. +// Codes are short, random, single use and expire after 10 minutes; the poll secret is kept only hashed. Asking is +// public, so pairings anonymous clients start are capped: `maxPending` in all (100 by default) and `perAddress` per +// client address (10). The trade: a party filling the caps locks pairing out for others for up to 10 minutes (behind a +// tunnel every remote client has the tunnel's address), instead of growing the server's memory without bound; a +// signed-in owner's pairings are not counted, so the owner can always pair. + +import { randomInt } from "node:crypto"; +import type { Principal } from "../../next/data/index.ts"; +import { defineFacet, type Facet } from "../../next/host/index.ts"; +import { Http, type HttpRequest, type HttpResponse } from "../http/index.ts"; +import { OWNER } from "./owner.ts"; +import { hashToken, newToken } from "./secrets.ts"; +import { Sessions } from "./sessions.ts"; + +/** How long a pairing code is valid: 10 minutes. */ +const PAIRING_MS = 10 * 60_000; + +/** The code's letters: no 0/O, 1/I/L. */ +const ALPHABET = "ABCDEFGHJKMNPQRSTUVWXYZ23456789"; + +/** Who starts a pairing: the signed-in owner (not counted), or an anonymous client at an address (counted). */ +export type PairingStarter = { readonly kind: "owner" } | { readonly kind: "anonymous"; readonly address: string }; + +/** One pairing under way: its code, the device's label, who started it, its expiry, its token once approved. */ +type Pairing = { + readonly code: string; + readonly label: string; + readonly by: PairingStarter; + readonly expires: number; + token?: string; +}; + +/** The limits of pairings under way: in all (100 by default) and per client address (10 by default). */ +export type PairingLimits = { readonly maxPending?: number; readonly perAddress?: number }; + +/** The pairings under way, by code and by their poll secret's hash, with the anonymous ones counted. */ +export class Pairings { + readonly #byCode = new Map(); + readonly #byPoll = new Map(); + readonly #perAddress = new Map(); + #anonymous = 0; + readonly #maxPending: number; + readonly #maxPerAddress: number; + + constructor(limits: PairingLimits = {}) { + this.#maxPending = limits.maxPending ?? 100; + this.#maxPerAddress = limits.perAddress ?? 10; + } + + /** + * A new pairing for a device labelled `label`, started `by` the owner or an anonymous client: its code (for the + * owner) and its poll secret (for the device); `undefined` when an anonymous client reached a limit. + */ + begin(label: string, by: PairingStarter): { readonly code: string; readonly poll: string } | undefined { + this.#expire(); + if (by.kind === "anonymous") { + const full = this.#anonymous >= this.#maxPending; + if (full || (this.#perAddress.get(by.address) ?? 0) >= this.#maxPerAddress) return undefined; + this.#anonymous++; + this.#perAddress.set(by.address, (this.#perAddress.get(by.address) ?? 0) + 1); + } + let code: string; + do code = `${letters(4)}-${letters(4)}`; + while (this.#byCode.has(code)); + const poll = newToken(); + const pairing: Pairing = { code, label, by, expires: Date.now() + PAIRING_MS }; + this.#byCode.set(code, pairing); + this.#byPoll.set(hashToken(poll), pairing); + return { code, poll }; + } + + /** The pairing waiting for approval under `code` (any case), unexpired. */ + waiting(code: string): Pairing | undefined { + this.#expire(); + const pairing = this.#byCode.get(code.toUpperCase()); + return pairing?.token === undefined ? pairing : undefined; + } + + /** What a poll with `poll` learns: still waiting, the device's token (once: the pairing ends), or nothing. */ + poll( + poll: string, + ): { readonly kind: "waiting" } | { readonly kind: "approved"; readonly token: string } | undefined { + this.#expire(); + const hash = hashToken(poll); + const pairing = this.#byPoll.get(hash); + if (pairing === undefined) return undefined; + if (pairing.token === undefined) return { kind: "waiting" }; + this.#end(hash, pairing); + return { kind: "approved", token: pairing.token }; + } + + /** Drops the expired pairings. */ + #expire(): void { + const now = Date.now(); + for (const [hash, pairing] of this.#byPoll) { + if (pairing.expires <= now) this.#end(hash, pairing); + } + } + + /** Ends `pairing` (its poll secret's hash `hash`). */ + #end(hash: string, pairing: Pairing): void { + this.#byPoll.delete(hash); + this.#byCode.delete(pairing.code); + const { by } = pairing; + if (by.kind === "owner") return; + this.#anonymous--; + const left = (this.#perAddress.get(by.address) ?? 1) - 1; + if (left === 0) this.#perAddress.delete(by.address); + else this.#perAddress.set(by.address, left); + } +} + +/** `count` random letters of `ALPHABET` (`crypto.randomInt`). */ +function letters(count: number): string { + let text = ""; + for (let index = 0; index < count; index++) text += ALPHABET[randomInt(ALPHABET.length)]; + return text; +} + +/** The pairing routes' prefix. */ +export const PAIR_PATH = "/_chord/pair"; + +/** + * The kit's facet for device pairing: `POST /_chord/pair` (public: `{ label? }` → `{ code, poll }`, 429 past a limit + * unless the owner asks), `POST /_chord/pair/poll` (public: `{ poll }` → `{ status: "waiting" }`, `{ status: + * "approved", token }` once, else 404), `GET` and `POST /_chord/pair/:code` (the owner's confirm page and approval: a + * device session of the owner). + */ +export function pairingFacet

    (pairings: Pairings): Facet

    { + return defineFacet

    ((_host, self) => + self.contribute(Http, { + routes: { + [`POST ${PAIR_PATH}`]: { + public: true, + handler: async (request, ctx): Promise => { + const label = await labelOf(request); + const by: PairingStarter = + ctx.principal?.kind === "owner" + ? { kind: "owner" } + : { kind: "anonymous", address: request.remoteAddress ?? "" }; + const begun = pairings.begin(label, by); + return begun === undefined ? { kind: "error", status: 429 } : { kind: "json", value: begun }; + }, + }, + [`POST ${PAIR_PATH}/poll`]: { + public: true, + handler: async (request): Promise => { + // A trust boundary: anyone may poll. + const body = (await request.json()) as { readonly poll?: unknown } | null; + const polled = typeof body?.poll === "string" ? pairings.poll(body.poll) : undefined; + if (polled === undefined) return { kind: "error", status: 404 }; + if (polled.kind === "waiting") return { kind: "json", value: { status: "waiting" } }; + return { kind: "json", value: { status: "approved", token: polled.token } }; + }, + }, + [`GET ${PAIR_PATH}/:code`]: async (request, ctx) => { + if (ctx.principal?.kind !== "owner") return { kind: "error", status: 403 }; + const pairing = pairings.waiting(request.params.code ?? ""); + if (pairing === undefined) + return { kind: "html", status: 404, body: "

    No such pairing (or it expired).

    " }; + return { kind: "html", body: confirmPage(pairing.code, pairing.label) }; + }, + [`POST ${PAIR_PATH}/:code`]: async (request, ctx) => { + if (ctx.principal?.kind !== "owner") return { kind: "error", status: 403 }; + const pairing = pairings.waiting(request.params.code ?? ""); + if (pairing === undefined) + return { kind: "html", status: 404, body: "

    No such pairing (or it expired).

    " }; + const sessions = await ctx.get(Sessions, {}); + const { token } = await sessions.issue({ principal: OWNER, label: pairing.label, kind: "device" }, ctx); + pairing.token = token; + return { + kind: "html", + body: `

    Approved: ${escapeHtml(pairing.label)} is signed in as the owner.

    `, + }; + }, + }, + }), + ); +} + +/** The device's label from a pairing request's body (`{ label }`, at most 100 characters), else `device`. */ +async function labelOf(request: HttpRequest): Promise { + const declared = Number(request.headers.get("content-length") ?? "0"); + if (declared === 0 && request.headers.get("transfer-encoding") === null) return "device"; + // A trust boundary: anyone may ask for a code. + const body = (await request.json(4096)) as { readonly label?: unknown } | null; + return typeof body?.label === "string" && body.label.trim() !== "" ? body.label.trim().slice(0, 100) : "device"; +} + +/** The owner's confirm page for pairing `code` (a form posting to itself: same origin, the session cookie). */ +function confirmPage(code: string, label: string): string { + return [ + "Pair a device", + `

    Sign in ${escapeHtml(label)} as the owner? Its code: ${escapeHtml(code)} (compare it with the device).

    `, + '
    ', + ].join("\n"); +} + +/** `text` safe in HTML. */ +function escapeHtml(text: string): string { + return text.replace(/[&<>"']/g, (char) => `&#${char.charCodeAt(0)};`); +} diff --git a/packages/chord/src/next-node/app/secrets.ts b/packages/chord/src/next-node/app/secrets.ts new file mode 100644 index 000000000..6de1d870f --- /dev/null +++ b/packages/chord/src/next-node/app/secrets.ts @@ -0,0 +1,75 @@ +// Secrets of the app kit (slice 13 §2): tokens from `crypto.randomBytes`, kept only as their SHA-256 hashes and +// compared in constant time, and secret files (the cookie key, the owner token file) created with mode 0600 in one +// step, so no other OS user and no half-written file ever reads them. + +import { createHash, randomBytes, timingSafeEqual } from "node:crypto"; +import { link, open, readFile, rename, unlink } from "node:fs/promises"; +import { dirname, join } from "node:path"; + +/** A fresh random token of `bytes` bytes, base64url. */ +export function newToken(bytes = 32): string { + return randomBytes(bytes).toString("base64url"); +} + +/** The hash a token is kept as (SHA-256, hex): what leaks from a store is no credential. */ +export function hashToken(token: string): string { + return createHash("sha256").update(token).digest("hex"); +} + +/** Whether `token` hashes to `hash`, compared in constant time. */ +export function matchesHash(token: string, hash: string): boolean { + const given = Buffer.from(hashToken(token), "hex"); + const kept = Buffer.from(hash, "hex"); + return given.length === kept.length && timingSafeEqual(given, kept); +} + +/** The secret in the file at `path`, created (mode 0600, in one step) with a fresh token when it does not exist yet. */ +export async function secretFile(path: string): Promise { + const existing = await readSecret(path); + if (existing !== undefined) return existing; + const secret = newToken(); + const written = await writeSecret(path, secret, "create"); + return written === "exists" ? ((await readSecret(path)) ?? secret) : secret; +} + +/** The text of the secret file at `path`, or `undefined` when it does not exist. */ +export async function readSecret(path: string): Promise { + try { + return (await readFile(path, "utf8")).trim(); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return undefined; + throw error; + } +} + +/** + * Writes `text` to `path` with mode 0600 in one step: a temporary file of the same directory (made exclusively, 0600) + * linked into place (`create`: refused when `path` exists, `"exists"` returned) or renamed over it (`replace`). + */ +export async function writeSecret( + path: string, + text: string, + mode: "create" | "replace", +): Promise<"written" | "exists"> { + const temporary = join(dirname(path), `.${newToken(9)}.tmp`); + const handle = await open(temporary, "wx", 0o600); + try { + await handle.writeFile(`${text}\n`); + await handle.sync(); + } finally { + await handle.close(); + } + try { + if (mode === "replace") { + await rename(temporary, path); + return "written"; + } + await link(temporary, path); + return "written"; + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "EEXIST") return "exists"; + throw error; + } finally { + await unlink(temporary).catch(() => {}); + } +} diff --git a/packages/chord/src/next-node/app/sessions.ts b/packages/chord/src/next-node/app/sessions.ts new file mode 100644 index 000000000..f42a4e5a1 --- /dev/null +++ b/packages/chord/src/next-node/app/sessions.ts @@ -0,0 +1,267 @@ +// Sessions (slice 13 §2, decision "The app kit, point 3: authentication"): the kit's object on the server that issues +// and revokes sessions, and its authenticator. A session is a principal behind a token `.` (random); only +// the secret's hash is kept. A browser carries its token in the kit's signed session cookie (`HttpOnly`, +// `SameSite=Strict`), a command-line client in `Authorization: Bearer `. Browser and device sessions persist in +// the server's store and expire after 400 days unused; ephemeral sessions (a pick-a-name login, a public route) live +// in memory only, at most `maxEphemeral` of them (past it the oldest ends), so a public login costs the same whatever +// came before, and none outlives the server's run. + +import type { Context } from "../../context/index.ts"; +import type { JsonValue } from "../../json.ts"; +import { arr, bool, local, map, method, num, obj, oneOf, opt, str, unknown } from "../../next/contracts/index.ts"; +import type { Principal } from "../../next/data/index.ts"; +import { defineState } from "../../next/documents/index.ts"; +import { defineFacet, type Facet } from "../../next/host/index.ts"; +import { + Authenticators, + cookie, + Http, + type HttpRequest, + type HttpWrap, + type Provenance, + readCookie, + readSignedCookie, +} from "../http/index.ts"; +import { hashToken, matchesHash, newToken } from "./secrets.ts"; + +/** What a session is for: a browser (its cookie), a device (a command-line client's token), an ephemeral login. */ +export type SessionKind = "browser" | "device" | "ephemeral"; + +/** A browser or device session as `Sessions.list` shows it: never its token or hash. */ +export type SessionInfo = { + readonly id: string; + readonly label: string; + readonly kind: "browser" | "device"; + readonly principal: unknown; + readonly created: number; + readonly lastUsed: number; +}; + +/** The kit's sessions (one per server): issue, verify, revoke, and the watchable list of persisted ones. */ +export const Sessions = local("chord.app.sessions", { + version: 1, + params: {}, + members: { + /** The browser and device sessions (ephemeral ones are not listed: there may be many, and they are short). */ + list: arr( + obj({ + id: str(), + label: str(), + kind: oneOf("browser", "device"), + principal: unknown(), + created: num(), + lastUsed: num(), + }), + ), + /** + * A new session for `principal`: its id, its token (shown once, kept only hashed) and the session cookie carrying + * it for a browser (`Set-Cookie`; `secure` for a remote one). + */ + issue: method( + obj({ + principal: unknown(), + label: str(), + kind: oneOf("browser", "device", "ephemeral"), + secure: opt(bool()), + }), + obj({ id: str(), token: str(), cookie: str() }), + ), + /** The principal `token` authenticates as, or `null` (unknown, revoked, expired, malformed). */ + verify: method(str(), unknown()), + /** Ends session `id` at once: its token authenticates nothing more. */ + revoke: method(str()), + }, +}); + +/** The name of the kit's session cookie. */ +export const SESSION_COOKIE = "chord-session"; + +/** The stored sessions: by id, each with its secret's hash and its principal (as JSON text: any of the app's). */ +const SessionsDocument = defineState("chord.app.sessions", { + schema: obj({ + sessions: map( + obj({ + hash: str(), + label: str(), + kind: oneOf("browser", "device"), + principal: str(), + created: num(), + lastUsed: num(), + }), + ), + }), + version: 1, + initial: () => ({ sessions: {} }), + persist: true, + checkpointWhen: (_value, _ops, { deltasSinceBase }) => deltasSinceBase >= 50, +}); + +/** An ephemeral session, in memory: its secret's hash and its principal. */ +type EphemeralSession = { readonly hash: string; readonly principal: Principal }; + +/** How often a session's `lastUsed` is written at most: once a minute (not one commit per request). */ +const TOUCH_MS = 60_000; + +/** How long a persisted session lives unused: 400 days, as its cookie. */ +const SESSION_IDLE_MS = 400 * 24 * 60 * 60_000; + +/** The sessions' limits: how many ephemeral sessions live at once (1,000 by default; past it the oldest ends). */ +export type SessionLimits = { readonly maxEphemeral?: number }; + +/** + * The kit's facet serving `Sessions` (a host-lifetime object) and contributing its authenticator: the bearer token + * first, else the session cookie signed with `cookieSecret`. + */ +export function sessionsFacet

    (cookieSecret: string, limits: SessionLimits = {}): Facet

    { + const maxEphemeral = limits.maxEphemeral ?? 1000; + return defineFacet

    ((host) => + host.serve(Sessions, (_params, self) => { + const document = self.state(SessionsDocument); + // Insertion order is age order: the first entry is the oldest. + const ephemeral = new Map(); + const change = (fn: (sessions: Record) => void, context: Context) => + // Not awaited by authentication: the request goes on; a write that fails is made again by a later request. + void document.change((draft) => fn(draft.sessions), context).catch(() => {}); + /** The principal of `token`, or `undefined`; touches or lazily removes the persisted session it names. */ + const authenticate = (token: string, context: Context): Principal | undefined => { + const dot = token.indexOf("."); + if (dot <= 0) return undefined; + const id = token.slice(0, dot); + const secret = token.slice(dot + 1); + const short = ephemeral.get(id); + if (short !== undefined) return matchesHash(secret, short.hash) ? short.principal : undefined; + // A trust boundary: `id` is the client's (`Object.hasOwn`: no inherited member is a session). + const sessions = document.value.sessions; + if (!Object.hasOwn(sessions, id)) return undefined; + const session = sessions[id]!; + if (!matchesHash(secret, session.hash)) return undefined; + const now = Date.now(); + if (now - session.lastUsed > SESSION_IDLE_MS) { + change((draft) => { + delete draft[id]; + }, context); + return undefined; + } + if (now - session.lastUsed > TOUCH_MS) { + change((draft) => { + if (Object.hasOwn(draft, id)) draft[id]!.lastUsed = now; + }, context); + } + // Stored by `issue` from a principal of the app's type. + return JSON.parse(session.principal) as Principal; + }; + self.contribute(Authenticators, (request, context) => { + const token = bearerToken(request) ?? readCookie(request, SESSION_COOKIE, cookieSecret); + return token === undefined ? undefined : authenticate(token, context); + }); + self.contribute(Http, { wraps: [rollingCookie(cookieSecret, authenticate)] }); + return { + list: self.derive([document], ([value]) => + Object.entries(value.sessions).map(([id, { label, kind, principal, created, lastUsed }]) => ({ + id, + label, + kind, + // Stored by `issue` as JSON text. + principal: JSON.parse(principal) as JsonValue, + created, + lastUsed, + })), + ), + issue: async ({ principal, label, kind, secure }, ctx) => { + const id = newToken(12); + const secret = newToken(32); + if (kind === "ephemeral") { + // Principals are the app's: `issue`'s callers pass one of its type. + ephemeral.set(id, { hash: hashToken(secret), principal: principal as Principal }); + for (const [oldest] of ephemeral) { + if (ephemeral.size <= maxEphemeral) break; + ephemeral.delete(oldest); + } + } else { + const now = Date.now(); + const stored = JSON.stringify(principal); + await document.change((draft) => { + draft.sessions[id] = { + hash: hashToken(secret), + label, + kind, + principal: stored, + created: now, + lastUsed: now, + }; + }, ctx); + } + const token = `${id}.${secret}`; + // Only whether the browser is remote matters to the cookie (`Secure`). + const provenance: Provenance = secure === true ? { kind: "remote", host: "" } : { kind: "local" }; + const set = sessionCookie(token, kind, { provenance }, cookieSecret); + return { id, token, cookie: set }; + }, + verify: async (token, ctx) => (authenticate(token, ctx) as JsonValue | undefined) ?? null, + revoke: async (id, ctx) => { + if (ephemeral.delete(id)) return null; + await document.change((draft) => { + delete draft.sessions[id]; + }, ctx); + return null; + }, + }; + }), + ); +} + +/** + * The session cookie carrying `token` (a session of `kind`) for a browser of `request`'s provenance: a persisted + * session's lives as long as the session does unused (400 days, renewed by `rollingCookie`), an ephemeral one's only + * while the browser runs. + */ +function sessionCookie( + token: string, + kind: SessionKind, + request: Pick, + secret: string, +): string { + const maxAge = kind === "ephemeral" ? undefined : SESSION_IDLE_MS / 1000; + return cookie(SESSION_COOKIE, token, { secret, request, maxAge }); +} + +/** How old a session cookie gets before a response renews it: a day. */ +const ROLL_MS = 24 * 60 * 60_000; + +/** + * The wrap around every route renewing the session cookie a request signed in with once it is older than a day: the + * response sets it again with a fresh signed expiry, so a browser used within every 400 days stays signed in. Not when + * the route sets a cookie itself, nor for an ephemeral session's (it has no expiry) or a token that no longer + * authenticates. + */ +function rollingCookie( + secret: string, + authenticate: (token: string, context: Context) => Principal | undefined, +): HttpWrap { + return { + name: "chord.app.rolling-cookie", + phase: "headers", + scope: "all", + wrap: async (request, next, ctx) => { + const response = await next(request); + if (ctx.principal === undefined || Object.keys(response.headers ?? {}).some(isSetCookie)) return response; + const signed = readSignedCookie(request, SESSION_COOKIE, secret); + if (signed?.expires === undefined || signed.expires - Date.now() > SESSION_IDLE_MS - ROLL_MS) return response; + if (authenticate(signed.value, ctx) === undefined) return response; + const renewed = sessionCookie(signed.value, "browser", request, secret); + return { ...response, headers: { ...response.headers, "Set-Cookie": renewed } }; + }, + }; +} + +/** Whether a header name is `Set-Cookie` (any case). */ +function isSetCookie(name: string): boolean { + return name.toLowerCase() === "set-cookie"; +} + +/** The token of an `Authorization: Bearer ` header. */ +function bearerToken(request: HttpRequest): string | undefined { + const header = request.headers.get("authorization"); + const match = header === null ? null : /^Bearer\s+(\S+)$/i.exec(header); + return match?.[1]; +} diff --git a/packages/chord/src/next-node/app/tar.ts b/packages/chord/src/next-node/app/tar.ts new file mode 100644 index 000000000..7c1cde0ad --- /dev/null +++ b/packages/chord/src/next-node/app/tar.ts @@ -0,0 +1,181 @@ +// A minimal ustar (POSIX tar) writer and reader for the kit's address archives (slice 13 §4): regular files and +// directories, paths of at most 255 bytes, nothing else. The reader is a trust boundary: an archive may come from +// anywhere, so it refuses every entry that is no regular file or directory, a path that is absolute, holds `..`, a +// backslash or NUL, or names an empty segment, and a header whose checksum does not hold. + +/** One entry to write: its path (`/`-separated, relative) and its bytes, as a size and the chunks that make it up. */ +export type TarEntry = { readonly path: string; readonly size: number; readonly data: AsyncIterable }; + +/** One entry read: a regular file (its data, to be consumed before the next entry) or a directory. */ +export type TarRead = + | { readonly kind: "file"; readonly path: string; readonly size: number; readonly data: AsyncIterable } + | { readonly kind: "directory"; readonly path: string }; + +/** A tar block. */ +const BLOCK = 512; + +/** The archive of `entries`, as the chunks of its bytes (two zero blocks end it). */ +export async function* writeTar(entries: AsyncIterable | Iterable): AsyncGenerator { + for await (const entry of entries) { + yield header(entry.path, entry.size); + let written = 0; + for await (const chunk of entry.data) { + written += chunk.length; + if (written > entry.size) throw new Error(`tar: ${entry.path} is larger than its size ${entry.size}`); + yield chunk; + } + if (written !== entry.size) throw new Error(`tar: ${entry.path} is ${written} bytes, not ${entry.size}`); + const pad = (BLOCK - (entry.size % BLOCK)) % BLOCK; + if (pad > 0) yield new Uint8Array(pad); + } + yield new Uint8Array(BLOCK * 2); +} + +/** The ustar header of a regular file at `path` of `size` bytes. */ +function header(path: string, size: number): Uint8Array { + checkPath(path); + const block = new Uint8Array(BLOCK); + const bytes = new TextEncoder().encode(path); + let name = bytes; + let prefix = new Uint8Array(); + if (bytes.length > 100) { + // Split at a `/` so the name has at most 100 bytes and the prefix at most 155. + const at = path.lastIndexOf("/", path.length - 1); + const split = new TextEncoder().encode(path.slice(0, at)).length; + if (at < 0 || split > 155 || bytes.length - split - 1 > 100) throw new Error(`tar: the path ${path} is too long`); + prefix = bytes.subarray(0, split); + name = bytes.subarray(split + 1); + } + block.set(name, 0); + field(block, 100, 8, "0000644"); + field(block, 108, 8, "0000000"); + field(block, 116, 8, "0000000"); + field(block, 124, 12, size.toString(8).padStart(11, "0")); + field( + block, + 136, + 12, + Math.floor(Date.now() / 1000) + .toString(8) + .padStart(11, "0"), + ); + block.fill(0x20, 148, 156); + block[156] = 0x30; + block.set(new TextEncoder().encode("ustar\u000000"), 257); + block.set(prefix, 345); + const sum = block.reduce((total, byte) => total + byte, 0); + field(block, 148, 8, `${sum.toString(8).padStart(6, "0")}\u0000 `); + return block; +} + +/** Writes `text` (ASCII) at `offset` of `block`, NUL-terminated within `length`. */ +function field(block: Uint8Array, offset: number, length: number, text: string): void { + const bytes = new TextEncoder().encode(text); + if (bytes.length > length) throw new Error(`tar: ${text} does not fit ${length} bytes`); + block.set(bytes, offset); +} + +/** Refuses a path an archive may not hold: absolute, with `..`, `.`, an empty segment, a backslash or a NUL. */ +function checkPath(path: string): void { + const segments = path.split("/"); + const bad = path === "" || path.includes("\\") || path.includes("\u0000") || path.startsWith("/"); + if (bad || segments.some((segment) => segment === "" || segment === "." || segment === "..")) { + throw new Error(`tar: the path ${JSON.stringify(path)} is refused`); + } +} + +/** + * Reads the archive `source` entry by entry (each file's data read fully before the next entry is asked for), refusing + * what it may not hold (see the file's header). + */ +export async function* readTar(source: AsyncIterable): AsyncGenerator { + const reader = new ByteReader(source); + for (;;) { + const block = await reader.read(BLOCK); + if (block.length < BLOCK) throw new Error("tar: the archive ends inside a header"); + if (block.every((byte) => byte === 0)) return; + const { path, size, type } = parseHeader(block); + const padded = Math.ceil(size / BLOCK) * BLOCK; + if (type === "directory") { + yield { kind: "directory", path }; + await reader.skip(padded); + continue; + } + let left = size; + const data = (async function* () { + while (left > 0) { + const chunk = await reader.read(Math.min(left, 64 * 1024)); + if (chunk.length === 0) throw new Error(`tar: the archive ends inside ${path}`); + left -= chunk.length; + yield chunk; + } + })(); + yield { kind: "file", path, size, data }; + // What the consumer left unread (also when it ended the data's iteration early), then the padding. + await reader.skip(left + padded - size); + } +} + +/** A header's path, size and type, its checksum checked; anything but a regular file or a directory refused. */ +function parseHeader(block: Uint8Array): { + readonly path: string; + readonly size: number; + readonly type: "file" | "directory"; +} { + const stored = Number.parseInt(text(block, 148, 8).trim(), 8); + let sum = 0; + for (let index = 0; index < BLOCK; index++) sum += index >= 148 && index < 156 ? 0x20 : block[index]!; + if (stored !== sum) throw new Error("tar: a header's checksum does not hold"); + const flag = String.fromCharCode(block[156]!); + const type = flag === "0" || flag === "\u0000" ? "file" : flag === "5" ? "directory" : undefined; + if (type === undefined) throw new Error(`tar: an entry of type ${JSON.stringify(flag)} is refused (files only)`); + const prefix = text(block, 345, 155); + const name = text(block, 0, 100); + const path = (prefix === "" ? name : `${prefix}/${name}`).replace(/\/$/, ""); + checkPath(path); + const size = Number.parseInt(text(block, 124, 12).trim() || "0", 8); + if (!Number.isSafeInteger(size) || size < 0) throw new Error(`tar: ${path} has no valid size`); + return { path, size: type === "directory" ? 0 : size, type }; +} + +/** The NUL-terminated text of a header field. */ +function text(block: Uint8Array, offset: number, length: number): string { + const bytes = block.subarray(offset, offset + length); + const end = bytes.indexOf(0); + return new TextDecoder().decode(end < 0 ? bytes : bytes.subarray(0, end)); +} + +/** Reads exactly as many bytes as asked from a stream of chunks (fewer only at its end). */ +class ByteReader { + readonly #source: AsyncIterator; + #buffer = new Uint8Array(); + + constructor(source: AsyncIterable) { + this.#source = source[Symbol.asyncIterator](); + } + + /** Up to `length` bytes: `length` unless the stream ends first. */ + async read(length: number): Promise { + while (this.#buffer.length < length) { + const next = await this.#source.next(); + if (next.done === true) break; + const joined = new Uint8Array(this.#buffer.length + next.value.length); + joined.set(this.#buffer); + joined.set(next.value, this.#buffer.length); + this.#buffer = joined; + } + const taken = this.#buffer.subarray(0, Math.min(length, this.#buffer.length)); + this.#buffer = this.#buffer.subarray(taken.length); + return taken; + } + + /** Skips `length` bytes. */ + async skip(length: number): Promise { + let left = length; + while (left > 0) { + const skipped = await this.read(Math.min(left, 64 * 1024)); + if (skipped.length === 0) throw new Error("tar: the archive ends early"); + left -= skipped.length; + } + } +} diff --git a/packages/chord/src/next-node/app/uploads.ts b/packages/chord/src/next-node/app/uploads.ts new file mode 100644 index 000000000..6ce14e0b0 --- /dev/null +++ b/packages/chord/src/next-node/app/uploads.ts @@ -0,0 +1,172 @@ +// Chunked uploads of one `Files` object (slice 13 §4, design §23 "Chunked transfers"): each upload one file handle in +// the run's directory, written at positions. An append is accepted at the offset the accepted appends reached, so a +// writer may send several without waiting (the call window allows it): each write lands at its own position whatever +// order the thread pool completes them in. `uploaded`, `finish` and expiry wait for an upload's writes in flight; a +// failed write ends its upload. + +import type { FileHandle } from "node:fs/promises"; +import { mkdir, open, rename, rm } from "node:fs/promises"; +import { join } from "node:path"; +import { ChordError } from "../../next/data/index.ts"; +import { invalid } from "../../next/data/internal.ts"; +import { newToken } from "./secrets.ts"; + +/** An upload under way: its file, the bytes accepted, its writes in flight, its first failed write, its end. */ +type Upload = { + readonly name: string; + readonly mime: string; + readonly handle: FileHandle; + size: number; + touched: number; + readonly writes: Set>; + failed: boolean; + ended: Promise | undefined; +}; + +/** A finished upload: its name, type and size, its bytes moved to the path `finish` was given. */ +export type Finished = { readonly name: string; readonly mime: string; readonly size: number }; + +/** + * The uploads under way of one `Files` object, in the directory `run` (this object's own, made on first use), at most + * `max` at once: each holds a file handle until it ends, so a caller beginning uploads without end would exhaust the + * process's descriptors. + */ +export class Uploads { + readonly #run: string; + readonly #max: number; + readonly #uploads = new Map(); + + constructor(run: string, max: number) { + this.#run = run; + this.#max = max; + } + + /** The bytes accepted by every upload under way (they count against the host's cap). */ + get size(): number { + let size = 0; + for (const upload of this.#uploads.values()) size += upload.size; + return size; + } + + /** A new upload of `name` and `mime`: its id; `unavailable` (`busy`) while `max` are under way. */ + async begin(name: string, mime: string): Promise { + if (this.#uploads.size >= this.#max) { + throw new ChordError({ + code: "unavailable", + reason: "busy", + message: `files: ${this.#max} uploads are under way; finish one, or wait for an idle one to expire`, + }); + } + const id = newToken(12); + await mkdir(this.#run, { recursive: true, mode: 0o700 }); + const handle = await open(join(this.#run, id), "wx", 0o600); + const upload: Upload = { + name, + mime, + handle, + size: 0, + touched: Date.now(), + writes: new Set(), + failed: false, + ended: undefined, + }; + this.#uploads.set(id, upload); + return id; + } + + /** + * Writes `bytes` at `offset` of upload `id`, which must be where the accepted appends reached (else `invalid`, its + * details the expected `{ offset }`); `room(adding, fileSize)` refuses what the caps do not allow, before anything + * is accepted. A failed write ends the upload. + */ + async append( + id: string, + offset: number, + bytes: Uint8Array, + room: (adding: number, fileSize: number) => void, + ): Promise { + const upload = this.#underway(id); + if (offset !== upload.size) { + throw invalid("args", `files: upload ${id} is at offset ${upload.size}, not ${offset}`, { + offset: upload.size, + }); + } + room(bytes.length, upload.size + bytes.length); + // Accepted now, with no await before the write starts: the next append may follow at once, its write going to + // its own position. + upload.size += bytes.length; + upload.touched = Date.now(); + const write = upload.handle.write(bytes, 0, bytes.length, offset); + upload.writes.add(write); + try { + await write; + } catch (error) { + upload.writes.delete(write); + if (!upload.failed) { + upload.failed = true; + this.#uploads.delete(id); + await this.#end(upload); + await rm(join(this.#run, id), { force: true }); + } + throw error; + } + upload.writes.delete(write); + } + + /** The bytes of upload `id` written so far, once its writes in flight landed: where a writer resumes. */ + async uploaded(id: string): Promise { + const upload = this.#underway(id); + await Promise.allSettled([...upload.writes]); + if (upload.failed) throw failedUpload(id); + return upload.size; + } + + /** Ends upload `id` once its writes landed and moves its file to `path`; refused when a write of it failed. */ + async finish(id: string, path: string): Promise { + const upload = this.#underway(id); + // Taken out first: an append arriving now finds no upload. + this.#uploads.delete(id); + await this.#end(upload); + if (upload.failed) throw failedUpload(id); + await rename(join(this.#run, id), path); + return { name: upload.name, mime: upload.mime, size: upload.size }; + } + + /** Drops the uploads unwritten for `idleMs`, once their writes landed. */ + async expire(idleMs: number): Promise { + const now = Date.now(); + for (const [id, upload] of this.#uploads) { + if (now - upload.touched >= idleMs) await this.#discard(id, upload); + } + } + + /** Drops every upload under way (their object ended). */ + async close(): Promise { + for (const [id, upload] of this.#uploads) await this.#discard(id, upload); + } + + /** The upload `id` under way; else `invalid`. */ + #underway(id: string): Upload { + const upload = this.#uploads.get(id); + if (upload === undefined) throw invalid("args", `files: no upload ${id} (finished, failed, expired or unknown)`); + return upload; + } + + /** Takes upload `id` out, closes its file once its writes landed and removes it. */ + async #discard(id: string, upload: Upload): Promise { + this.#uploads.delete(id); + await this.#end(upload); + await rm(join(this.#run, id), { force: true }); + } + + /** Closes `upload`'s file once its writes in flight landed; once, whoever ends it first. */ + #end(upload: Upload): Promise { + upload.ended ??= Promise.allSettled([...upload.writes]).then(() => upload.handle.close()); + return upload.ended; + } +} + +/** The refusal of an upload one of whose writes failed. */ +function failedUpload(id: string): Error { + return invalid("args", `files: upload ${id} failed to write and was dropped`); +} diff --git a/packages/chord/src/next-node/app/users.ts b/packages/chord/src/next-node/app/users.ts new file mode 100644 index 000000000..2461ad407 --- /dev/null +++ b/packages/chord/src/next-node/app/users.ts @@ -0,0 +1,68 @@ +// The users plugin (slice 13 §3, decision "The app kit, point 2: principals and users"): accounts keyed by the user's +// id with a profile the app describes (a schema it passes in: a Slack-like profile or just a name), kept in the +// server's store. User data never rides in the principal: a principal names the user, `Users` holds the rest. +// An exported facet, not a plugin directory: it takes the app's schema as an argument (settings are arguments), and a +// plugin directory could not import it (plugins import only the app's API). The app loads it in its root by +// re-exporting it through its API (`export const users = defineUsers(Profile)` in `/v3/server`, a one-line plugin). + +import type { JsonValue } from "../../json.ts"; +import { arr, local, map, method, nul, obj, type Schema, str, union } from "../../next/contracts/index.ts"; +import type { Principal } from "../../next/data/index.ts"; +import { invalid } from "../../next/data/internal.ts"; +import { defineState } from "../../next/documents/index.ts"; +import { defineFacet, type Facet } from "../../next/host/index.ts"; + +/** The users plugin for profiles of schema `profile`: its contract and the facet serving it on the server. */ +export function defineUsers(profile: Schema) { + const Users = local("chord.app.users", { + version: 1, + params: {}, + members: { + /** Every user: id and profile. */ + list: arr(obj({ id: str(), profile })), + /** The profile of user `id`, or `null`. */ + get: method(str(), union([profile, nul()])), + /** Sets the profile of user `id` (a new user: an account made). */ + update: method(obj({ id: str(), profile })), + /** Removes user `id`'s account. */ + remove: method(str()), + }, + }); + // Stored as JSON: the document's types cannot follow a generic profile, the contract's members do. + const stored = profile as Schema; + const Accounts = defineState("chord.app.users", { + schema: obj({ users: map(stored) }), + version: 1, + initial: () => ({ users: {} }), + persist: true, + checkpointWhen: (_value, _ops, { deltasSinceBase }) => deltasSinceBase >= 50, + }); + const facet: Facet

    = defineFacet

    ((host) => + host.serve(Users, (_params, self) => { + const accounts = self.state(Accounts); + return { + list: self.derive([accounts], ([value]) => + Object.entries(value.users).map(([id, each]) => ({ id, profile: each as T })), + ), + // A trust boundary: `id` is the caller's (`Object.hasOwn`: no inherited member is a user). Written by + // `update` from a `T`. + get: async (id) => (Object.hasOwn(accounts.value.users, id) ? (accounts.value.users[id] as T) : null), + update: async ({ id, profile: next }, ctx) => { + // As a key it would set the record's prototype, not add a user. + if (id === "__proto__") throw invalid("args", "users: __proto__ is no user id"); + await accounts.change((draft) => { + draft.users[id] = next as JsonValue; + }, ctx); + return null; + }, + remove: async (id, ctx) => { + await accounts.change((draft) => { + delete draft.users[id]; + }, ctx); + return null; + }, + }; + }), + ); + return { Users, facet }; +} diff --git a/packages/chord/src/next-node/app/worker.ts b/packages/chord/src/next-node/app/worker.ts new file mode 100644 index 000000000..edb398434 --- /dev/null +++ b/packages/chord/src/next-node/app/worker.ts @@ -0,0 +1,72 @@ +// `startWorker` (slice 13 §1): a worker of an app built with the kit, in its entry module. `workerMain` plus the kit's +// part: the worker's directory under its kind and address with its store, the host at the worker's address acting as +// the principal the server gave it, connected to the server first, its plugins loaded with the kit's worker facets +// (`WorkerInfo`, `Files`). + +import { join } from "node:path"; +import type { JsonValue } from "../../json.ts"; +import type { Principal } from "../../next/data/index.ts"; +import { createHost } from "../../next/host/index.ts"; +import { openStore, sqliteBackend } from "../../next/store/index.ts"; +import { openNodeSqliteDatabase } from "../../sqlite/node.ts"; +import type { Watchable } from "../../watch.ts"; +import { workerMain } from "../workers/index.ts"; +import type { AppApi, AppKind, KitStart } from "./app.ts"; +import { filesFacet } from "./files.ts"; +import { workerInfoFacet } from "./info.ts"; +import { privateDirectory, privateStore } from "./layout.ts"; + +/** What `startWorker` takes: the app's API and kinds as the server's `startApp` has them, this process's modules. */ +export type WorkerOptions = { + readonly api: AppApi; + readonly kinds: Readonly>; + /** Extra keep-alive counts of the app (a record of numbers, all zero when idle). */ + readonly busy?: Watchable>>; + /** The worker host's errors (`HostError`) and a failed start's. */ + readonly onError: (error: unknown) => void; +}; + +/** Runs this process as a worker of the app: once the start message arrived, its host is up and its plugins loaded. */ +export function startWorker

    (options: WorkerOptions): void { + // The worker's facet kind comes with the start message: every kind's packages are offered, its own are imported. + const packages = Object.assign({}, ...Object.values(options.kinds).map((kind) => kind.packages ?? {})); + workerMain( + async ({ address, start, parent, plugins, onClose }, context) => { + // The server's `startApp` sends `{ kit, app }`. + const { kit, app } = start as { readonly kit: KitStart; readonly app: JsonValue }; + await privateDirectory(kit.dir); + await privateStore(join(kit.dir, "store.sqlite")); + const store = await openStore( + sqliteBackend(await openNodeSqliteDatabase(join(kit.dir, "store.sqlite"))), + context, + ); + onClose((ctx) => store.close(ctx)); + // Principals are the app's: the server made these with its `principal` options. + const host = createHost

    ({ + address, + principal: kit.principal as unknown as P, + store, + onError: options.onError, + }); + onClose((ctx) => host.close(ctx)); + // First: the server sees the worker started, then the load runs. + host.connect(parent, { principal: kit.server as unknown as P, trusted: true }, context); + const info = { + address: { kind: address.kind, key: address.key, params: address.params as JsonValue }, + start: app, + }; + const core = [ + { name: "chord.app.worker-info", facet: workerInfoFacet

    (info) }, + { name: "chord.app.files", facet: filesFacet

    (join(kit.dir, "files"), kit.files, kit.uploads) }, + ]; + await plugins.load(host, { kind: kit.facet, core }, context); + return options.busy === undefined ? {} : { busy: options.busy }; + }, + { + hostPackages: { ...options.api.modules, ...packages }, + // Worker errors are public (decision "worker errors in public"): a fixed message, never the error's own. + redact: () => ({ message: "worker failed" }), + onError: options.onError, + }, + ); +} diff --git a/packages/chord/src/next-node/http/README.md b/packages/chord/src/next-node/http/README.md index 266ba00d0..8d482c464 100644 --- a/packages/chord/src/next-node/http/README.md +++ b/packages/chord/src/next-node/http/README.md @@ -35,12 +35,16 @@ await server.close(ctx); // a plugin's server facet self.contribute(Http, { routes: { - "GET /notes/:id": async (request, ctx) => ({ kind: "json", value: … }), + "GET /notes/:id": async (request, ctx) => ({ kind: "json", value: … }), // needs a principal (else 401) + "GET /status": { public: true, handler: async () => ({ kind: "json", value: "ok" }) }, "GET /": { page: true, handler: async (request) => ({ kind: "html", body: `` }; + "GET /page": { + public: true, + handler: async (request) => { + nonce = request.nonce; + return { kind: "html", body: `` }; + }, }, }, framers: ["https://shell.example.com"], @@ -273,16 +319,22 @@ describe("routes and wraps failing", () => { const server = await served({ "@app/a": contributing({ routes: { - "GET /throws": async () => { - throw new Error("secret"); + "GET /throws": { + public: true, + handler: async () => { + throw new Error("secret"); + }, }, - "GET /failure": async () => { - throw new HttpFailure(422, "no"); + "GET /failure": { + public: true, + handler: async () => { + throw new HttpFailure(422, "no"); + }, }, }, }), "@app/b": contributing({ - routes: { "GET /wrapped": async () => ({ kind: "json", value: 1 }) }, + routes: { "GET /wrapped": { public: true, handler: async () => ({ kind: "json", value: 1 }) } }, wraps: [ { name: "broken", @@ -310,9 +362,12 @@ describe("routes and wraps failing", () => { const server = await served({ "@app/a": contributing({ routes: { - "GET /r": async (_request, ctx) => { - order.push(`route as ${ctx.principal?.kind ?? "anonymous"}`); - return { kind: "json", value: 1 }; + "GET /r": { + public: true, + handler: async (_request, ctx) => { + order.push(`route as ${ctx.principal?.kind ?? "anonymous"}`); + return { kind: "json", value: 1 }; + }, }, }, wraps: [ @@ -347,16 +402,21 @@ describe("reloads and close", () => { const gate = Promise.withResolvers(); const old = contributing({ routes: { - "GET /v": async () => { - await gate.promise; - return { kind: "html", body: "old" }; + "GET /v": { + public: true, + handler: async () => { + await gate.promise; + return { kind: "html", body: "old" }; + }, }, }, }); const server = await served({ "@app/v": old }); const running = send(server.port, "GET", "/v"); await new Promise((resolve) => setTimeout(resolve, 50)); - const next = contributing({ routes: { "GET /v": async () => ({ kind: "html", body: "new" }) } }); + const next = contributing({ + routes: { "GET /v": { public: true, handler: async () => ({ kind: "html", body: "new" }) } }, + }); await server.host.reload(server.facets({ "@app/v": next }), context); expect((await send(server.port, "GET", "/v")).body.toString()).toBe("new"); gate.resolve(); @@ -370,13 +430,19 @@ describe("reloads and close", () => { { "@app/slow": contributing({ routes: { - "GET /quick": async () => { - await new Promise((resolve) => setTimeout(resolve, 30)); - return { kind: "html", body: "done" }; + "GET /quick": { + public: true, + handler: async () => { + await new Promise((resolve) => setTimeout(resolve, 30)); + return { kind: "html", body: "done" }; + }, }, - "GET /stuck": async (_request, ctx: Ctx) => { - sealed = ctx.abortSignal; - return new Promise(() => {}); + "GET /stuck": { + public: true, + handler: async (_request, ctx: Ctx) => { + sealed = ctx.abortSignal; + return new Promise(() => {}); + }, }, }, }), @@ -405,9 +471,12 @@ describe("reloads and close", () => { { "@app/r": contributing({ routes: { - "GET /": async () => { - calls++; - return { kind: "json", value: 1 }; + "GET /": { + public: true, + handler: async () => { + calls++; + return { kind: "json", value: 1 }; + }, }, }, }), @@ -446,7 +515,9 @@ describe("cancellation", () => { let later = 0; const server = await served( { - "@app/r": contributing({ routes: { "GET /": async () => ({ kind: "html", body: "ok" }) } }), + "@app/r": contributing({ + routes: { "GET /": { public: true, handler: async () => ({ kind: "html", body: "ok" }) } }, + }), "@app/z-later": defineFacet((_host, self) => self.contribute(Authenticators, () => { later++; @@ -476,7 +547,11 @@ describe("cancellation", () => { it("asks no authenticator for a request aborted before it arrived", async () => { let calls = 0; const server = await served( - { "@app/r": contributing({ routes: { "GET /": async () => ({ kind: "html", body: "ok" }) } }) }, + { + "@app/r": contributing({ + routes: { "GET /": { public: true, handler: async () => ({ kind: "html", body: "ok" }) } }, + }), + }, { config: { listen: undefined }, authenticate: () => { @@ -503,18 +578,21 @@ describe("cancellation", () => { { "@app/b": contributing({ routes: { - "GET /": async (_request, ctx) => { - entered.resolve(); - if (when === "inside the route") abort.abort(new Error("client gone")); - bornSealed = ctx.abortSignal?.aborted; - return { - kind: "bytes", - mime: "application/octet-stream", - read: async () => { - reads++; - return new Uint8Array([1]); - }, - }; + "GET /": { + public: true, + handler: async (_request, ctx) => { + entered.resolve(); + if (when === "inside the route") abort.abort(new Error("client gone")); + bornSealed = ctx.abortSignal?.aborted; + return { + kind: "bytes", + mime: "application/octet-stream", + read: async () => { + reads++; + return new Uint8Array([1]); + }, + }; + }, }, }, }), @@ -539,11 +617,14 @@ describe("cancellation", () => { { "@app/b": contributing({ routes: { - "GET /": async () => ({ - kind: "bytes", - mime: "application/octet-stream", - read: async () => new Uint8Array([1]), - }), + "GET /": { + public: true, + handler: async () => ({ + kind: "bytes", + mime: "application/octet-stream", + read: async () => new Uint8Array([1]), + }), + }, }, }), }, @@ -587,10 +668,13 @@ describe("cancellation", () => { { "@app/wait": contributing({ routes: { - "GET /wait": async () => { - entered.resolve(); - await gate.promise; - return { kind: "html", body: "done" }; + "GET /wait": { + public: true, + handler: async () => { + entered.resolve(); + await gate.promise; + return { kind: "html", body: "done" }; + }, }, }, }), @@ -612,7 +696,11 @@ describe("cancellation", () => { const entered = Promise.withResolvers(); const gate = Promise.withResolvers(); const server = await served( - { "@app/r": contributing({ routes: { "GET /": async () => ({ kind: "html", body: "ok" }) } }) }, + { + "@app/r": contributing({ + routes: { "GET /": { public: true, handler: async () => ({ kind: "html", body: "ok" }) } }, + }), + }, { config: { listen: undefined, closeTimeout: 10 }, authenticate: async () => { @@ -640,15 +728,18 @@ describe("cancellation", () => { { "@app/bytes": contributing({ routes: { - "GET /stream": async () => ({ - kind: "bytes", - mime: "application/octet-stream", - read: async (_offset, _length, ctx) => { - reads++; - readAfterSeal ||= ctx.abortSignal?.aborted ?? false; - return new Uint8Array([reads]); - }, - }), + "GET /stream": { + public: true, + handler: async () => ({ + kind: "bytes", + mime: "application/octet-stream", + read: async (_offset, _length, ctx) => { + reads++; + readAfterSeal ||= ctx.abortSignal?.aborted ?? false; + return new Uint8Array([reads]); + }, + }), + }, }, }), }, @@ -672,10 +763,13 @@ describe("cancellation", () => { { "@app/body": contributing({ routes: { - "POST /body": async (request) => { - const value = await request.json(); - writes++; - return { kind: "json", value }; + "POST /body": { + public: true, + handler: async (request) => { + const value = await request.json(); + writes++; + return { kind: "json", value }; + }, }, }, }), @@ -700,7 +794,11 @@ describe("answers", () => { // Sol's recheck N8: `Response` threw on a body for these statuses. it.each([204, 205, 304])("answers %s without a body", async (status) => { const server = await served( - { "@app/s": contributing({ routes: { "GET /": async () => ({ kind: "json", value: null, status }) } }) }, + { + "@app/s": contributing({ + routes: { "GET /": { public: true, handler: async () => ({ kind: "json", value: null, status }) } }, + }), + }, { config: { listen: undefined } }, ); const response = await server.http.fetch(loopback("/"), LOOPBACK); @@ -714,8 +812,11 @@ describe("answers", () => { { "@app/r": contributing({ routes: { - "GET /": async () => { - throw new ChordError({ code: "forbidden", message: "no" }); + "GET /": { + public: true, + handler: async () => { + throw new ChordError({ code: "forbidden", message: "no" }); + }, }, }, }), @@ -731,7 +832,12 @@ describe("answers", () => { it("sends several Set-Cookie headers", async () => { const server = await served({ "@app/r": contributing({ - routes: { "GET /": async () => ({ kind: "json", value: 1, headers: { "Set-Cookie": ["a=1", "b=2"] } }) }, + routes: { + "GET /": { + public: true, + handler: async () => ({ kind: "json", value: 1, headers: { "Set-Cookie": ["a=1", "b=2"] } }), + }, + }, }), }); expect((await send(server.port, "GET", "/")).headers["set-cookie"]).toEqual(["a=1", "b=2"]); @@ -752,9 +858,16 @@ describe("GC", () => { { "@app/b": contributing({ routes: { - "GET /": async (_request, ctx) => { - refs.push(new WeakRef(ctx)); - return { kind: "bytes", mime: "application/octet-stream", read: async () => new Uint8Array([1]) }; + "GET /": { + public: true, + handler: async (_request, ctx) => { + refs.push(new WeakRef(ctx)); + return { + kind: "bytes", + mime: "application/octet-stream", + read: async () => new Uint8Array([1]), + }; + }, }, }, }), @@ -777,10 +890,17 @@ describe("GC", () => { { "@app/b": contributing({ routes: { - "GET /": async (_request, ctx) => { - refs.push(new WeakRef(ctx)); - aborts.at(-1)?.abort(new Error("client gone")); - return { kind: "bytes", mime: "application/octet-stream", read: async () => new Uint8Array([1]) }; + "GET /": { + public: true, + handler: async (_request, ctx) => { + refs.push(new WeakRef(ctx)); + aborts.at(-1)?.abort(new Error("client gone")); + return { + kind: "bytes", + mime: "application/octet-stream", + read: async () => new Uint8Array([1]), + }; + }, }, }, }), @@ -804,9 +924,12 @@ describe("GC", () => { const server = await served({ "@app/a": contributing({ routes: { - "GET /r": async (request, ctx) => { - refs.push(new WeakRef(ctx), new WeakRef(request)); - return { kind: "json", value: 1 }; + "GET /r": { + public: true, + handler: async (request, ctx) => { + refs.push(new WeakRef(ctx), new WeakRef(request)); + return { kind: "json", value: 1 }; + }, }, }, }), diff --git a/packages/chord/test/next-node/http/node.test.ts b/packages/chord/test/next-node/http/node.test.ts index 8349bd751..c8aaf0de6 100644 --- a/packages/chord/test/next-node/http/node.test.ts +++ b/packages/chord/test/next-node/http/node.test.ts @@ -27,8 +27,8 @@ async function raw(port: number, text: string): Promise<{ readonly status: strin /** A route answering its path and params, under `GET /who` and `GET /n/:name`. */ const echo = contributing({ routes: { - "GET /who": async (request) => ({ kind: "json", value: request.path }), - "GET /n/:name": async (request) => ({ kind: "json", value: request.params }), + "GET /who": { public: true, handler: async (request) => ({ kind: "json", value: request.path }) }, + "GET /n/:name": { public: true, handler: async (request) => ({ kind: "json", value: request.params }) }, }, }); @@ -118,9 +118,15 @@ describe("Web requests", () => { { "@app/r": contributing({ routes: { - "GET /plain": async (request) => ({ kind: "json", value: request.path }), - "GET /headers": async (request) => ({ kind: "json", value: request.headers.get("x-thing") }), - "POST /body": async (request) => ({ kind: "json", value: await request.json() }), + "GET /plain": { public: true, handler: async (request) => ({ kind: "json", value: request.path }) }, + "GET /headers": { + public: true, + handler: async (request) => ({ kind: "json", value: request.headers.get("x-thing") }), + }, + "POST /body": { + public: true, + handler: async (request) => ({ kind: "json", value: await request.json() }), + }, }, }), }, @@ -157,9 +163,12 @@ describe("the request's scope", () => { { "@app/large": contributing({ routes: { - "GET /large": async (_request, ctx) => { - signal = ctx.abortSignal; - return { kind: "html", body: "x".repeat(size) }; + "GET /large": { + public: true, + handler: async (_request, ctx) => { + signal = ctx.abortSignal; + return { kind: "html", body: "x".repeat(size) }; + }, }, }, }), @@ -195,9 +204,12 @@ describe("the request's scope", () => { const server = await served({ "@app/slow": contributing({ routes: { - "GET /slow": async (_request, ctx) => { - entered.resolve(ctx.abortSignal!); - return new Promise(() => {}); + "GET /slow": { + public: true, + handler: async (_request, ctx) => { + entered.resolve(ctx.abortSignal!); + return new Promise(() => {}); + }, }, }, }), diff --git a/packages/chord/test/next-node/http/request.test.ts b/packages/chord/test/next-node/http/request.test.ts index a48a2429e..9bbfb491a 100644 --- a/packages/chord/test/next-node/http/request.test.ts +++ b/packages/chord/test/next-node/http/request.test.ts @@ -12,9 +12,12 @@ describe("the request a route gets", () => { const server = await served({ "@app/a": contributing({ routes: { - "GET /notes/:id": async (request) => { - seen = request; - return { kind: "json", value: null }; + "GET /notes/:id": { + public: true, + handler: async (request) => { + seen = request; + return { kind: "json", value: null }; + }, }, }, }), @@ -39,11 +42,17 @@ describe("bodies", () => { served({ "@app/a": contributing({ routes: { - "POST /json": async (request) => ({ kind: "json", value: await request.json(16) }), - "POST /body": async (request) => { - let bytes = 0; - for await (const chunk of request.body(16)) bytes += chunk.byteLength; - return { kind: "json", value: bytes }; + "POST /json": { + public: true, + handler: async (request) => ({ kind: "json", value: await request.json(16) }), + }, + "POST /body": { + public: true, + handler: async (request) => { + let bytes = 0; + for await (const chunk of request.body(16)) bytes += chunk.byteLength; + return { kind: "json", value: bytes }; + }, }, }, }), @@ -91,7 +100,9 @@ describe("bodies", () => { it("drains a small body nobody read and keeps the connection; a large one closes it", async () => { const server = await served({ - "@app/a": contributing({ routes: { "POST /ignore": async () => ({ kind: "json", value: 1 }) } }), + "@app/a": contributing({ + routes: { "POST /ignore": { public: true, handler: async () => ({ kind: "json", value: 1 }) } }, + }), }); const agent = new Agent({ keepAlive: true, maxSockets: 1 }); const post = (body: string) => diff --git a/packages/chord/test/next-node/http/response.test.ts b/packages/chord/test/next-node/http/response.test.ts index 3c953ba85..8913f459b 100644 --- a/packages/chord/test/next-node/http/response.test.ts +++ b/packages/chord/test/next-node/http/response.test.ts @@ -11,7 +11,7 @@ import { contributing, send, served } from "./support.ts"; /** A server answering `GET /r` with `response`. */ async function answering(response: HttpResponse) { - return served({ "@app/r": contributing({ routes: { "GET /r": async () => response } }) }); + return served({ "@app/r": contributing({ routes: { "GET /r": { public: true, handler: async () => response } } }) }); } /** A `bytes` response over `data` that records its reads. */ @@ -64,8 +64,11 @@ describe("descriptions", () => { const server = await served({ "@app/r": contributing({ routes: { - "GET /header": async () => ({ kind: "json", value: 1, headers: { "X-Split": "a\r\nSet-Cookie: x=1" } }), - "GET /bigint": async () => ({ kind: "json", value: { n: 1n } as never }), + "GET /header": { + public: true, + handler: async () => ({ kind: "json", value: 1, headers: { "X-Split": "a\r\nSet-Cookie: x=1" } }), + }, + "GET /bigint": { public: true, handler: async () => ({ kind: "json", value: { n: 1n } as never }) }, }, }), }); @@ -185,21 +188,27 @@ describe("bytes", () => { const server = await served({ "@app/r": contributing({ routes: { - "GET /short": async () => ({ - kind: "bytes", - mime: "text/plain", - size: 10, - read: async () => Buffer.from("abc"), - }), + "GET /short": { + public: true, + handler: async () => ({ + kind: "bytes", + mime: "text/plain", + size: 10, + read: async () => Buffer.from("abc"), + }), + }, // Astra's build review: a throwing read hung the response, unreported. - "GET /throws": async () => ({ - kind: "bytes", - mime: "text/plain", - size: 10, - read: async () => { - throw new Error("read failed"); - }, - }), + "GET /throws": { + public: true, + handler: async () => ({ + kind: "bytes", + mime: "text/plain", + size: 10, + read: async () => { + throw new Error("read failed"); + }, + }), + }, }, }), }); diff --git a/packages/chord/test/next-node/http/router.test.ts b/packages/chord/test/next-node/http/router.test.ts index 9854820c6..8bcff127e 100644 --- a/packages/chord/test/next-node/http/router.test.ts +++ b/packages/chord/test/next-node/http/router.test.ts @@ -104,18 +104,25 @@ describe("patterns and precedence (D8)", () => { expect(matched(router, "GET", "/v/b")).toEqual({ pattern: "GET /:name/b", params: { name: "v" } }); }); - it("takes a route with options: page and crossSite", () => { + it("takes a route with options: page, crossSite and public (a cross-site route is public: always anonymous)", () => { const { router } = compose([ "@app/a", { routes: { "GET /p": { handler: route("p"), page: true }, "POST /hook": { handler: route("h"), crossSite: true }, + "GET /open": { handler: route("o"), public: true }, + "GET /closed": route("c"), }, + sockets: { "/s": { handler: () => {}, public: true }, "/t": () => {} }, }, ]); - expect(router.route("GET", "/p")).toMatchObject({ page: true, crossSite: false }); - expect(router.route("POST", "/hook")).toMatchObject({ page: false, crossSite: true }); + expect(router.route("GET", "/p")).toMatchObject({ page: true, crossSite: false, public: false }); + expect(router.route("POST", "/hook")).toMatchObject({ page: false, crossSite: true, public: true }); + expect(router.route("GET", "/open")).toMatchObject({ public: true }); + expect(router.route("GET", "/closed")).toMatchObject({ public: false }); + expect(router.socket("/s")).toMatchObject({ public: true }); + expect(router.socket("/t")).toMatchObject({ public: false }); expect(router.route("GET", "/notes")).toBeUndefined(); }); }); diff --git a/packages/chord/test/next-node/http/upgrades.test.ts b/packages/chord/test/next-node/http/upgrades.test.ts index f4de24124..616099779 100644 --- a/packages/chord/test/next-node/http/upgrades.test.ts +++ b/packages/chord/test/next-node/http/upgrades.test.ts @@ -75,7 +75,7 @@ describe("the links endpoint", () => { await server.close(); }); - it("refuses a socket without a principal, from another Origin or Host, foreign, or whose links.options throws (403)", async () => { + it("refuses a socket without a principal (401), from another Origin or Host, foreign, or whose links.options throws (403)", async () => { const server = await served( { "@app/hosts": contributing({ hosts: ["a.example.com"] }) }, { @@ -92,8 +92,9 @@ describe("the links endpoint", () => { ); const { port } = server; const owner = { "x-principal": "owner" }; + // Nobody: the links endpoint needs a principal (deny by default, slice 13). + await expect(socketTo(port, "/chord")).rejects.toThrow("refused 401"); const refusals = [ - socketTo(port, "/chord"), socketTo(port, "/chord", { ...owner, origin: "http://evil.com" }), socketTo(port, "/chord", { ...owner, origin: "null" }), socketTo(port, "/chord", { ...owner, origin: `HTTP://127.0.0.1:${port}` }), @@ -139,7 +140,7 @@ describe("the links endpoint", () => { describe("plugin sockets", () => { it("hands the socket and its request to the handler in a scope with its principal; no principal is refused", async () => { const server = await served({ "@app/echo": echo }); - await expect(socketTo(server.port, "/echo/lobby")).rejects.toThrow("refused 403"); + await expect(socketTo(server.port, "/echo/lobby")).rejects.toThrow("refused 401"); await expect(socketTo(server.port, "/nope", { "x-principal": "owner" })).rejects.toThrow("refused 404"); const socket = await socketTo(server.port, "/echo/lobby", { "x-principal": "owner" }); expect(await nextMessage(socket)).toBe("lobby as owner"); @@ -149,6 +150,22 @@ describe("plugin sockets", () => { await server.close(); }); + it("accepts a public socket without a principal, its scope anonymous (deny by default, slice 13 §2)", async () => { + const lobby = contributing({ + sockets: { + "/lobby": { + public: true, + handler: (socket, _request, ctx) => socket.send(`as ${ctx.principal?.kind ?? "nobody"}`), + }, + }, + }); + const server = await served({ "@app/lobby": lobby }); + const socket = await socketTo(server.port, "/lobby"); + expect(await nextMessage(socket)).toBe("as nobody"); + socket.close(); + await server.close(); + }); + it("an oversized frame closes only that socket (1009), never the server", async () => { const server = await served({ "@app/echo": echo }); const [big, other] = await Promise.all([ diff --git a/packages/chord/test/next/examples/40-http/plugins/notes/server.ts b/packages/chord/test/next/examples/40-http/plugins/notes/server.ts index 48e6b9388..d784376e0 100644 --- a/packages/chord/test/next/examples/40-http/plugins/notes/server.ts +++ b/packages/chord/test/next/examples/40-http/plugins/notes/server.ts @@ -10,6 +10,7 @@ export default defineFacet((_host, self) => routes: { // A page: a link from another site may open it. Its inline script carries the server's nonce (the CSP). "GET /": { + public: true, page: true, handler: async (request) => ({ kind: "html", @@ -17,13 +18,19 @@ export default defineFacet((_host, self) => }), }, // `:name` is one path segment, decoded. - "GET /notes/:name": async (request) => ({ kind: "json", value: { note: request.params.name } }), + "GET /notes/:name": { + public: true, + handler: async (request) => ({ kind: "json", value: { note: request.params.name } }), + }, // A file of this plugin, saved under its name, with ranges. - "GET /notes/:name/download": async (request) => - fileResponse(pluginFile(`files/${request.params.name}.md`), { - download: `${request.params.name}.md`, - ranges: true, - }), + "GET /notes/:name/download": { + public: true, + handler: async (request) => + fileResponse(pluginFile(`files/${request.params.name}.md`), { + download: `${request.params.name}.md`, + ranges: true, + }), + }, }, }), ); diff --git a/packages/chord/test/next/examples/44-http-wraps/plugins/notes/server.ts b/packages/chord/test/next/examples/44-http-wraps/plugins/notes/server.ts index 0c2a83103..db5441651 100644 --- a/packages/chord/test/next/examples/44-http-wraps/plugins/notes/server.ts +++ b/packages/chord/test/next/examples/44-http-wraps/plugins/notes/server.ts @@ -5,7 +5,9 @@ import { Http } from "@earendil-works/chord/next/http"; export default defineFacet((_host, self) => self.contribute(Http, { - routes: { "GET /notes": async () => ({ kind: "json", value: ["buy milk", "call mom"] }) }, + routes: { + "GET /notes": { public: true, handler: async () => ({ kind: "json", value: ["buy milk", "call mom"] }) }, + }, wraps: [ { name: "cache", diff --git a/packages/chord/test/next/examples/46-allowed-hosts/plugins/tunnel/server.ts b/packages/chord/test/next/examples/46-allowed-hosts/plugins/tunnel/server.ts index 2e1239c10..da5783e73 100644 --- a/packages/chord/test/next/examples/46-allowed-hosts/plugins/tunnel/server.ts +++ b/packages/chord/test/next/examples/46-allowed-hosts/plugins/tunnel/server.ts @@ -12,7 +12,12 @@ export const Tunnel = local("example.tunnel", { export default defineFacet((host, self) => { self.contribute(Http, { - routes: { "GET /hello": async (request) => ({ kind: "json", value: { origin: request.origin } }) }, + routes: { + "GET /hello": { + public: true, + handler: async (request) => ({ kind: "json", value: { origin: request.origin } }), + }, + }, }); host.serve(Tunnel, (_params, tunnel) => { const allowed = tunnel.writable({ hosts: [] }); diff --git a/packages/chord/test/next/examples/47-http-own-server/plugins/notes/server.ts b/packages/chord/test/next/examples/47-http-own-server/plugins/notes/server.ts index 2b7a0abd9..f08442bd0 100644 --- a/packages/chord/test/next/examples/47-http-own-server/plugins/notes/server.ts +++ b/packages/chord/test/next/examples/47-http-own-server/plugins/notes/server.ts @@ -6,7 +6,10 @@ import { Http } from "@earendil-works/chord/next/http"; export default defineFacet((_host, self) => self.contribute(Http, { routes: { - "GET /notes": async (request) => ({ kind: "json", value: { notes: ["buy milk"], via: request.origin } }), + "GET /notes": { + public: true, + handler: async (request) => ({ kind: "json", value: { notes: ["buy milk"], via: request.origin } }), + }, }, }), ); diff --git a/packages/chord/test/next/examples/48-http-baseline/main.ts b/packages/chord/test/next/examples/48-http-baseline/main.ts index 764fcce21..ee1483cd7 100644 --- a/packages/chord/test/next/examples/48-http-baseline/main.ts +++ b/packages/chord/test/next/examples/48-http-baseline/main.ts @@ -1,7 +1,8 @@ // The security baseline (slice 10): on by default, the app decides only who someone is. Here the app signs people in // with a signed cookie (`cookie`, `readCookie`) and the notes plugin contributes a page, routes and a webhook. Requests -// another site's page makes in a browser (Fetch Metadata says so) are never authenticated, a foreign `Origin` is -// refused, but a link to a page still opens it signed in, and a cross-site webhook runs anonymous. +// another site's page makes in a browser (Fetch Metadata says so) are never authenticated (so a route not marked +// public answers them 401: deny by default, slice 13), a foreign `Origin` is refused, but a link to a page still opens +// it signed in, and a cross-site webhook runs anonymous. // Run from packages/chord: // node --conditions=source --experimental-strip-types test/next/examples/48-http-baseline/main.ts import { request } from "node:http"; @@ -21,9 +22,12 @@ const http = chordHttp.createHttp(server, { listen: { host: "127.0.0.1", port: 0 const app = defineFacet((_host, self) => { self.contribute(chordHttp.Http, { routes: { - "POST /sign-in": async (request) => { - const set = chordHttp.cookie("user", String(await request.json()), { secret, request, sameSite: "Lax" }); - return { kind: "json", value: "signed in", headers: { "Set-Cookie": set } }; + "POST /sign-in": { + public: true, + handler: async (request) => { + const set = chordHttp.cookie("user", String(await request.json()), { secret, request, sameSite: "Lax" }); + return { kind: "json", value: "signed in", headers: { "Set-Cookie": set } }; + }, }, }, }); diff --git a/packages/chord/test/next/examples/48-http-baseline/plugins/notes/server.ts b/packages/chord/test/next/examples/48-http-baseline/plugins/notes/server.ts index 0980b62f1..52b8c142f 100644 --- a/packages/chord/test/next/examples/48-http-baseline/plugins/notes/server.ts +++ b/packages/chord/test/next/examples/48-http-baseline/plugins/notes/server.ts @@ -1,5 +1,6 @@ // The notes plugin's server facet: the notes page (a link from another site may open it), the notes as JSON, adding -// a note, and a webhook another site calls (cross-site: it always runs anonymous). +// a note, and a webhook another site calls (cross-site: it always runs anonymous, so it is public). Nothing else is +// public: a request nobody signed in gets 401 before this code runs (deny by default). import { defineFacet } from "@earendil-works/chord/next"; import { Http } from "@earendil-works/chord/next/http"; @@ -14,8 +15,7 @@ export default defineFacet((_host, self) => handler: async (_request, ctx) => ({ kind: "html", body: `hi ${who(ctx.principal)}` }), }, "GET /notes": async (_request, ctx) => ({ kind: "json", value: { notes, as: who(ctx.principal) } }), - "POST /notes": async (request, ctx) => { - if (ctx.principal === undefined) return { kind: "error", status: 401 }; + "POST /notes": async (request) => { notes.push(String(await request.json())); return { kind: "json", value: notes }; }, diff --git a/packages/chord/test/next/examples/53-app/app/principals.ts b/packages/chord/test/next/examples/53-app/app/principals.ts new file mode 100644 index 000000000..cde89e19a --- /dev/null +++ b/packages/chord/test/next/examples/53-app/app/principals.ts @@ -0,0 +1,8 @@ +// The notes app's principals: its server and workers, the kit's owner, and the users its plugins may sign in. + +/** Who someone is in the notes app. */ +export type NotesPrincipal = + | { readonly kind: "server" } + | { readonly kind: "owner" } + | { readonly kind: "worker"; readonly project: string } + | { readonly kind: "user"; readonly id: string; readonly name: string }; diff --git a/packages/chord/test/next/examples/53-app/app/v1.ts b/packages/chord/test/next/examples/53-app/app/v1.ts new file mode 100644 index 000000000..45cb45764 --- /dev/null +++ b/packages/chord/test/next/examples/53-app/app/v1.ts @@ -0,0 +1,31 @@ +// The notes app's API, version 1, shared by every host kind (`@notes/app/v1`): what its plugins import from their +// host. The part of Chord the app supports, the kit's start-data objects and files, the app's own host kind (a +// project, one worker per project) and its registry of home-page sections. The API only declares the registry: its +// provider is the home plugin of the app's required root, not base code. Examples 53 to 55 run this app; a real app +// keeps it in `src/api/v1.ts`. + +import { defineRegistry, hostKind, str } from "../../../../../src/next/index.ts"; + +export { + arr, + constant, + defineFacet, + defineRegistry, + defineState, + defineTable, + int, + local, + method, + obj, + remote, + str, +} from "../../../../../src/next/index.ts"; +export { AppInfo, Files, WorkerInfo } from "../../../../../src/next-node/app/index.ts"; + +/** A project of the app: its own worker process, store and files. */ +export const Project = hostKind("project", { id: str() }); + +/** The home page's sections, one per contributing plugin (title and the path it links to). */ +export const Sections = defineRegistry<{ readonly title: string; readonly path: string }, string[]>("notes.sections", { + version: 1, +}); diff --git a/packages/chord/test/next/examples/53-app/app/v1/server.ts b/packages/chord/test/next/examples/53-app/app/v1/server.ts new file mode 100644 index 000000000..de7f5a42b --- /dev/null +++ b/packages/chord/test/next/examples/53-app/app/v1/server.ts @@ -0,0 +1,5 @@ +// The notes app's API v1 for server facets only (`@notes/app/v1/server`): the HTTP registry plugins contribute routes +// to, and the kit's sessions. + +export { Sessions } from "../../../../../../src/next-node/app/index.ts"; +export { Http } from "../../../../../../src/next-node/http/index.ts"; diff --git a/packages/chord/test/next/examples/53-app/client.ts b/packages/chord/test/next/examples/53-app/client.ts new file mode 100644 index 000000000..2c26b82d6 --- /dev/null +++ b/packages/chord/test/next/examples/53-app/client.ts @@ -0,0 +1,44 @@ +// A small HTTP client for examples 53 to 55, standing in for a browser and a command-line client: a request to the +// app on loopback, its status, `Set-Cookie` and JSON body. + +import { request } from "node:http"; + +/** An answer: status, the cookie it set (name=value), JSON body (or text). */ +export type Answer = { readonly status: number; readonly cookie: string | undefined; readonly body: unknown }; + +/** `method path` on the app at `port`, with `headers` and a JSON `body`. */ +export function call( + port: number, + method: string, + path: string, + headers: Readonly> = {}, + body?: unknown, +): Promise { + return new Promise((resolve, reject) => { + const sent = request( + { + host: "127.0.0.1", + port, + method, + path, + headers: { ...headers, ...(body === undefined ? {} : { "content-type": "application/json" }) }, + }, + (response) => { + const chunks: Buffer[] = []; + response.on("data", (chunk: Buffer) => chunks.push(chunk)); + response.on("end", () => { + const text = Buffer.concat(chunks).toString(); + const type = response.headers["content-type"] ?? ""; + const cookie = response.headers["set-cookie"]?.[0]?.split(";")[0]; + resolve({ + status: response.statusCode ?? 0, + cookie, + body: type.includes("json") ? JSON.parse(text) : text, + }); + }); + }, + ); + sent.on("error", reject); + sent.end(body === undefined ? undefined : JSON.stringify(body)); + }); +} diff --git a/packages/chord/test/next/examples/53-app/main.ts b/packages/chord/test/next/examples/53-app/main.ts new file mode 100644 index 000000000..f99fbd3ac --- /dev/null +++ b/packages/chord/test/next/examples/53-app/main.ts @@ -0,0 +1,56 @@ +// The app kit (slice 13): an app is its API (`app/`), its built-in plugins (`plugins/`: home and notes) and a start +// script (`start.ts`) calling `startApp`; its workers' entry (`worker.ts`) calls `startWorker`. The kit makes the data +// directory, the server's store and host, HTTP (deny by default: no session, 401), one worker per project with its own +// store and files, and the owner's sign-in: a one-time owner link for the browser, an owner token file for command-line +// clients on the same machine. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/53-app/main.ts +import { mkdtemp, readdir, readFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { BACKGROUND_CONTEXT as context } from "../../../../src/context/index.ts"; +import { call } from "./client.ts"; +import { startNotes } from "./start.ts"; + +const dataDir = join(await mkdtemp(join(tmpdir(), "notes-")), "data"); + +// 1. Start the app. The start script prints the owner link (single use, 10 minutes, a new one at each start). +const app = await startNotes(dataDir, context); +const link = new URL(app.ownerLink()); +const port = Number(link.port); +console.log("1. owner link:", `${link.origin}${link.pathname}#…`); +console.log(" data directory:", (await readdir(dataDir)).sort().join(", ")); + +// 2. Nobody is signed in: every plugin route answers 401 before the plugin runs. +console.log("2. anonymous GET /:", (await call(port, "GET", "/")).status); + +// 3. The owner opens the link: the page posts the token from the URL's fragment (never sent in a URL, so never in a +// log) and gets the session cookie. The link is used up. +const page = await call(port, "GET", link.pathname); +console.log( + "3. the link's page:", + page.status, + String(page.body).includes("location.hash") ? "(reads the fragment)" : "", +); +const signedIn = await call(port, "POST", link.pathname, {}, { token: link.hash.slice(1) }); +const browser = { cookie: signedIn.cookie ?? "" }; +console.log(" signed in:", signedIn.status, signedIn.body); +console.log(" the link again:", (await call(port, "POST", link.pathname, {}, { token: link.hash.slice(1) })).status); + +// 4. Signed in, the home plugin reads the kit's `AppInfo` and lists the sections of its registry (the notes plugin's); +// the notes plugin's routes reach the project's worker (it starts on the first request) with its store and the +// start data the app gave it. +console.log("4. GET /:", JSON.stringify((await call(port, "GET", "/", browser)).body)); +await call(port, "POST", "/notes/p1", browser, "milk, eggs"); +console.log(" GET /notes/p1:", JSON.stringify((await call(port, "GET", "/notes/p1", browser)).body)); +console.log(" worker directories:", (await readdir(join(dataDir, "workers", "project"))).join(", ")); + +// 5. A command-line client on the same machine reads the owner token file (mode 0600) and sends it as a bearer token. +const token = (await readFile(app.ownerTokenFile, "utf8")).trim(); +const cli = { authorization: `Bearer ${token}` }; +console.log("5. CLI GET /:", JSON.stringify((await call(port, "GET", "/", cli)).body)); + +// 6. Close: HTTP, workers, host, store, in that order. +await app.close(context); +await rm(join(dataDir, ".."), { recursive: true, force: true }); +console.log("6. closed"); diff --git a/packages/chord/test/next/examples/53-app/plugins/home/package.json b/packages/chord/test/next/examples/53-app/plugins/home/package.json new file mode 100644 index 000000000..c64b0510f --- /dev/null +++ b/packages/chord/test/next/examples/53-app/plugins/home/package.json @@ -0,0 +1,5 @@ +{ + "name": "@notes/home", + "private": true, + "type": "module" +} diff --git a/packages/chord/test/next/examples/53-app/plugins/home/server.ts b/packages/chord/test/next/examples/53-app/plugins/home/server.ts new file mode 100644 index 000000000..608e869b8 --- /dev/null +++ b/packages/chord/test/next/examples/53-app/plugins/home/server.ts @@ -0,0 +1,27 @@ +// The home plugin's server facet: it serves the app's `Sections` registry (the provider of a core registry is a plugin +// of the app's required root, so it reloads like any plugin) and an object reading it, and `GET /` says who is signed +// in to which app (the kit's `AppInfo`, the start data every server plugin can read) with the sections plugins +// contributed. + +import { AppInfo, arr, defineFacet, local, Sections, str } from "@notes/app/v1"; +import { Http } from "@notes/app/v1/server"; + +/** The home page's data: the composed sections. */ +const Home = local("notes.home", { version: 1, params: {}, members: { sections: arr(str()) } }); + +export default defineFacet((host, self) => { + host.serve(Sections, { compose: (table) => table.map(({ value }) => `${value.title}: ${value.path}`) }); + host.serve(Home, (_params, home) => ({ + sections: home.derive([home.registry(Sections)], ([sections]) => sections), + })); + self.contribute(Http, { + routes: { + "GET /": async (_request, ctx) => { + const info = await ctx.get(AppInfo, {}); + const page = await ctx.get(Home, {}); + const app = `${info.name.value} ${info.version.value}`; + return { kind: "json", value: { app, you: ctx.principal ?? null, sections: page.sections.value } }; + }, + }, + }); +}); diff --git a/packages/chord/test/next/examples/53-app/plugins/notes/contract.ts b/packages/chord/test/next/examples/53-app/plugins/notes/contract.ts new file mode 100644 index 000000000..3304ba83f --- /dev/null +++ b/packages/chord/test/next/examples/53-app/plugins/notes/contract.ts @@ -0,0 +1,22 @@ +// A project's notes: served by the project's worker (`Project({ id })`), reached from the server by id. + +import { method, obj, Project, remote, str } from "@notes/app/v1"; + +export const Notes = remote("notes.notes", { + version: 1, + params: { id: str() }, + hostAddress: ({ id }) => Project({ id }), + variants: { + main: { + state: { text: str(), start: str() }, + methods: { + /** Replaces the project's text. */ + write: method(str()), + /** Stores an attachment in the project's files; its id. */ + attach: method(obj({ name: str(), body: str() }), str()), + /** An attachment's text. */ + attachment: method(str(), str()), + }, + }, + }, +}); diff --git a/packages/chord/test/next/examples/53-app/plugins/notes/package.json b/packages/chord/test/next/examples/53-app/plugins/notes/package.json new file mode 100644 index 000000000..c345893f7 --- /dev/null +++ b/packages/chord/test/next/examples/53-app/plugins/notes/package.json @@ -0,0 +1,5 @@ +{ + "name": "@notes/notes", + "private": true, + "type": "module" +} diff --git a/packages/chord/test/next/examples/53-app/plugins/notes/server.ts b/packages/chord/test/next/examples/53-app/plugins/notes/server.ts new file mode 100644 index 000000000..535c60157 --- /dev/null +++ b/packages/chord/test/next/examples/53-app/plugins/notes/server.ts @@ -0,0 +1,23 @@ +// The notes plugin's server facet: routes reading and writing a project's notes, and its section of the home page. +// No route is public: the kit refuses anyone without a session (401) before they run. + +import { defineFacet, Sections } from "@notes/app/v1"; +import { Http } from "@notes/app/v1/server"; +import { Notes } from "./contract.ts"; + +export default defineFacet((_host, self) => { + self.contribute(Sections, { title: "Notes", path: "/notes/:id" }); + self.contribute(Http, { + routes: { + "GET /notes/:id": async (request, ctx) => { + const notes = await ctx.get(Notes, { id: request.params.id ?? "" }); + return { kind: "json", value: { text: notes.text.value, start: notes.start.value } }; + }, + "POST /notes/:id": async (request, ctx) => { + const notes = await ctx.get(Notes, { id: request.params.id ?? "" }); + await notes.write(String(await request.json()), ctx); + return { kind: "json", value: "saved" }; + }, + }, + }); +}); diff --git a/packages/chord/test/next/examples/53-app/plugins/notes/worker.ts b/packages/chord/test/next/examples/53-app/plugins/notes/worker.ts new file mode 100644 index 000000000..f034c7171 --- /dev/null +++ b/packages/chord/test/next/examples/53-app/plugins/notes/worker.ts @@ -0,0 +1,49 @@ +// The notes plugin's worker facet: the project's text in a persisted document (kept by a local object with empty +// params, the worker's one copy: a persisted state needs a stable identity), its attachments in the project's files +// (the kit's `Files`), and the start data the app gave the worker (`WorkerInfo`). + +import { defineFacet, defineState, Files, local, method, obj, str, WorkerInfo } from "@notes/app/v1"; +import { Notes } from "./contract.ts"; + +const Text = defineState("notes.text", { + schema: obj({ text: str() }), + version: 1, + initial: () => ({ text: "" }), + persist: true, + checkpointWhen: (_value, _ops, { deltasSinceBase }) => deltasSinceBase >= 20, +}); + +/** The project's text, one per worker. */ +const Store = local("notes.store", { version: 1, params: {}, members: { text: str(), write: method(str()) } }); + +export default defineFacet((host) => { + host.serve(Store, (_params, self) => { + const text = self.state(Text); + return { + text: self.derive([text], ([value]) => value.text), + write: async (next, ctx) => { + await text.change((draft) => { + draft.text = next; + }, ctx); + return null; + }, + }; + }); + host.serve(Notes, (_params, _principal, self) => { + const store = self.get(Store, {}); + const info = self.get(WorkerInfo, {}); + const files = self.get(Files, {}); + return { + variant: "main", + members: { + text: self.derive([store.text], ([text]) => text), + start: self.derive([info.start], ([start]) => JSON.stringify(start)), + write: (next, ctx) => store.write(next, ctx), + attach: ({ name, body }, ctx) => + files.put({ name, mime: "text/plain", bytes: new TextEncoder().encode(body) }, ctx), + attachment: async (id, ctx) => + new TextDecoder().decode(await files.read({ id, offset: 0, length: 65536 }, ctx)), + }, + }; + }); +}); diff --git a/packages/chord/test/next/examples/53-app/start.ts b/packages/chord/test/next/examples/53-app/start.ts new file mode 100644 index 000000000..382f5db29 --- /dev/null +++ b/packages/chord/test/next/examples/53-app/start.ts @@ -0,0 +1,50 @@ +// The notes app's start: everything the app itself decides, given to `startApp`. The kit does the rest: the data +// directory, the store, the host, HTTP with sessions and the owner link, a workers component for projects, and the +// plugins (`plugins/` here, the app's required root, and `/plugins/`, the user's). +// Examples 53 to 55 call `startNotes`; a real app's `start.ts` calls it with `process.argv[2]` and prints the link. + +import { fileURLToPath } from "node:url"; +import type { Context } from "../../../../src/context/index.ts"; +import { type App, startApp } from "../../../../src/next-node/app/index.ts"; +import type { NotesPrincipal } from "./app/principals.ts"; +import * as v1server from "./app/v1/server.ts"; +import * as v1 from "./app/v1.ts"; + +/** The app's API: its name, versions (newest first) and this process's modules (the worker's entry has the same). */ +export const API = { + name: "@notes/app", + versions: ["v1"], + modules: { "@notes/app/v1": v1, "@notes/app/v1/server": v1server }, +}; + +/** The app's facet kinds. */ +export const KINDS = { server: { platform: "node" }, worker: { platform: "node" } } as const; + +/** Starts the notes app on a free loopback port with its data in `dataDir`. */ +export function startNotes(dataDir: string, context: Context): Promise> { + return startApp( + { + name: "notes", + dataDir, + root: fileURLToPath(new URL("./plugins/", import.meta.url)), + listen: { host: "127.0.0.1", port: 0 }, + api: API, + kinds: KINDS, + principal: { server: { kind: "server" }, worker: (address) => ({ kind: "worker", project: address.key }) }, + workers: [ + { + kind: v1.Project, + entry: fileURLToPath(new URL("./worker.ts", import.meta.url)), + execArgv: ["--conditions=source", "--experimental-strip-types", "--no-warnings"], + // What each project's worker gets as `WorkerInfo.start`. + start: (address) => ({ project: address.key }), + // Who may reach a project: the owner (through the server's routes) and the server itself. + authorize: (principal) => principal.kind === "owner" || principal.kind === "server", + idleMs: () => 60_000, + }, + ], + onError: (error) => console.error("app:", error.kind, error.error), + }, + context, + ); +} diff --git a/packages/chord/test/next/examples/53-app/worker.ts b/packages/chord/test/next/examples/53-app/worker.ts new file mode 100644 index 000000000..bffa65fd2 --- /dev/null +++ b/packages/chord/test/next/examples/53-app/worker.ts @@ -0,0 +1,7 @@ +// The notes app's worker entry: one process per project. `startWorker` opens the project's store in its directory, +// connects to the server and loads the plugins' worker facets. + +import { startWorker } from "../../../../src/next-node/app/index.ts"; +import { API, KINDS } from "./start.ts"; + +startWorker({ api: API, kinds: KINDS, onError: (error) => console.error("worker:", error) }); diff --git a/packages/chord/test/next/examples/54-app-sessions/main.ts b/packages/chord/test/next/examples/54-app-sessions/main.ts new file mode 100644 index 000000000..775a54d84 --- /dev/null +++ b/packages/chord/test/next/examples/54-app-sessions/main.ts @@ -0,0 +1,62 @@ +// Sessions and device pairing (slice 13 §2), with example 53's notes app. A command-line client on another machine +// has no owner token file: it asks for a pairing code (`POST /_chord/pair`, public), the owner approves the code in a +// signed-in browser (`/_chord/pair/`, owner only), and the client's next poll returns its device token, once. +// The owner lists the sessions (`Sessions`, the kit's object; never a token or its hash) and revokes the device's. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/54-app-sessions/main.ts +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { BACKGROUND_CONTEXT as context } from "../../../../src/context/index.ts"; +import { Sessions } from "../../../../src/next-node/app/index.ts"; +import { call } from "../53-app/client.ts"; +import { startNotes } from "../53-app/start.ts"; + +const base = await mkdtemp(join(tmpdir(), "notes-")); +const app = await startNotes(join(base, "data"), context); +const link = new URL(app.ownerLink()); +const port = Number(link.port); + +// 1. The owner signs in in the browser with the owner link (example 53). +const browser = { cookie: (await call(port, "POST", link.pathname, {}, { token: link.hash.slice(1) })).cookie ?? "" }; + +// 2. The client asks for a code and shows it; it keeps the poll secret to itself. +const { code, poll } = (await call(port, "POST", "/_chord/pair", {}, { label: "laptop CLI" })).body as { + code: string; + poll: string; +}; +console.log("1. pairing code:", code.replace(/[A-Z0-9]/g, "X")); +console.log(" poll:", JSON.stringify((await call(port, "POST", "/_chord/pair/poll", {}, { poll })).body)); + +// 3. Only the owner may see and approve it: an anonymous request is refused before the kit's route runs. +console.log("2. anonymous approval:", (await call(port, "POST", `/_chord/pair/${code}`)).status); +const confirm = await call(port, "GET", `/_chord/pair/${code}`, browser); +console.log( + " the owner's confirm page:", + confirm.status, + String(confirm.body).includes("laptop CLI") ? "(names the device)" : "", +); +console.log(" approved:", (await call(port, "POST", `/_chord/pair/${code}`, browser)).status); + +// 4. The client's next poll returns its token, once; it signs in with it as a bearer token. +const approved = (await call(port, "POST", "/_chord/pair/poll", {}, { poll })).body as { + status: string; + token: string; +}; +console.log("3. poll:", approved.status); +console.log(" poll again:", (await call(port, "POST", "/_chord/pair/poll", {}, { poll })).status); +const device = { authorization: `Bearer ${approved.token}` }; +console.log(" CLI GET /:", JSON.stringify((await call(port, "GET", "/", device)).body)); + +// 5. The owner lists the sessions (a server plugin would do this with `self.get(Sessions, {})`) and revokes the +// device's: its next request is refused. +const sessions = await app.host.get(Sessions, {}, context); +console.log("4. sessions:"); +for (const { kind, label } of sessions.list.value) console.log(` ${kind}: ${label}`); +const laptop = sessions.list.value.find(({ label }) => label === "laptop CLI"); +await sessions.revoke(laptop?.id ?? "", context); +console.log(" revoked; CLI GET /:", (await call(port, "GET", "/", device)).status); +sessions.release(); + +await app.close(context); +await rm(base, { recursive: true, force: true }); diff --git a/packages/chord/test/next/examples/55-app-export/main.ts b/packages/chord/test/next/examples/55-app-export/main.ts new file mode 100644 index 000000000..749617947 --- /dev/null +++ b/packages/chord/test/next/examples/55-app-export/main.ts @@ -0,0 +1,66 @@ +// Export and import per address (slice 13 §4), with example 53's notes app. `app.exportAddress` stops a project's +// worker and writes one archive: a manifest (app, API version, format), the project's store export and its files. +// `app.importAddress` makes another address from it (refusing one that has data) and the next request starts its +// worker on the imported store and files. Archives of another app are refused. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/55-app-export/main.ts +import { mkdtemp, rm, stat } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { BACKGROUND_CONTEXT as context } from "../../../../src/context/index.ts"; +import { method, obj, Project, remote, str } from "../53-app/app/v1.ts"; +import { startNotes } from "../53-app/start.ts"; + +// The notes plugin's contract, declared again with the API's builders (contracts are data: the same name, version and +// schema bind). The plugin's own files import `@notes/app/v1`, a host package this process has no module path for. +const Notes = remote("notes.notes", { + version: 1, + params: { id: str() }, + hostAddress: ({ id }) => Project({ id }), + variants: { + main: { + state: { text: str(), start: str() }, + methods: { + write: method(str()), + attach: method(obj({ name: str(), body: str() }), str()), + attachment: method(str(), str()), + }, + }, + }, +}); + +const base = await mkdtemp(join(tmpdir(), "notes-")); +const app = await startNotes(join(base, "data"), context); + +// 1. Project p1 gets a text and an attachment (its worker's store and files). The server reaches it as `Notes`. +const p1 = await app.host.get(Notes, { id: "p1" }, context); +await p1.write("the plan", context); +const attachment = await p1.attach({ name: "plan.txt", body: "step 1, step 2" }, context); +p1.release(); + +// 2. Export p1: its worker stops, the archive is written (mode 0600), and p1 goes on (the next hold starts it). +const archive = join(base, "p1.tar"); +await app.exportAddress(Project({ id: "p1" }), archive, context); +console.log( + "1. exported:", + `${(await stat(archive)).size} bytes`, + `mode ${((await stat(archive)).mode & 0o777).toString(8)}`, +); + +// 3. Import it as p2: a copy with p1's text and attachment, its own worker and directory. +await app.importAddress(archive, Project({ id: "p2" }), context); +const p2 = await app.host.get(Notes, { id: "p2" }, context); +console.log("2. p2:", JSON.stringify(p2.text.value), JSON.stringify(await p2.attachment(attachment, context))); +await p2.write("p2's own plan", context); +p2.release(); +const again = await app.host.get(Notes, { id: "p1" }, context); +console.log(" p1 unchanged:", JSON.stringify(again.text.value)); +again.release(); + +// 4. An address that has data is never overwritten. +await app + .importAddress(archive, Project({ id: "p1" }), context) + .catch((error: Error) => console.log("3. refused:", error.message.replace(base, "…"))); + +await app.close(context); +await rm(base, { recursive: true, force: true }); diff --git a/tsconfig.json b/tsconfig.json index a6a826276..e8e536464 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -12,8 +12,10 @@ "@earendil-works/chord/next/plugins": ["./packages/chord/src/next-node/plugins/index.ts"], "@earendil-works/chord/next/plugins/runtime": ["./packages/chord/src/next-node/plugins/runtime.ts"], "@acme/app/*": ["./packages/chord/test/next/examples/49-plugin-api/app/*.ts"], + "@notes/app/*": ["./packages/chord/test/next/examples/53-app/app/*.ts"], "@earendil-works/chord/next/http": ["./packages/chord/src/next-node/http/index.ts"], "@earendil-works/chord/next/workers": ["./packages/chord/src/next-node/workers/index.ts"], + "@earendil-works/chord/next/app": ["./packages/chord/src/next-node/app/index.ts"], "@earendil-works/chord/node": ["./packages/chord/src/node.ts"], "@earendil-works/chord/output": ["./packages/chord/src/output.ts"], "@earendil-works/chord/telemetry-sqlite": ["./packages/chord/src/telemetry-sqlite/index.ts"], From d6af4d18e55b86ddef249d5bf8a4939601707945 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 15:23:22 +0000 Subject: [PATCH 157/175] docs(chord): looks and themes decided; slice 11 spec gains themes --- packages/chord/docs/next/decisions.md | 9 +++++++++ packages/chord/docs/next/slice-11.md | 16 +++++++++++++++- 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index d510d2c2d..8291b56fd 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2318,6 +2318,15 @@ validation and copies the v2 kernel added around Delta. or `github:bob/plugins#`, installed by plain npm), imported by path into it (`@bob/plugins/notes/contract.ts`) or through its `exports`. Removed: the dependency form of `chord.plugins` (`"name": "github:...#ref:subfolder"`) and its own clone-and-link installer (it could delete outside `node_modules`). +- **Looks and themes (owner, 2026-11-02: "i suppose", then "do all the things"; proposal `docs/next/theming/`):** + shared components come from the app's per-kind API modules (one instance in the base bundle); looks are design + tokens, CSS custom properties on `:root` on the web, which the app's components and plugins' CSS read; plugins scope + their CSS with CSS modules (`*.module.css`); the terminal gets the app's components and a theme lookup by role. A + theme is data in one registry, `Themes`, shipped by the kit in its shared module with the code that writes the active + theme as CSS variables (web) and the lookup (TUI): a base set of roles the kit defines (accent, bg, fg, muted, error, + warning, success), apps may add roles; a user picks one dark and one light theme, the system preference chooses; + `web` extras may name fonts and assets of the plugin under the page's CSP. Plugins contribute themes; they cannot + change the app's base CSS or components. The open questions' recommendations taken. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a diff --git a/packages/chord/docs/next/slice-11.md b/packages/chord/docs/next/slice-11.md index e13899000..efb2cbb43 100644 --- a/packages/chord/docs/next/slice-11.md +++ b/packages/chord/docs/next/slice-11.md @@ -81,6 +81,20 @@ releases, presence is the app's. No components, no styling. the bridged link, in-place reloads through a `TuiScripts` object as the web. - **`TuiScope`** (P9): `new TuiScope(host, changed)` with `hold`, `watch`, `registry`, `dispose`; no pi-tui dependency. +## 4b. Looks and themes (decision "Looks and themes", `docs/next/theming/`) + +- The kit ships `Themes` (`defineRegistry`, in the shared module the kit's apps re-export), + `Theme = { id, name, mode: "dark" | "light", colors: { accent, bg, fg, muted, error, warning, success, ...app roles }, + web?, tui? }`, and a `themes` preference per user (`{ dark, light }` theme ids) stored by the kit's users plugin or a + local object for single-user apps. +- Web: `startPage` watches `Themes` and the preference with `prefers-color-scheme`, and writes the active theme as CSS + variables on `:root` (`--accent`, ..., plus `web` extras), with no reload; the base bundle's stylesheet is the app's. +- TUI: `TuiScope`'s host gets a `theme` lookup by role (true colour, else the nearest of 256 colours, `tui` overrides + first). +- CSS modules (`*.module.css`) work in plugin builds (esbuild's local-css loader); documented in `plugin-authoring.md`. +- Version 1 of `Themes`; a later role is added by a version bump mapping old entries (missing roles from the default + theme). + ## 5. In the app kit `startApp` gains `web: { entry, page }` and `tui: { entry }` (optional); with them it creates the surfaces and adds @@ -96,4 +110,4 @@ their facets. Without, an app has no web or TUI surface (an API-only app). TUI: the sandbox cannot read a file or spawn a process (a test plugin that tries), the bridge, an in-place reload, `TuiScope` disposal; pairing a TUI client against a kit app. - Examples: `56-web` (a kit app with a page, two web plugins, an edit applied in place), `57-tui` (a kit app with a - TUI client, sandboxed). + TUI client, sandboxed), `58-themes` (a theme plugin restyling a page and a TUI live). From cf74b67653633c565cf2033e78c19c5525ed2432 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 15:49:56 +0000 Subject: [PATCH 158/175] docs(chord): p3 meeting summary, session sync and index --- packages/chord/docs/next/decisions.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 8291b56fd..314984ba5 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2327,6 +2327,15 @@ validation and copies the v2 kernel added around Delta. warning, success), apps may add roles; a user picks one dark and one light theme, the system preference chooses; `web` extras may name fonts and assets of the plugin under the page's CSP. Plugins contribute themes; they cannot change the app's base CSS or components. The open questions' recommendations taken. +- **p3 meeting, 2026-11-02 (Mario and Armin; Armin's summary, relayed by the owner):** session sync: the session + repository talks to the server process to enumerate sessions; the session index in the server is updated by the + worker from events and sent to the server by RPC. Sync to R2 goes through Radius: Radius keys each session by id and + keeps its last sequence number, tells pi which sessions it has with their last sequence numbers (optimization: time + of last sync), and pi sends the delta commit log from that sequence number; R2 stores the SQLite file and the commit + log (SQLite DO to R2); sessions have an owner id. The index is a Chord table replicated like sessions; its entry is + updated only on a session metadata change, a session idle-state change and a subagent idle-state change. p3: each + host (server, worker, tui, web) has a kernel loading the plugins with contribution points via registries; p2's look + and feel made great; new session storage; hands (machine daemons) good, secure and connecting proactively. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From e64f313bd61b9ff2cf173491751997682f099225 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 15:58:52 +0000 Subject: [PATCH 159/175] docs(chord): p3 feature set and work packages, meeting prep --- packages/chord/docs/next/p3-features/book.md | 329 +++++++ .../chord/docs/next/p3-features/index.html | 832 ++++++++++++++++++ 2 files changed, 1161 insertions(+) create mode 100644 packages/chord/docs/next/p3-features/book.md create mode 100644 packages/chord/docs/next/p3-features/index.html diff --git a/packages/chord/docs/next/p3-features/book.md b/packages/chord/docs/next/p3-features/book.md new file mode 100644 index 000000000..4d45c3c26 --- /dev/null +++ b/packages/chord/docs/next/p3-features/book.md @@ -0,0 +1,329 @@ +--- +title: p3: Feature Set and Work Packages (prep for the design meeting) +cover: p3 / Features +kicker: Meeting prep · p3 on Chord v2 · 2026-11-03 +subtitle: What p2 does today, what p3 should ship first, how hands, sync and the plugin API fit, and a first cut into work packages +stats: + - 28 | feature areas in p2 today + - 21 | model-facing tools in p2 + - 78 | coding-agent extension examples checked against the p3 plugin API + - 12 | proposed work packages +note: | + Written by the redesign session for the meeting on 2026-11-03, from three research passes over p2 + (`apps/p2` at `b7af55ace`), the coding agent's extension examples and p2's environments and Radius code + (`/tmp/p3-prep/features.md`, `extensions.md`, `envs.md`), the owner's notes and Armin's summary of the 2026-11-02 + meeting. Chord v2's slices 1-13 are built (the app kit included); slice 11 (web and TUI) is building. Everything + marked "proposed" is for the meeting to decide. +colophon: p3 Feature Set and Work Packages · meeting prep · 2026-11-03 +--- + +# Before we begin {.preface} + +p2 grew feature by feature on the phone: 33 plugins, about 65,000 lines, a web shell and a TUI. p3 is its rewrite on +Chord v2, with an app kit that already gives hosts, workers, HTTP with sessions and pairing, plugins with versioned +APIs, files and export. The meeting's job is to decide what p3 ships first, what waits, and how the work splits. + +Some of it was settled yesterday (Armin's summary): sessions sync to R2 through Radius by commit log, the session index +is a replicated Chord table, every host kind loads plugins through registries, hands connect proactively. The owner +added: voice waits; environments turn around into hands; continuing a local session on Radius is core; subagents and +handoff are in, artifacts are questionable; the plugin API stays small enough to evolve. + +## Ten things to know + +:::findings +1. **Proposed base:** sessions (with delete), the agent and conversations, subagents, history and handoff, code mode, + shell commands, models and login, prompt resources, workspace (read and review), file index, status and notices, + search through the index, the web shell and the TUI, hands, session sync and continue-on-Radius. +2. **Proposed later:** voice, terminals (need channels and hands), artifacts, port forwards, chat mode and guest + sharing, MCP and `tool_search`. +3. **Hands replace SSH-dialled daemons:** a long-lived daemon dials the server (or Radius), is enrolled once by a + pairing code, and carries p2's existing daemon protocol over one multiplexed WebSocket. +4. **The sync handshake needs a driver identity.** Two machines writing one session both push "from Radius's last + seq" and silently diverge; an epoch or lease per session (Chord's rule S7) closes it. +5. **The index has two writers** once a session continues on Radius; three options, one recommended (derive rows from + session stores). +6. **Tools in base:** `read`, `write`, `edit`, `bash`, `codemode`, `subagent`, `team`, `agent`, `conversations`, + `history_*`, location tools, `rename_session`, `send_file`; artifacts, ports and `reload_plugins` wait or change. +7. **One CLI, no environment variables:** `p3 serve`, `p3 tui`, `p3 hand`, `p3 owner-link`, `p3 install`; settings as + arguments and files, per the app kit. +8. **The plugin API v1 is seven clusters,** almost all already existing: Durable's extension, a `Conversation` + contract, input transforms, environments and models, questions, UI registries, objects for state. +9. **Litmus test:** all 78 coding-agent extension examples are expressible; Doom needs two small fixes (frame splitting + in the TUI bridge, a path for large plugin assets); streaming frames from a worker and raw terminal ownership do + not fit, by design. +10. **Twelve work packages,** with p3's skeleton on the app kit and the session/index/sync core first. +::: + +## Words used throughout + +A **hand** is a daemon on a machine that connects out to a p3 server or to Radius and offers execution (commands, +files, watching, terminals). A **driver** is the single host allowed to write a session's store at a time; a +**replica** follows its commits. An **epoch** numbers driver changes. The **index** is the table of session metadata +(title, status, idle state) the session list reads. The **app kit** is Chord's `startApp`/`startWorker` layer. + +# p2 Today, and the Proposed Cut {#features lede="Every feature area, what it is, and where it goes in p3."} + +Sizes are non-test lines in p2. "Base" means p3's first release; "later" means designed for, built after; "drop" means +not carried over. + +| Area | What it does | p2 size | Proposed | +| --- | --- | ---: | --- | +| Sessions | list by machine and directory, create, rename, pin, archive, relocate, clone, fork, unread, automatic titles | 2,186 | **base**, plus delete; clone and fork in the web too | +| Conversations and agent | transcript, send, steer, follow-up, outbox, stop, model and thinking, cost, compaction | 3,682 + web 2,700 | **base** | +| Subagents, one-shot agents, cross-session messages | named background agents, `team`, `agent` from scripts, `conversations` | 2,182 | **base** (owner: "awesome") | +| History and handoff | re-read earlier entries after compaction, hand off with a continuation | 533 | **base** | +| Code mode | scripts calling tools as nested durable calls | 155 + Durable | **base** (small, Durable owns it) | +| Shell commands | `!cmd` and `!!cmd` in the conversation's environment | in chat | **base**, web too (TUI only today) | +| Chat mode | people talk without a turn, `@pi` mentions | 876 | later | +| Sharing and guests | share links, guest names, revocation | 1,047 | later (the app kit's users and invites cover the start) | +| Devices and owner login | owner links, device cookies | 684 | **base**, replaced by the app kit's sessions and pairing | +| Models, credentials, provider login | picker, scope, API keys, OAuth flows | 3,828 | **base** | +| Settings | `settings.json`, hot reload | 87 | **base** | +| Prompt resources | AGENTS.md, skills, templates, read in the session's environment | 958 | **base** | +| Workspace | files, git status, diffs, search, review comments | 2,655 + web 1,341 | **base** (read and review; uploads later) | +| File index and `@` | ranked file completion | 242 | **base** | +| Terminals | detached PTYs, ghostty-vt in the web | 3,518 | later (channels, hands) | +| Environments and locations | SSH environments, directory picker, agents moving themselves | 2,546 | **base, redesigned as hands** | +| Artifacts and files | attachments, `send_file`, versioned artifacts with previews | 1,713 | attachments and `send_file` base (app kit `Files`); artifacts later | +| Search | sidebar search over titles and messages (FTS5) | 775 | **base** over the index; message text search through the same sync mechanism later | +| Status and presence | busy, tasks, usage, viewers, worker failures | 566 | **base** (feeds the index) | +| Logs and notices | owner log viewer, notices | 487 | **base** | +| Voice | rooms, barge-in, speech service | 3,770 + Python | later (owner) | +| Radius relay and port forwards | brain WebSocket, up to 50 forwards | 2,000 | relay **base** (as the hands/sync link), forwards later | +| Web shell | PWA, sheets, navigation, reconnect | 9,252 | **base**, rebuilt on slice 11 ("look and feel great") | +| TUI | launcher, sandbox, 40 commands, themes, keybindings | 6,720 + launcher | **base** on slice 11 | +| Dev and reload | supervisor, dirty marks, reload tool | 468 + scripts | **base** as the app kit's "apply changes"; the supervisor goes | +| Fake provider, `PI_MOBILE_*` aliases, `P2_DEV_RESUME` | test and legacy leftovers | 218 | drop | +| `tool_search`, MCP | not installed in p2 | - | later | + +Rough edges p3 should not inherit: features only in the TUI (shell commands, clone, fork, compact, new context), no +session delete, terminals and voice without a TUI side, docs that drifted from code (voice routes, TUI notes), the +plugin loader's module leak on reload (solved by Chord's plugin component). + +# Hands {#hands lede="Environments turned around: the machine connects to the server, not the other way."} + +## What p2 does + +p2's server and workers start a daemon (`pi-env`) over SSH on demand, one connection per session and per use (four +connection pools: each worker, the directory picker, the connection test, the terminal host). The daemon is +stateless and kills its commands 30 seconds after its connection goes quiet. Its frame protocol (exec, files, watch, +file listing, PTYs) does not depend on SSH. + +## What a hand is (proposed) + +| | p2's daemon | hand | +| --- | --- | --- | +| who dials | the server, over SSH | the hand, over WSS, to the server or to Radius | +| needs | sshd, a key, a network path to the machine | outbound HTTPS only | +| lifetime | per connection | a service (systemd, launchd, Termux:Boot), reconnecting | +| install | deployed and verified at each start | installed once, signed updates | +| machine identity | SSH host key | a hand key pair, enrolled once | +| connections | one per session and use | one per hand and server, multiplexed | + +- **Enrollment:** `p3 hand enroll ` shows a code; the owner approves it in the UI (the app kit's device + pairing); the server pins the hand's public key; fingerprints shown on both sides. Revocation in a Hands panel. +- **Reachability:** a hand reaches the owner's own server directly (LAN, own tunnel) or through Radius, which relays + hands to whichever host drives a session (the upsell: no tunnel to set up; Radius can also offer sandboxed machines + and hands of its own). +- **Protocol:** p2's daemon protocol over the WebSocket, unchanged in shape; two changes: several sessions share one + connection (cancellation, handles and backpressure per session), and a disconnect starts a grace period instead of + killing everything (detached commands and PTYs reattach on reconnect; this moves the terminal host's job into the + hand). +- **Binding:** a session's location becomes `{ hand, cwd }`; one server object holds the hand connections and the + `HarnessEnvironments` entry resolves a hand id to an execution environment; workers reach hands through the server's + links. An SSH machine stays possible as a hand kind the server dials. +- **Offline:** an operation needing an offline hand fails with a clear error the model sees; the hand's status shows + in the session list. + +# Session Sync and the Index {#sync lede="What was decided yesterday, and the three gaps it leaves."} + +## Decided + +The worker updates the session index from events and sends it to the server; sessions sync to R2 through Radius: +Radius reports per session its last sequence number, pi uploads the commit log from there; R2 holds the SQLite file +and the commit log, a SQLite Durable Object applies them; sessions have an owner id; the index is a Chord table +replicated like a session, its row updated only on metadata, session idle and subagent idle changes. + +Chord v2 has the pieces: a versioned outbox written in each commit's transaction, export archives, batch apply that +refuses gaps and divergence, export and import per address (app kit). Missing (the sync layer): upload and trim, +epochs, pull-back after a power loss, the split, starting from an archive. + +## Gap 1: who is the driver + +Two machines both write session X (the laptop resumed it while Radius continued it). Both ask Radius for its last +seq, both upload from there; Radius applies whichever arrives first and refuses the other as "not following". The +copies have now diverged and nobody knows. + +**Proposed:** a driver epoch per session, handed out by Radius. An upload names its epoch; an old epoch is refused; +the refused copy becomes a new session (Chord's rule S7: a split, never a rollback, because tools already ran). +"Continue on Radius" is an epoch change: the local driver waits for idle, uploads the tail and the session's files, +Radius bumps the epoch and starts the agent from its store, the local side becomes a replica or closes. Back is the +reverse. + +## Gap 2: two writers of the index + +When a session continues on Radius, Radius writes its index row, and so does the local server. + +| Option | How | Cost | +| --- | --- | --- | +| a. one index per writer | each host keeps its own index, Radius merges per owner, newest epoch wins per row | a second merge rule | +| b. rows derived from session stores | every host computes a session's row from the session's own commits | no index sync at all; each host needs the session's commits | +| c. rows tagged `{ epoch, seq }` | Radius keeps the highest per row | rows must carry their source position | + +**Recommended:** (b) where the session's commits are present (Radius has them all), with (c)'s tag on the rows a local +server writes for sessions it drives, so a list without the session store still sorts correctly. + +## Gap 3: what else moves + +Files outside the store (uploads, sent files, the app kit's `Files`) must sync as blobs before the batches naming +them, R2 as their home. Model access and secrets for a session continued on Radius: Radius's own models, or the +owner's provider keys stored on Radius (decide). Local prompt resources (the device's `AGENTS.md`, skills) must be read +through a hand or synced. Closed sessions have no worker: the server reads a session's outbox without starting it, or +starts it to sync. + +# Tools Built In {#tools lede="p2's 21 model-facing tools, and which ship with p3."} + +| Tool | p2 plugin | Proposed | +| --- | --- | --- | +| `read`, `write`, `edit`, `bash` | coding-tools (Durable) | base | +| `powershell` | powershell | base (deferred tool, Windows hands) | +| `codemode` | codemode | base | +| `agent` (from scripts) | agent | base | +| `subagent`, `team` | subagents | base | +| `conversations` | conversations | base | +| `history_search`, `history_read`, `history_handoff` | history | base | +| `location_envs`, `location_dirs`, `location_move` | locations | base, renamed around hands (`hands`, `hand_dirs`, `move`) | +| `rename_session` | sessions | base | +| `send_file` | send-file | base (on the app kit's `Files`) | +| `artifact` | artifacts | later (owner: maybe not as useful) | +| `ports` | ports | later (with forwards) | +| `reload_plugins` | dev-tools | base as an "apply changes" tool for the owner, not tied to a checkout | +| `tool_search` | Durable | later (with MCP) | + +Every tool is a plugin in the app's own root, so any of them can be removed, replaced or added without touching base +code (the app kit's rule: base code is the API, bootstrap and a start-data facet). + +# Running p3 {#running lede="Starting the server, the CLI, themes and keybindings."} + +## The CLI (proposed) + +One binary, `p3`, and no environment variables (app kit decision); settings are arguments or files in the data +directory. + +| Command | Does | +| --- | --- | +| `p3 serve [--data-dir

    ] [--host ] [--port ]` | starts the server (`startApp`); prints the one-time owner link | +| `p3 owner-link` | prints a new owner link for a running server (through the owner token file) | +| `p3 tui [url] [--pair]` | the sandboxed TUI; on the same machine it uses the owner token file, elsewhere device pairing | +| `p3 hand enroll `, `p3 hand run` | enrolls this machine as a hand, runs it (or installs a service) | +| `p3 radius login` | signs the server into Radius (device code) | +| `p3 install ` | installs a plugin into the user's root | +| `p3 export `, `p3 import ` | the app kit's export per address | + +The data directory defaults to `~/.p3`. Its files: `settings.json`, `models.json`, `auth.json` (locked), +`AGENTS.md`, `skills/`, `prompts/`, `plugins/` (the user's root), the server store, a directory per session worker, +`owner-token`, `logs/`. Plugin-specific configuration moves into plugin objects (documents) instead of more files. + +## Themes and keybindings + +Themes follow the looks-and-themes proposal (decided 2026-11-02): design tokens as CSS variables on the web, roles in +the TUI, one `Themes` registry plugins contribute to, one dark and one light choice per user. Keybindings: commands are +registry entries with default keys (p2's `TuiCommands` with `when` conditions, 78 binding ids); the user's overrides +live in a preferences document (per user, synced like any data), not in files on each client; the web gets the same +command registry with its own key handling. + +# The Plugin API {#api lede="The smallest surface that still expresses everything the coding agent's extensions do."} + +## The rule + +Offer few things, each stable: what an API offers, it must keep (versioned in the import path, `@p3/app/v1`). Every +cluster below is a registry or a contract that already exists in Chord, Durable or p2; p3 v1 publishes those and +nothing else. Legacy concepts of the old extension API are not carried: CLI flags, an event bus, RPC-mode UI, `hasUI` +checks, "destructive switch" gates, `pi.exec` running locally regardless of the environment. + +## Seven clusters + +| Cluster | Host | Mechanism | Covers | +| --- | --- | --- | --- | +| A. The agent's turn | worker | Durable's `Extension` in `HarnessExtensions`: tools, `beforeTool`/`afterTool`, `beforeRequest`, sections, wraps, `onYield`, `afterRun`, `beforeCompact` | 20 examples: hello, permission-gate, protected-paths, tool-override, plan-mode, todo, custom-compaction, ... | +| B. Conversation control | worker object | a `Conversation` contract: `submit`, `configure`, `append` (typed entries), `compact`, `fork`, `rename`; state: busy, agent, usage | send-user-message, file-trigger, session-name, preset, handoff, qna, ... | +| C. Input | worker | `InputTransforms` registry (rewrite or answer input before the model) | input-transform, inline-bash | +| D. Environments and models | worker, server | `HarnessEnvironments` (ssh, gondolin, hands as kinds), `HarnessModels` (providers, virtual models), `LoginProviders` | ssh, gondolin, custom providers, jev-router | +| E. Asking the user | server object, UI renderers | `Questions.ask` from worker code; `QuestionKinds` per UI kind | permission-gate, question, questionnaire, timed-confirm, project-trust | +| F. UI | tui, web | registries: `Commands` (slash, palette, keys), `ToolRenderers`, `EntryRenderers`, `Slots` (header, footer, status, widgets), overlays through the UI context, editor extension points | every UI-only example, the games | +| G. State and standing behaviour | any | `local` objects owning natives, documents for state | file-trigger's watcher, gondolin's VM, game saves | + +```ts +// a plugin with a tool, a gate and a renderer: plugins/todo/ +// worker.ts +self.contribute(HarnessExtensions, { name: "todo", tools: [todoTool], hooks: [hook(ToolTask, { beforeTool: guard })] }); +// tui.ts +self.contribute(TuiToolRenderers, { todo: { call: (args) => todoCall(args), result: (r) => todoList(r.details) } }); +self.contribute(TuiCommands, { "todo.show": { slash: { name: "todos" }, run: (ui) => ui.overlay("todos", new TodoView(ui)) } }); +// web.tsx: the same registries with React components +``` + +## The litmus test + +All 78 examples map onto the seven clusters (table in `/tmp/p3-prep/extensions.md`). The examples that mix agent and +UI split into a worker facet and a tui/web facet sharing a contract file: a tool that wants a dialog asks the +`Questions` object, which works from any client, several clients or none. + +- **Snake, Space Invaders, tic-tac-toe:** a command plus an overlay in `tui.ts` and a canvas in `web.tsx`, engine in a + shared file, saves in a user-keyed document. Fit today. +- **Doom:** fits with two fixes in the TUI: the sandbox bridge caps a frame at 64 KiB and a Doom frame is about 160 KB + (split writes at escape boundaries), and the sandbox has no files or network, so large plugin assets (a 4 MB WAD) + need a documented "asset of my plugin" path (chunked reads through the plugin's server facet). +- **Does not fit, by design:** streaming 35 FPS frames from a worker to clients (objects carry state; needs channels), + engine state surviving an in-place reload (needs an explicit save), raw terminal ownership (`interactive-shell` + becomes a terminal view; terminal notifications need a launcher action). + +## What v1 leaves out + +Editor replacement (extension points only), raw provider stream events and response headers, a fork hook, a +`turn_start` hook, an event bus. Each has one or two examples; add when a real plugin needs it. + +# Work Packages {#packages lede="A first cut, in dependency order, for the meeting to rearrange."} + +| # | Package | Contains | Depends on | +| --- | --- | --- | --- | +| 1 | p3 skeleton | `startApp` with p3's API v1 modules (shared, web, tui, worker), core plugins in the app's root, `p3` CLI | Chord slices 11-13 | +| 2 | Sessions and conversations | Durable on the Chord store, session lifecycle with delete, the conversation contract, subagents, history and handoff, code mode, shell commands | 1 | +| 3 | Session index | the index table, worker-to-server updates on metadata and idle changes, the session list, search over it | 2 | +| 4 | Session sync to Radius | outbox upload and trim, epochs, pull-back, split, files as blobs, continue on Radius and back | 2, 3, Radius side | +| 5 | Radius side | per-session SQLite DO, commit log and SQLite on R2, epochs, the sync handshake, hand relay | with 4 | +| 6 | Hands | daemon as an outbound service, enrollment by pairing, multiplexed protocol, reattach after disconnects, server hand pool, `HarnessEnvironments` kind, SSH as a hand kind | 1 | +| 7 | Web shell | p2's layout and flows rebuilt on slice 11 ("look and feel great"), themes | 1, slice 11 | +| 8 | TUI | p2's TUI on slice 11's sandbox, frame splitting, plugin assets path, keybindings and themes | 1, slice 11 | +| 9 | Models and settings | catalog, credentials, provider login flows, settings, prompt resources | 1 | +| 10 | Workspace | files, git, diffs, search, review comments, file index and `@`, through hands | 6 | +| 11 | Plugin API v1 and docs | the seven clusters published in `@p3/app/v1`, authoring guide, ported examples as the litmus (hello, permission-gate, todo, plan-mode, snake, doom) | 1, 2, 7, 8 | +| 12 | Access and status | owner, devices and pairing (app kit), notices, logs, status and presence | 1 | + +Later: voice, terminals (channels and hands), artifacts, port forwards, chat mode and guest sharing, MCP and +`tool_search`. + +# Open Questions {.epilogue lede="To decide in the meeting."} + +1. The proposed cut: base versus later per area (first table). +2. Driver identity: epochs per session, handed out by Radius, with splits on conflict? +3. The index with two writers: derived rows plus `{ epoch, seq }` tags, or one of the other options? +4. Who syncs a closed session (the server reading its outbox, or starting its worker)? +5. Files outside the store as blobs on R2, uploaded before their batch? +6. Models and secrets for sessions continued on Radius: Radius's own models, or the owner's keys stored there? +7. Hands: through Radius as a relay to the driver, and may one hand serve the local server and Radius at once? +8. Do hands keep commands and terminals alive across disconnects (absorbing p2's terminal host)? +9. Hand install and updates: signed binary, service install, version negotiation. +10. Is SSH kept as a hand kind for machines without a service? +11. Plugin API v1: the seven clusters as listed, and what is explicitly left out? +12. Work package owners and order. + +# Sources {.appendix} + +| Source | What | +| --- | --- | +| `/tmp/p3-prep/features.md` | p2's feature areas with files, sizes, tools, rough edges; the CLI, environment variables and data files | +| `/tmp/p3-prep/extensions.md` | all 78 extension examples, the capability matrix, the clusters with sketches, the hard cases | +| `/tmp/p3-prep/envs.md` | p2's environments and Radius link, Chord's sync pieces, the sync and hands sketches, open questions | +| `apps/p2/docs/p3-functions/` | the functionality map (partly outdated) | +| `packages/chord/docs/next/` | design, decisions (2026-11-02 meeting summary), app kit and plugin model, theming | diff --git a/packages/chord/docs/next/p3-features/index.html b/packages/chord/docs/next/p3-features/index.html new file mode 100644 index 000000000..352a6d1ed --- /dev/null +++ b/packages/chord/docs/next/p3-features/index.html @@ -0,0 +1,832 @@ + + +p3: Feature Set and Work Packages (prep for the design meeting) + + + + +
    +
    +
    Meeting prep · p3 on Chord v2 · 2026-11-03
    +

    p3
    Features

    +

    What p2 does today, what p3 should ship first, how hands, sync and the plugin API fit, and a first cut into work packages

    +
    28feature areas in p2 today
    21model-facing tools in p2
    78coding-agent extension examples checked against the p3 plugin API
    12proposed work packages
    +

    Written by the redesign session for the meeting on 2026-11-03, from three research passes over p2 +(apps/p2 at b7af55ace), the coding agent's extension examples and p2's environments and Radius code +(/tmp/p3-prep/features.md, extensions.md, envs.md), the owner's notes and Armin's summary of the 2026-11-02 +meeting. Chord v2's slices 1-13 are built (the app kit included); slice 11 (web and TUI) is building. Everything +marked "proposed" is for the meeting to decide.

    +
    +
    + + +
    +

    Before we begin

    +

    p2 grew feature by feature on the phone: 33 plugins, about 65,000 lines, a web shell and a TUI. p3 is its rewrite on +Chord v2, with an app kit that already gives hosts, workers, HTTP with sessions and pairing, plugins with versioned +APIs, files and export. The meeting's job is to decide what p3 ships first, what waits, and how the work splits.

    +

    Some of it was settled yesterday (Armin's summary): sessions sync to R2 through Radius by commit log, the session index +is a replicated Chord table, every host kind loads plugins through registries, hands connect proactively. The owner +added: voice waits; environments turn around into hands; continuing a local session on Radius is core; subagents and +handoff are in, artifacts are questionable; the plugin API stays small enough to evolve.

    +

    Ten things to know

    +
      +
    1. Proposed base: sessions (with delete), the agent and conversations, subagents, history and handoff, code mode, +shell commands, models and login, prompt resources, workspace (read and review), file index, status and notices, +search through the index, the web shell and the TUI, hands, session sync and continue-on-Radius.
      2. +
    2. Proposed later: voice, terminals (need channels and hands), artifacts, port forwards, chat mode and guest +sharing, MCP and tool_search.
      3. +
    3. Hands replace SSH-dialled daemons: a long-lived daemon dials the server (or Radius), is enrolled once by a +pairing code, and carries p2's existing daemon protocol over one multiplexed WebSocket.
      4. +
    4. The sync handshake needs a driver identity. Two machines writing one session both push "from Radius's last +seq" and silently diverge; an epoch or lease per session (Chord's rule S7) closes it.
      5. +
    5. The index has two writers once a session continues on Radius; three options, one recommended (derive rows from +session stores).
      6. +
    6. Tools in base: read, write, edit, bash, codemode, subagent, team, agent, conversations, +history_*, location tools, rename_session, send_file; artifacts, ports and reload_plugins wait or change.
      7. +
    7. One CLI, no environment variables: p3 serve, p3 tui, p3 hand, p3 owner-link, p3 install; settings as +arguments and files, per the app kit.
      8. +
    8. The plugin API v1 is seven clusters, almost all already existing: Durable's extension, a Conversation +contract, input transforms, environments and models, questions, UI registries, objects for state.
      9. +
    9. Litmus test: all 78 coding-agent extension examples are expressible; Doom needs two small fixes (frame splitting +in the TUI bridge, a path for large plugin assets); streaming frames from a worker and raw terminal ownership do +not fit, by design.
      10. +
    10. Twelve work packages, with p3's skeleton on the app kit and the session/index/sync core first.
      11. +
    +

    Words used throughout

    +

    A hand is a daemon on a machine that connects out to a p3 server or to Radius and offers execution (commands, +files, watching, terminals). A driver is the single host allowed to write a session's store at a time; a +replica follows its commits. An epoch numbers driver changes. The index is the table of session metadata +(title, status, idle state) the session list reads. The app kit is Chord's startApp/startWorker layer.

    +
    + +
    I

    p2 Today, and the Proposed Cut

    Every feature area, what it is, and where it goes in p3.

    +

    Sizes are non-test lines in p2. "Base" means p3's first release; "later" means designed for, built after; "drop" means +not carried over.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    AreaWhat it doesp2 sizeProposed
    Sessionslist by machine and directory, create, rename, pin, archive, relocate, clone, fork, unread, automatic titles2,186base, plus delete; clone and fork in the web too
    Conversations and agenttranscript, send, steer, follow-up, outbox, stop, model and thinking, cost, compaction3,682 + web 2,700base
    Subagents, one-shot agents, cross-session messagesnamed background agents, team, agent from scripts, conversations2,182base (owner: "awesome")
    History and handoffre-read earlier entries after compaction, hand off with a continuation533base
    Code modescripts calling tools as nested durable calls155 + Durablebase (small, Durable owns it)
    Shell commands!cmd and !!cmd in the conversation's environmentin chatbase, web too (TUI only today)
    Chat modepeople talk without a turn, @pi mentions876later
    Sharing and guestsshare links, guest names, revocation1,047later (the app kit's users and invites cover the start)
    Devices and owner loginowner links, device cookies684base, replaced by the app kit's sessions and pairing
    Models, credentials, provider loginpicker, scope, API keys, OAuth flows3,828base
    Settingssettings.json, hot reload87base
    Prompt resourcesAGENTS.md, skills, templates, read in the session's environment958base
    Workspacefiles, git status, diffs, search, review comments2,655 + web 1,341base (read and review; uploads later)
    File index and @ranked file completion242base
    Terminalsdetached PTYs, ghostty-vt in the web3,518later (channels, hands)
    Environments and locationsSSH environments, directory picker, agents moving themselves2,546base, redesigned as hands
    Artifacts and filesattachments, send_file, versioned artifacts with previews1,713attachments and send_file base (app kit Files); artifacts later
    Searchsidebar search over titles and messages (FTS5)775base over the index; message text search through the same sync mechanism later
    Status and presencebusy, tasks, usage, viewers, worker failures566base (feeds the index)
    Logs and noticesowner log viewer, notices487base
    Voicerooms, barge-in, speech service3,770 + Pythonlater (owner)
    Radius relay and port forwardsbrain WebSocket, up to 50 forwards2,000relay base (as the hands/sync link), forwards later
    Web shellPWA, sheets, navigation, reconnect9,252base, rebuilt on slice 11 ("look and feel great")
    TUIlauncher, sandbox, 40 commands, themes, keybindings6,720 + launcherbase on slice 11
    Dev and reloadsupervisor, dirty marks, reload tool468 + scriptsbase as the app kit's "apply changes"; the supervisor goes
    Fake provider, PI_MOBILE_* aliases, P2_DEV_RESUMEtest and legacy leftovers218drop
    tool_search, MCPnot installed in p2-later
    +

    Rough edges p3 should not inherit: features only in the TUI (shell commands, clone, fork, compact, new context), no +session delete, terminals and voice without a TUI side, docs that drifted from code (voice routes, TUI notes), the +plugin loader's module leak on reload (solved by Chord's plugin component).

    +
    + +
    II

    Hands

    Environments turned around: the machine connects to the server, not the other way.

    +

    What p2 does

    +

    p2's server and workers start a daemon (pi-env) over SSH on demand, one connection per session and per use (four +connection pools: each worker, the directory picker, the connection test, the terminal host). The daemon is +stateless and kills its commands 30 seconds after its connection goes quiet. Its frame protocol (exec, files, watch, +file listing, PTYs) does not depend on SSH.

    +

    What a hand is (proposed)

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    p2's daemonhand
    who dialsthe server, over SSHthe hand, over WSS, to the server or to Radius
    needssshd, a key, a network path to the machineoutbound HTTPS only
    lifetimeper connectiona service (systemd, launchd, Termux:Boot), reconnecting
    installdeployed and verified at each startinstalled once, signed updates
    machine identitySSH host keya hand key pair, enrolled once
    connectionsone per session and useone per hand and server, multiplexed
    +
      +
    • Enrollment: p3 hand enroll <server> shows a code; the owner approves it in the UI (the app kit's device +pairing); the server pins the hand's public key; fingerprints shown on both sides. Revocation in a Hands panel.
      • +
    • Reachability: a hand reaches the owner's own server directly (LAN, own tunnel) or through Radius, which relays +hands to whichever host drives a session (the upsell: no tunnel to set up; Radius can also offer sandboxed machines +and hands of its own).
      • +
    • Protocol: p2's daemon protocol over the WebSocket, unchanged in shape; two changes: several sessions share one +connection (cancellation, handles and backpressure per session), and a disconnect starts a grace period instead of +killing everything (detached commands and PTYs reattach on reconnect; this moves the terminal host's job into the +hand).
      • +
    • Binding: a session's location becomes { hand, cwd }; one server object holds the hand connections and the +HarnessEnvironments entry resolves a hand id to an execution environment; workers reach hands through the server's +links. An SSH machine stays possible as a hand kind the server dials.
      • +
    • Offline: an operation needing an offline hand fails with a clear error the model sees; the hand's status shows +in the session list.
      • +
    +
    + +
    III

    Session Sync and the Index

    What was decided yesterday, and the three gaps it leaves.

    +

    Decided

    +

    The worker updates the session index from events and sends it to the server; sessions sync to R2 through Radius: +Radius reports per session its last sequence number, pi uploads the commit log from there; R2 holds the SQLite file +and the commit log, a SQLite Durable Object applies them; sessions have an owner id; the index is a Chord table +replicated like a session, its row updated only on metadata, session idle and subagent idle changes.

    +

    Chord v2 has the pieces: a versioned outbox written in each commit's transaction, export archives, batch apply that +refuses gaps and divergence, export and import per address (app kit). Missing (the sync layer): upload and trim, +epochs, pull-back after a power loss, the split, starting from an archive.

    +

    Gap 1: who is the driver

    +

    Two machines both write session X (the laptop resumed it while Radius continued it). Both ask Radius for its last +seq, both upload from there; Radius applies whichever arrives first and refuses the other as "not following". The +copies have now diverged and nobody knows.

    +

    Proposed: a driver epoch per session, handed out by Radius. An upload names its epoch; an old epoch is refused; +the refused copy becomes a new session (Chord's rule S7: a split, never a rollback, because tools already ran). +"Continue on Radius" is an epoch change: the local driver waits for idle, uploads the tail and the session's files, +Radius bumps the epoch and starts the agent from its store, the local side becomes a replica or closes. Back is the +reverse.

    +

    Gap 2: two writers of the index

    +

    When a session continues on Radius, Radius writes its index row, and so does the local server.

    + + + + + + + + + + + + + + + + + + + + + + + +
    OptionHowCost
    a. one index per writereach host keeps its own index, Radius merges per owner, newest epoch wins per rowa second merge rule
    b. rows derived from session storesevery host computes a session's row from the session's own commitsno index sync at all; each host needs the session's commits
    c. rows tagged { epoch, seq }Radius keeps the highest per rowrows must carry their source position
    +

    Recommended: (b) where the session's commits are present (Radius has them all), with (c)'s tag on the rows a local +server writes for sessions it drives, so a list without the session store still sorts correctly.

    +

    Gap 3: what else moves

    +

    Files outside the store (uploads, sent files, the app kit's Files) must sync as blobs before the batches naming +them, R2 as their home. Model access and secrets for a session continued on Radius: Radius's own models, or the +owner's provider keys stored on Radius (decide). Local prompt resources (the device's AGENTS.md, skills) must be read +through a hand or synced. Closed sessions have no worker: the server reads a session's outbox without starting it, or +starts it to sync.

    +
    + +
    IV

    Tools Built In

    p2's 21 model-facing tools, and which ship with p3.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Toolp2 pluginProposed
    read, write, edit, bashcoding-tools (Durable)base
    powershellpowershellbase (deferred tool, Windows hands)
    codemodecodemodebase
    agent (from scripts)agentbase
    subagent, teamsubagentsbase
    conversationsconversationsbase
    history_search, history_read, history_handoffhistorybase
    location_envs, location_dirs, location_movelocationsbase, renamed around hands (hands, hand_dirs, move)
    rename_sessionsessionsbase
    send_filesend-filebase (on the app kit's Files)
    artifactartifactslater (owner: maybe not as useful)
    portsportslater (with forwards)
    reload_pluginsdev-toolsbase as an "apply changes" tool for the owner, not tied to a checkout
    tool_searchDurablelater (with MCP)
    +

    Every tool is a plugin in the app's own root, so any of them can be removed, replaced or added without touching base +code (the app kit's rule: base code is the API, bootstrap and a start-data facet).

    +
    + +
    V

    Running p3

    Starting the server, the CLI, themes and keybindings.

    +

    The CLI (proposed)

    +

    One binary, p3, and no environment variables (app kit decision); settings are arguments or files in the data +directory.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    CommandDoes
    p3 serve [--data-dir <dir>] [--host <addr>] [--port <n>]starts the server (startApp); prints the one-time owner link
    p3 owner-linkprints a new owner link for a running server (through the owner token file)
    p3 tui [url] [--pair]the sandboxed TUI; on the same machine it uses the owner token file, elsewhere device pairing
    p3 hand enroll <url>, p3 hand runenrolls this machine as a hand, runs it (or installs a service)
    p3 radius loginsigns the server into Radius (device code)
    `p3 install <git:...npm:...>`
    p3 export <session> <file>, p3 import <file>the app kit's export per address
    +

    The data directory defaults to ~/.p3. Its files: settings.json, models.json, auth.json (locked), +AGENTS.md, skills/, prompts/, plugins/ (the user's root), the server store, a directory per session worker, +owner-token, logs/. Plugin-specific configuration moves into plugin objects (documents) instead of more files.

    +

    Themes and keybindings

    +

    Themes follow the looks-and-themes proposal (decided 2026-11-02): design tokens as CSS variables on the web, roles in +the TUI, one Themes registry plugins contribute to, one dark and one light choice per user. Keybindings: commands are +registry entries with default keys (p2's TuiCommands with when conditions, 78 binding ids); the user's overrides +live in a preferences document (per user, synced like any data), not in files on each client; the web gets the same +command registry with its own key handling.

    +
    + +
    VI

    The Plugin API

    The smallest surface that still expresses everything the coding agent's extensions do.

    +

    The rule

    +

    Offer few things, each stable: what an API offers, it must keep (versioned in the import path, @p3/app/v1). Every +cluster below is a registry or a contract that already exists in Chord, Durable or p2; p3 v1 publishes those and +nothing else. Legacy concepts of the old extension API are not carried: CLI flags, an event bus, RPC-mode UI, hasUI +checks, "destructive switch" gates, pi.exec running locally regardless of the environment.

    +

    Seven clusters

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    ClusterHostMechanismCovers
    A. The agent's turnworkerDurable's Extension in HarnessExtensions: tools, beforeTool/afterTool, beforeRequest, sections, wraps, onYield, afterRun, beforeCompact20 examples: hello, permission-gate, protected-paths, tool-override, plan-mode, todo, custom-compaction, ...
    B. Conversation controlworker objecta Conversation contract: submit, configure, append (typed entries), compact, fork, rename; state: busy, agent, usagesend-user-message, file-trigger, session-name, preset, handoff, qna, ...
    C. InputworkerInputTransforms registry (rewrite or answer input before the model)input-transform, inline-bash
    D. Environments and modelsworker, serverHarnessEnvironments (ssh, gondolin, hands as kinds), HarnessModels (providers, virtual models), LoginProvidersssh, gondolin, custom providers, jev-router
    E. Asking the userserver object, UI renderersQuestions.ask from worker code; QuestionKinds per UI kindpermission-gate, question, questionnaire, timed-confirm, project-trust
    F. UItui, webregistries: Commands (slash, palette, keys), ToolRenderers, EntryRenderers, Slots (header, footer, status, widgets), overlays through the UI context, editor extension pointsevery UI-only example, the games
    G. State and standing behaviouranylocal objects owning natives, documents for statefile-trigger's watcher, gondolin's VM, game saves
    +
    // a plugin with a tool, a gate and a renderer: plugins/todo/
+// worker.ts
+self.contribute(HarnessExtensions, { name: "todo", tools: [todoTool], hooks: [hook(ToolTask, { beforeTool: guard })] });
+// tui.ts
+self.contribute(TuiToolRenderers, { todo: { call: (args) => todoCall(args), result: (r) => todoList(r.details) } });
+self.contribute(TuiCommands, { "todo.show": { slash: { name: "todos" }, run: (ui) => ui.overlay("todos", new TodoView(ui)) } });
+// web.tsx: the same registries with React components
    +

    The litmus test

    +

    All 78 examples map onto the seven clusters (table in /tmp/p3-prep/extensions.md). The examples that mix agent and +UI split into a worker facet and a tui/web facet sharing a contract file: a tool that wants a dialog asks the +Questions object, which works from any client, several clients or none.

    +
      +
    • Snake, Space Invaders, tic-tac-toe: a command plus an overlay in tui.ts and a canvas in web.tsx, engine in a +shared file, saves in a user-keyed document. Fit today.
      • +
    • Doom: fits with two fixes in the TUI: the sandbox bridge caps a frame at 64 KiB and a Doom frame is about 160 KB +(split writes at escape boundaries), and the sandbox has no files or network, so large plugin assets (a 4 MB WAD) +need a documented "asset of my plugin" path (chunked reads through the plugin's server facet).
      • +
    • Does not fit, by design: streaming 35 FPS frames from a worker to clients (objects carry state; needs channels), +engine state surviving an in-place reload (needs an explicit save), raw terminal ownership (interactive-shell +becomes a terminal view; terminal notifications need a launcher action).
      • +
    +

    What v1 leaves out

    +

    Editor replacement (extension points only), raw provider stream events and response headers, a fork hook, a +turn_start hook, an event bus. Each has one or two examples; add when a real plugin needs it.

    +
    + +
    VII

    Work Packages

    A first cut, in dependency order, for the meeting to rearrange.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    #PackageContainsDepends on
    1p3 skeletonstartApp with p3's API v1 modules (shared, web, tui, worker), core plugins in the app's root, p3 CLIChord slices 11-13
    2Sessions and conversationsDurable on the Chord store, session lifecycle with delete, the conversation contract, subagents, history and handoff, code mode, shell commands1
    3Session indexthe index table, worker-to-server updates on metadata and idle changes, the session list, search over it2
    4Session sync to Radiusoutbox upload and trim, epochs, pull-back, split, files as blobs, continue on Radius and back2, 3, Radius side
    5Radius sideper-session SQLite DO, commit log and SQLite on R2, epochs, the sync handshake, hand relaywith 4
    6Handsdaemon as an outbound service, enrollment by pairing, multiplexed protocol, reattach after disconnects, server hand pool, HarnessEnvironments kind, SSH as a hand kind1
    7Web shellp2's layout and flows rebuilt on slice 11 ("look and feel great"), themes1, slice 11
    8TUIp2's TUI on slice 11's sandbox, frame splitting, plugin assets path, keybindings and themes1, slice 11
    9Models and settingscatalog, credentials, provider login flows, settings, prompt resources1
    10Workspacefiles, git, diffs, search, review comments, file index and @, through hands6
    11Plugin API v1 and docsthe seven clusters published in @p3/app/v1, authoring guide, ported examples as the litmus (hello, permission-gate, todo, plan-mode, snake, doom)1, 2, 7, 8
    12Access and statusowner, devices and pairing (app kit), notices, logs, status and presence1
    +

    Later: voice, terminals (channels and hands), artifacts, port forwards, chat mode and guest sharing, MCP and +tool_search.

    +
    + +
    ∴

    Open Questions

    To decide in the meeting.

    +
      +
    1. The proposed cut: base versus later per area (first table).
      2. +
    2. Driver identity: epochs per session, handed out by Radius, with splits on conflict?
      3. +
    3. The index with two writers: derived rows plus { epoch, seq } tags, or one of the other options?
      4. +
    4. Who syncs a closed session (the server reading its outbox, or starting its worker)?
      5. +
    5. Files outside the store as blobs on R2, uploaded before their batch?
      6. +
    6. Models and secrets for sessions continued on Radius: Radius's own models, or the owner's keys stored there?
      7. +
    7. Hands: through Radius as a relay to the driver, and may one hand serve the local server and Radius at once?
      8. +
    8. Do hands keep commands and terminals alive across disconnects (absorbing p2's terminal host)?
      9. +
    9. Hand install and updates: signed binary, service install, version negotiation.
      10. +
    10. Is SSH kept as a hand kind for machines without a service?
      11. +
    11. Plugin API v1: the seven clusters as listed, and what is explicitly left out?
      12. +
    12. Work package owners and order.
      13. +
    +
    + +
    A

    Sources

    + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    SourceWhat
    /tmp/p3-prep/features.mdp2's feature areas with files, sizes, tools, rough edges; the CLI, environment variables and data files
    /tmp/p3-prep/extensions.mdall 78 extension examples, the capability matrix, the clusters with sketches, the hard cases
    /tmp/p3-prep/envs.mdp2's environments and Radius link, Chord's sync pieces, the sync and hands sketches, open questions
    apps/p2/docs/p3-functions/the functionality map (partly outdated)
    packages/chord/docs/next/design, decisions (2026-11-02 meeting summary), app kit and plugin model, theming
    +
    + +

    p3 Feature Set and Work Packages · meeting prep · 2026-11-03

    +
    + + From 95485d32f19d5ab9dca81bf2d7299661347d3238 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 16:15:54 +0000 Subject: [PATCH 160/175] docs(chord): startServer, exportData without stopping; p3 notes on subagent context and tree --- packages/chord/docs/next/decisions.md | 15 +++++++++++++++ packages/chord/docs/next/slices.md | 7 +++++++ 2 files changed, 22 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 314984ba5..1898085ce 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2336,6 +2336,21 @@ validation and copies the v2 kernel added around Delta. updated only on a session metadata change, a session idle-state change and a subagent idle-state change. p3: each host (server, worker, tui, web) has a kernel loading the plugins with contribution points via registries; p2's look and feel made great; new session storage; hands (machine daemons) good, secure and connecting proactively. +- **App kit naming and export (owner, 2026-11-02: "startApp > startServer?", "am i exporting an address? i'm exporting + the store a host created", "export should stop nothing before the export, it slots into the commit queue"; refines + slice 13):** `startApp` is `startServer` (the pair `startServer`/`startWorker`; an app always has one server). + `exportAddress`/`importAddress` are `exportData(address, path)`/`importData(path, address)`: the data a host created + (its store and files directory). Export stops nothing: a running worker exports its own store (`store.export()` holds + the commit queue while it reads, so the archive is one consistent point) and its finished files to the given path, + asked through a kit object on the worker; with no worker running, the server takes the worker's process lock (no + worker can start meanwhile) and reads the store itself. Import still takes the lock and refuses an address whose + directory exists (nothing is overwritten; replacing a session is delete, then import; a copy is an import under a new + address); registering the imported host in the app's index is the app's. +- **p3 notes, 2026-11-02 (owner):** subagents may start with the main agent's context: `context: "inherit"` forks the + parent conversation (Durable's `conversation.fork`) at the last entry before the assistant message with the spawning + call; model, tools and location may differ; the tool's description names the cost (prompt cache only with the same + model). pi's `/tree` (in-session branches) is wanted for p3: proposed as forks inside the session's store (edit an + earlier message to fork, a branch list to switch; a full navigator and branch summaries later). - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a diff --git a/packages/chord/docs/next/slices.md b/packages/chord/docs/next/slices.md index 972d721b6..a6472a274 100644 --- a/packages/chord/docs/next/slices.md +++ b/packages/chord/docs/next/slices.md @@ -165,6 +165,13 @@ Rules (owner, 2026-10-22): whoever defers something (this session, a subagent, a change; a slice's spec includes every entry of its section, and its build and review check them off; a done entry is deleted (the slice's spec and code are the record). Moved here from `carry-over.md` (owner, 2026-10-24). + +### Slice 11 (with the app kit's follow-ups) + +- **App kit renames and export** (decision "App kit naming and export"): `startApp` → `startServer`; + `exportAddress`/`importAddress` → `exportData`/`importData`; export without stopping the worker (a kit object on the + worker exports its store and finished files; without a worker the server takes the lock and reads the store); import + unchanged apart from the name. Examples 53-55, the app README and the guide's app kit chapter follow. ### Any slice (protocol and checks) - **Bump `PROTOCOL_VERSION`** for any message or reason change once p3 has shipped (slice 3b added `retry`, `loading` From 1f8604f84f968814fc84523557c01d704a6a8f83 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 16:47:09 +0000 Subject: [PATCH 161/175] docs(chord): sync applies batches with the host stopped --- packages/chord/docs/next/decisions.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 1898085ce..70bf483f2 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2351,6 +2351,11 @@ validation and copies the v2 kernel added around Delta. call; model, tools and location may differ; the tool's description names the cost (prompt cache only with the same model). pi's `/tree` (in-session branches) is wanted for p3: proposed as forks inside the session's store (edit an earlier message to fork, a branch list to switch; a full navigator and branch summaries later). +- **Sync applies batches with the host stopped (owner, 2026-11-02: "sure"; sync example `60-session-sync`):** + `applyBatch` refuses while a host holds claims on the store; the sync layer stops the session's worker, applies, + restarts it. Usually nothing runs: Radius receiving uploads runs no host for a session it does not drive, and taking + a session back pulls before its worker starts. A live view of a replica (watching a session continue elsewhere) is a + later refinement: fewer pulls while watched, or a read-only view without the worker. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 7dffeaf5213e17718ffc2a2dfb11420c4c989eb1 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 16:58:24 +0000 Subject: [PATCH 162/175] docs(chord): i18n proposal --- packages/chord/docs/next/i18n/book.md | 182 +++++++++++++ packages/chord/docs/next/i18n/index.html | 317 +++++++++++++++++++++++ 2 files changed, 499 insertions(+) create mode 100644 packages/chord/docs/next/i18n/book.md create mode 100644 packages/chord/docs/next/i18n/index.html diff --git a/packages/chord/docs/next/i18n/book.md b/packages/chord/docs/next/i18n/book.md new file mode 100644 index 000000000..bcc275cc2 --- /dev/null +++ b/packages/chord/docs/next/i18n/book.md @@ -0,0 +1,182 @@ +--- +title: Speaking Every Language: i18n in the app kit +cover: i18n / in the App Kit +kicker: A proposal · Chord v2 app kit · 2026-11-02 +subtitle: How an app and its plugins write their texts once, in code, and show them in each viewer's language +stats: + - 1 | function a plugin calls: t(key, ...args) + - 0 | message files outside the code + - 0 | page reloads to switch a language +note: | + A proposal by the redesign session, asked for by the owner on 2026-11-02 ("i18n also needs to be part of the app + kit"; "need code example"). It starts from mini-lit's `src/i18n.ts` (typed catalogs in code) and adapts it to + plugins that are built separately, reload live and run on several hosts. Not built; it would land in the app kit + with slice 11. +colophon: Speaking Every Language · proposal · 2026-11-02 +--- + +# Before we begin {.preface} + +Lena uses p3 in German; her colleague Tom, a guest in her session, reads English. The notes plugin shows "New note" +to Tom and "Neue Notiz" to Lena, on the same page served by the same server. A French translator later adds French to +the app and to every plugin, without touching any of them. And when Lena switches to French, nothing reloads: the +text on her screen changes, the half-written message stays. + +mini-lit already does the first part well: messages are TypeScript objects, keyed and type-checked, so a missing +German text is a compile error. Three things do not carry over: its messages are one global table extended by +declaration merging (which separately built plugins cannot share), the language lives in `localStorage` and switching +reloads the page, and every text is formatted where it is shown, which breaks when the server formats a message for +two viewers with two languages. + +## Five things to know + +:::findings +1. **Messages stay in code,** per plugin: `defineMessages("notes", { en: {...}, de: {...} })` returns a typed `t`. +2. **Values may be functions,** for plurals and parameters; dates and numbers go through `Intl`. +3. **Languages are contributions:** a `Translations` registry adds a language to any plugin, the app included. +4. **The language is a user preference,** from the browser or the system locale by default; switching re-renders live. +5. **Servers and workers never format text for people:** they send a key and its parameters, and each viewer's client + renders them in its own language. +::: + +## Words used throughout + +A **message** is one text with a key (`newNote`). A **catalog** is the messages of one plugin in one language. A +**namespace** is the plugin a catalog belongs to. A **locale** is a language tag (`de`, `pt-BR`). + +# Messages in Code {#messages lede="Each plugin defines its own messages and gets a typed function back."} + +```ts +// plugins/notes/messages.ts +import { defineMessages } from "@p3/app/v1"; + +export const t = defineMessages("notes", { + en: { + newNote: "New note", + empty: "No notes yet", + count: (n: number) => (n === 1 ? "1 note" : `${n} notes`), + saved: (title: string, at: Date) => `Saved "${title}" at ${formatTime(at)}`, + }, + de: { + newNote: "Neue Notiz", + empty: "Noch keine Notizen", + count: (n) => (n === 1 ? "1 Notiz" : `${n} Notizen`), + saved: (title, at) => `„${title}“ um ${formatTime(at)} gespeichert`, + }, +}); +``` + +```tsx +// plugins/notes/web.tsx +import { t } from "./messages.ts"; + +function NotesHeader({ count }: { count: number }) { + return

    {t("count", count)}

    ; +} +``` + +The first catalog (`en`) defines the keys and their argument types; every other catalog must have the same keys with +the same signatures, so a missing German text, a typo in a key or a wrong argument is a type error, not a blank in +production. `t("count", "3")` does not compile. The namespace (`"notes"`) keeps two plugins' `empty` apart. + +Functions cover what plain strings cannot: plurals (`count`), parameters in the right word order (`saved` puts the +title first in English and German alike, but another language may not), and gender or case where a language needs it. +Plural rules beyond one and many use `Intl.PluralRules`; dates, times and numbers use `Intl.DateTimeFormat` and +`Intl.NumberFormat` with the viewer's locale, through two small helpers (`formatTime`, `formatNumber`) that read the +current locale like `t`. + +# Languages as Contributions {#translations lede="Adding French to the app and every plugin without editing them."} + +A plugin ships the languages its author speaks. Everything else can come from anyone, as a contribution: + +```ts +// plugins/french/web.ts (a translation plugin) +import { defineFacet, Translations } from "@p3/app/v1"; +import type { MessagesOf } from "@p3/app/v1"; +import type { t as notes } from "../notes/messages.ts"; // the notes plugin's keys, for the type check + +const notesFr: MessagesOf = { + newNote: "Nouvelle note", + empty: "Aucune note", + count: (n) => (n <= 1 ? `${n} note` : `${n} notes`), + saved: (title, at) => `« ${title} » enregistrée à ${formatTime(at)}`, +}; + +export default defineFacet((_host, self) => { + self.contribute(Translations, { namespace: "notes", locale: "fr", messages: notesFr }); + self.contribute(Translations, { namespace: "app", locale: "fr", messages: appFr }); +}); +``` + +- `MessagesOf` gives a translation plugin the exact keys and signatures to fill, from the plugin it + translates (an imported type, erased at build). +- A catalog may be partial: a missing key falls back to the plugin's own catalog for that language's base (`pt-BR` to + `pt`), then to the first catalog (`en`), key by key. A fallback is reported once in the logs, so gaps are visible. +- Two contributions for the same namespace and locale: the first by plugin name wins, the other is reported (the + registry rule for conflicts). +- A plugin's own catalogs are its built-in contributions; a translation plugin can also override one of them (a + better German), contributed explicitly with `override: true`. + +# Choosing a Language {#locale lede="Per viewer, defaulted from the device, switched live."} + +The language is a user preference, kept with the user's other preferences (the kit's per-principal document, like +the theme choice), so it follows a person across devices: + +- **Default:** the browser's `navigator.languages` on the web, the system locale in the TUI, matched against the + locales any catalog offers. +- **Switching:** the app's settings show the locales available (the union of all catalogs); choosing one writes the + preference. `t` reads the current locale from a watchable; React components re-render through a hook, the TUI + redraws through its scope. Nothing reloads. + +```tsx +// any web component +const { t, locale } = useMessages(notesMessages); // re-renders when the locale or a translation changes +``` + +When a translation plugin is edited and rolled out, the registry recomposes and the open pages show the new text, as +with themes. + +# Text from the Server {#server lede="Why the server sends keys, not sentences."} + +The server and the workers also produce text people read: a notice ("Worker failed to start"), a tool card's +summary, a status line, an error. If the worker formatted "Notiz gespeichert" for Lena, Tom would read German too: +both look at the same object state. + +So objects carry messages as data, and each client renders them: + +```ts +// plugins/notes/worker.ts: state and notices carry a key and its parameters +self.notify({ message: { namespace: "notes", key: "saved", params: [note.title, new Date(note.at)] } }); + +// plugins/notes/web.tsx: rendered in the viewer's language +const text = t.render(notice.message); // typed against the notes catalog +``` + +Parameters are plain data (strings, numbers, ISO dates), so they cross links like any state. What is never +translated: the model's prompts and the model's own answers (the conversation's language is the user's business with +the model), log lines (for developers, in English), error codes. Error *messages* shown to people follow the same rule +as notices: a code plus parameters, rendered by the client. + +# Open Questions {.epilogue lede="To decide."} + +1. **Keys or English text as keys.** mini-lit keys by the English text (`"Copy"`), which reads well and needs no key + names; named keys (`newNote`) survive rewording the English. Recommendation: named keys, with the English catalog + as the source of truth. +2. **A file format for translators.** Code catalogs are best for developers and agents; human translators may want + JSON or PO files. Recommendation: no files now; a small tool can export a plugin's catalogs to JSON and import them + back into a translation plugin when someone needs it. +3. **Right-to-left languages.** The page sets `dir` from the locale; the app's CSS uses logical properties + (`margin-inline-start`). Recommendation: the kit sets `lang` and `dir` on the root; components are the app's. +4. **The TUI's locale.** From the system locale, overridden by the user's preference, like the web. + +# API in Brief {.appendix} + +| Piece | Where | What | +| --- | --- | --- | +| `defineMessages(namespace, catalogs)` | the app's API (shared) | typed `t(key, ...args)`, `t.render(message)` | +| `MessagesOf` | the app's API (type) | the keys and signatures a translation must fill | +| `Translations` | the app's API (shared registry) | `{ namespace, locale, messages, override? }` | +| `useMessages(t)` | the app's web module | the hook re-rendering on locale or translation changes | +| `formatTime`, `formatNumber` | the app's API | `Intl` helpers in the viewer's locale | +| the locale preference | the kit's per-principal preferences | default from the device, switched live | +| `{ namespace, key, params }` | object state and notices | a message as data, rendered by each client | diff --git a/packages/chord/docs/next/i18n/index.html b/packages/chord/docs/next/i18n/index.html new file mode 100644 index 000000000..f635243ae --- /dev/null +++ b/packages/chord/docs/next/i18n/index.html @@ -0,0 +1,317 @@ + + +Speaking Every Language: i18n in the app kit + + + + +
    +
    +
    A proposal · Chord v2 app kit · 2026-11-02
    +

    i18n
    in the App Kit

    +

    How an app and its plugins write their texts once, in code, and show them in each viewer's language

    +
    1function a plugin calls: t(key, ...args)
    0message files outside the code
    0page reloads to switch a language
    +

    A proposal by the redesign session, asked for by the owner on 2026-11-02 ("i18n also needs to be part of the app +kit"; "need code example"). It starts from mini-lit's src/i18n.ts (typed catalogs in code) and adapts it to +plugins that are built separately, reload live and run on several hosts. Not built; it would land in the app kit +with slice 11.

    +
    +
    + + +
    +

    Before we begin

    +

    Lena uses p3 in German; her colleague Tom, a guest in her session, reads English. The notes plugin shows "New note" +to Tom and "Neue Notiz" to Lena, on the same page served by the same server. A French translator later adds French to +the app and to every plugin, without touching any of them. And when Lena switches to French, nothing reloads: the +text on her screen changes, the half-written message stays.

    +

    mini-lit already does the first part well: messages are TypeScript objects, keyed and type-checked, so a missing +German text is a compile error. Three things do not carry over: its messages are one global table extended by +declaration merging (which separately built plugins cannot share), the language lives in localStorage and switching +reloads the page, and every text is formatted where it is shown, which breaks when the server formats a message for +two viewers with two languages.

    +

    Five things to know

    +
      +
    1. Messages stay in code, per plugin: defineMessages("notes", { en: {...}, de: {...} }) returns a typed t.
      2. +
    2. Values may be functions, for plurals and parameters; dates and numbers go through Intl.
      3. +
    3. Languages are contributions: a Translations registry adds a language to any plugin, the app included.
      4. +
    4. The language is a user preference, from the browser or the system locale by default; switching re-renders live.
      5. +
    5. Servers and workers never format text for people: they send a key and its parameters, and each viewer's client +renders them in its own language.
      6. +
    +

    Words used throughout

    +

    A message is one text with a key (newNote). A catalog is the messages of one plugin in one language. A +namespace is the plugin a catalog belongs to. A locale is a language tag (de, pt-BR).

    +
    + +
    I

    Messages in Code

    Each plugin defines its own messages and gets a typed function back.

    +
    // plugins/notes/messages.ts
+import { defineMessages } from "@p3/app/v1";
+
+export const t = defineMessages("notes", {
+	en: {
+		newNote: "New note",
+		empty: "No notes yet",
+		count: (n: number) => (n === 1 ? "1 note" : `${n} notes`),
+		saved: (title: string, at: Date) => `Saved "${title}" at ${formatTime(at)}`,
+	},
+	de: {
+		newNote: "Neue Notiz",
+		empty: "Noch keine Notizen",
+		count: (n) => (n === 1 ? "1 Notiz" : `${n} Notizen`),
+		saved: (title, at) => `„${title}“ um ${formatTime(at)} gespeichert`,
+	},
+});
    +
    // plugins/notes/web.tsx
+import { t } from "./messages.ts";
+
+function NotesHeader({ count }: { count: number }) {
+	return <h2>{t("count", count)} <button>{t("newNote")}</button></h2>;
+}
    +

    The first catalog (en) defines the keys and their argument types; every other catalog must have the same keys with +the same signatures, so a missing German text, a typo in a key or a wrong argument is a type error, not a blank in +production. t("count", "3") does not compile. The namespace ("notes") keeps two plugins' empty apart.

    +

    Functions cover what plain strings cannot: plurals (count), parameters in the right word order (saved puts the +title first in English and German alike, but another language may not), and gender or case where a language needs it. +Plural rules beyond one and many use Intl.PluralRules; dates, times and numbers use Intl.DateTimeFormat and +Intl.NumberFormat with the viewer's locale, through two small helpers (formatTime, formatNumber) that read the +current locale like t.

    +
    + +
    II

    Languages as Contributions

    Adding French to the app and every plugin without editing them.

    +

    A plugin ships the languages its author speaks. Everything else can come from anyone, as a contribution:

    +
    // plugins/french/web.ts (a translation plugin)
+import { defineFacet, Translations } from "@p3/app/v1";
+import type { MessagesOf } from "@p3/app/v1";
+import type { t as notes } from "../notes/messages.ts";   // the notes plugin's keys, for the type check
+
+const notesFr: MessagesOf<typeof notes> = {
+	newNote: "Nouvelle note",
+	empty: "Aucune note",
+	count: (n) => (n <= 1 ? `${n} note` : `${n} notes`),
+	saved: (title, at) => `« ${title} » enregistrée à ${formatTime(at)}`,
+};
+
+export default defineFacet((_host, self) => {
+	self.contribute(Translations, { namespace: "notes", locale: "fr", messages: notesFr });
+	self.contribute(Translations, { namespace: "app", locale: "fr", messages: appFr });
+});
    +
      +
    • MessagesOf<typeof t> gives a translation plugin the exact keys and signatures to fill, from the plugin it +translates (an imported type, erased at build).
      • +
    • A catalog may be partial: a missing key falls back to the plugin's own catalog for that language's base (pt-BR to +pt), then to the first catalog (en), key by key. A fallback is reported once in the logs, so gaps are visible.
      • +
    • Two contributions for the same namespace and locale: the first by plugin name wins, the other is reported (the +registry rule for conflicts).
      • +
    • A plugin's own catalogs are its built-in contributions; a translation plugin can also override one of them (a +better German), contributed explicitly with override: true.
      • +
    +
    + +
    III

    Choosing a Language

    Per viewer, defaulted from the device, switched live.

    +

    The language is a user preference, kept with the user's other preferences (the kit's per-principal document, like +the theme choice), so it follows a person across devices:

    +
      +
    • Default: the browser's navigator.languages on the web, the system locale in the TUI, matched against the +locales any catalog offers.
      • +
    • Switching: the app's settings show the locales available (the union of all catalogs); choosing one writes the +preference. t reads the current locale from a watchable; React components re-render through a hook, the TUI +redraws through its scope. Nothing reloads.
      • +
    +
    // any web component
+const { t, locale } = useMessages(notesMessages);   // re-renders when the locale or a translation changes
    +

    When a translation plugin is edited and rolled out, the registry recomposes and the open pages show the new text, as +with themes.

    +
    + +
    IV

    Text from the Server

    Why the server sends keys, not sentences.

    +

    The server and the workers also produce text people read: a notice ("Worker failed to start"), a tool card's +summary, a status line, an error. If the worker formatted "Notiz gespeichert" for Lena, Tom would read German too: +both look at the same object state.

    +

    So objects carry messages as data, and each client renders them:

    +
    // plugins/notes/worker.ts: state and notices carry a key and its parameters
+self.notify({ message: { namespace: "notes", key: "saved", params: [note.title, new Date(note.at)] } });
+
+// plugins/notes/web.tsx: rendered in the viewer's language
+const text = t.render(notice.message);   // typed against the notes catalog
    +

    Parameters are plain data (strings, numbers, ISO dates), so they cross links like any state. What is never +translated: the model's prompts and the model's own answers (the conversation's language is the user's business with +the model), log lines (for developers, in English), error codes. Error messages shown to people follow the same rule +as notices: a code plus parameters, rendered by the client.

    +
    + +
    ∴

    Open Questions

    To decide.

    +
      +
    1. Keys or English text as keys. mini-lit keys by the English text ("Copy"), which reads well and needs no key +names; named keys (newNote) survive rewording the English. Recommendation: named keys, with the English catalog +as the source of truth.
      2. +
    2. A file format for translators. Code catalogs are best for developers and agents; human translators may want +JSON or PO files. Recommendation: no files now; a small tool can export a plugin's catalogs to JSON and import them +back into a translation plugin when someone needs it.
      3. +
    3. Right-to-left languages. The page sets dir from the locale; the app's CSS uses logical properties +(margin-inline-start). Recommendation: the kit sets lang and dir on the root; components are the app's.
      4. +
    4. The TUI's locale. From the system locale, overridden by the user's preference, like the web.
      5. +
    +
    + +
    A

    API in Brief

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    PieceWhereWhat
    defineMessages(namespace, catalogs)the app's API (shared)typed t(key, ...args), t.render(message)
    MessagesOf<typeof t>the app's API (type)the keys and signatures a translation must fill
    Translationsthe app's API (shared registry){ namespace, locale, messages, override? }
    useMessages(t)the app's web modulethe hook re-rendering on locale or translation changes
    formatTime, formatNumberthe app's APIIntl helpers in the viewer's locale
    the locale preferencethe kit's per-principal preferencesdefault from the device, switched live
    { namespace, key, params }object state and noticesa message as data, rendered by each client
    +
    + +

    Speaking Every Language · proposal · 2026-11-02

    +
    + + From 48f9f1b423bea5275c96274f3442a805e0c57c17 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 17:03:46 +0000 Subject: [PATCH 163/175] docs(chord): i18n proposal, messages as object state --- packages/chord/docs/next/i18n/book.md | 17 +++++++++++++---- packages/chord/docs/next/i18n/index.html | 16 ++++++++++++---- 2 files changed, 25 insertions(+), 8 deletions(-) diff --git a/packages/chord/docs/next/i18n/book.md b/packages/chord/docs/next/i18n/book.md index bcc275cc2..32a268354 100644 --- a/packages/chord/docs/next/i18n/book.md +++ b/packages/chord/docs/next/i18n/book.md @@ -145,13 +145,22 @@ both look at the same object state. So objects carry messages as data, and each client renders them: ```ts -// plugins/notes/worker.ts: state and notices carry a key and its parameters -self.notify({ message: { namespace: "notes", key: "saved", params: [note.title, new Date(note.at)] } }); +// plugins/notes/contract.ts: a member of the Notes object's state holds a message as data +lastSaved: opt(obj({ namespace: str(), key: str(), params: arr(unknown()) })), -// plugins/notes/web.tsx: rendered in the viewer's language -const text = t.render(notice.message); // typed against the notes catalog +// plugins/notes/worker.ts: the object writes the key and its parameters, never a sentence +await self.tx(async (tx) => { + (await tx.draft(status)).lastSaved = { namespace: "notes", key: "saved", params: [note.title, note.at] }; +}, ctx); + +// plugins/notes/web.tsx: each viewer renders it in its own language +const notes = useHold(Notes, { session }); +const text = notes.lastSaved && t.render(notes.lastSaved); // typed against the notes catalog ``` +The same holds for anything else that reaches people: an app's notices object stores `{ namespace, key, params }` +per notice, a tool card's summary carries one, an error shown to a user carries a code and parameters. + Parameters are plain data (strings, numbers, ISO dates), so they cross links like any state. What is never translated: the model's prompts and the model's own answers (the conversation's language is the user's business with the model), log lines (for developers, in English), error codes. Error *messages* shown to people follow the same rule diff --git a/packages/chord/docs/next/i18n/index.html b/packages/chord/docs/next/i18n/index.html index f635243ae..b568aa8a7 100644 --- a/packages/chord/docs/next/i18n/index.html +++ b/packages/chord/docs/next/i18n/index.html @@ -239,11 +239,19 @@ with themes.

    summary, a status line, an error. If the worker formatted "Notiz gespeichert" for Lena, Tom would read German too: both look at the same object state.

    So objects carry messages as data, and each client renders them:

    -
    // plugins/notes/worker.ts: state and notices carry a key and its parameters
-self.notify({ message: { namespace: "notes", key: "saved", params: [note.title, new Date(note.at)] } });
+
    // plugins/notes/contract.ts: a member of the Notes object's state holds a message as data
+lastSaved: opt(obj({ namespace: str(), key: str(), params: arr(unknown()) })),
 
-// plugins/notes/web.tsx: rendered in the viewer's language
-const text = t.render(notice.message);   // typed against the notes catalog
    
+// plugins/notes/worker.ts: the object writes the key and its parameters, never a sentence
+await self.tx(async (tx) => {
+	(await tx.draft(status)).lastSaved = { namespace: "notes", key: "saved", params: [note.title, note.at] };
+}, ctx);
+
+// plugins/notes/web.tsx: each viewer renders it in its own language
+const notes = useHold(Notes, { session });
+const text = notes.lastSaved && t.render(notes.lastSaved);   // typed against the notes catalog
    +

    The same holds for anything else that reaches people: an app's notices object stores { namespace, key, params } +per notice, a tool card's summary carries one, an error shown to a user carries a code and parameters.

    Parameters are plain data (strings, numbers, ISO dates), so they cross links like any state. What is never translated: the model's prompts and the model's own answers (the conversation's language is the user's business with the model), log lines (for developers, in English), error codes. Error messages shown to people follow the same rule From 1746cfd51556c15096b3a27d890f7bb3c3b395b0 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 17:09:59 +0000 Subject: [PATCH 164/175] docs(chord): i18n proposal, t.message and one file per host --- packages/chord/docs/next/i18n/book.md | 29 ++++++++++++++++++++++-- packages/chord/docs/next/i18n/index.html | 22 ++++++++++++++++-- 2 files changed, 47 insertions(+), 4 deletions(-) diff --git a/packages/chord/docs/next/i18n/book.md b/packages/chord/docs/next/i18n/book.md index 32a268354..b431cb030 100644 --- a/packages/chord/docs/next/i18n/book.md +++ b/packages/chord/docs/next/i18n/book.md @@ -117,6 +117,24 @@ export default defineFacet((_host, self) => { - A plugin's own catalogs are its built-in contributions; a translation plugin can also override one of them (a better German), contributed explicitly with `override: true`. +## One file, every host + +Registries are per host: a contribution lives on the host whose facet made it. A French catalog contributed from the +web facet exists in browser pages only. So a plugin keeps its messages in one shared file, `messages.ts`, which every +facet that renders text imports (the web and TUI facets; the worker's too when it builds messages with `t.message`), +and a translation plugin keeps its catalog in one shared file too and contributes it from each facet that renders: + +```ts +// plugins/french/fr.ts: the catalogs, written once +export const notesFr: MessagesOf = { newNote: "Nouvelle note", /* ... */ }; +// plugins/french/web.ts and plugins/french/tui.ts: the same two lines each +export default defineFacet((_host, self) => self.contribute(Translations, { namespace: "notes", locale: "fr", messages: notesFr })); +``` + +The server and workers need catalogs only when they produce final text themselves (an email, a push notification); +then their facets contribute them too, and the text is rendered in the recipient's language from their preference. +Catalogs never travel over links: they contain functions. + # Choosing a Language {#locale lede="Per viewer, defaulted from the device, switched live."} The language is a user preference, kept with the user's other preferences (the kit's per-principal document, like @@ -148,9 +166,10 @@ So objects carry messages as data, and each client renders them: // plugins/notes/contract.ts: a member of the Notes object's state holds a message as data lastSaved: opt(obj({ namespace: str(), key: str(), params: arr(unknown()) })), -// plugins/notes/worker.ts: the object writes the key and its parameters, never a sentence +// plugins/notes/worker.ts: the object writes the key and its parameters, never a sentence; the same `t` builds it +import { t } from "./messages.ts"; await self.tx(async (tx) => { - (await tx.draft(status)).lastSaved = { namespace: "notes", key: "saved", params: [note.title, note.at] }; + (await tx.draft(status)).lastSaved = t.message("saved", note.title, note.at); // typed: key and arguments checked }, ctx); // plugins/notes/web.tsx: each viewer renders it in its own language @@ -158,6 +177,12 @@ const notes = useHold(Notes, { session }); const text = notes.lastSaved && t.render(notes.lastSaved); // typed against the notes catalog ``` +`t.message` returns a `Message<"notes">`, plain data `{ namespace, key, params }`, typed against the first catalog: +`t.message("saved", 42)` and `t.message("svaed", ...)` are type errors. `t.render` takes only `Message<"notes">`. Across a +link the contract's member is a generic message schema, so `t.render` checks the namespace and key at runtime and +shows the key for an unknown one. The writer and the reader are type-checked; the hop between them is checked at +runtime. + The same holds for anything else that reaches people: an app's notices object stores `{ namespace, key, params }` per notice, a tool card's summary carries one, an error shown to a user carries a code and parameters. diff --git a/packages/chord/docs/next/i18n/index.html b/packages/chord/docs/next/i18n/index.html index b568aa8a7..7a2fb4888 100644 --- a/packages/chord/docs/next/i18n/index.html +++ b/packages/chord/docs/next/i18n/index.html @@ -216,6 +216,18 @@ registry rule for conflicts).

  • A plugin's own catalogs are its built-in contributions; a translation plugin can also override one of them (a better German), contributed explicitly with override: true.
    • +

    One file, every host

    +

    Registries are per host: a contribution lives on the host whose facet made it. A French catalog contributed from the +web facet exists in browser pages only. So a plugin keeps its messages in one shared file, messages.ts, which every +facet that renders text imports (the web and TUI facets; the worker's too when it builds messages with t.message), +and a translation plugin keeps its catalog in one shared file too and contributes it from each facet that renders:

    +
    // plugins/french/fr.ts: the catalogs, written once
+export const notesFr: MessagesOf<typeof notes> = { newNote: "Nouvelle note", /* ... */ };
+// plugins/french/web.ts and plugins/french/tui.ts: the same two lines each
+export default defineFacet((_host, self) => self.contribute(Translations, { namespace: "notes", locale: "fr", messages: notesFr }));
    +

    The server and workers need catalogs only when they produce final text themselves (an email, a push notification); +then their facets contribute them too, and the text is rendered in the recipient's language from their preference. +Catalogs never travel over links: they contain functions.

    III

    Choosing a Language

    Per viewer, defaulted from the device, switched live.

    @@ -242,14 +254,20 @@ both look at the same object state.

    // plugins/notes/contract.ts: a member of the Notes object's state holds a message as data
 lastSaved: opt(obj({ namespace: str(), key: str(), params: arr(unknown()) })),
 
-// plugins/notes/worker.ts: the object writes the key and its parameters, never a sentence
+// plugins/notes/worker.ts: the object writes the key and its parameters, never a sentence; the same `t` builds it
+import { t } from "./messages.ts";
 await self.tx(async (tx) => {
-	(await tx.draft(status)).lastSaved = { namespace: "notes", key: "saved", params: [note.title, note.at] };
+	(await tx.draft(status)).lastSaved = t.message("saved", note.title, note.at);   // typed: key and arguments checked
 }, ctx);
 
 // plugins/notes/web.tsx: each viewer renders it in its own language
 const notes = useHold(Notes, { session });
 const text = notes.lastSaved && t.render(notes.lastSaved);   // typed against the notes catalog
    +

    t.message returns a Message<"notes">, plain data { namespace, key, params }, typed against the first catalog: +t.message("saved", 42) and t.message("svaed", ...) are type errors. t.render takes only Message<"notes">. Across a +link the contract's member is a generic message schema, so t.render checks the namespace and key at runtime and +shows the key for an unknown one. The writer and the reader are type-checked; the hop between them is checked at +runtime.

    The same holds for anything else that reaches people: an app's notices object stores { namespace, key, params } per notice, a tool card's summary carries one, an error shown to a user carries a code and parameters.

    Parameters are plain data (strings, numbers, ISO dates), so they cross links like any state. What is never From 642e30b68930a644050794011555a04ed9027eb4 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 17:18:31 +0000 Subject: [PATCH 165/175] docs(chord): p3 meeting prep with pictures, sync example, branches --- .../chord/docs/next/p3-features/README.md | 6 + packages/chord/docs/next/p3-features/book.md | 159 +++- .../chord/docs/next/p3-features/index.html | 832 ------------------ 3 files changed, 160 insertions(+), 837 deletions(-) create mode 100644 packages/chord/docs/next/p3-features/README.md delete mode 100644 packages/chord/docs/next/p3-features/index.html diff --git a/packages/chord/docs/next/p3-features/README.md b/packages/chord/docs/next/p3-features/README.md new file mode 100644 index 000000000..c7fd1d2e0 --- /dev/null +++ b/packages/chord/docs/next/p3-features/README.md @@ -0,0 +1,6 @@ +# p3 features (meeting prep) + +`book.md` references screenshots and clips as `{{img:|caption}}` / `{{clip:|caption}}`; they were captured +from p2 on 2026-11-02 (kept outside the repository: `/tmp/p3-prep/shots/` on slayer). Build: inline them with +`/tmp/p3-doc/inline.py book.md book.inlined.md`, then the explainer build. Published privately on Radius +(01m6efr04nezms4zx32p7pp0w1). diff --git a/packages/chord/docs/next/p3-features/book.md b/packages/chord/docs/next/p3-features/book.md index 4d45c3c26..3bdfba676 100644 --- a/packages/chord/docs/next/p3-features/book.md +++ b/packages/chord/docs/next/p3-features/book.md @@ -2,7 +2,7 @@ title: p3: Feature Set and Work Packages (prep for the design meeting) cover: p3 / Features kicker: Meeting prep · p3 on Chord v2 · 2026-11-03 -subtitle: What p2 does today, what p3 should ship first, how hands, sync and the plugin API fit, and a first cut into work packages +subtitle: What p2 does today (in pictures), what p3 should ship first, how hands, sync and the plugin API fit, and a first cut into work packages stats: - 28 | feature areas in p2 today - 21 | model-facing tools in p2 @@ -61,6 +61,86 @@ files, watching, terminals). A **driver** is the single host allowed to write a **replica** follows its commits. An **epoch** numbers driver changes. The **index** is the table of session metadata (title, status, idle state) the session list reads. The **app kit** is Chord's `startApp`/`startWorker` layer. +# p2 in Pictures {#pictures lede="What each feature area looks like today, on the web, the phone and in the terminal."} + +All content is invented (a small shop project, the owner Dana, a guest), the models are a fake provider's, tools ran +for real on a demo repository. Stills are from the web on a desktop and a phone and from the TUI; the clips loop. + +## Sessions + +{{img:sessions-list-desktop.webp|The sidebar: sessions grouped by directory, subagent counts, footer buttons.}} +{{img:sessions-drawer-phone.webp|The drawer on the phone, a subagent expanded.}} {{img:sessions-create-phone.webp|A new session on the phone.}} +{{img:sessions-create-directory-picker.webp|The directory picker for a new session: environment, path, folders, recent.}} +{{img:sessions-options-sheet.webp|The session sheet: title, share, working directory, pin, archive.}} +{{img:tui-session-selector.webp|The TUI's session selector, subagents threaded under their session.}} + +## Conversations and the agent + +{{clip:clip-streaming.mp4|Sending a question: thinking, a read card, a table and code stream in.}} +{{img:conversation-thinking.webp|A turn with thinking expanded and read, write and edit cards.}} +{{img:conversation-edit-diff-card.webp|An edit card with its diff.}} +{{img:conversation-bash-card.webp|A bash card with its output, then the answer.}} +{{img:conversation-streaming.webp|While an answer streams: the busy dot, the Steer composer, the stop button.}} +{{img:conversation-transcript-phone.webp|The transcript on the phone.}} {{img:conversation-model-picker-phone.webp|The model sheet on the phone.}} +{{img:conversation-cost-sheet.webp|The cost sheet: totals, cost and cache hits per response.}} +{{img:tui-transcript-tools.webp|The same agent in the TUI: thinking, tools, the answer, editor and footer.}} + +## Subagents, history and handoff, code mode + +{{img:subagents-spawn-and-report.webp|A spawned subagent in the parent's transcript, and its report.}} +{{img:subagents-conversation.webp|The subagent's own conversation, selected in the sidebar.}} +{{img:history-handoff-card.webp|A handoff: the continuation text, then a new context.}} +{{img:history-search-card.webp|history_search with its hits.}} +{{img:codemode-card.webp|A code mode script with three nested tool calls.}} + +## Shell commands and chat mode + +{{img:shell-command-tui.webp|A shell command run from the TUI with its output (the web can only show it).}} +{{img:chatmode-two-people.webp|Chat mode: owner and guest talk, an @pi mention gets a digest and an answer.}} + +## Sharing, devices, providers + +{{img:sharing-sheet.webp|The share sheet: read-only and read/write links, a connected guest.}} +{{img:sharing-join-page.webp|Joining through a share link.}} {{img:sharing-guest-view-phone.webp|The session as the guest sees it.}} +{{img:devices-sheet.webp|The devices sheet (the one-time token masked).}} +{{img:providers-login.webp|A provider's login methods.}} + +## Workspace, @ completion, terminals + +{{clip:clip-workspace-review.mp4|Changes, a diff, a line comment, Send all; the agent answers the review.}} +{{img:workspace-files.webp|Files: tree, sizes, upload.}} +{{img:workspace-changes.webp|Changes: branch, uncommitted files, commits.}} +{{img:workspace-search.webp|Text search across the project.}} +{{clip:clip-at-completion.mp4|@ completion in the composer.}} +{{img:terminals-web.webp|A web terminal panel with tabs and a key row.}} +{{img:tui-slash-changes-diff.webp|The TUI's /changes diff (read-only).}} + +## Environments, artifacts, attachments, search + +{{img:environments-panel.webp|The environments sheet: SSH environments and the ports entry.}} +{{img:artifacts-preview.webp|An artifact in the sandboxed preview.}} +{{img:attachments-message.webp|A message with an image attachment.}} +{{img:search-results.webp|Sidebar search: message hits for "rounding".}} + +## Status, notices, logs, voice, ports + +{{img:status-footer-viewers.webp|The composer footer: model, viewers, cache hit rate, cost, context.}} +{{img:notices-owner-notice.webp|An owner notice after a worker was killed, with Show logs.}} +{{img:logs-sheet.webp|The logs sheet following the server's log.}} +{{img:voice-sheet.webp|The voice sheet (later in p3).}} +{{img:ports-panel.webp|The ports panel without Radius (later in p3).}} + +## The web shell and the TUI + +{{clip:clip-phone-sheets.mp4|The phone: drawer, switching sessions, the session sheet, the share sheet, Back.}} +{{img:webshell-phone-sheet-2-directory.webp|The sheet stack on the phone: a second sheet pushed on the first.}} +{{clip:clip-tui-palette.mp4|The TUI's F1 palette: filter, run a command, the model selector.}} +{{img:tui-slash-commands.webp|/ command completion in the TUI.}} +{{img:tui-theme-picker.webp|The TUI's theme picker with preview.}} {{img:tui-theme-light.webp|The light theme.}} + +Not shown: clone and fork (TUI only, not run), SSH environments connected (no network in the capture), Radius and +public ports (off), voice in use (no microphone), dev mode reload pills, provider OAuth flows. + # p2 Today, and the Proposed Cut {#features lede="Every feature area, what it is, and where it goes in p3."} Sizes are non-test lines in p2. "Base" means p3's first release; "later" means designed for, built after; "drop" means @@ -68,7 +148,7 @@ not carried over. | Area | What it does | p2 size | Proposed | | --- | --- | ---: | --- | -| Sessions | list by machine and directory, create, rename, pin, archive, relocate, clone, fork, unread, automatic titles | 2,186 | **base**, plus delete; clone and fork in the web too | +| Sessions | list by machine and directory, create, rename, pin, archive, relocate, clone, fork, unread, automatic titles | 2,186 | **base**, plus delete and in-session branches (`/tree`); clone and fork in the web too | | Conversations and agent | transcript, send, steer, follow-up, outbox, stop, model and thinking, cost, compaction | 3,682 + web 2,700 | **base** | | Subagents, one-shot agents, cross-session messages | named background agents, `team`, `agent` from scripts, `conversations` | 2,182 | **base** (owner: "awesome") | | History and handoff | re-read earlier entries after compaction, hand off with a continuation | 533 | **base** | @@ -181,6 +261,75 @@ owner's provider keys stored on Radius (decide). Local prompt resources (the dev through a hand or synced. Closed sessions have no worker: the server reads a session's outbox without starting it, or starts it to sync. +## The sync example + +`packages/chord/test/next/examples/60-session-sync/` (uncommitted, in `/home/badlogic/pi-sync-ex`) plays the decided +protocol with Chord's real store pieces (the outbox, `applyBatch`, export archives) and a small in-memory stand-in for +Radius. Its output, step by step: + +```text +1. laptop s1: lastSeq 4 outbox floor 0 +2. sync: uploaded 1..4, Radius at 4, trimmed through it +3. laptop s1: lastSeq 6 - sync: uploaded 5..6, Radius at 6, trimmed through it +4. laptop s2: lastSeq 3 outbox floor 3 - sync: archive uploaded (737 bytes), Radius imported it at 3 + laptop s2: lastSeq 4 - sync: uploaded 4..4 +5. Radius takes over, epoch 2; Radius writes seq 7 + stale laptop: lastSeq 7 - push: refused for an old epoch: this copy is now s1~split@6 +5b. no epoch check - stale push: refused: does not follow seq 7 + stale sync: nothing to send, Radius at 7 - outbox floor 7 (its own commit 7 trimmed) + stale: ... 5 assistant: (stale laptop) added a different test + Radius: ... 5 assistant: (on Radius) added a test (same seq, different data) +6. laptop takes epoch 3, pulls 7..7: same values: true; its next commit syncs at epoch 3 +``` + +:::sequence animate | The decided handshake (steps 1-3), then continuing on Radius (5) and back (6). +actor laptop "laptop" "driver, epoch 1" accent +actor radius "Radius" "replica per session" +laptop -> radius: which sessions do you have? +radius --> laptop: s1 at seq 4, epoch 1 +laptop => radius: commits 5..6 (epoch 1) +radius --> laptop: s1 at seq 6 (acknowledged: trim through 6) +divider "continue on Radius" +radius -> radius: epoch 2, Radius drives, writes seq 7 +laptop -> radius: commit 7 (epoch 1) +radius --> laptop: refused: old epoch (the copy splits) +divider "back to the laptop" +laptop -> radius: take epoch 3 +laptop => radius: pull 7..7, apply +::: + +What it shows: the handshake works and only the tail moves; a session Radius lacks starts from an archive; without +epochs a second writer's commit is lost silently (5b), with them it splits; taking a session back is a pull plus an +epoch. One more finding: `applyBatch` refuses while a host holds the store open, so the sync layer stops the session's +worker, applies, restarts it (decided 2026-11-02: usually nothing runs at that moment). The example lists nine things +its stand-ins fake that the sync layer must build (authenticated handshake, persisted epochs, the batch wire format, +retries, streaming bootstrap with files, the DO and R2 side, the continue fence, the split's new id, pull-back after a +power loss). + +# Branches and Subagent Context {#branches lede="pi's /tree, fork and clone, and subagents that start from the parent's context."} + +pi branches three ways; p2 has two, in the TUI only: + +| pi | What it does | p2 | +| --- | --- | --- | +| `/tree` | branch inside the session: pick an earlier message, edit and resubmit; the old branch stays; leaving a branch can summarize it into the one entered | missing | +| `/fork` | a new session from an earlier user message | TUI only, main conversation only | +| `/clone` | a copy of the active branch as a new session | TUI only | + +Durable already has the building block: `conversation.fork(at)` creates a conversation in the same session store that +shares every entry up to `at`, without copying. Proposed for p3: + +- **Base:** editing an earlier message forks the conversation there (an in-session branch); a branch list switches + between them; fork and clone to a new session; all in the web and the TUI. +- **Later:** a full tree navigator and branch summaries. +- **Index and sync:** branches are conversations inside one session store, so sync carries them for free; the index + marks the active branch and tells branches from subagents. + +**Subagents with the parent's context** (owner, 2026-11-02): `subagent({ action: "spawn", ..., context: "inherit" })` +forks the parent conversation at the last entry before the assistant message holding the spawning call. The child +knows everything the parent knew; its model, tools and location may differ; with the same model the shared prefix is +usually a prompt-cache hit, otherwise it is paid in full (the tool's description says so). Base in p3. + # Tools Built In {#tools lede="p2's 21 model-facing tools, and which ship with p3."} | Tool | p2 plugin | Proposed | @@ -212,13 +361,13 @@ directory. | Command | Does | | --- | --- | -| `p3 serve [--data-dir

    ] [--host ] [--port ]` | starts the server (`startApp`); prints the one-time owner link | +| `p3 serve [--data-dir ] [--host ] [--port ]` | starts the server (`startServer`); prints the one-time owner link | | `p3 owner-link` | prints a new owner link for a running server (through the owner token file) | | `p3 tui [url] [--pair]` | the sandboxed TUI; on the same machine it uses the owner token file, elsewhere device pairing | | `p3 hand enroll `, `p3 hand run` | enrolls this machine as a hand, runs it (or installs a service) | | `p3 radius login` | signs the server into Radius (device code) | | `p3 install ` | installs a plugin into the user's root | -| `p3 export `, `p3 import ` | the app kit's export per address | +| `p3 export `, `p3 import ` | the app kit's `exportData`/`importData` (a running session is not stopped) | The data directory defaults to `~/.p3`. Its files: `settings.json`, `models.json`, `auth.json` (locked), `AGENTS.md`, `skills/`, `prompts/`, `plugins/` (the user's root), the server store, a directory per session worker, @@ -287,7 +436,7 @@ Editor replacement (extension points only), raw provider stream events and respo | # | Package | Contains | Depends on | | --- | --- | --- | --- | -| 1 | p3 skeleton | `startApp` with p3's API v1 modules (shared, web, tui, worker), core plugins in the app's root, `p3` CLI | Chord slices 11-13 | +| 1 | p3 skeleton | `startServer` with p3's API v1 modules (shared, web, tui, worker), core plugins in the app's root, `p3` CLI | Chord slices 11-13 | | 2 | Sessions and conversations | Durable on the Chord store, session lifecycle with delete, the conversation contract, subagents, history and handoff, code mode, shell commands | 1 | | 3 | Session index | the index table, worker-to-server updates on metadata and idle changes, the session list, search over it | 2 | | 4 | Session sync to Radius | outbox upload and trim, epochs, pull-back, split, files as blobs, continue on Radius and back | 2, 3, Radius side | diff --git a/packages/chord/docs/next/p3-features/index.html b/packages/chord/docs/next/p3-features/index.html deleted file mode 100644 index 352a6d1ed..000000000 --- a/packages/chord/docs/next/p3-features/index.html +++ /dev/null @@ -1,832 +0,0 @@ - - -p3: Feature Set and Work Packages (prep for the design meeting) - - - - -
    -
    -
    Meeting prep · p3 on Chord v2 · 2026-11-03
    -

    p3
    Features

    -

    What p2 does today, what p3 should ship first, how hands, sync and the plugin API fit, and a first cut into work packages

    -
    28feature areas in p2 today
    21model-facing tools in p2
    78coding-agent extension examples checked against the p3 plugin API
    12proposed work packages
    -

    Written by the redesign session for the meeting on 2026-11-03, from three research passes over p2 -(apps/p2 at b7af55ace), the coding agent's extension examples and p2's environments and Radius code -(/tmp/p3-prep/features.md, extensions.md, envs.md), the owner's notes and Armin's summary of the 2026-11-02 -meeting. Chord v2's slices 1-13 are built (the app kit included); slice 11 (web and TUI) is building. Everything -marked "proposed" is for the meeting to decide.

    -
    -
    - - -
    -

    Before we begin

    -

    p2 grew feature by feature on the phone: 33 plugins, about 65,000 lines, a web shell and a TUI. p3 is its rewrite on -Chord v2, with an app kit that already gives hosts, workers, HTTP with sessions and pairing, plugins with versioned -APIs, files and export. The meeting's job is to decide what p3 ships first, what waits, and how the work splits.

    -

    Some of it was settled yesterday (Armin's summary): sessions sync to R2 through Radius by commit log, the session index -is a replicated Chord table, every host kind loads plugins through registries, hands connect proactively. The owner -added: voice waits; environments turn around into hands; continuing a local session on Radius is core; subagents and -handoff are in, artifacts are questionable; the plugin API stays small enough to evolve.

    -

    Ten things to know

    -
      -
    1. Proposed base: sessions (with delete), the agent and conversations, subagents, history and handoff, code mode, -shell commands, models and login, prompt resources, workspace (read and review), file index, status and notices, -search through the index, the web shell and the TUI, hands, session sync and continue-on-Radius.
      2. -
    2. Proposed later: voice, terminals (need channels and hands), artifacts, port forwards, chat mode and guest -sharing, MCP and tool_search.
      3. -
    3. Hands replace SSH-dialled daemons: a long-lived daemon dials the server (or Radius), is enrolled once by a -pairing code, and carries p2's existing daemon protocol over one multiplexed WebSocket.
      4. -
    4. The sync handshake needs a driver identity. Two machines writing one session both push "from Radius's last -seq" and silently diverge; an epoch or lease per session (Chord's rule S7) closes it.
      5. -
    5. The index has two writers once a session continues on Radius; three options, one recommended (derive rows from -session stores).
      6. -
    6. Tools in base: read, write, edit, bash, codemode, subagent, team, agent, conversations, -history_*, location tools, rename_session, send_file; artifacts, ports and reload_plugins wait or change.
      7. -
    7. One CLI, no environment variables: p3 serve, p3 tui, p3 hand, p3 owner-link, p3 install; settings as -arguments and files, per the app kit.
      8. -
    8. The plugin API v1 is seven clusters, almost all already existing: Durable's extension, a Conversation -contract, input transforms, environments and models, questions, UI registries, objects for state.
      9. -
    9. Litmus test: all 78 coding-agent extension examples are expressible; Doom needs two small fixes (frame splitting -in the TUI bridge, a path for large plugin assets); streaming frames from a worker and raw terminal ownership do -not fit, by design.
      10. -
    10. Twelve work packages, with p3's skeleton on the app kit and the session/index/sync core first.
      11. -
    -

    Words used throughout

    -

    A hand is a daemon on a machine that connects out to a p3 server or to Radius and offers execution (commands, -files, watching, terminals). A driver is the single host allowed to write a session's store at a time; a -replica follows its commits. An epoch numbers driver changes. The index is the table of session metadata -(title, status, idle state) the session list reads. The app kit is Chord's startApp/startWorker layer.

    -
    - -
    I

    p2 Today, and the Proposed Cut

    Every feature area, what it is, and where it goes in p3.

    -

    Sizes are non-test lines in p2. "Base" means p3's first release; "later" means designed for, built after; "drop" means -not carried over.

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    AreaWhat it doesp2 sizeProposed
    Sessionslist by machine and directory, create, rename, pin, archive, relocate, clone, fork, unread, automatic titles2,186base, plus delete; clone and fork in the web too
    Conversations and agenttranscript, send, steer, follow-up, outbox, stop, model and thinking, cost, compaction3,682 + web 2,700base
    Subagents, one-shot agents, cross-session messagesnamed background agents, team, agent from scripts, conversations2,182base (owner: "awesome")
    History and handoffre-read earlier entries after compaction, hand off with a continuation533base
    Code modescripts calling tools as nested durable calls155 + Durablebase (small, Durable owns it)
    Shell commands!cmd and !!cmd in the conversation's environmentin chatbase, web too (TUI only today)
    Chat modepeople talk without a turn, @pi mentions876later
    Sharing and guestsshare links, guest names, revocation1,047later (the app kit's users and invites cover the start)
    Devices and owner loginowner links, device cookies684base, replaced by the app kit's sessions and pairing
    Models, credentials, provider loginpicker, scope, API keys, OAuth flows3,828base
    Settingssettings.json, hot reload87base
    Prompt resourcesAGENTS.md, skills, templates, read in the session's environment958base
    Workspacefiles, git status, diffs, search, review comments2,655 + web 1,341base (read and review; uploads later)
    File index and @ranked file completion242base
    Terminalsdetached PTYs, ghostty-vt in the web3,518later (channels, hands)
    Environments and locationsSSH environments, directory picker, agents moving themselves2,546base, redesigned as hands
    Artifacts and filesattachments, send_file, versioned artifacts with previews1,713attachments and send_file base (app kit Files); artifacts later
    Searchsidebar search over titles and messages (FTS5)775base over the index; message text search through the same sync mechanism later
    Status and presencebusy, tasks, usage, viewers, worker failures566base (feeds the index)
    Logs and noticesowner log viewer, notices487base
    Voicerooms, barge-in, speech service3,770 + Pythonlater (owner)
    Radius relay and port forwardsbrain WebSocket, up to 50 forwards2,000relay base (as the hands/sync link), forwards later
    Web shellPWA, sheets, navigation, reconnect9,252base, rebuilt on slice 11 ("look and feel great")
    TUIlauncher, sandbox, 40 commands, themes, keybindings6,720 + launcherbase on slice 11
    Dev and reloadsupervisor, dirty marks, reload tool468 + scriptsbase as the app kit's "apply changes"; the supervisor goes
    Fake provider, PI_MOBILE_* aliases, P2_DEV_RESUMEtest and legacy leftovers218drop
    tool_search, MCPnot installed in p2-later
    -

    Rough edges p3 should not inherit: features only in the TUI (shell commands, clone, fork, compact, new context), no -session delete, terminals and voice without a TUI side, docs that drifted from code (voice routes, TUI notes), the -plugin loader's module leak on reload (solved by Chord's plugin component).

    -
    - -
    II

    Hands

    Environments turned around: the machine connects to the server, not the other way.

    -

    What p2 does

    -

    p2's server and workers start a daemon (pi-env) over SSH on demand, one connection per session and per use (four -connection pools: each worker, the directory picker, the connection test, the terminal host). The daemon is -stateless and kills its commands 30 seconds after its connection goes quiet. Its frame protocol (exec, files, watch, -file listing, PTYs) does not depend on SSH.

    -

    What a hand is (proposed)

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    p2's daemonhand
    who dialsthe server, over SSHthe hand, over WSS, to the server or to Radius
    needssshd, a key, a network path to the machineoutbound HTTPS only
    lifetimeper connectiona service (systemd, launchd, Termux:Boot), reconnecting
    installdeployed and verified at each startinstalled once, signed updates
    machine identitySSH host keya hand key pair, enrolled once
    connectionsone per session and useone per hand and server, multiplexed
    -
      -
    • Enrollment: p3 hand enroll <server> shows a code; the owner approves it in the UI (the app kit's device -pairing); the server pins the hand's public key; fingerprints shown on both sides. Revocation in a Hands panel.
      • -
    • Reachability: a hand reaches the owner's own server directly (LAN, own tunnel) or through Radius, which relays -hands to whichever host drives a session (the upsell: no tunnel to set up; Radius can also offer sandboxed machines -and hands of its own).
      • -
    • Protocol: p2's daemon protocol over the WebSocket, unchanged in shape; two changes: several sessions share one -connection (cancellation, handles and backpressure per session), and a disconnect starts a grace period instead of -killing everything (detached commands and PTYs reattach on reconnect; this moves the terminal host's job into the -hand).
      • -
    • Binding: a session's location becomes { hand, cwd }; one server object holds the hand connections and the -HarnessEnvironments entry resolves a hand id to an execution environment; workers reach hands through the server's -links. An SSH machine stays possible as a hand kind the server dials.
      • -
    • Offline: an operation needing an offline hand fails with a clear error the model sees; the hand's status shows -in the session list.
      • -
    -
    - -
    III

    Session Sync and the Index

    What was decided yesterday, and the three gaps it leaves.

    -

    Decided

    -

    The worker updates the session index from events and sends it to the server; sessions sync to R2 through Radius: -Radius reports per session its last sequence number, pi uploads the commit log from there; R2 holds the SQLite file -and the commit log, a SQLite Durable Object applies them; sessions have an owner id; the index is a Chord table -replicated like a session, its row updated only on metadata, session idle and subagent idle changes.

    -

    Chord v2 has the pieces: a versioned outbox written in each commit's transaction, export archives, batch apply that -refuses gaps and divergence, export and import per address (app kit). Missing (the sync layer): upload and trim, -epochs, pull-back after a power loss, the split, starting from an archive.

    -

    Gap 1: who is the driver

    -

    Two machines both write session X (the laptop resumed it while Radius continued it). Both ask Radius for its last -seq, both upload from there; Radius applies whichever arrives first and refuses the other as "not following". The -copies have now diverged and nobody knows.

    -

    Proposed: a driver epoch per session, handed out by Radius. An upload names its epoch; an old epoch is refused; -the refused copy becomes a new session (Chord's rule S7: a split, never a rollback, because tools already ran). -"Continue on Radius" is an epoch change: the local driver waits for idle, uploads the tail and the session's files, -Radius bumps the epoch and starts the agent from its store, the local side becomes a replica or closes. Back is the -reverse.

    -

    Gap 2: two writers of the index

    -

    When a session continues on Radius, Radius writes its index row, and so does the local server.

    - - - - - - - - - - - - - - - - - - - - - - - -
    OptionHowCost
    a. one index per writereach host keeps its own index, Radius merges per owner, newest epoch wins per rowa second merge rule
    b. rows derived from session storesevery host computes a session's row from the session's own commitsno index sync at all; each host needs the session's commits
    c. rows tagged { epoch, seq }Radius keeps the highest per rowrows must carry their source position
    -

    Recommended: (b) where the session's commits are present (Radius has them all), with (c)'s tag on the rows a local -server writes for sessions it drives, so a list without the session store still sorts correctly.

    -

    Gap 3: what else moves

    -

    Files outside the store (uploads, sent files, the app kit's Files) must sync as blobs before the batches naming -them, R2 as their home. Model access and secrets for a session continued on Radius: Radius's own models, or the -owner's provider keys stored on Radius (decide). Local prompt resources (the device's AGENTS.md, skills) must be read -through a hand or synced. Closed sessions have no worker: the server reads a session's outbox without starting it, or -starts it to sync.

    -
    - -
    IV

    Tools Built In

    p2's 21 model-facing tools, and which ship with p3.

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Toolp2 pluginProposed
    read, write, edit, bashcoding-tools (Durable)base
    powershellpowershellbase (deferred tool, Windows hands)
    codemodecodemodebase
    agent (from scripts)agentbase
    subagent, teamsubagentsbase
    conversationsconversationsbase
    history_search, history_read, history_handoffhistorybase
    location_envs, location_dirs, location_movelocationsbase, renamed around hands (hands, hand_dirs, move)
    rename_sessionsessionsbase
    send_filesend-filebase (on the app kit's Files)
    artifactartifactslater (owner: maybe not as useful)
    portsportslater (with forwards)
    reload_pluginsdev-toolsbase as an "apply changes" tool for the owner, not tied to a checkout
    tool_searchDurablelater (with MCP)
    -

    Every tool is a plugin in the app's own root, so any of them can be removed, replaced or added without touching base -code (the app kit's rule: base code is the API, bootstrap and a start-data facet).

    -
    - -
    V

    Running p3

    Starting the server, the CLI, themes and keybindings.

    -

    The CLI (proposed)

    -

    One binary, p3, and no environment variables (app kit decision); settings are arguments or files in the data -directory.

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    CommandDoes
    p3 serve [--data-dir <dir>] [--host <addr>] [--port <n>]starts the server (startApp); prints the one-time owner link
    p3 owner-linkprints a new owner link for a running server (through the owner token file)
    p3 tui [url] [--pair]the sandboxed TUI; on the same machine it uses the owner token file, elsewhere device pairing
    p3 hand enroll <url>, p3 hand runenrolls this machine as a hand, runs it (or installs a service)
    p3 radius loginsigns the server into Radius (device code)
    `p3 install <git:...npm:...>`
    p3 export <session> <file>, p3 import <file>the app kit's export per address
    -

    The data directory defaults to ~/.p3. Its files: settings.json, models.json, auth.json (locked), -AGENTS.md, skills/, prompts/, plugins/ (the user's root), the server store, a directory per session worker, -owner-token, logs/. Plugin-specific configuration moves into plugin objects (documents) instead of more files.

    -

    Themes and keybindings

    -

    Themes follow the looks-and-themes proposal (decided 2026-11-02): design tokens as CSS variables on the web, roles in -the TUI, one Themes registry plugins contribute to, one dark and one light choice per user. Keybindings: commands are -registry entries with default keys (p2's TuiCommands with when conditions, 78 binding ids); the user's overrides -live in a preferences document (per user, synced like any data), not in files on each client; the web gets the same -command registry with its own key handling.

    -
    - -
    VI

    The Plugin API

    The smallest surface that still expresses everything the coding agent's extensions do.

    -

    The rule

    -

    Offer few things, each stable: what an API offers, it must keep (versioned in the import path, @p3/app/v1). Every -cluster below is a registry or a contract that already exists in Chord, Durable or p2; p3 v1 publishes those and -nothing else. Legacy concepts of the old extension API are not carried: CLI flags, an event bus, RPC-mode UI, hasUI -checks, "destructive switch" gates, pi.exec running locally regardless of the environment.

    -

    Seven clusters

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    ClusterHostMechanismCovers
    A. The agent's turnworkerDurable's Extension in HarnessExtensions: tools, beforeTool/afterTool, beforeRequest, sections, wraps, onYield, afterRun, beforeCompact20 examples: hello, permission-gate, protected-paths, tool-override, plan-mode, todo, custom-compaction, ...
    B. Conversation controlworker objecta Conversation contract: submit, configure, append (typed entries), compact, fork, rename; state: busy, agent, usagesend-user-message, file-trigger, session-name, preset, handoff, qna, ...
    C. InputworkerInputTransforms registry (rewrite or answer input before the model)input-transform, inline-bash
    D. Environments and modelsworker, serverHarnessEnvironments (ssh, gondolin, hands as kinds), HarnessModels (providers, virtual models), LoginProvidersssh, gondolin, custom providers, jev-router
    E. Asking the userserver object, UI renderersQuestions.ask from worker code; QuestionKinds per UI kindpermission-gate, question, questionnaire, timed-confirm, project-trust
    F. UItui, webregistries: Commands (slash, palette, keys), ToolRenderers, EntryRenderers, Slots (header, footer, status, widgets), overlays through the UI context, editor extension pointsevery UI-only example, the games
    G. State and standing behaviouranylocal objects owning natives, documents for statefile-trigger's watcher, gondolin's VM, game saves
    -
    // a plugin with a tool, a gate and a renderer: plugins/todo/
-// worker.ts
-self.contribute(HarnessExtensions, { name: "todo", tools: [todoTool], hooks: [hook(ToolTask, { beforeTool: guard })] });
-// tui.ts
-self.contribute(TuiToolRenderers, { todo: { call: (args) => todoCall(args), result: (r) => todoList(r.details) } });
-self.contribute(TuiCommands, { "todo.show": { slash: { name: "todos" }, run: (ui) => ui.overlay("todos", new TodoView(ui)) } });
-// web.tsx: the same registries with React components
    -

    The litmus test

    -

    All 78 examples map onto the seven clusters (table in /tmp/p3-prep/extensions.md). The examples that mix agent and -UI split into a worker facet and a tui/web facet sharing a contract file: a tool that wants a dialog asks the -Questions object, which works from any client, several clients or none.

    -
      -
    • Snake, Space Invaders, tic-tac-toe: a command plus an overlay in tui.ts and a canvas in web.tsx, engine in a -shared file, saves in a user-keyed document. Fit today.
      • -
    • Doom: fits with two fixes in the TUI: the sandbox bridge caps a frame at 64 KiB and a Doom frame is about 160 KB -(split writes at escape boundaries), and the sandbox has no files or network, so large plugin assets (a 4 MB WAD) -need a documented "asset of my plugin" path (chunked reads through the plugin's server facet).
      • -
    • Does not fit, by design: streaming 35 FPS frames from a worker to clients (objects carry state; needs channels), -engine state surviving an in-place reload (needs an explicit save), raw terminal ownership (interactive-shell -becomes a terminal view; terminal notifications need a launcher action).
      • -
    -

    What v1 leaves out

    -

    Editor replacement (extension points only), raw provider stream events and response headers, a fork hook, a -turn_start hook, an event bus. Each has one or two examples; add when a real plugin needs it.

    -
    - -
    VII

    Work Packages

    A first cut, in dependency order, for the meeting to rearrange.

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    #PackageContainsDepends on
    1p3 skeletonstartApp with p3's API v1 modules (shared, web, tui, worker), core plugins in the app's root, p3 CLIChord slices 11-13
    2Sessions and conversationsDurable on the Chord store, session lifecycle with delete, the conversation contract, subagents, history and handoff, code mode, shell commands1
    3Session indexthe index table, worker-to-server updates on metadata and idle changes, the session list, search over it2
    4Session sync to Radiusoutbox upload and trim, epochs, pull-back, split, files as blobs, continue on Radius and back2, 3, Radius side
    5Radius sideper-session SQLite DO, commit log and SQLite on R2, epochs, the sync handshake, hand relaywith 4
    6Handsdaemon as an outbound service, enrollment by pairing, multiplexed protocol, reattach after disconnects, server hand pool, HarnessEnvironments kind, SSH as a hand kind1
    7Web shellp2's layout and flows rebuilt on slice 11 ("look and feel great"), themes1, slice 11
    8TUIp2's TUI on slice 11's sandbox, frame splitting, plugin assets path, keybindings and themes1, slice 11
    9Models and settingscatalog, credentials, provider login flows, settings, prompt resources1
    10Workspacefiles, git, diffs, search, review comments, file index and @, through hands6
    11Plugin API v1 and docsthe seven clusters published in @p3/app/v1, authoring guide, ported examples as the litmus (hello, permission-gate, todo, plan-mode, snake, doom)1, 2, 7, 8
    12Access and statusowner, devices and pairing (app kit), notices, logs, status and presence1
    -

    Later: voice, terminals (channels and hands), artifacts, port forwards, chat mode and guest sharing, MCP and -tool_search.

    -
    - -
    ∴

    Open Questions

    To decide in the meeting.

    -
      -
    1. The proposed cut: base versus later per area (first table).
      2. -
    2. Driver identity: epochs per session, handed out by Radius, with splits on conflict?
      3. -
    3. The index with two writers: derived rows plus { epoch, seq } tags, or one of the other options?
      4. -
    4. Who syncs a closed session (the server reading its outbox, or starting its worker)?
      5. -
    5. Files outside the store as blobs on R2, uploaded before their batch?
      6. -
    6. Models and secrets for sessions continued on Radius: Radius's own models, or the owner's keys stored there?
      7. -
    7. Hands: through Radius as a relay to the driver, and may one hand serve the local server and Radius at once?
      8. -
    8. Do hands keep commands and terminals alive across disconnects (absorbing p2's terminal host)?
      9. -
    9. Hand install and updates: signed binary, service install, version negotiation.
      10. -
    10. Is SSH kept as a hand kind for machines without a service?
      11. -
    11. Plugin API v1: the seven clusters as listed, and what is explicitly left out?
      12. -
    12. Work package owners and order.
      13. -
    -
    - -
    A

    Sources

    - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    SourceWhat
    /tmp/p3-prep/features.mdp2's feature areas with files, sizes, tools, rough edges; the CLI, environment variables and data files
    /tmp/p3-prep/extensions.mdall 78 extension examples, the capability matrix, the clusters with sketches, the hard cases
    /tmp/p3-prep/envs.mdp2's environments and Radius link, Chord's sync pieces, the sync and hands sketches, open questions
    apps/p2/docs/p3-functions/the functionality map (partly outdated)
    packages/chord/docs/next/design, decisions (2026-11-02 meeting summary), app kit and plugin model, theming
    -
    - -

    p3 Feature Set and Work Packages · meeting prep · 2026-11-03

    -
    - - From 15f00a786a3d2900004b401c6f725e3bfed2d772 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 20:13:38 +0000 Subject: [PATCH 166/175] docs(chord): example 60, session sync played with the store's outbox, applyBatch and archives --- .../next/examples/60-session-sync/main.ts | 180 ++++++++++++++++++ .../next/examples/60-session-sync/radius.ts | 159 ++++++++++++++++ .../next/examples/60-session-sync/session.ts | 83 ++++++++ .../next/examples/60-session-sync/sync.ts | 101 ++++++++++ 4 files changed, 523 insertions(+) create mode 100644 packages/chord/test/next/examples/60-session-sync/main.ts create mode 100644 packages/chord/test/next/examples/60-session-sync/radius.ts create mode 100644 packages/chord/test/next/examples/60-session-sync/session.ts create mode 100644 packages/chord/test/next/examples/60-session-sync/sync.ts diff --git a/packages/chord/test/next/examples/60-session-sync/main.ts b/packages/chord/test/next/examples/60-session-sync/main.ts new file mode 100644 index 000000000..04c922d8f --- /dev/null +++ b/packages/chord/test/next/examples/60-session-sync/main.ts @@ -0,0 +1,180 @@ +// Session sync as decided (decisions S3, S5, S7, W9, "The outbox is optional, internal and versioned", "p3 meeting, +// 2026-11-02"), played with Chord's real pieces: a laptop store with an outbox, `store.outbox.read`/`trim`, +// `store.export` with `encodeJsonl`, `importStore`, and batch apply, `applyBatch` (internal: exported from +// `chord/next` marked internal, imported here from `src/next/store/apply-batch.ts`; it takes the `ChordStore` behind +// `openStore`'s `Store`). Radius and the laptop's sync client are stand-ins written for this example (`radius.ts`, +// `sync.ts`), not Chord APIs: the sync layer is not built yet (slices.md, "The sync layer"). +// Steps: 1. a session on the laptop; 2. the handshake and a first upload; 3. a second sync moves only the tail; +// 4. a bootstrap from an archive for a session whose outbox was turned on late; 5. "continue on Radius" and the epoch +// fence (the split), then the same without the fence (silent divergence); 6. back to the laptop with a pull. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/60-session-sync/main.ts +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { Readable } from "node:stream"; +import { BACKGROUND_CONTEXT as context } from "../../../../src/context/index.ts"; +import { decodeJsonl, importStore, openStore, type Store, sqliteBackend } from "../../../../src/next/store/index.ts"; +import { openNodeSqliteDatabase } from "../../../../src/sqlite/node.ts"; +import { RadiusStandIn } from "./radius.ts"; +import { describe, withSession } from "./session.ts"; +import { archiveOf, type LocalSession, pull, push, type Synced, sync } from "./sync.ts"; + +const radius = new RadiusStandIn(); +const directory = await mkdtemp(join(tmpdir(), "chord-sync-")); + +/** One sync's result as a line. */ +function shown(result: Synced): string { + switch (result.kind) { + case "uploaded": + return `uploaded ${result.from}..${result.to}, Radius at ${result.lastSeq}, trimmed through it`; + case "nothing": + return `nothing to send, Radius at ${result.lastSeq}`; + case "bootstrapped": + return `archive uploaded (${result.bytes} bytes), Radius imported it at ${result.lastSeq}`; + case "behind": + return `Radius is ahead at ${result.lastSeq}`; + case "split": + return `refused for an old epoch: this copy is now ${result.session.id}`; + case "refused": + return `refused: ${result.message}`; + } +} + +/** A new in-memory store (outbox on) imported from archive bytes. */ +async function copyOf(bytes: Uint8Array): Promise { + const backend = sqliteBackend(await openNodeSqliteDatabase(":memory:")); + return importStore(backend, decodeJsonl(Readable.from([bytes])), context, { outbox: true }); +} + +/** Radius's view of a session as a line. */ +function atRadius(id: string): string { + const known = radius.handshake()[id]!; + return `Radius ${id}: lastSeq ${known.lastSeq}, log ${radius.logRange(id)}, epoch ${known.epoch} (${known.driver})`; +} + +try { + // 1. The laptop: SQLite in memory with the outbox on from its first commit; one session, a few commits. + const store = await openStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), context, { outbox: true }); + const s1: LocalSession = { id: "s1", store, epoch: 1, acked: 0 }; + await withSession(store, async (session) => { + await session.rename("Fix the parser", context); + await session.say({ role: "user", text: "the parser drops comments" }, context); + await session.say({ role: "assistant", text: "found it in lexer.ts" }, context); + }); + console.log("1. laptop s1: lastSeq", store.lastSeq, "outbox floor", store.outbox!.floor); + + // 2. The handshake: Radius has no s1 (lastSeq 0), so the whole outbox goes, 1..lastSeq; then trimmed. + console.log("2. sync:", shown(await sync(s1, radius))); + console.log(" ", atRadius("s1"), "- laptop outbox floor", store.outbox!.floor); + + // 3. Two more commits; the next sync sends only them. + await withSession(store, async (session) => { + await session.say({ role: "user", text: "and strings too" }, context); + await session.say({ role: "assistant", text: "fixed both" }, context); + }); + console.log("3. laptop s1: lastSeq", store.lastSeq, "- sync:", shown(await sync(s1, radius))); + console.log(" ", atRadius("s1")); + console.log(" laptop:", await describe(store)); + console.log(" Radius:", await describe(radius.replica("s1"))); + + // 4. Session s2 ran without an outbox (a file here, since it is reopened), then sync was turned on: the outbox starts + // after its lastSeq, so Radius, which has nothing, needs an archive first; batches go on from there. + const path = join(directory, "s2.sqlite"); + const before = await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context); + await withSession(before, async (session) => { + await session.rename("Release notes", context); + await session.say({ role: "user", text: "draft the notes for 1.4" }, context); + }); + await before.close(context); + const store2 = await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context, { outbox: true }); + const s2: LocalSession = { id: "s2", store: store2, epoch: 1, acked: 0 }; + console.log( + "4. laptop s2: lastSeq", + store2.lastSeq, + "outbox floor", + store2.outbox!.floor, + "- sync:", + shown(await sync(s2, radius)), + ); + await withSession(store2, async (session) => { + await session.say({ role: "assistant", text: "here is a draft" }, context); + }); + console.log(" laptop s2: lastSeq", store2.lastSeq, "- sync:", shown(await sync(s2, radius))); + console.log(" ", atRadius("s2")); + console.log(" Radius:", await describe(radius.replica("s2"))); + await store2.close(context); + + // 5. "Continue on Radius": the laptop uploads its tail (nothing left here), Radius bumps the epoch and drives, + // writing a commit on its replica. The laptop keeps its copy as a replica and writes nothing. + const handoff = await archiveOf(store); + console.log( + "5. laptop s1 sync:", + shown(await sync(s1, radius)), + "- Radius takes over, epoch", + radius.takeOver("s1"), + ); + await radius.drive("s1", (replica) => + withSession(replica, async (session) => { + await session.say({ role: "assistant", text: "(on Radius) added a test" }, context); + }), + ); + console.log(" ", atRadius("s1")); + // A laptop process that missed the handoff (here a copy of the laptop's store at it, so the laptop itself can take + // the session back in step 6) writes at epoch 1 and uploads the new commit after its last acknowledgement. + const stale: LocalSession = { id: "s1", store: await copyOf(handoff), epoch: 1, acked: 6 }; + await withSession(stale.store, async (session) => { + await session.say({ role: "assistant", text: "(stale laptop) added a different test" }, context); + }); + const fenced = await push(stale, radius); + console.log(" stale laptop: lastSeq", stale.store.lastSeq, "- push:", shown(fenced)); + // The split (S7): no rollback; the copy is a new session with its origin, synced on its own from an archive. + if (fenced.kind === "split") { + const split = fenced.session; + console.log( + " split", + split.id, + "origin", + JSON.stringify(split.origin), + "- sync:", + shown(await sync(split, radius)), + ); + console.log(" ", atRadius(split.id)); + } + await stale.store.close(context); + + // 5b. The same without the epoch fence: the stale upload of seq 7 does not follow Radius's 7 and is refused, so far + // so good; but the next handshake finds both at 7, sends nothing and trims the laptop's commit 7 as acknowledged. + radius.checkEpochs = false; + const unfenced: LocalSession = { id: "s1", store: await copyOf(handoff), epoch: 1, acked: 6 }; + await withSession(unfenced.store, async (session) => { + await session.say({ role: "assistant", text: "(stale laptop) added a different test" }, context); + }); + console.log("5b. no epoch check - stale push:", shown(await push(unfenced, radius))); + console.log(" stale sync:", shown(await sync(unfenced, radius)), "- outbox floor", unfenced.store.outbox!.floor); + console.log(" stale: ", await describe(unfenced.store)); + console.log(" Radius:", await describe(radius.replica("s1"))); + await unfenced.store.close(context); + radius.checkEpochs = true; + + // 6. Back to the laptop: it takes the epoch (Radius stops driving), pulls Radius's commits after its lastSeq + // (batch apply on the laptop's store; no host open on it), and drives again. + s1.epoch = radius.handBack("s1", "laptop"); + const pulled = await pull(s1, radius); + console.log(`6. laptop takes epoch ${s1.epoch}, pulls ${pulled.from}..${pulled.to}: lastSeq ${store.lastSeq}`); + const [laptopView, radiusView] = [await describe(store), await describe(radius.replica("s1"))]; + console.log(" laptop:", laptopView); + console.log(" Radius:", radiusView); + console.log(" same values:", laptopView === radiusView); + await withSession(store, async (session) => { + await session.rename("Fix the parser (done)", context); + }); + console.log(" laptop writes, sync:", shown(await sync(s1, radius))); + console.log(" ", atRadius("s1")); + console.log(" same values:", (await describe(store)) === (await describe(radius.replica("s1")))); + + await store.close(context); +} finally { + await radius.close(); + await rm(directory, { recursive: true, force: true }); +} diff --git a/packages/chord/test/next/examples/60-session-sync/radius.ts b/packages/chord/test/next/examples/60-session-sync/radius.ts new file mode 100644 index 000000000..177ca5cf9 --- /dev/null +++ b/packages/chord/test/next/examples/60-session-sync/radius.ts @@ -0,0 +1,159 @@ +// STAND-IN for Radius, not a Chord API: what example 60 needs of the server side of session sync, in memory. Per +// session id it keeps a replica store (SQLite in memory, as a Durable Object's SQLite would be), the commit log (what +// R2 would hold; the replica's outbox is moved into it and trimmed after every change), the driver and its epoch. +// Batches and archives cross a "wire" as JSON and JSONL bytes, so the replica checks them as a trust boundary. +// `checkEpochs = false` turns the epoch fence off, to show what the gap analysis warned about. +import { Readable } from "node:stream"; +import { BACKGROUND_CONTEXT as context } from "../../../../src/context/index.ts"; +import { ChordError } from "../../../../src/next/data/index.ts"; +// Internal: the sync layer's batch apply (exported from `chord/next` marked internal) takes the `ChordStore` that +// `openStore` returns behind the app-facing `Store` type. +import { applyBatch } from "../../../../src/next/store/apply-batch.ts"; +import { + decodeJsonl, + importStore, + type OutboxCommit, + openStore, + type Store, + sqliteBackend, +} from "../../../../src/next/store/index.ts"; +import type { ChordStore } from "../../../../src/next/store/store.ts"; +import { openNodeSqliteDatabase } from "../../../../src/sqlite/node.ts"; + +/** What the handshake reports per session: Radius's applied head, the epoch and who drives. */ +export type Known = { readonly lastSeq: number; readonly epoch: number; readonly driver: string }; + +/** A batch as it travels: the commits `from..to` and the uploader's epoch (S5: every batch carries `from` and `to`). */ +export type Batch = { + readonly from: number; + readonly to: number; + readonly epoch: number; + readonly commits: readonly OutboxCommit[]; +}; + +/** An upload's answer: applied (the new head is the acknowledgement) or refused, for an old epoch or by the replica. */ +export type Uploaded = + | { readonly kind: "applied"; readonly lastSeq: number; readonly epoch: number } + | { readonly kind: "refused"; readonly reason: "epoch"; readonly epoch: number } + | { readonly kind: "refused"; readonly reason: "batch"; readonly message: string }; + +type Entry = { + readonly replica: Store; + /** The commit log (R2's part): every commit after `logFloor` (the seq it was bootstrapped at). */ + readonly log: OutboxCommit[]; + readonly logFloor: number; + epoch: number; + driver: string; +}; + +/** The stand-in: sessions by id, for one owner. */ +export class RadiusStandIn { + /** The epoch fence (S7); off only to show the divergence without it. */ + checkEpochs = true; + readonly #sessions = new Map(); + + /** The handshake: every session Radius has, with its `lastSeq`, epoch and driver. */ + handshake(): Record { + const known: Record = {}; + for (const [id, { replica, epoch, driver }] of this.#sessions) + known[id] = { lastSeq: replica.lastSeq, epoch, driver }; + return known; + } + + /** Applies an uploaded batch (JSON) to the session's replica; a first batch from seq 1 makes the session. */ + async upload(id: string, driver: string, wire: string): Promise { + const batch = JSON.parse(wire) as Batch; + let entry = this.#sessions.get(id); + if (entry === undefined && batch.from === 1) entry = this.#create(id, driver, await openReplica()); + if (entry === undefined) return { kind: "refused", reason: "batch", message: "unknown session: bootstrap first" }; + if (this.checkEpochs && batch.epoch !== entry.epoch) + return { kind: "refused", reason: "epoch", epoch: entry.epoch }; + try { + await applyBatch(entry.replica as ChordStore, batch.commits, context); + } catch (error) { + if (!(error instanceof ChordError)) throw error; + return { kind: "refused", reason: "batch", message: error.message }; + } + await this.#moveToLog(entry); + return { kind: "applied", lastSeq: entry.replica.lastSeq, epoch: entry.epoch }; + } + + /** Makes the session from an archive (JSONL bytes): its replica imported, the log starting at its `lastSeq`. */ + async bootstrap(id: string, driver: string, bytes: Uint8Array): Promise { + const backend = sqliteBackend(await openNodeSqliteDatabase(":memory:")); + const replica = await importStore(backend, decodeJsonl(Readable.from([bytes])), context, { outbox: true }); + const entry = this.#create(id, driver, replica); + return { lastSeq: replica.lastSeq, epoch: entry.epoch, driver }; + } + + /** "Continue on Radius": a new epoch, Radius drives; an upload at the old epoch is refused from now on. */ + takeOver(id: string): number { + const entry = this.#entry(id); + entry.epoch += 1; + entry.driver = "radius"; + return entry.epoch; + } + + /** Radius as the driver: `work` writes on the replica (a host on it, closed again), its commits go to the log. */ + async drive(id: string, work: (replica: Store) => Promise): Promise { + const entry = this.#entry(id); + if (entry.driver !== "radius") throw new Error(`radius: ${id} is driven by ${entry.driver}`); + await work(entry.replica); + await this.#moveToLog(entry); + } + + /** "Back to the laptop": a new epoch for `driver`; Radius stops writing. */ + handBack(id: string, driver: string): number { + const entry = this.#entry(id); + entry.epoch += 1; + entry.driver = driver; + return entry.epoch; + } + + /** The log's commits after `afterSeq` (JSON), for a pull. */ + pull(id: string, afterSeq: number): string { + const entry = this.#entry(id); + if (afterSeq < entry.logFloor) throw new Error(`radius: ${id}'s log starts after ${entry.logFloor}`); + return JSON.stringify(entry.log.filter((commit) => commit.seq > afterSeq)); + } + + /** The session's replica, to read it in the example. */ + replica(id: string): Store { + return this.#entry(id).replica; + } + + /** The seqs the log holds, as `from..to`. */ + logRange(id: string): string { + const { log, logFloor } = this.#entry(id); + return log.length === 0 ? `empty after ${logFloor}` : `${log[0]!.seq}..${log.at(-1)!.seq}`; + } + + /** Closes every replica. */ + async close(): Promise { + for (const { replica } of this.#sessions.values()) await replica.close(context); + } + + #create(id: string, driver: string, replica: Store): Entry { + const entry: Entry = { replica, log: [], logFloor: replica.lastSeq, epoch: 1, driver }; + this.#sessions.set(id, entry); + return entry; + } + + #entry(id: string): Entry { + const entry = this.#sessions.get(id); + if (entry === undefined) throw new Error(`radius: no session ${id}`); + return entry; + } + + /** The replica's new commits into the log (the DO writing its commit log to R2), then trimmed from its outbox. */ + async #moveToLog(entry: Entry): Promise { + const outbox = entry.replica.outbox!; + entry.log.push(...(await outbox.read(outbox.floor, 10_000, context))); + await outbox.trim(entry.replica.lastSeq, context); + } +} + +/** A replica as Radius makes it for a session uploaded from seq 1: empty, its outbox on. */ +async function openReplica(): Promise { + return openStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), context, { outbox: true }); +} diff --git a/packages/chord/test/next/examples/60-session-sync/session.ts b/packages/chord/test/next/examples/60-session-sync/session.ts new file mode 100644 index 000000000..b1d4c9e30 --- /dev/null +++ b/packages/chord/test/next/examples/60-session-sync/session.ts @@ -0,0 +1,83 @@ +// The session example 60 syncs: one object per host with a persisted title document and a `messages` table, and a +// helper running a host on a store for the length of some work (one open host per store; a closed host holds no +// claims, so a batch may apply to its store afterwards). +import { BACKGROUND_CONTEXT as context } from "../../../../src/context/index.ts"; +import { arr, int, local, method, obj, str } from "../../../../src/next/contracts/index.ts"; +import { defineState } from "../../../../src/next/documents/index.ts"; +import { createHost, defineFacet, type Reference } from "../../../../src/next/host/index.ts"; +import { defineTable, query, type Store } from "../../../../src/next/store/index.ts"; + +const Messages = defineTable("messages", { + version: 1, + columns: { n: int(), role: str(), text: str() }, + key: ["n"], +}); + +const Title = defineState("title", { + schema: obj({ text: str() }), + version: 1, + initial: () => ({ text: "untitled" }), + persist: true, + checkpointWhen: () => false, +}); + +/** The session's contract: its title, a message (one commit each), a rename (one commit) and the transcript. */ +export const Session = local("example.session", { + version: 1, + params: {}, + members: { + title: obj({ text: str() }), + say: method(obj({ role: str(), text: str() }), int()), + rename: method(str()), + transcript: method(obj({}), arr(str())), + }, +}); + +const facet = defineFacet((host) => { + host.serve(Session, (_, self) => { + const messages = self.table(Messages); + const title = self.state(Title); + const all = query(Messages).orderBy((m) => m.n, "asc"); + return { + title, + say: async ({ role, text }, ctx) => { + const n = (await messages.count(all, ctx)) + 1; + await self.tx(async (tx) => tx.put(messages, { n, role, text }), ctx); + return n; + }, + rename: async (text, ctx) => { + await title.change((draft) => { + draft.text = text; + }, ctx); + return null; + }, + transcript: async (_, ctx) => { + const lines: string[] = []; + for await (const m of messages.rows(all, ctx)) lines.push(`${m.n} ${m.role}: ${m.text}`); + return lines; + }, + }; + }); +}); + +/** Runs a host serving the session on `store` until `work` is done, then closes the host (the store stays open). */ +export async function withSession( + store: Store, + work: (session: Reference) => Promise, +): Promise { + const host = createHost({ store, onError: (error) => console.error("background error:", error) }); + try { + await host.load([{ name: "@example/session", facet }], context); + return await work(await host.get(Session, {}, context)); + } finally { + await host.close(context); + } +} + +/** The session as one line: its title and its messages. */ +export function describe(store: Store): Promise { + return withSession(store, async (session) => { + const lines = await session.transcript({}, context); + return `"${session.title.value.text}" [${lines.join(" | ")}]`; + }); +} diff --git a/packages/chord/test/next/examples/60-session-sync/sync.ts b/packages/chord/test/next/examples/60-session-sync/sync.ts new file mode 100644 index 000000000..e88c7e0b9 --- /dev/null +++ b/packages/chord/test/next/examples/60-session-sync/sync.ts @@ -0,0 +1,101 @@ +// STAND-IN for the laptop's sync layer, not a Chord API: the decided handshake (ask Radius for `lastSeq`, send the +// outbox from `lastSeq + 1`, trim through the answer), the bootstrap from an archive, the eager upload of new commits +// after the last acknowledgement (S5), the split on an old epoch (S7) and the pull of Radius's commits. Built on +// Chord's real pieces: `store.outbox.read`/`trim`, `store.export`, `encodeJsonl`, and `applyBatch` (internal). +import { BACKGROUND_CONTEXT as context } from "../../../../src/context/index.ts"; +import { applyBatch } from "../../../../src/next/store/apply-batch.ts"; +import { encodeJsonl, type OutboxCommit, type Store } from "../../../../src/next/store/index.ts"; +import type { ChordStore } from "../../../../src/next/store/store.ts"; +import type { Batch, RadiusStandIn } from "./radius.ts"; + +/** A session on the laptop: its id, store, the epoch it drives in (a new session: 1) and the `lastSeq` Radius last acknowledged. */ +export type LocalSession = { + readonly id: string; + readonly store: Store; + epoch: number; + acked: number; + /** A split-off copy's origin: the session it was and the last seq they share (S7). */ + readonly origin?: { readonly id: string; readonly seq: number }; +}; + +/** What a sync or a push did. */ +export type Synced = + | { readonly kind: "uploaded"; readonly from: number; readonly to: number; readonly lastSeq: number } + | { readonly kind: "nothing"; readonly lastSeq: number } + | { readonly kind: "bootstrapped"; readonly lastSeq: number; readonly bytes: number } + | { readonly kind: "behind"; readonly lastSeq: number } + | { readonly kind: "split"; readonly session: LocalSession } + | { readonly kind: "refused"; readonly message: string }; + +const DRIVER = "laptop"; + +/** The handshake, then the outbox from Radius's `lastSeq + 1`; an archive when Radius has less than the outbox. */ +export async function sync(session: LocalSession, radius: RadiusStandIn): Promise { + const { store } = session; + const outbox = store.outbox!; + const known = radius.handshake()[session.id]; + if (known === undefined && outbox.floor > 0) return bootstrap(session, radius); + if (known !== undefined && known.lastSeq < outbox.floor) return bootstrap(session, radius); + // Radius ahead: a pull-back (S3, same epoch) or a split (S7); step 6 pulls explicitly. + if (known !== undefined && known.lastSeq > store.lastSeq) return { kind: "behind", lastSeq: known.lastSeq }; + session.acked = known?.lastSeq ?? 0; + await outbox.trim(session.acked, context); + return push(session, radius); +} + +/** Uploads the outbox after the last acknowledgement, without asking first (an eager or resubmitted upload, S5). */ +export async function push(session: LocalSession, radius: RadiusStandIn): Promise { + const { store } = session; + const commits = await store.outbox!.read(session.acked, 10_000, context); + if (commits.length === 0) return { kind: "nothing", lastSeq: session.acked }; + const batch: Batch = { from: commits[0]!.seq, to: commits.at(-1)!.seq, epoch: session.epoch, commits }; + const answer = await radius.upload(session.id, DRIVER, JSON.stringify(batch)); + switch (answer.kind) { + case "applied": + session.acked = answer.lastSeq; + session.epoch = answer.epoch; + await store.outbox!.trim(answer.lastSeq, context); + return { kind: "uploaded", from: batch.from, to: batch.to, lastSeq: answer.lastSeq }; + case "refused": + if (answer.reason === "batch") return { kind: "refused", message: answer.message }; + // S7: no rollback; this copy goes on as a new session that shares the old one's commits through `acked`. + return { + kind: "split", + session: { + id: `${session.id}~split@${session.acked}`, + store, + epoch: 1, + acked: 0, + origin: { id: session.id, seq: session.acked }, + }, + }; + } +} + +/** Uploads the store's archive; Radius imports it and batches go on from its `lastSeq`. */ +async function bootstrap(session: LocalSession, radius: RadiusStandIn): Promise { + const bytes = await archiveOf(session.store); + const known = await radius.bootstrap(session.id, DRIVER, bytes); + session.acked = known.lastSeq; + session.epoch = known.epoch; + await session.store.outbox!.trim(known.lastSeq, context); + return { kind: "bootstrapped", lastSeq: known.lastSeq, bytes: bytes.length }; +} + +/** Applies Radius's commits after the local `lastSeq` (no host may be open on the store: its claims refuse a batch). */ +export async function pull(session: LocalSession, radius: RadiusStandIn): Promise<{ from: number; to: number }> { + const commits = JSON.parse(radius.pull(session.id, session.store.lastSeq)) as OutboxCommit[]; + const from = session.store.lastSeq + 1; + await applyBatch(session.store as ChordStore, commits, context); + // Batch apply writes the commits to this store's outbox too; Radius has them, so they are acknowledged. + session.acked = session.store.lastSeq; + await session.store.outbox!.trim(session.acked, context); + return { from, to: session.store.lastSeq }; +} + +/** The store's archive as JSONL bytes (the commit queue held while it is read). */ +export async function archiveOf(store: Store): Promise { + const parts: Uint8Array[] = []; + for await (const part of encodeJsonl(store.export(context))) parts.push(part); + return Buffer.concat(parts); +} From 342514bdf87b9fcc094b49820e2d636ec95b0a8f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Petr=20Hou=C5=A1ka?= Date: Mon, 2 Nov 2026 21:21:23 +0100 Subject: [PATCH 167/175] feat(coding-agent): list updated packages with version changes in pi update (#11707) ## Problem `pi update`, `pi update --all` and `pi update --extensions` print only `Updating ...` progress lines and a final `Updated packages`. Users cannot see which extensions changed or to which version. Several outdated npm packages in the same scope are installed as one batch, so the progress output does not even name them. ## Change `pi update` now prints a summary of what actually changed: ``` Updated 2 packages: npm:foo 1.2.0 -> 1.3.0 git:github.com/user/bar (project) abc1234 -> def5678 ``` When nothing changed it prints `Packages are already up to date` (or ` is already up to date`). - npm: compares the installed `package.json` version before and after the batch install. The "before" version comes from the existing `shouldUpdateNpmSource` check, so this adds no extra registry calls. - git: compares `git rev-parse HEAD` before and after `updateGit`. Commits are shortened to 7 characters. - Packages whose version or commit did not change are omitted, including npm installs that hit stale registry metadata (those already get the existing warning). ## API `PackageManager.update()` still returns `Promise`. `ProgressEvent.type` gains a `"result"` member. Each changed package is reported through the existing progress callback as: ```ts { type: "result", action: "update", source: "npm:foo", message: "npm:foo 1.2.0 -> 1.3.0" } ``` The CLI collects these messages to print the summary. A separate type keeps the existing contract that each `start` is followed by exactly one `complete` or `error` for the same source. Reusing `complete` would have produced a second `complete` for git packages and a `complete` without a `start` for batched npm packages. The only compatibility cost is for TypeScript consumers with an exhaustive `switch` on `event.type`. ## Tests - `test/package-manager.test.ts`: npm update emits `start`, `complete`, then a `result` with `npm:example@^1.0.0 (project) 1.0.0 -> 1.2.0`; an up-to-date package emits no result. - `test/git-update.test.ts`: git update emits a result with the old and new commit; an up-to-date clone emits no result. `should expand ~ in exclusion and exact patterns` fails locally on Windows. It tests path resolution and is unrelated to this change. --- packages/coding-agent/CHANGELOG.md | 4 ++ .../coding-agent/src/core/package-manager.ts | 70 +++++++++++++++++-- .../coding-agent/src/package-manager-cli.ts | 20 ++++-- packages/coding-agent/test/git-update.test.ts | 11 +++ .../coding-agent/test/package-manager.test.ts | 19 ++++- 5 files changed, 114 insertions(+), 10 deletions(-) diff --git a/packages/coding-agent/CHANGELOG.md b/packages/coding-agent/CHANGELOG.md index 36af238b5..839fe6ae4 100644 --- a/packages/coding-agent/CHANGELOG.md +++ b/packages/coding-agent/CHANGELOG.md @@ -2,6 +2,10 @@ ## [Unreleased] +### Added + +- `pi update` now lists each updated package with its npm version or git commit change, for example `npm:foo 1.2.0 -> 1.3.0`, and prints `Packages are already up to date` when nothing changed. `PackageManager.update()` reports each changed package as a new `result` progress event whose `message` holds that line ([#11707](https://github.com/earendil-works/pi/pull/11707)) + ## [1.5.0] - 2026-11-02 ### New Features diff --git a/packages/coding-agent/src/core/package-manager.ts b/packages/coding-agent/src/core/package-manager.ts index 5f4229b24..b83381a9e 100644 --- a/packages/coding-agent/src/core/package-manager.ts +++ b/packages/coding-agent/src/core/package-manager.ts @@ -108,7 +108,13 @@ export interface PackageInstallFailure { export type MissingSourceAction = "install" | "skip" | "error"; export interface ProgressEvent { - type: "start" | "progress" | "complete" | "error"; + /** + * `start` is followed by exactly one `complete` or `error` for the same source. + * `progress` carries an intermediate message or warning. + * `result` reports an outcome outside that pairing, e.g. one changed package + * after update(): `npm:foo 1.2.0 -> 1.3.0`. + */ + type: "start" | "progress" | "complete" | "error" | "result"; action: "install" | "remove" | "update" | "clone" | "pull"; source: string; message?: string; @@ -193,6 +199,15 @@ interface GitUpdateTarget extends ConfiguredUpdateSource { parsed: GitSource; } +/** A configured package whose installed npm version or git commit changed during update(). */ +interface UpdatedPackage extends ConfiguredUpdateSource { + type: "npm" | "git"; + /** npm version or git commit before the update; undefined if it was not installed. */ + from?: string; + /** npm version or git commit after the update. */ + to?: string; +} + interface ResourceAccumulator { extensions: Map; skills: Map; @@ -1204,10 +1219,8 @@ export class DefaultPackageManager implements PackageManager { const npmCheckResults = await this.runWithConcurrency(npmCheckTasks, UPDATE_CHECK_CONCURRENCY); const userNpmUpdates: NpmUpdateTarget[] = []; const projectNpmUpdates: NpmUpdateTarget[] = []; - for (const result of npmCheckResults) { - if (!result.shouldUpdate) { - continue; - } + const npmUpdates = npmCheckResults.filter((result) => result.shouldUpdate); + for (const result of npmUpdates) { if (result.entry.scope === "user") { userNpmUpdates.push(result.entry); } else { @@ -1222,17 +1235,64 @@ export class DefaultPackageManager implements PackageManager { if (projectNpmUpdates.length > 0) { tasks.push(this.updateNpmBatch(projectNpmUpdates, "project")); } + const gitUpdates: UpdatedPackage[] = []; if (gitCandidates.length > 0) { const gitTasks = gitCandidates.map( (entry) => async () => this.withProgress("update", entry.source, `Updating ${entry.source}...`, async () => { + const targetDir = this.getGitInstallPath(entry.parsed, entry.scope); + const from = await this.getGitHead(targetDir); await this.updateGit(entry.parsed, entry.scope); + const to = await this.getGitHead(targetDir); + if (from !== to) { + gitUpdates.push({ source: entry.source, type: "git", scope: entry.scope, from, to }); + } }), ); tasks.push(this.runWithConcurrency(gitTasks, GIT_UPDATE_CONCURRENCY).then(() => {})); } await Promise.all(tasks); + + const updated: UpdatedPackage[] = []; + for (const { entry } of npmUpdates) { + const installedPath = this.getManagedNpmInstallPath(entry.parsed, entry.scope); + const to = existsSync(installedPath) ? this.getInstalledNpmVersion(installedPath) : undefined; + if (to !== entry.installedVersion) { + updated.push({ source: entry.source, type: "npm", scope: entry.scope, from: entry.installedVersion, to }); + } + } + // Keep git results in configured order, independent of completion order. + for (const entry of gitCandidates) { + const update = gitUpdates.find((u) => u.source === entry.source && u.scope === entry.scope); + if (update) updated.push(update); + } + + // Report each changed package, e.g. "npm:foo 1.2.0 -> 1.3.0". + for (const pkg of updated) { + const format = (value: string | undefined) => + value === undefined ? "(none)" : pkg.type === "git" ? value.slice(0, 7) : value; + const scope = pkg.scope === "project" ? " (project)" : ""; + this.emitProgress({ + type: "result", + action: "update", + source: pkg.source, + message: `${pkg.source}${scope} ${format(pkg.from)} -> ${format(pkg.to)}`, + }); + } + } + + private async getGitHead(targetDir: string): Promise { + if (!existsSync(targetDir)) return undefined; + try { + const head = await this.runCommandCapture("git", ["rev-parse", "HEAD"], { + cwd: targetDir, + timeoutMs: NETWORK_TIMEOUT_MS, + }); + return head.trim() || undefined; + } catch { + return undefined; + } } private async shouldUpdateNpmSource( diff --git a/packages/coding-agent/src/package-manager-cli.ts b/packages/coding-agent/src/package-manager-cli.ts index eecf962ed..911d0c84f 100644 --- a/packages/coding-agent/src/package-manager-cli.ts +++ b/packages/coding-agent/src/package-manager-cli.ts @@ -1027,8 +1027,12 @@ export async function handlePackageCommand( const packageManager = new DefaultPackageManager({ cwd, agentDir, settingsManager }); + // Update result events describe a changed package, e.g. "npm:foo 1.2.0 -> 1.3.0". + const updatedPackages: string[] = []; packageManager.setProgressCallback((event) => { - if (event.type === "start") { + if (event.type === "result" && event.action === "update" && event.message) { + updatedPackages.push(event.message); + } else if (event.type === "start") { process.stdout.write(chalk.dim(`${event.message}\n`)); } else if (event.type === "progress" && event.message) { process.stdout.write(chalk.yellow(`${event.message}\n`)); @@ -1099,10 +1103,18 @@ export async function handlePackageCommand( if (updateTargetIncludesExtensions(target)) { const updateSource = target.type === "extensions" ? target.source : undefined; await packageManager.update(updateSource); - if (updateSource) { - console.log(chalk.green(`Updated ${updateSource}`)); + if (updatedPackages.length === 0) { + console.log( + chalk.green( + updateSource ? `${updateSource} is already up to date` : "Packages are already up to date", + ), + ); } else { - console.log(chalk.green("Updated packages")); + const count = updatedPackages.length; + console.log(chalk.green(`Updated ${count} package${count === 1 ? "" : "s"}:`)); + for (const message of updatedPackages) { + console.log(` ${message}`); + } } } if (updateTargetIncludesSelf(target)) { diff --git a/packages/coding-agent/test/git-update.test.ts b/packages/coding-agent/test/git-update.test.ts index 8b771f688..849383aa2 100644 --- a/packages/coding-agent/test/git-update.test.ts +++ b/packages/coding-agent/test/git-update.test.ts @@ -150,8 +150,13 @@ describe("DefaultPackageManager git update", () => { } }; + const updated: string[] = []; + packageManager.setProgressCallback((event) => { + if (event.type === "result" && event.action === "update" && event.message) updated.push(event.message); + }); await packageManager.update(); + expect(updated).toEqual([]); expect(executedCommands).toContain( "git fetch --prune --no-tags origin +refs/heads/main:refs/remotes/origin/main", ); @@ -165,14 +170,20 @@ describe("DefaultPackageManager git update", () => { it("should update to latest commit when remote has new commits", async () => { setupRemoteAndInstall(); expect(getFileContent(installedDir, "extension.ts")).toBe("// v1"); + const oldCommit = getCurrentCommit(installedDir); // Add a new commit to remote const newCommit = createCommit(remoteDir, "extension.ts", "// v2", "Second commit"); // Update via package manager (no args = uses settings) + const updated: string[] = []; + packageManager.setProgressCallback((event) => { + if (event.type === "result" && event.action === "update" && event.message) updated.push(event.message); + }); await packageManager.update(); // Verify update succeeded + expect(updated).toEqual([`${gitSource} ${oldCommit.slice(0, 7)} -> ${newCommit.slice(0, 7)}`]); expect(getCurrentCommit(installedDir)).toBe(newCommit); expect(getFileContent(installedDir, "extension.ts")).toBe("// v2"); }); diff --git a/packages/coding-agent/test/package-manager.test.ts b/packages/coding-agent/test/package-manager.test.ts index b5895b331..7d01a4ece 100644 --- a/packages/coding-agent/test/package-manager.test.ts +++ b/packages/coding-agent/test/package-manager.test.ts @@ -2612,10 +2612,21 @@ export default function(api) { api.registerTool({ name: "test", description: "te const runCommandCaptureSpy = vi .spyOn(packageManager as any, "runCommandCapture") .mockResolvedValue('["1.0.0","1.2.0"]'); - const runCommandSpy = vi.spyOn(packageManager as any, "runCommand").mockResolvedValue(undefined); + const runCommandSpy = vi.spyOn(packageManager as any, "runCommand").mockImplementation(async () => { + writeFileSync(join(installedPath, "package.json"), JSON.stringify({ name: "example", version: "1.2.0" })); + }); + const events: ProgressEvent[] = []; + packageManager.setProgressCallback((event) => events.push(event)); await packageManager.update("npm:example"); + // The result is a separate event, so each start still has exactly one complete. + expect(events.map((event) => [event.type, event.source, event.message])).toEqual([ + ["start", "npm:example@^1.0.0", "Updating npm:example@^1.0.0..."], + ["complete", "npm:example@^1.0.0", undefined], + ["result", "npm:example@^1.0.0", "npm:example@^1.0.0 (project) 1.0.0 -> 1.2.0"], + ]); + expect(runCommandCaptureSpy).toHaveBeenCalledWith( "npm", ["view", "example@^1.0.0", "version", "--json"], @@ -2647,8 +2658,14 @@ export default function(api) { api.registerTool({ name: "test", description: "te .mockResolvedValue('["1.0.0","1.3.1","1.0.2"]'); const runCommandSpy = vi.spyOn(packageManager as any, "runCommand").mockResolvedValue(undefined); + const updated: string[] = []; + packageManager.setProgressCallback((event) => { + if (event.type === "result" && event.action === "update" && event.message) updated.push(event.message); + }); await packageManager.update("npm:example"); + expect(updated).toEqual([]); + expect(runCommandCaptureSpy).toHaveBeenCalledWith( "npm", ["view", "example@^1.0.0", "version", "--json"], From 128912d246d078e9f59a61ba634bbfb996612e14 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 20:26:39 +0000 Subject: [PATCH 168/175] docs(chord): the terminal's sandbox is a child process --- packages/chord/docs/next/decisions.md | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 70bf483f2..d3b47e00a 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2299,7 +2299,8 @@ validation and copies the v2 kernel added around Delta. `useHold(Type, params)`, `useWatch(watchable)`, `useRegistry(R)`, `useRefState(ref)`; releasing holds on unmount, and a component unmounted by a reload never counts as leaving (design §21). No components, no styling; non-React apps never load it. -- **Slice 11, P8: the terminal client always runs sandboxed (owner, 2026-11-01: "the terminal should always use the +- **Slice 11, P8: the terminal client always runs sandboxed (**superseded in part** by "The terminal's sandbox is a + child process", 2026-11-02: no longer a worker thread) (owner, 2026-11-01: "the terminal should always use the sandbox approach"):** `chord/next/tui` is a Node client host that connects with a token (decision "The app kit, point 3", (c)), downloads the base bundle and the TUI plugin scripts from the server and runs the app's TUI entry, always inside p2's sandbox: a permission-dropped worker thread with a bridge terminal, the trusted launcher outside it @@ -2356,6 +2357,22 @@ validation and copies the v2 kernel added around Delta. restarts it. Usually nothing runs: Radius receiving uploads runs no host for a session it does not drive, and taking a session back pulls before its worker starts. A live view of a replica (watching a session continue elsewhere) is a later refinement: fewer pulls while watched, or a read-only view without the worker. +- **The terminal's sandbox is a child process (owner, 2026-11-02: "yes"; refines "Slice 11, P8"; slice 11 recheck, + Opus E1, E2):** the hardened worker thread was escaped twice in one review round, because downloaded code shares + one JavaScript realm with Node's internals: E1 replaced `Set.prototype.has`, which the `require` allowlist used, and + `require("tty")` wrote raw to the terminal; E2 patched `EventEmitter.prototype.emit` and caught Node's real + `process` when Node emitted a warning on it (environment, `kill`, every builtin). Each fix closes one road and the + next reachable internal is another. So the sandbox is a child process of the launcher's Node binary: stdin, stdout + and stderr are pipes to the launcher (no descriptor is the terminal), its environment empty (only + `LD_LIBRARY_PATH` copied when the launcher has one: needed to run the same binary, set by the user), its directory + a private empty one, `--permission` with nothing granted (no file system, network, child processes, workers, + addons, inspector, WASI), `--no-experimental-sqlite` (SQLite opens files past the permission model, Node gist + https://gist.github.com/badlogic/a83cd80d0669dbcf77e029ac7b25ac96). Messages are length-prefixed binary frames on + the pipes, not Node's IPC. The JavaScript layer stays as a second barrier (evaluation only through the bootstrap, + the builtin allowlist, a frozen substitute `process`); the V8 code-generation toggle goes (the bridge evaluates + strings anyway). Known gaps, accepted until Node closes them: the child can signal the user's processes + (`process.kill`) and allocate memory outside its heap. Cost: one more Node process per terminal session (measured + by the build against the worker). Back to a worker only if Node ships per-worker permissions. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 10cf3264fbbb42d02f7dee45a8b19adcf9085e4e Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 20:26:54 +0000 Subject: [PATCH 169/175] =?UTF-8?q?docs(chord):=20design=20=C2=A723,=20the?= =?UTF-8?q?=20terminal's=20sandbox=20is=20a=20child=20process?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- packages/chord/docs/next/design.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/packages/chord/docs/next/design.md b/packages/chord/docs/next/design.md index 4c56352f2..69ebfd0aa 100644 --- a/packages/chord/docs/next/design.md +++ b/packages/chord/docs/next/design.md @@ -2305,8 +2305,10 @@ Decision "Slice 11's points" and "Slice 11, P3" to "P9". the page); pages learn of them through an object the web component serves (the current set); a base code change reloads the page. - `chord/next/react`: `HostProvider`, `useHold`, `useWatch`, `useRegistry`, `useRefState`. -- `chord/next/tui`: a Node client connecting with a token, always running the downloaded code in a sandbox (a - permission-dropped worker thread, a bridge terminal, the trusted launcher outside); a component scope (`hold`, +- `chord/next/tui`: a Node client connecting with a token, always running the downloaded code in a sandbox (decision + "The terminal's sandbox is a child process": a child process on pipes with an empty environment, `--permission` + granting nothing and no SQLite; the builtin allowlist and a frozen substitute `process` inside; a bridge terminal; + the trusted launcher outside, confirming copy, open and attach on its own prompt); a component scope (`hold`, `watch`, `registry`, released on dispose, one `changed` callback for the app's redraw), no pi-tui dependency. ### Channels From 65c85bda2c7195d33078e1bdb28508882133bfde Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 20:47:13 +0000 Subject: [PATCH 170/175] docs(chord): slice 11 spec, the sandbox is a child process --- packages/chord/docs/next/slice-11.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/packages/chord/docs/next/slice-11.md b/packages/chord/docs/next/slice-11.md index efb2cbb43..43403bd20 100644 --- a/packages/chord/docs/next/slice-11.md +++ b/packages/chord/docs/next/slice-11.md @@ -72,8 +72,9 @@ releases, presence is the app's. No components, no styling. - **Launcher** (trusted, a Node process): `startTerminal({ url, token | tokenFile, pairing?, ... })` connects (a token from the owner token file, slice 13, or device pairing), downloads the base bundle and the TUI kind's plugin scripts - over HTTP, and starts the **sandbox**: a worker thread with dropped permissions (Node's permission model: no file - system, no child processes, no network except the bridge; p2's `tui-sandbox.ts`), always (P8), whose runtime gets a + over HTTP, and starts the **sandbox**: a child process on pipes (decision "The terminal's sandbox is a child + process", replacing p2's worker thread: empty environment, Node's permission model granting nothing, no SQLite; no + terminal, file system, child processes or network except the bridge), always (P8), whose runtime gets a bridge terminal (input events in, frames out), the Chord link (messages relayed by the launcher) and the evaluated scripts. The launcher keeps credentials, the real terminal and local preferences, and exposes a small set of actions to the sandbox (clipboard, open URL), each confirmed by the launcher's own UI when it touches the machine. From 806cdd68e312f8cb80efe367c661bea0bd3ee537 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Petr=20Hou=C5=A1ka?= Date: Mon, 2 Nov 2026 21:49:07 +0100 Subject: [PATCH 171/175] fix(coding-agent): only report packages as up to date when pi update checked them (#11711) ## Problem #11707 prints `Packages are already up to date` whenever no package changed. In two cases that claim is false. **1. npm left the old version installed.** Extension updates install `@latest` or the configured range. If npm resolves stale registry metadata, it reinstalls the old version and exits 0. The existing check warns, then the summary contradicts it: ``` Updating npm:foo... npm:foo is still 1.2.0 (registry reports 1.3.0); npm registry metadata may be stale, try again in a few minutes Packages are already up to date ``` **2. Offline mode.** With `--offline` or `PI_OFFLINE=1`, `updateConfiguredSources()` returns before checking anything, and the CLI still prints `Packages are already up to date`. ## Change The CLI only prints "already up to date" when the update ran and printed no warning: | Situation | Output | |---|---| | Packages changed | `Updated N packages:` and the list (unchanged) | | Nothing changed, no warnings | `Packages are already up to date` (unchanged) | | Nothing changed, a warning was printed | `No packages were updated, see the warnings above` (or ` was not updated, ...`) | | Offline mode | `Skipped package updates (offline mode)` | The CLI counts `progress` update events, which are always warnings, and does not try to map them to packages. That is deliberate: the git warning about untracked files that could not be cleaned is printed even though the commit did change, so a warning does not mean "this package was not updated". In offline mode, `update()` still runs, so `pi update ` still fails with "No matching package". `isOfflineModeEnabled()` is now exported from `core/package-manager.ts` (not from the package index) so the CLI uses the same check. The changelog entry from #11707 is adjusted, since it is unreleased. ## Tests `test/package-command-paths.test.ts`, both through `main()`: - npm `view` reports 1.3.0 but `install` leaves 1.2.0: the stale-metadata warning is printed, then `No packages were updated, see the warnings above`, and no "up to date". - `PI_OFFLINE=1`: `Skipped package updates (offline mode)`, no "up to date". Both fail without the fix. `cycles project package overrides in config local mode`, `prints a pnpm metadata hint when self-update fails` and `should expand ~ in exclusion and exact patterns` fail locally on Windows, on `origin/main` as well. --- packages/coding-agent/CHANGELOG.md | 2 +- .../coding-agent/src/core/package-manager.ts | 3 +- .../coding-agent/src/package-manager-cli.ts | 18 ++++++- .../test/package-command-paths.test.ts | 48 +++++++++++++++++++ 4 files changed, 67 insertions(+), 4 deletions(-) diff --git a/packages/coding-agent/CHANGELOG.md b/packages/coding-agent/CHANGELOG.md index 839fe6ae4..c987db53b 100644 --- a/packages/coding-agent/CHANGELOG.md +++ b/packages/coding-agent/CHANGELOG.md @@ -4,7 +4,7 @@ ### Added -- `pi update` now lists each updated package with its npm version or git commit change, for example `npm:foo 1.2.0 -> 1.3.0`, and prints `Packages are already up to date` when nothing changed. `PackageManager.update()` reports each changed package as a new `result` progress event whose `message` holds that line ([#11707](https://github.com/earendil-works/pi/pull/11707)) +- `pi update` now lists each updated package with its npm version or git commit change, for example `npm:foo 1.2.0 -> 1.3.0`, and prints `Packages are already up to date` when nothing changed. When nothing changed but an update printed a warning, for example because npm left the old version installed after resolving stale registry metadata, it prints `No packages were updated, see the warnings above` instead; in offline mode it prints `Skipped package updates (offline mode)`. `PackageManager.update()` reports each changed package as a new `result` progress event whose `message` holds that line ([#11707](https://github.com/earendil-works/pi/pull/11707), [#11711](https://github.com/earendil-works/pi/pull/11711)) ## [1.5.0] - 2026-11-02 diff --git a/packages/coding-agent/src/core/package-manager.ts b/packages/coding-agent/src/core/package-manager.ts index b83381a9e..5fd93754e 100644 --- a/packages/coding-agent/src/core/package-manager.ts +++ b/packages/coding-agent/src/core/package-manager.ts @@ -60,7 +60,8 @@ const NETWORK_TIMEOUT_MS = 10000; const UPDATE_CHECK_CONCURRENCY = 4; const GIT_UPDATE_CONCURRENCY = 4; -function isOfflineModeEnabled(): boolean { +/** Whether PI_OFFLINE is set; package installs and updates then skip all network access. */ +export function isOfflineModeEnabled(): boolean { const value = process.env.PI_OFFLINE; if (!value) return false; return value === "1" || value.toLowerCase() === "true" || value.toLowerCase() === "yes"; diff --git a/packages/coding-agent/src/package-manager-cli.ts b/packages/coding-agent/src/package-manager-cli.ts index 911d0c84f..93a8fd4f8 100644 --- a/packages/coding-agent/src/package-manager-cli.ts +++ b/packages/coding-agent/src/package-manager-cli.ts @@ -32,7 +32,7 @@ import { } from "./config.ts"; import type { InlineExtension } from "./core/extensions/types.ts"; import { ModelRuntime } from "./core/model-runtime.ts"; -import { DefaultPackageManager } from "./core/package-manager.ts"; +import { DefaultPackageManager, isOfflineModeEnabled } from "./core/package-manager.ts"; import { type AppMode, resolveProjectTrusted } from "./core/project-trust.ts"; import { DefaultResourceLoader, formatPackageInstallWarning, isBuiltinExtension } from "./core/resource-loader.ts"; import { SettingsManager } from "./core/settings-manager.ts"; @@ -1029,12 +1029,15 @@ export async function handlePackageCommand( // Update result events describe a changed package, e.g. "npm:foo 1.2.0 -> 1.3.0". const updatedPackages: string[] = []; + // Progress messages are warnings, e.g. npm reinstalled the old version from stale registry metadata. + let warningCount = 0; packageManager.setProgressCallback((event) => { if (event.type === "result" && event.action === "update" && event.message) { updatedPackages.push(event.message); } else if (event.type === "start") { process.stdout.write(chalk.dim(`${event.message}\n`)); } else if (event.type === "progress" && event.message) { + warningCount++; process.stdout.write(chalk.yellow(`${event.message}\n`)); } }); @@ -1103,7 +1106,18 @@ export async function handlePackageCommand( if (updateTargetIncludesExtensions(target)) { const updateSource = target.type === "extensions" ? target.source : undefined; await packageManager.update(updateSource); - if (updatedPackages.length === 0) { + // Only claim "up to date" when the update actually checked and nothing warned. + if (isOfflineModeEnabled()) { + console.log(chalk.yellow("Skipped package updates (offline mode)")); + } else if (updatedPackages.length === 0 && warningCount > 0) { + console.log( + chalk.yellow( + updateSource + ? `${updateSource} was not updated, see the warnings above` + : "No packages were updated, see the warnings above", + ), + ); + } else if (updatedPackages.length === 0) { console.log( chalk.green( updateSource ? `${updateSource} is already up to date` : "Packages are already up to date", diff --git a/packages/coding-agent/test/package-command-paths.test.ts b/packages/coding-agent/test/package-command-paths.test.ts index a12acd856..711cf3f96 100644 --- a/packages/coding-agent/test/package-command-paths.test.ts +++ b/packages/coding-agent/test/package-command-paths.test.ts @@ -369,6 +369,54 @@ if (process.platform !== "win32") fs.chmodSync(piPath, 0o755); } }); + /** Install `npm:fake-package` at `installedVersion` with an npm whose registry reports `registryVersion`. */ + function prepareStaleNpmPackage(installedVersion: string, registryVersion: string): void { + const installedPath = join(agentDir, "npm", "node_modules", "fake-package"); + mkdirSync(installedPath, { recursive: true }); + writeFileSync( + join(installedPath, "package.json"), + JSON.stringify({ name: "fake-package", version: installedVersion }), + ); + // `npm view` reports the newer version, but `npm install` succeeds without changing anything. + const fakeNpmPath = join(tempDir, "fake-stale-npm.cjs"); + writeFileSync( + fakeNpmPath, + `if (process.argv.includes("view")) console.log(${JSON.stringify(JSON.stringify(registryVersion))});`, + ); + writeFileSync( + join(agentDir, "settings.json"), + JSON.stringify({ packages: ["npm:fake-package"], npmCommand: [originalExecPath, fakeNpmPath] }), + ); + } + + it("does not report packages as up to date when npm left an older version installed", async () => { + prepareStaleNpmPackage("1.2.0", "1.3.0"); + const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); + const stdoutSpy = vi.spyOn(process.stdout, "write").mockImplementation(() => true); + + await expect(main(["update", "--extensions"])).resolves.toBeUndefined(); + + const progress = stdoutSpy.mock.calls.map(([chunk]) => String(chunk)).join(""); + const stdout = logSpy.mock.calls.map(([message]) => String(message)).join("\n"); + expect(progress).toContain("npm:fake-package is still 1.2.0 (registry reports 1.3.0)"); + expect(stdout).toContain("No packages were updated, see the warnings above"); + expect(stdout).not.toContain("up to date"); + }); + + it("does not report packages as up to date in offline mode", async () => { + prepareStaleNpmPackage("1.2.0", "1.3.0"); + // main() writes both variables in offline mode; stubbing them lets afterEach restore them. + vi.stubEnv("PI_OFFLINE", "1"); + vi.stubEnv("PI_SKIP_VERSION_CHECK", "1"); + const logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); + + await expect(main(["update", "--extensions"])).resolves.toBeUndefined(); + + const stdout = logSpy.mock.calls.map(([message]) => String(message)).join("\n"); + expect(stdout).toContain("Skipped package updates (offline mode)"); + expect(stdout).not.toContain("up to date"); + }); + it("uses saved project trust during update", async () => { mkdirSync(join(projectDir, ".pi"), { recursive: true }); const fakeNpmPath = join(tempDir, "fake-trusted-project-npm.cjs"); From 9c248717daee3e1d349e0c6858e3d294e9df5da1 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 21:28:52 +0000 Subject: [PATCH 172/175] docs(chord): messages and languages in the app kit decided --- packages/chord/docs/next/decisions.md | 23 +++++++++++++++++++++++ packages/chord/docs/next/design.md | 15 +++++++++++++++ packages/chord/docs/next/slices.md | 5 +++++ 3 files changed, 43 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index d3b47e00a..20049a107 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2373,6 +2373,29 @@ validation and copies the v2 kernel added around Delta. strings anyway). Known gaps, accepted until Node closes them: the child can signal the user's processes (`process.kill`) and allocate memory outside its heap. Cost: one more Node process per terminal session (measured by the build against the worker). Back to a worker only if Node ships per-worker permissions. +- **Messages and languages in the app kit (owner, 2026-11-03: "i18n proposal is a go"; proposal `docs/next/i18n/`):** + text people read is translated per viewer, never per server. (1) **Messages in code, per plugin:** + `defineMessages(namespace, { en: {...}, de: {...} })` in the app's shared API returns a typed `t`; the first catalog + defines the keys and argument types, every other catalog must match them (a missing key or a wrong argument is a + type error); values may be functions (plurals, parameters in each language's order); dates and numbers through + `Intl` helpers (`formatTime`, `formatNumber`) in the viewer's locale. Named keys (`newNote`), the English catalog the + source of truth. A plugin keeps its messages in one shared file (`messages.ts`) every facet that renders imports. + (2) **Languages are contributions:** a `Translations` registry (`{ namespace, locale, messages, override? }`) in the + shared API; `MessagesOf` types a translation; a missing key falls back to the locale's base (`pt-BR` to + `pt`), then to the first catalog, key by key, reported once; two contributions for one namespace and locale: the first + by plugin name wins, the other reported; a plugin's own catalogs are its built-in contributions, `override: true` + replaces one. Contributions are per host, so a translation plugin keeps its catalogs in a shared file and contributes + them from each facet that renders (web, tui; server and worker facets only when they render final text, an email or + a push notification). Catalogs never cross links (they hold functions). (3) **The locale is a user preference** + (kept per principal like the theme choice), defaulting to the browser's `navigator.languages` or the system locale + in the terminal, matched against the locales the catalogs offer; switching re-renders live (`useMessages(t)` on the + web, the scope's redraw in the terminal), nothing reloads; the kit sets `lang` and `dir` on the page's root + (right-to-left through logical CSS properties, the app's components). (4) **Servers and workers send keys, not + text:** `t.message(key, ...args)` builds typed data `{ namespace, key, params }` (a `Message<"notes">`) for object + state, notices, tool summaries and errors shown to people (a code plus parameters); each client renders it with + `t.render` in its own language (checked at runtime across a link: an unknown key shows the key). Never translated: + the model's prompts and answers, log lines, error codes. No translator file format now; an export/import tool to JSON + when someone needs it. Built after slice 11 (carry-over "After slice 11"). - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a diff --git a/packages/chord/docs/next/design.md b/packages/chord/docs/next/design.md index 69ebfd0aa..da87d32a7 100644 --- a/packages/chord/docs/next/design.md +++ b/packages/chord/docs/next/design.md @@ -2311,6 +2311,21 @@ Decision "Slice 11's points" and "Slice 11, P3" to "P9". the trusted launcher outside, confirming copy, open and attach on its own prompt); a component scope (`hold`, `watch`, `registry`, released on dispose, one `changed` callback for the app's redraw), no pi-tui dependency. +### Messages and languages (after slice 11) + +Decision "Messages and languages in the app kit". Text people read is rendered per viewer, never by a server. + +- **Messages in code:** `defineMessages(namespace, { en, de, ... })` per plugin, in a shared `messages.ts` every + rendering facet imports; typed `t(key, ...args)`, keys and signatures from the first catalog; functions for plurals + and word order; `formatTime`, `formatNumber` through `Intl` in the viewer's locale. +- **Translations** registry (`{ namespace, locale, messages, override? }`), per host: a translation plugin contributes + its shared catalog file from each facet that renders. Fallback key by key: the locale's base, then the first catalog + (reported once); conflicts: the first by plugin name wins. Catalogs never cross links. +- **The locale** is a per-principal preference (device default, live switch, `lang` and `dir` on the page). +- **Messages as data:** `t.message(key, ...args)` gives `{ namespace, key, params }` for state, notices and errors; + each client renders it with `t.render` (checked at runtime across a link). Prompts, model output, logs and error + codes are never translated. + ### Channels WebSocket byte channels with acknowledged windows, separate from object links, wait for a second user after the diff --git a/packages/chord/docs/next/slices.md b/packages/chord/docs/next/slices.md index a6472a274..808044528 100644 --- a/packages/chord/docs/next/slices.md +++ b/packages/chord/docs/next/slices.md @@ -201,6 +201,11 @@ deleted (the slice's spec and code are the record). Moved here from `carry-over. ### After slice 11 +- **Messages and languages in the app kit** (decision "Messages and languages in the app kit", proposal + `docs/next/i18n/`): `defineMessages`, `t.message`/`t.render`, `MessagesOf`, the `Translations` registry with + fallbacks and conflicts, the locale preference per principal (default from the device, live switch, `lang`/`dir` on + the page), `useMessages` and the terminal scope's redraw, `formatTime`/`formatNumber`; tests and an example (the + notes plugin in English and German, a French translation plugin, a server message rendered per viewer). - **Small examples** (owner, 2026-10-30: the 7c examples are complete but complex): once HTTP, workers, web and TUI exist, a pass of small, easy-to-read examples, each showing one thing a plugin author or app does. From 5bc41451654be22e79cc1a89ccc54a37f6d56691 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 21:15:32 +0000 Subject: [PATCH 173/175] feat(chord): Chord v2 web and terminal surfaces (slice 11) - Web: base bundles built from the app's entry at start (content-hashed, served forever), plugin scripts that register themselves and one combined file, WebScripts/TuiScripts showing the current set, startPage with the connection loop, in-place plugin reloads and a reload on a new base bundle. - React bindings: HostProvider, useHold, useWatch, useRegistry, useRefState. - Terminal: startTerminal, a launcher that signs in with a token, a token file or device pairing and always runs the app's terminal code in a sandboxed child process (pipes, empty environment, --permission granting nothing, no SQLite, a builtin allowlist and a frozen process inside); output filter, confirmed copy/open/attach actions; startTerminalRuntime and TuiScope inside. - Themes: the Themes registry, ThemeChoice per principal, CSS variables on the page, terminal colours by role; CSS modules in plugin builds. - App kit: startServer gains web and tui; startApp renamed startServer, exportAddress/importAddress renamed exportData/importData and export no longer stops the worker; ephemeral sessions end with their client's last link. - Examples 56-web, 57-tui, 58-themes, 59-chat. --- package-lock.json | 65 +++ packages/chord/CHANGELOG.md | 3 +- packages/chord/docs/next/design.md | 8 +- .../docs/next/guide-plugins-hosts/book.md | 19 +- .../docs/next/guide-plugins-hosts/index.html | 19 +- packages/chord/docs/next/plugin-authoring.md | 12 + packages/chord/docs/next/slices.md | 26 +- packages/chord/package.json | 8 + packages/chord/src/next-node/app/README.md | 54 +- packages/chord/src/next-node/app/app.ts | 205 +++++--- packages/chord/src/next-node/app/archive.ts | 129 +++-- packages/chord/src/next-node/app/data.ts | 52 ++ .../src/next-node/app/ephemeral-sessions.ts | 80 +++ packages/chord/src/next-node/app/index.ts | 4 +- packages/chord/src/next-node/app/layout.ts | 4 +- packages/chord/src/next-node/app/sessions.ts | 45 +- packages/chord/src/next-node/app/worker.ts | 9 +- packages/chord/src/next-node/http/README.md | 7 +- packages/chord/src/next-node/http/headers.ts | 9 +- packages/chord/src/next-node/http/http.ts | 17 +- packages/chord/src/next-node/http/router.ts | 11 + packages/chord/src/next-node/http/upgrades.ts | 6 +- .../chord/src/next-node/plugins/README.md | 7 +- packages/chord/src/next-node/plugins/build.ts | 35 +- .../src/next-node/plugins/css-modules.ts | 65 +++ .../chord/src/next-node/plugins/evaluate.ts | 20 +- .../chord/src/next-node/plugins/runtime.ts | 155 +----- .../chord/src/next-node/surfaces/README.md | 40 ++ .../chord/src/next-node/surfaces/bundle.ts | 71 +++ .../chord/src/next-node/surfaces/current.ts | 70 +++ .../chord/src/next-node/surfaces/index.ts | 14 + .../chord/src/next-node/surfaces/published.ts | 159 ++++++ .../chord/src/next-node/surfaces/served.ts | 50 ++ packages/chord/src/next-node/surfaces/tui.ts | 67 +++ packages/chord/src/next-node/surfaces/web.ts | 137 +++++ packages/chord/src/next-node/tui/README.md | 75 +++ packages/chord/src/next-node/tui/actions.ts | 212 ++++++++ .../chord/src/next-node/tui/credentials.ts | 71 +++ packages/chord/src/next-node/tui/filter.ts | 75 +++ packages/chord/src/next-node/tui/index.ts | 8 + packages/chord/src/next-node/tui/launcher.ts | 466 ++++++++++++++++++ packages/chord/src/next-node/tui/pipe.ts | 132 +++++ packages/chord/src/next-node/tui/sandbox.ts | 450 +++++++++++++++++ packages/chord/src/next-react/README.md | 14 + packages/chord/src/next-react/hooks.ts | 79 +++ packages/chord/src/next-react/index.ts | 5 + packages/chord/src/next-react/stores.ts | 45 ++ packages/chord/src/next-tui/README.md | 29 ++ packages/chord/src/next-tui/bridge.ts | 69 +++ packages/chord/src/next-tui/channel.ts | 163 ++++++ packages/chord/src/next-tui/frames.ts | 51 ++ packages/chord/src/next-tui/index.ts | 14 + packages/chord/src/next-tui/runtime.ts | 155 ++++++ packages/chord/src/next-tui/scope.ts | 81 +++ packages/chord/src/next-tui/terminal.ts | 45 ++ packages/chord/src/next-tui/theme.ts | 44 ++ packages/chord/src/next-web/README.md | 25 + packages/chord/src/next-web/environment.ts | 37 ++ packages/chord/src/next-web/index.ts | 6 + packages/chord/src/next-web/page.ts | 85 ++++ packages/chord/src/next-web/scripts.ts | 79 +++ packages/chord/src/next-web/themes.ts | 38 ++ packages/chord/src/next/index.ts | 1 + packages/chord/src/next/link/values.ts | 14 +- packages/chord/src/next/surfaces/README.md | 37 ++ .../chord/src/next/surfaces/active-theme.ts | 63 +++ packages/chord/src/next/surfaces/client.ts | 216 ++++++++ .../chord/src/next/surfaces/connection.ts | 191 +++++++ packages/chord/src/next/surfaces/index.ts | 42 ++ packages/chord/src/next/surfaces/loaded.ts | 152 ++++++ .../chord/src/next/surfaces/registered.ts | 77 +++ packages/chord/src/next/surfaces/scripts.ts | 43 ++ .../chord/src/next/surfaces/theme-choice.ts | 82 +++ packages/chord/src/next/surfaces/themes.ts | 136 +++++ packages/chord/test/next-node/app/app.test.ts | 69 ++- .../chord/test/next-node/app/archive.test.ts | 26 +- .../next-node/app/ephemeral-sessions.test.ts | 51 ++ packages/chord/test/next-node/app/support.ts | 4 +- .../chord/test/next-node/fixtures/tui/api.ts | 7 + .../test/next-node/fixtures/tui/drive.ts | 17 + .../test/next-node/fixtures/tui/escapes.ts | 262 ++++++++++ .../chord/test/next-node/fixtures/tui/main.ts | 126 +++++ .../test/next-node/fixtures/tui/os-layer.ts | 95 ++++ .../next-node/fixtures/tui/sandbox-run.ts | 14 + .../chord/test/next-node/fixtures/web/api.ts | 14 + .../chord/test/next-node/fixtures/web/main.ts | 81 +++ .../chord/test/next-node/http/http.test.ts | 14 + .../test/next-node/http/upgrades.test.ts | 25 + .../test/next-node/plugins/build.test.ts | 59 +++ .../chord/test/next-node/react/hooks.test.ts | 45 ++ .../chord/test/next-node/react/stores.test.ts | 114 +++++ .../chord/test/next-node/surfaces/tui.test.ts | 56 +++ .../chord/test/next-node/surfaces/web.test.ts | 218 ++++++++ .../chord/test/next-node/tui/actions.test.ts | 228 +++++++++ .../test/next-node/tui/credentials.test.ts | 36 ++ .../chord/test/next-node/tui/filter.test.ts | 38 ++ .../chord/test/next-node/tui/launcher.test.ts | 263 ++++++++++ .../chord/test/next-node/tui/pipe.test.ts | 60 +++ .../chord/test/next-node/tui/sandbox.test.ts | 112 +++++ packages/chord/test/next-node/tui/support.ts | 116 +++++ packages/chord/test/next-node/web/browser.ts | 181 +++++++ .../chord/test/next-node/web/page.test.ts | 83 ++++ packages/chord/test/next-node/web/support.ts | 79 +++ .../chord/test/next/examples/53-app/main.ts | 2 +- .../chord/test/next/examples/53-app/start.ts | 6 +- .../test/next/examples/55-app-export/main.ts | 15 +- .../next/examples/56-web/app/principals.ts | 7 + .../chord/test/next/examples/56-web/app/v1.ts | 12 + .../test/next/examples/56-web/app/v1/tui.ts | 6 + .../test/next/examples/56-web/app/v1/web.ts | 10 + .../test/next/examples/56-web/headless.ts | 69 +++ .../chord/test/next/examples/56-web/main.ts | 58 +++ .../56-web/plugins/clock/package.json | 5 + .../next/examples/56-web/plugins/clock/web.ts | 6 + .../56-web/plugins/notes/package.json | 5 + .../next/examples/56-web/plugins/notes/tui.ts | 6 + .../next/examples/56-web/plugins/notes/web.ts | 6 + .../chord/test/next/examples/56-web/start.ts | 57 +++ .../test/next/examples/56-web/tui/main.ts | 30 ++ .../test/next/examples/56-web/web/main.ts | 33 ++ .../chord/test/next/examples/57-tui/main.ts | 68 +++ .../test/next/examples/57-tui/terminal.ts | 10 + .../test/next/examples/58-themes/main.ts | 124 +++++ .../58-themes/plugins/midnight.web.ts | 20 + .../58-themes/plugins/solarized/package.json | 5 + .../58-themes/plugins/solarized/theme.ts | 21 + .../58-themes/plugins/solarized/tui.ts | 6 + .../58-themes/plugins/solarized/web.ts | 6 + .../next/examples/59-chat/app/principals.ts | 8 + .../test/next/examples/59-chat/app/v1.ts | 73 +++ .../next/examples/59-chat/app/v1/server.ts | 10 + .../next/examples/59-chat/app/v1/worker.ts | 3 + .../chord/test/next/examples/59-chat/main.ts | 128 +++++ .../59-chat/plugins/chat/package.json | 5 + .../examples/59-chat/plugins/chat/server.ts | 45 ++ .../examples/59-chat/plugins/chat/worker.ts | 78 +++ .../examples/59-chat/plugins/login.server.ts | 5 + .../chord/test/next/examples/59-chat/start.ts | 76 +++ .../test/next/examples/59-chat/tui/main.ts | 90 ++++ .../test/next/examples/59-chat/web/main.ts | 95 ++++ .../test/next/examples/59-chat/web/style.css | 7 + .../test/next/examples/59-chat/worker.ts | 6 + packages/chord/test/next/link/values.test.ts | 8 + .../test/next/surfaces/active-theme.test.ts | 123 +++++ .../chord/test/next/surfaces/client.test.ts | 179 +++++++ .../test/next/surfaces/connection.test.ts | 214 ++++++++ .../test/next/surfaces/registered.test.ts | 57 +++ .../chord/test/next/surfaces/themes.test.ts | 70 +++ packages/chord/test/next/tui/frames.test.ts | 26 + packages/chord/test/next/tui/scope.test.ts | 80 +++ tsconfig.json | 7 + 151 files changed, 9376 insertions(+), 392 deletions(-) create mode 100644 packages/chord/src/next-node/app/data.ts create mode 100644 packages/chord/src/next-node/app/ephemeral-sessions.ts create mode 100644 packages/chord/src/next-node/plugins/css-modules.ts create mode 100644 packages/chord/src/next-node/surfaces/README.md create mode 100644 packages/chord/src/next-node/surfaces/bundle.ts create mode 100644 packages/chord/src/next-node/surfaces/current.ts create mode 100644 packages/chord/src/next-node/surfaces/index.ts create mode 100644 packages/chord/src/next-node/surfaces/published.ts create mode 100644 packages/chord/src/next-node/surfaces/served.ts create mode 100644 packages/chord/src/next-node/surfaces/tui.ts create mode 100644 packages/chord/src/next-node/surfaces/web.ts create mode 100644 packages/chord/src/next-node/tui/README.md create mode 100644 packages/chord/src/next-node/tui/actions.ts create mode 100644 packages/chord/src/next-node/tui/credentials.ts create mode 100644 packages/chord/src/next-node/tui/filter.ts create mode 100644 packages/chord/src/next-node/tui/index.ts create mode 100644 packages/chord/src/next-node/tui/launcher.ts create mode 100644 packages/chord/src/next-node/tui/pipe.ts create mode 100644 packages/chord/src/next-node/tui/sandbox.ts create mode 100644 packages/chord/src/next-react/README.md create mode 100644 packages/chord/src/next-react/hooks.ts create mode 100644 packages/chord/src/next-react/index.ts create mode 100644 packages/chord/src/next-react/stores.ts create mode 100644 packages/chord/src/next-tui/README.md create mode 100644 packages/chord/src/next-tui/bridge.ts create mode 100644 packages/chord/src/next-tui/channel.ts create mode 100644 packages/chord/src/next-tui/frames.ts create mode 100644 packages/chord/src/next-tui/index.ts create mode 100644 packages/chord/src/next-tui/runtime.ts create mode 100644 packages/chord/src/next-tui/scope.ts create mode 100644 packages/chord/src/next-tui/terminal.ts create mode 100644 packages/chord/src/next-tui/theme.ts create mode 100644 packages/chord/src/next-web/README.md create mode 100644 packages/chord/src/next-web/environment.ts create mode 100644 packages/chord/src/next-web/index.ts create mode 100644 packages/chord/src/next-web/page.ts create mode 100644 packages/chord/src/next-web/scripts.ts create mode 100644 packages/chord/src/next-web/themes.ts create mode 100644 packages/chord/src/next/surfaces/README.md create mode 100644 packages/chord/src/next/surfaces/active-theme.ts create mode 100644 packages/chord/src/next/surfaces/client.ts create mode 100644 packages/chord/src/next/surfaces/connection.ts create mode 100644 packages/chord/src/next/surfaces/index.ts create mode 100644 packages/chord/src/next/surfaces/loaded.ts create mode 100644 packages/chord/src/next/surfaces/registered.ts create mode 100644 packages/chord/src/next/surfaces/scripts.ts create mode 100644 packages/chord/src/next/surfaces/theme-choice.ts create mode 100644 packages/chord/src/next/surfaces/themes.ts create mode 100644 packages/chord/test/next-node/app/ephemeral-sessions.test.ts create mode 100644 packages/chord/test/next-node/fixtures/tui/api.ts create mode 100644 packages/chord/test/next-node/fixtures/tui/drive.ts create mode 100644 packages/chord/test/next-node/fixtures/tui/escapes.ts create mode 100644 packages/chord/test/next-node/fixtures/tui/main.ts create mode 100644 packages/chord/test/next-node/fixtures/tui/os-layer.ts create mode 100644 packages/chord/test/next-node/fixtures/tui/sandbox-run.ts create mode 100644 packages/chord/test/next-node/fixtures/web/api.ts create mode 100644 packages/chord/test/next-node/fixtures/web/main.ts create mode 100644 packages/chord/test/next-node/react/hooks.test.ts create mode 100644 packages/chord/test/next-node/react/stores.test.ts create mode 100644 packages/chord/test/next-node/surfaces/tui.test.ts create mode 100644 packages/chord/test/next-node/surfaces/web.test.ts create mode 100644 packages/chord/test/next-node/tui/actions.test.ts create mode 100644 packages/chord/test/next-node/tui/credentials.test.ts create mode 100644 packages/chord/test/next-node/tui/filter.test.ts create mode 100644 packages/chord/test/next-node/tui/launcher.test.ts create mode 100644 packages/chord/test/next-node/tui/pipe.test.ts create mode 100644 packages/chord/test/next-node/tui/sandbox.test.ts create mode 100644 packages/chord/test/next-node/tui/support.ts create mode 100644 packages/chord/test/next-node/web/browser.ts create mode 100644 packages/chord/test/next-node/web/page.test.ts create mode 100644 packages/chord/test/next-node/web/support.ts create mode 100644 packages/chord/test/next/examples/56-web/app/principals.ts create mode 100644 packages/chord/test/next/examples/56-web/app/v1.ts create mode 100644 packages/chord/test/next/examples/56-web/app/v1/tui.ts create mode 100644 packages/chord/test/next/examples/56-web/app/v1/web.ts create mode 100644 packages/chord/test/next/examples/56-web/headless.ts create mode 100644 packages/chord/test/next/examples/56-web/main.ts create mode 100644 packages/chord/test/next/examples/56-web/plugins/clock/package.json create mode 100644 packages/chord/test/next/examples/56-web/plugins/clock/web.ts create mode 100644 packages/chord/test/next/examples/56-web/plugins/notes/package.json create mode 100644 packages/chord/test/next/examples/56-web/plugins/notes/tui.ts create mode 100644 packages/chord/test/next/examples/56-web/plugins/notes/web.ts create mode 100644 packages/chord/test/next/examples/56-web/start.ts create mode 100644 packages/chord/test/next/examples/56-web/tui/main.ts create mode 100644 packages/chord/test/next/examples/56-web/web/main.ts create mode 100644 packages/chord/test/next/examples/57-tui/main.ts create mode 100644 packages/chord/test/next/examples/57-tui/terminal.ts create mode 100644 packages/chord/test/next/examples/58-themes/main.ts create mode 100644 packages/chord/test/next/examples/58-themes/plugins/midnight.web.ts create mode 100644 packages/chord/test/next/examples/58-themes/plugins/solarized/package.json create mode 100644 packages/chord/test/next/examples/58-themes/plugins/solarized/theme.ts create mode 100644 packages/chord/test/next/examples/58-themes/plugins/solarized/tui.ts create mode 100644 packages/chord/test/next/examples/58-themes/plugins/solarized/web.ts create mode 100644 packages/chord/test/next/examples/59-chat/app/principals.ts create mode 100644 packages/chord/test/next/examples/59-chat/app/v1.ts create mode 100644 packages/chord/test/next/examples/59-chat/app/v1/server.ts create mode 100644 packages/chord/test/next/examples/59-chat/app/v1/worker.ts create mode 100644 packages/chord/test/next/examples/59-chat/main.ts create mode 100644 packages/chord/test/next/examples/59-chat/plugins/chat/package.json create mode 100644 packages/chord/test/next/examples/59-chat/plugins/chat/server.ts create mode 100644 packages/chord/test/next/examples/59-chat/plugins/chat/worker.ts create mode 100644 packages/chord/test/next/examples/59-chat/plugins/login.server.ts create mode 100644 packages/chord/test/next/examples/59-chat/start.ts create mode 100644 packages/chord/test/next/examples/59-chat/tui/main.ts create mode 100644 packages/chord/test/next/examples/59-chat/web/main.ts create mode 100644 packages/chord/test/next/examples/59-chat/web/style.css create mode 100644 packages/chord/test/next/examples/59-chat/worker.ts create mode 100644 packages/chord/test/next/surfaces/active-theme.test.ts create mode 100644 packages/chord/test/next/surfaces/client.test.ts create mode 100644 packages/chord/test/next/surfaces/connection.test.ts create mode 100644 packages/chord/test/next/surfaces/registered.test.ts create mode 100644 packages/chord/test/next/surfaces/themes.test.ts create mode 100644 packages/chord/test/next/tui/frames.test.ts create mode 100644 packages/chord/test/next/tui/scope.test.ts diff --git a/package-lock.json b/package-lock.json index a0749b93b..5655e8690 100644 --- a/package-lock.json +++ b/package-lock.json @@ -2598,6 +2598,26 @@ "@types/retry": "*" } }, + "node_modules/@types/react": { + "version": "19.2.14", + "resolved": "https://registry.npmjs.org/@types/react/-/react-19.2.14.tgz", + "integrity": "sha512-ilcTH/UniCkMdtexkoCN0bI7pMcJDvmQFPvuPvmEaYA/NSfFTAgdUSLAoVjaRJm7+6PvcM+q1zYOwS4wTYMF9w==", + "dev": true, + "license": "MIT", + "dependencies": { + "csstype": "^3.2.2" + } + }, + "node_modules/@types/react-dom": { + "version": "19.2.3", + "resolved": "https://registry.npmjs.org/@types/react-dom/-/react-dom-19.2.3.tgz", + "integrity": "sha512-jp2L/eY6fn+KgVVQAOqYItbF0VY/YApe5Mz2F0aykSO8gx31bYCZyvSeYxCHKvzHG5eZjc+zyaS5BrBWya2+kQ==", + "dev": true, + "license": "MIT", + "peerDependencies": { + "@types/react": "^19.2.0" + } + }, "node_modules/@types/retry": { "version": "0.12.5", "resolved": "https://registry.npmjs.org/@types/retry/-/retry-0.12.5.tgz", @@ -3431,6 +3451,13 @@ "node": ">= 8" } }, + "node_modules/csstype": { + "version": "3.2.3", + "resolved": "https://registry.npmjs.org/csstype/-/csstype-3.2.3.tgz", + "integrity": "sha512-z1HGKcYy2xA8AGQfwrn0PAy+PB7X/GSj3UVJW9qKyn43xWa+gl5nXmU4qqLMRzWVLFC8KusUX8T/0kCiOYpAIQ==", + "dev": true, + "license": "MIT" + }, "node_modules/data-uri-to-buffer": { "version": "4.0.1", "resolved": "https://registry.npmjs.org/data-uri-to-buffer/-/data-uri-to-buffer-4.0.1.tgz", @@ -4781,6 +4808,29 @@ "rc": "cli.js" } }, + "node_modules/react": { + "version": "19.3.0", + "resolved": "https://registry.npmjs.org/react/-/react-19.3.0.tgz", + "integrity": "sha512-E8LUcbtBWt20bbl2YoHfx4ZDBdxVTfOKtCZn9cDSJ4l6/nuoApcpIBcj47t2wZoVX8g2ZHuMHbiShgCR1T5Sog==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/react-dom": { + "version": "19.3.0", + "resolved": "https://registry.npmjs.org/react-dom/-/react-dom-19.3.0.tgz", + "integrity": "sha512-JDk8dgif51OjFoDE70+OT9ICyYr+69HlmihNwp1+Nsfbna3t5sIiCa9ZJktDmQ4/1b/rn26hIAR2uYXDMr5r0Q==", + "dev": true, + "license": "MIT", + "dependencies": { + "scheduler": "^0.28.0" + }, + "peerDependencies": { + "react": "^19.3.0" + } + }, "node_modules/readable-stream": { "version": "3.6.2", "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz", @@ -4875,6 +4925,13 @@ "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==", "license": "MIT" }, + "node_modules/scheduler": { + "version": "0.28.0", + "resolved": "https://registry.npmjs.org/scheduler/-/scheduler-0.28.0.tgz", + "integrity": "sha512-juorfCmIkIw8tT+p5BXSm6PJjQF/ycEYmKyzURCIt/RaZIhL+PulbQ9Yu2z1HdOJDdqDTlxA1+xKBmHXJsczAw==", + "dev": true, + "license": "MIT" + }, "node_modules/semver": { "version": "7.8.5", "resolved": "https://registry.npmjs.org/semver/-/semver-7.8.5.tgz", @@ -5773,7 +5830,11 @@ "@earendil-works/pi-telemetry": "1.5.0" }, "devDependencies": { + "@types/react": "19.2.14", + "@types/react-dom": "19.2.3", "@types/ws": "8.18.2", + "react": "19.3.0", + "react-dom": "19.3.0", "vitest": "4.1.12", "ws": "8.21.0" }, @@ -5782,12 +5843,16 @@ }, "peerDependencies": { "esbuild": "0.28.2", + "react": "19.3.0", "ws": "8.21.0" }, "peerDependenciesMeta": { "esbuild": { "optional": true }, + "react": { + "optional": true + }, "ws": { "optional": true } diff --git a/packages/chord/CHANGELOG.md b/packages/chord/CHANGELOG.md index fb3520709..8ea8cb930 100644 --- a/packages/chord/CHANGELOG.md +++ b/packages/chord/CHANGELOG.md @@ -5,7 +5,8 @@ ### Added - Chord v2's plugins component gains the plugin model (slice 12): `createPlugins({ api: { name, versions } })` checks every import of the app's versioned API (`@acme/app/v3`, per-kind modules such as `@acme/app/v3/web`): an unlisted version, another kind's module, two versions in one facet and a direct import of Chord are build errors naming the file and line. Roots may be `{ path, required }`: a required root's plugin failing at `load` rejects it with a `PluginError` after the load settled. Files named `..` (`contract.web.tsx`) are of that kind, refused in facets of another kind with the chain of imports named. Single-file plugins (`hello.server.ts` at a root's top, grouped by name, host packages only). `installPlugin({ git, ref } | { npm, version }, root, ctx)` installs plugins from git (a shallow clone of a branch, tag or commit; a repository of several plugins too) or npm; a package lists the plugin folders it provides in its `package.json` (`"chord": { "plugins": ["./notes", "plugins/*", "!plugins/draft"] }`), used by discovery and `installPlugin` (without it: one plugin when the package's top has facet files, else its subfolders with a `package.json`). `testPlugin(dir, { kind, modules, stubs, principal }, ctx)` builds a plugin as production does and loads it on an in-memory host beside stub facets, collecting its contributions to registries nobody serves and listing the contracts it waits for. Agents' guide: `docs/next/plugin-authoring.md`. -- Chord v2 gains the app kit (slice 13, `src/next-node/app`, to be exported as `@earendil-works/chord/next/app`): `startApp

    ({ name, dataDir?, root?, listen?, api, kinds, principal, workers?, files?, install?, onError }, ctx)` lays out the data directory (`server/store.sqlite`, `workers//

    /`, `plugins/`, `logs/`, `owner-token`; one server per data directory), creates the server's store and host, HTTP, one workers component per worker kind and the plugins component over the app's required root and the user's root, and loads the server's plugins with the kit's objects; `startWorker({ api, kinds, busy?, onError })` is a worker's entry (its directory and store, the link to the server first, its plugins). The start-data objects `AppInfo` and `WorkerInfo` serve process values to plugins. Sessions (`Sessions`: `issue`, `verify`, `revoke`, `list`; tokens kept as SHA-256 hashes, compared in constant time; persisted ones end after 400 days unused and their browser cookie is renewed daily while used, ephemeral ones live in memory, at most `limits.ephemeralSessions`) authenticate a signed `HttpOnly`, `SameSite=Strict` cookie or `Authorization: Bearer`; a one-time owner link per start (`app.ownerLink()`, the token in the URL's fragment, single use, 10 minutes), an owner token file (mode 0600, written atomically) and device pairing (`POST /_chord/pair`, owner approval, a poll returning the token once; bounded in all and per client address) sign the owner in. The data directory, its stores and secret files are made private (0700, 0600) at every start; the home directory, a file system's root and another user's directory are refused. `Files` per host: small and chunked writes at a stored offset, ranged reads, a watchable index, caps, at most `limits.uploads` uploads under way. `app.exportAddress(address, path, ctx)` and `app.importAddress(archive, address, ctx)` move a worker's store and files as a tar, holding the worker's process lock meanwhile (refusing an existing address, another app's archive, every path leaving the target, and archives past `limits.importBytes` or `limits.importEntries`). `app.applyChanges(ctx)` and `app.plugins.install(source, ctx)`. Optional plugins as exported facets: `defineUsers(profile)`, `inviteFacet()`, `ephemeralLoginFacet()`. +- Chord v2 gains the app kit (slice 13, `src/next-node/app`, to be exported as `@earendil-works/chord/next/app`): `startServer

    ({ name, dataDir?, root?, listen?, api, kinds, principal, workers?, files?, install?, onError }, ctx)` lays out the data directory (`server/store.sqlite`, `workers//

    /`, `plugins/`, `logs/`, `owner-token`; one server per data directory), creates the server's store and host, HTTP, one workers component per worker kind and the plugins component over the app's required root and the user's root, and loads the server's plugins with the kit's objects; `startWorker({ api, kinds, busy?, onError })` is a worker's entry (its directory and store, the link to the server first, its plugins). The start-data objects `AppInfo` and `WorkerInfo` serve process values to plugins. Sessions (`Sessions`: `issue`, `verify`, `revoke`, `list`; tokens kept as SHA-256 hashes, compared in constant time; persisted ones end after 400 days unused and their browser cookie is renewed daily while used, ephemeral ones live in memory, at most `limits.ephemeralSessions`) authenticate a signed `HttpOnly`, `SameSite=Strict` cookie or `Authorization: Bearer`; a one-time owner link per start (`app.ownerLink()`, the token in the URL's fragment, single use, 10 minutes), an owner token file (mode 0600, written atomically) and device pairing (`POST /_chord/pair`, owner approval, a poll returning the token once; bounded in all and per client address) sign the owner in. The data directory, its stores and secret files are made private (0700, 0600) at every start; the home directory, a file system's root and another user's directory are refused. `Files` per host: small and chunked writes at a stored offset, ranged reads, a watchable index, caps, at most `limits.uploads` uploads under way. `app.exportData(address, path, ctx)` and `app.importData(archive, address, ctx)` move the data a worker created (its store and files) as a tar without stopping anything: a running worker writes its own store at one consistent point, else the server reads it under the worker's process lock (import refusing an existing address, another app's archive, every path leaving the target, and archives past `limits.importBytes` or `limits.importEntries`). `app.applyChanges(ctx)` and `app.plugins.install(source, ctx)`. Optional plugins as exported facets: `defineUsers(profile)`, `inviteFacet()`, `ephemeralLoginFacet()`. +- Chord v2 gains web and terminal surfaces (slice 11): `webSurface` and `tuiSurface` (`src/next-node/surfaces`) build a client kind's base bundle from the app's entry at start (content-hashed, served forever) and publish its plugins' facets as scripts that register themselves (`chordFacet("", "", function (require, module, exports) {…})`) and one combined file, with `WebScripts` and `TuiScripts` showing the current set (a rollout step publishes without waiting for clients) and the page served through the app's own template (`PageAssets`, `pageScripts`). `startPage` (`src/next-web`) runs a page's client host: the connection loop (`keepConnected`: 250 ms doubling to 5 s, wakes, silent sockets, `canRetry`), plugins applied in place on a new set (a failed one keeps its old facet), a reload when the server's base bundle changed, and the active theme as CSS variables. React bindings (`src/next-react`): `HostProvider`, `useHold`, `useWatch`, `useRegistry`, `useRefState` (React is an optional peer dependency). The terminal: `startTerminal` (`src/next-node/tui`) signs in with a token, a token file or device pairing and runs the app's terminal code always in a sandbox (a child process on pipes with Node's permission model: no file system, terminal, processes, workers, network or SQLite; Node 26.5.1 or later), filtering its output, relaying its link and confirming its actions (copy, open a URL, attach a file the user names); `startTerminalRuntime` and `TuiScope` (`src/next-tui`) inside it. Themes: the `Themes` registry, `ThemeChoice` (one dark and one light pick per principal), `followActiveTheme`, `themeVariables`, `terminalColor`. `startServer` gains `web` and `tui`. Plugin builds scope `*.module.css` files per plugin and file. Ephemeral sessions end with their client's last link (after a minute). The HTTP component's links endpoint gains an `opened` hook. - Chord v2's `HttpRequest` gains `remoteAddress`, the peer's address as the listener knows it (for limits per client). - Chord v2's HTTP component gains `readSignedCookie(request, name, secret)`: `readCookie`'s value with its signed expiry, for renewing a cookie before it expires. diff --git a/packages/chord/docs/next/design.md b/packages/chord/docs/next/design.md index da87d32a7..e48701e53 100644 --- a/packages/chord/docs/next/design.md +++ b/packages/chord/docs/next/design.md @@ -2272,11 +2272,11 @@ fails closed when stale; a start waits for a draining old holder without polling ### The app kit (`chord/next/app`; slice 13) -Decisions "The app kit, point 1" to "point 5". `startApp(options)` on the server, `startWorker(options)` in a worker +Decisions "The app kit, point 1" to "point 5". `startServer(options)` on the server, `startWorker(options)` in a worker entry. Opinionated and small: an app is its API, its built-in plugins and a start script. - **Shape:** the app's layout is fixed: `src/api/v/` (shared and per-kind modules) and the built-in plugins root - `plugins/`. The data directory is a `startApp` option (no environment variables), default a dot directory named after + `plugins/`. The data directory is a `startServer` option (no environment variables), default a dot directory named after the app in the home directory. One server; worker kinds are a list the app defines, each a host kind with its own params (sharding is the app's) and how it starts (entry, start data, principal, idle time); zero kinds is a server with clients only. @@ -2290,7 +2290,9 @@ entry. Opinionated and small: an app is its API, its built-in plugins and a star - **Storage:** one store for the server, a directory per worker under its kind and address with its store; plugins never use data-directory paths. A files plugin per address (`put`, chunked transfers, a watchable index, caps) is the default file storage; raw directories only where a real path is needed, owned by one object, under the worker's - directory. Export and import per address: the store's archive plus the worker's directory. + directory. Export and import the data a host created (`exportData`, `importData`): the store's archive plus its + files; export stops nothing (a running worker packs its own store; without one the server reads it under the + worker's lock), import refuses an address that has data (decision "App kit naming and export"). - **Plugins:** the app's root (required) and the user's root in the data directory; installs from git or npm; one "apply changes" call builds and rolls out to the server, then every running worker, exposed by the app as it likes. diff --git a/packages/chord/docs/next/guide-plugins-hosts/book.md b/packages/chord/docs/next/guide-plugins-hosts/book.md index ad2d511f2..5f9576170 100644 --- a/packages/chord/docs/next/guide-plugins-hosts/book.md +++ b/packages/chord/docs/next/guide-plugins-hosts/book.md @@ -268,7 +268,7 @@ a `remote` object is made per principal, so a guest can get a read-only variant :::note | The app kit does this wiring Starting workers, routing a host kind to them, the WebSocket entry and authentication are slice 10's components (`createWorkers`, `chord.http`). An app built with the app kit (`chord/next/app`, the last chapter) does not wire them: -`startApp(...)` on the server and `startWorker(...)` in a worker's entry do. +`startServer(...)` on the server and `startWorker(...)` in a worker's entry do. ::: # Client Hosts {#clients lede="Browser tabs and terminal UIs are hosts too: what they can do, and what they cannot."} @@ -633,7 +633,7 @@ same name, the first root wins, and the other is reported. as production does, loads it on an in-memory host beside stub facets serving what it uses, collects what it contributes to registries nobody serves, and lists the contracts it waits for (`docs/next/plugin-authoring.md`). -# The App Kit {#kit lede="startApp and startWorker: the hosts, stores, sign-in, files and plugin roots every app needs, so an app is its API, its plugins and a start script."} +# The App Kit {#kit lede="startServer and startWorker: the hosts, stores, sign-in, files and plugin roots every app needs, so an app is its API, its plugins and a start script."} The chapters so far wired an app by hand: a store, a host, `createHttp`, `createWorkers`, `createPlugins`, an authenticator, a worker entry connecting to its parent. Every app needs the same wiring, and most of it has one right @@ -646,7 +646,7 @@ The notes app of example 53 is four files of base code and two plugins: ```ts // start.ts -const app = await startApp({ +const app = await startServer({ name: "notes", // the data directory: ~/.notes unless dataDir says otherwise root: "./plugins", // the app's own plugins: a required root api: { name: "@notes/app", versions: ["v1"], modules: { "@notes/app/v1": v1, "@notes/app/v1/server": v1server } }, @@ -666,7 +666,7 @@ console.log(`Open ${app.ownerLink()}`); startWorker({ api: API, kinds: KINDS, onError: (error) => log(error) }); ``` -`startApp` creates the data directory (`server/store.sqlite`, `workers//
    /`, `plugins/`, `logs/`, +`startServer` creates the data directory (`server/store.sqlite`, `workers//
    /`, `plugins/`, `logs/`, `owner-token`), opens the server's store, creates the server host, HTTP, a workers component per worker kind and the plugins component over two roots: the app's `plugins/` and the user's `/plugins/`. Then it loads the server's plugins with the kit's own facets beside them and resolves. `startWorker` does the worker's half: the project's @@ -707,9 +707,10 @@ Users other than the owner come from optional plugins the app adds: `defineUsers The kit never reloads by itself. An edit or an install marks plugins dirty; `app.plugins.state` drives the app's "apply changes" button, and `app.applyChanges(ctx)` builds and rolls out to the server, then the workers. -A project's data is its directory. `app.exportAddress(Project({ id }), path, ctx)` stops its worker and writes one -archive (a manifest, the store's export, the files); `app.importAddress(path, Project({ id: "copy" }), ctx)` makes a new -project from it and refuses one that already has data (example 55). +A project's data is what its worker created: its store and its files. `app.exportData(Project({ id }), path, ctx)` +writes them as one archive (a manifest, the store's export, the files) and stops nothing: a running worker writes its +own store at one consistent point. `app.importData(path, Project({ id: "copy" }), ctx)` makes a new project from it and +refuses one that already has data (example 55). # Reference {.appendix} @@ -725,8 +726,8 @@ project from it and refuses one that already has data (example 55). | `plugins.load(host, { kind, core, modules }, ctx)`, `build(names)`, `rollout(steps, built)` | load, build, roll out | | `plugins.installPlugin(source, root, ctx)`, `testPlugin(dir, options, ctx)` | install a plugin, test one alone | | `hostKind(name, params?)`, `Kind(params)` | host kinds and addresses | -| `startApp

    ({ name, dataDir?, root?, listen?, api, kinds, principal, workers?, files?, onError }, ctx)` | an app's server, with the kit | +| `startServer

    ({ name, dataDir?, root?, listen?, api, kinds, principal, workers?, files?, onError }, ctx)` | an app's server, with the kit | | `startWorker({ api, kinds, busy?, onError })` | a worker's entry, with the kit | | `app.ownerLink()`, `app.applyChanges(ctx)`, `app.plugins.install(source, ctx)` | sign in, apply changes, install | -| `app.exportAddress(address, path, ctx)`, `app.importAddress(path, address, ctx)` | move a worker's data | +| `app.exportData(address, path, ctx)`, `app.importData(path, address, ctx)` | move a worker's data | | `AppInfo`, `WorkerInfo`, `Files`, `Sessions` | the kit's objects plugins get | diff --git a/packages/chord/docs/next/guide-plugins-hosts/index.html b/packages/chord/docs/next/guide-plugins-hosts/index.html index beac6027f..d0ec02a3c 100644 --- a/packages/chord/docs/next/guide-plugins-hosts/index.html +++ b/packages/chord/docs/next/guide-plugins-hosts/index.html @@ -309,7 +309,7 @@ tab.connect(webSocketLink(remote object is made per principal, so a guest can get a read-only variant of the same notes.

    The app kit does this wiring

    Starting workers, routing a host kind to them, the WebSocket entry and authentication are slice 10's components (createWorkers, chord.http). An app built with the app kit (chord/next/app, the last chapter) does not wire them: -startApp(...) on the server and startWorker(...) in a worker's entry do.

    +startServer(...) on the server and startWorker(...) in a worker's entry do.

    @@ -594,7 +594,7 @@ contributes to registries nobody serves, and lists the contracts it waits for (< -
    X

    The App Kit

    startApp and startWorker: the hosts, stores, sign-in, files and plugin roots every app needs, so an app is its API, its plugins and a start script.

    +
    X

    The App Kit

    startServer and startWorker: the hosts, stores, sign-in, files and plugin roots every app needs, so an app is its API, its plugins and a start script.

    The chapters so far wired an app by hand: a store, a host, createHttp, createWorkers, createPlugins, an authenticator, a worker entry connecting to its parent. Every app needs the same wiring, and most of it has one right answer. The app kit (chord/next/app) is that answer, opinionated on purpose: one data directory layout, one way @@ -602,7 +602,7 @@ to sign in, one place for files.

    A whole app

    The notes app of example 53 is four files of base code and two plugins:

    // start.ts
-const app = await startApp<NotesPrincipal>({
+const app = await startServer<NotesPrincipal>({
 	name: "notes",                                  // the data directory: ~/.notes unless dataDir says otherwise
 	root: "./plugins",                              // the app's own plugins: a required root
 	api: { name: "@notes/app", versions: ["v1"], modules: { "@notes/app/v1": v1, "@notes/app/v1/server": v1server } },
@@ -620,7 +620,7 @@ console.log(`Open ${app.ownerLink()
 
 // worker.ts
 startWorker({ api: API, kinds: KINDS, onError: (error) => log(error) });
    -

    startApp creates the data directory (server/store.sqlite, workers/<kind>/<address>/, plugins/, logs/, +

    startServer creates the data directory (server/store.sqlite, workers/<kind>/<address>/, plugins/, logs/, owner-token), opens the server's store, creates the server host, HTTP, a workers component per worker kind and the plugins component over two roots: the app's plugins/ and the user's <dataDir>/plugins/. Then it loads the server's plugins with the kit's own facets beside them and resolves. startWorker does the worker's half: the project's @@ -652,9 +652,10 @@ returns its token (example 54).

    Applying changes, moving projects

    The kit never reloads by itself. An edit or an install marks plugins dirty; app.plugins.state drives the app's "apply changes" button, and app.applyChanges(ctx) builds and rolls out to the server, then the workers.

    -

    A project's data is its directory. app.exportAddress(Project({ id }), path, ctx) stops its worker and writes one -archive (a manifest, the store's export, the files); app.importAddress(path, Project({ id: "copy" }), ctx) makes a new -project from it and refuses one that already has data (example 55).

    +

    A project's data is what its worker created: its store and its files. app.exportData(Project({ id }), path, ctx) +writes them as one archive (a manifest, the store's export, the files) and stops nothing: a running worker writes its +own store at one consistent point. app.importData(path, Project({ id: "copy" }), ctx) makes a new project from it and +refuses one that already has data (example 55).

    A

    Reference

    @@ -706,7 +707,7 @@ project from it and refuses one that already has data (example 55).

    host kinds and addresses -startApp<P>({ name, dataDir?, root?, listen?, api, kinds, principal, workers?, files?, onError }, ctx) +startServer<P>({ name, dataDir?, root?, listen?, api, kinds, principal, workers?, files?, onError }, ctx) an app's server, with the kit @@ -718,7 +719,7 @@ project from it and refuses one that already has data (example 55).

    sign in, apply changes, install -app.exportAddress(address, path, ctx), app.importAddress(path, address, ctx) +app.exportData(address, path, ctx), app.importData(path, address, ctx) move a worker's data diff --git a/packages/chord/docs/next/plugin-authoring.md b/packages/chord/docs/next/plugin-authoring.md index c21f477a3..64c2063b4 100644 --- a/packages/chord/docs/next/plugin-authoring.md +++ b/packages/chord/docs/next/plugin-authoring.md @@ -114,6 +114,18 @@ No type check, no build hooks, no `dist` needed; generated files are committed. process of its kind with that process's API modules. A failed build or evaluation leaves the plugin out (at startup) or keeps its old code (at a reload), reported with the error. +## Styles and themes (web and terminal facets) + +- A web facet's CSS is served with its script and swapped when the plugin reloads. Scope it with a **CSS module**: a + file named `*.module.css` has its class names made unique to the plugin and the file, and its import is the mapping + (`import styles from "./notes.module.css";
    `). Plain CSS files still work; their class + names should then start with the plugin's name. +- Read the app's look from its CSS variables (`var(--accent)`, `var(--bg)`, `var(--fg)`, `var(--muted)`, `var(--error)`, + `var(--warning)`, `var(--success)`, and the app's own roles), never fixed colours: the active theme sets them. +- In a terminal facet, ask the app's theme lookup for a role (`theme.fg("accent", "Notes")`); never write raw colours. +- A theme is data: contribute a `Theme` (`{ id, name, mode, colors, web?, tui? }`) to the kit's `Themes` registry, + from a web facet and a terminal facet to reach both. Users pick one dark and one light theme. + ## Installing - `plugins.installPlugin({ git: "https://github.com/bob/stats.git", ref: "v1.2.0" }, root, ctx)`: a shallow clone of diff --git a/packages/chord/docs/next/slices.md b/packages/chord/docs/next/slices.md index 808044528..33a7c8ffe 100644 --- a/packages/chord/docs/next/slices.md +++ b/packages/chord/docs/next/slices.md @@ -37,6 +37,13 @@ slice spec's section citations are for orientation, not a reading limit (owner, - **Slice 10** (built 2026-10-31, `slice-10.md`): `src/next-node/http/` (a `fetch` core with the security baseline, Chord's Node listener, `nodeRequest`, plugin sockets, links), `src/next-node/workers/`, `createPluginRuntime`, kinds, the kernel's `RouteHandle` and anonymous `host.root`, origins only from relays; examples 40-48. +- **Slice 12** (committed 2026-11-01, `slice-12.md`): the plugin model in `src/next-node/plugins/` (the app's versioned + API, per-kind files, single-file plugins, installs from git and npm, `testPlugin`); examples 49-52. +- **Slice 13** (committed 2026-11-02, `slice-13.md`): the app kit, `src/next-node/app/` (`startServer`, `startWorker`, + sessions, owner link, pairing, `Files`, `exportData`/`importData`); examples 53-55. +- **Slice 11** (committed 2026-11-03, `slice-11.md`): `src/next/surfaces/` (scripts objects, registered scripts, the + connection loop, clients, themes), `src/next-web/`, `src/next-react/`, `src/next-tui/`, `src/next-node/surfaces/`, + `src/next-node/tui/` (the launcher and its child-process sandbox); examples 56-59. - **Chord's shared code** (upstream pi): `src/watch.ts` (`Watch`, `NativeWatch`, `WatchHandle`, `WatchEnd`, `NativeWatchable`), Delta (`track`, `diffRevisions`, `applyImmutable`, encoders, `PendingOps`, `appendText`), context, JSON, output, SQLite drivers. @@ -64,7 +71,7 @@ telemetry last, since it instruments the app kit too; 2026-11-01: the plugin mod | 7c | **Plugins** | `chord/next/plugins` (discovery, `buildFacet`, `evaluateFacet`, `pluginFile`, `installDependencies`, the watcher, `rollout`, `mapStack`; no dev/prod split; the kernel's `load`/`reload` per plugin; `slice-7c.md`) | | 10 | **HTTP and workers** | `chord/next/http` (server, `Http` registry, responses as descriptions, WebSocket entry) and `chord/next/workers` (a worker per address, IPC, idle stop) | | 12 | **The plugin model** | imports (relative within a repository, package names pinned across), versioned app API with Chord re-exported, per-kind API modules and contract files with the build's import check, single-file plugins, git installs, `testPlugin`, a required root (`slice-12.md`) | -| 13 | **The app kit** | `chord/next/app`: `startApp`, `startWorker`, layout, worker kinds, owner link, deny by default, sessions, CLI tokens, storage, the files plugin, export per address, plugin roots and "apply changes", base code as API, bootstrap and a start-data facet (`slice-13.md`) | +| 13 | **The app kit** | `chord/next/app`: `startServer`, `startWorker`, layout, worker kinds, owner link, deny by default, sessions, CLI tokens, storage, the files plugin, export per address, plugin roots and "apply changes", base code as API, bootstrap and a start-data facet (`slice-13.md`) | | 11 | **Web and TUI** | `chord/next/web` (page host, React hooks) and `chord/next/tui` (launcher, sandbox), from p2's code | | 9 | **Telemetry** | spans, events, metrics across hosts | @@ -147,7 +154,8 @@ pattern, no code. Decided cuts are noted where they apply. | 22 | relays, size limits per part, compression at the origin, adapted bindings get diffs | 4a (relays), 4b (limits, compression), 7b | | 22 | trace carriers (R5) | 9 | | 23 | HTTP (routes, wraps, responses as descriptions, request bodies, plugin sockets), workers; channels and chunked transfers stay owner patterns until a second user | 10 | -| 23 | the app kit: `startApp`, `startWorker`, start-data objects, deny by default, sessions, owner link and token file, device pairing, files (with chunked transfers), export and import per address, plugin roots and "apply changes", optional users, invite and ephemeral-login plugins | 13 | +| 23 | the app kit: `startServer`, `startWorker`, start-data objects, deny by default, sessions, owner link and token file, device pairing, files (with chunked transfers), export and import per address, plugin roots and "apply changes", optional users, invite and ephemeral-login plugins | 13 | +| 23 | web and TUI: base bundles, registered plugin scripts and the combined file, `WebScripts`/`TuiScripts`, `startPage` with the connection loop and in-place reloads, React bindings, the sandboxed terminal (launcher, sandbox, `TuiScope`), themes (`Themes`, `ThemeChoice`, CSS variables, terminal colours), CSS modules | 11 | | 24 | Durable integration: `chord(fn, context)` (Q6), a `RegistryReader` over composed registries; the harness object and its model and env registries are p3's (base code, owner, 2026-10-27) | 8 (Chord parts), p3 | | 24 | `StoreStorage` | 6b | | 27 | versions table: contracts (L1 done), registries 8, documents 6, jobs 5, link protocol 3a | as listed | @@ -166,12 +174,6 @@ change; a slice's spec includes every entry of its section, and its build and re deleted (the slice's spec and code are the record). Moved here from `carry-over.md` (owner, 2026-10-24). -### Slice 11 (with the app kit's follow-ups) - -- **App kit renames and export** (decision "App kit naming and export"): `startApp` → `startServer`; - `exportAddress`/`importAddress` → `exportData`/`importData`; export without stopping the worker (a kit object on the - worker exports its store and finished files; without a worker the server takes the lock and reads the store); import - unchanged apart from the name. Examples 53-55, the app README and the guide's app kit chapter follow. ### Any slice (protocol and checks) - **Bump `PROTOCOL_VERSION`** for any message or reason change once p3 has shipped (slice 3b added `retry`, `loading` @@ -191,14 +193,6 @@ deleted (the slice's spec and code are the record). Moved here from `carry-over. - **`rebuild` of tables** (key and column-type changes): `evolveTable` has no edit for them; build one only when a real case needs it. (Slice 6a's second spec review; out of slice 7b's scope, `slice-7b.md` §12.) -### Slice 11: web and TUI - -- **Ephemeral sessions end with the connection** (slice 13 §3): the kit's ephemeral login drops its sessions when the - server starts and gives the browser a cookie without a lifetime; ending a session when its client's last link - closes needs the clients' connection loop, which slice 11 builds. (Slice 13 build.) -- **`startApp` gains the web surface** (slice 13 §6): serving the clients' bundles from the kit, behind its sessions. - (Slice 13 spec.) - ### After slice 11 - **Messages and languages in the app kit** (decision "Messages and languages in the app kit", proposal diff --git a/packages/chord/package.json b/packages/chord/package.json index 6e8e12a4d..c61aef777 100644 --- a/packages/chord/package.json +++ b/packages/chord/package.json @@ -95,18 +95,26 @@ }, "peerDependencies": { "esbuild": "0.28.2", + "react": "19.3.0", "ws": "8.21.0" }, "peerDependenciesMeta": { "esbuild": { "optional": true }, + "react": { + "optional": true + }, "ws": { "optional": true } }, "devDependencies": { + "@types/react": "19.2.14", + "@types/react-dom": "19.2.3", "@types/ws": "8.18.2", + "react": "19.3.0", + "react-dom": "19.3.0", "vitest": "4.1.12", "ws": "8.21.0" } diff --git a/packages/chord/src/next-node/app/README.md b/packages/chord/src/next-node/app/README.md index 7fcd9e4a0..254e5da54 100644 --- a/packages/chord/src/next-node/app/README.md +++ b/packages/chord/src/next-node/app/README.md @@ -1,7 +1,7 @@ # app The app kit (slice 13, design §23 "The app kit", decisions "The app kit, point 1" to "point 5"): an opinionated layer -over `http`, `workers` and `plugins` so an app is its API, its built-in plugins and a start script. `startApp` on the +over `http`, `workers` and `plugins` so an app is its API, its built-in plugins and a start script. `startServer` on the server, `startWorker` in a worker's entry; the kit makes the data directory, the stores, the hosts, HTTP with deny by default, sessions and the owner's sign-in, device pairing, each host's files, export and import per address, and the plugin roots with "apply changes". Node-only. Exported as `@earendil-works/chord/next/app` once v2 has its entry @@ -11,7 +11,7 @@ point. Examples 53 (`53-app/`, a whole small app), 54 (sessions and pairing) and ```ts // start.ts (the app's start script) -const app = await startApp

    ({ +const app = await startServer

    ({ name: "acme", // the default data directory: ~/.acme dataDir: process.argv[2], // optional; no environment variables root: "./plugins", // the app's built-in plugins (required root); this is the default @@ -31,14 +31,16 @@ const app = await startApp

    ({ install: { npmCommand: ["npm"] }, // how installs run npm limits: { ephemeralSessions: 1000, pendingPairings: 100, pairingsPerAddress: 10, // the defaults uploads: 64, importBytes: 4 * 2 ** 30, importEntries: 100_000 }, + web: { entry: "./src/web/main.tsx", page: (request, assets) => shell(request.nonce, assets) }, // slice 11 + tui: { entry: "./src/tui/main.ts" }, onError: (error) => log(error), // AppError: { kind: "host" | "http" | "workers" | "plugins", error } }, ctx); console.log(`Open ${app.ownerLink()}`); app.host; app.http; app.workers.get("project"); app.plugins.state; app.dataDir; app.ownerTokenFile; await app.plugins.install({ git: url, ref: "v1" }, ctx); // into /plugins/ await app.applyChanges(ctx); // build the dirty plugins, roll out: server, then workers -await app.exportAddress(Project({ id }), "p.tar", ctx); -await app.importAddress("p.tar", Project({ id: "copy" }), ctx); +await app.exportData(Project({ id }), "p.tar", ctx); +await app.importData("p.tar", Project({ id: "copy" }), ctx); await app.close(ctx); // http, workers, host, store, plugins, the lock // worker.ts @@ -50,7 +52,7 @@ startWorker

    ({ api, kinds, busy, onError }); - **The API's modules** are named by convention: `/` for every kind and `//` for a kind's own; a kind's `packages` (React for a web kind) are host packages besides. The modules are this process's instances; the worker's entry passes the same `api` and `kinds`. -- **`startApp`**, in order: the data directory, by its real path (refused when it is the home directory, a file +- **`startServer`**, in order: the data directory, by its real path (refused when it is the home directory, a file system's root, or another user's: the kit would make it private), its lock (one server per data directory, also through a symbolic link: a second is refused after a second's wait), the layout made private (below), the cookie secret, the server's store, the host (address `Server()`, `principal.server`), `createHttp` (the links endpoint at `/chord`), the plugins component with the two roots, one @@ -94,9 +96,12 @@ and reads `AppInfo`. an ephemeral one only while the browser runs); a malformed token or an id that is no session (a prototype key included) is a plain 401. Browser and device sessions persist in the server's store; `lastUsed` is written at most once a minute, and a session unused for 400 days ends (checked when it is used, removed then). Ephemeral - sessions (a public login) live in memory only, never listed: at most `limits.ephemeralSessions`, none outliving the - server's run, so a login costs the same whatever came before. Past the cap the oldest ends: a flood of logins signs - out the earliest ephemeral users (the trade for bounded memory). + sessions (a public login) live in memory only (`ephemeral-sessions.ts`), never listed: at most + `limits.ephemeralSessions`, none outliving the server's run, so a login costs the same whatever came before. Past + the cap the oldest ends: a flood of logins signs out the earliest ephemeral users (the trade for bounded memory). An + ephemeral session ends with its client's connection (slice 11): once the last Chord link opened with it closed and + none opened for a minute (a reload, a reconnect), or when none opened within a minute of the login; the links + endpoint's `opened` hook counts them. - **The owner link:** each start makes a one-time token (single use, 10 minutes, the previous start's no longer valid), `app.ownerLink()` is `http://:/_chord/owner#` (`127.0.0.1` for `0.0.0.0`, `[::1]` for `::`, an IPv6 address in brackets). The page at `GET /_chord/owner` posts @@ -170,13 +175,17 @@ plugin's own remote methods, which call it. ## Export and import -`exportAddress(address, path, ctx)` stops the address's worker (`workers.stop`) and takes its process lock (the one a -worker takes at start) for the whole pack, writes the archive to `path` (mode 0600, staged and renamed), then releases -the lock and lets the address start again (`workers.retry`): a hold arriving meanwhile starts a worker that waits for -the lock, then opens the unchanged state. `importAddress` holds the same lock while it unpacks. A failed address (a -crashed worker nobody retried) is not stopped or retried: it stays failed. The archive is a ustar of +`exportData(address, path, ctx)` writes the data the worker at `address` created (its store and finished files) to +`path` (mode 0600, staged and renamed), stopping nothing (decision "App kit naming and export"): a running worker packs +its own open store (`HostData`, `data.ts`, a kit object only the server's principal may use; the store's export holds +the commit queue while it reads, so the archive is one consistent point) and its files; with no worker running, the +server takes the worker's process lock (no worker starts meanwhile) and packs the store itself. A file added or removed +while the export runs may be outside that point (the index is the store's): each file is opened when the archive +comes to it (one descriptor at a time), one deleted meanwhile skipped or read from its open handle. `importData` takes the same lock (held: a +worker runs, so the address has data: refused) and changes nothing else. Registering an imported host in the app's own +index is the app's. The archive is a ustar of `manifest.json` (`{ format: 1, app, api, address }`), `store.archive` (the store's export, JSONL) and `files/…`. -`importAddress(archive, address, ctx)` refuses an address that has a directory, unpacks beside it and renames into +`importData(archive, address, ctx)` refuses an address that has a directory, unpacks beside it and renames into place: the manifest must come first and name this app and format 1; only regular files and directories under `manifest.json`, `store.archive` and `files/` are taken; absolute paths, `..`, `.`, empty segments, backslashes, NULs, links, devices and bad checksums are refused (the kit's own tar reader, `tar.ts`); past `limits.importBytes` unpacked @@ -185,14 +194,25 @@ archive leaves nothing. The next hold starts the worker on the imported store an ## Plugins -Roots: the app's `root` (required: a plugin of it failing stops `startApp`) and `/plugins/` (the user's: +Roots: the app's `root` (required: a plugin of it failing stops `startServer`) and `/plugins/` (the user's: installs go there). `app.plugins.state` is the plugins component's marks for the app's "apply changes" button; `applyChanges` builds the dirty plugins (and installed ones) and rolls out to the server, then each worker kind; nothing is applied by itself. ## Internals -`app.ts` (`startApp`, the `App`), `worker.ts` (`startWorker`), `layout.ts` (the layout, private directories and +`app.ts` (`startServer`, the `App`), `worker.ts` (`startWorker`), `layout.ts` (the layout, private directories and stores), `secrets.ts` (tokens, hashes, files of mode 0600), `sessions.ts`, `owner.ts` (owner link and token file), `pairing.ts`, `files.ts` and `uploads.ts` (chunked uploads), `info.ts` (start-data objects), `archive.ts` and `tar.ts` -(export and import), and the optional plugins `users.ts`, `invite.ts`, `ephemeral.ts`. +(export and import), `ephemeral-sessions.ts`, and the optional plugins `users.ts`, `invite.ts`, `ephemeral.ts`. + +## Web and terminal (slice 11) + +With `web` the kit builds the page's base bundle from `web.entry`, publishes the `web` kind's plugin scripts and serves +the page through `web.page` at `web.paths` (`["/"]`), signed in unless `web.public` (`webSurface`, +`next-node/surfaces`); with `tui` the terminal's base bundle and the `tui` kind's scripts for launchers +(`tuiSurface`). Plugin assets are served under `/_chord/web/assets/` (the plugins component's `publicPath`). +`applyChanges` rolls out to the server, the workers, then the surfaces (`web`, `tui`: the new set published, clients +not waited for). `app.web` and `app.tui` give the base bundles' hashes. The server also serves the `Themes` registry and +`ThemeChoice` (each principal's dark and light pick, kept in its store). Without `web` and `tui` an app has no client +surface (API only). diff --git a/packages/chord/src/next-node/app/app.ts b/packages/chord/src/next-node/app/app.ts index f9796714d..5879dd652 100644 --- a/packages/chord/src/next-node/app/app.ts +++ b/packages/chord/src/next-node/app/app.ts @@ -1,4 +1,4 @@ -// `startApp` (slice 13 §1, §5; decisions "The app kit, point 1" to "point 5"): the server of an app built with the kit. +// `startServer` (slice 13 §1, §5; decisions "The app kit, point 1" to "point 5"): the server of an app built with the kit. // It lays out the data directory, opens the server's store, creates the server host, the HTTP component and one // workers component per worker kind, the plugins component over the app's required root and the user's root, loads // the server's plugins with the kit's own facets (the start-data object, sessions, the owner link, pairing, files), @@ -9,8 +9,10 @@ import { type Context, withAbortSignal } from "../../context/index.ts"; import type { JsonValue } from "../../json.ts"; import { type Address, type HostKind, hostKind, type KindShape } from "../../next/contracts/index.ts"; import type { Principal } from "../../next/data/index.ts"; -import { createHost, type Host, type HostError, type PluginFacet } from "../../next/host/index.ts"; +import { canonicalJson } from "../../next/data/principal.ts"; +import { createHost, type Facet, type Host, type HostError, type PluginFacet } from "../../next/host/index.ts"; import { openStore, type Store, sqliteBackend } from "../../next/store/index.ts"; +import { themeChoiceFacet, themesFacet } from "../../next/surfaces/index.ts"; import { openNodeSqliteDatabase } from "../../sqlite/node.ts"; import type { Watchable } from "../../watch.ts"; import { createHttp, type HttpComponent, type HttpError, type ListenAddress } from "../http/index.ts"; @@ -23,16 +25,19 @@ import { type PluginsError, type PluginsState, type RolloutResult, + type RolloutStep, } from "../plugins/index.ts"; +import { tuiSurface, WEB_PREFIX, type WebSurfaceOptions, webSurface } from "../surfaces/index.ts"; import { createWorkers, type ProcessLock, processLock, type Workers, type WorkersError } from "../workers/index.ts"; -import { ARCHIVE_FORMAT, type ArchiveManifest, packAddress, unpackAddress } from "./archive.ts"; +import { ARCHIVE_FORMAT, type ArchiveManifest, packData, unpackData } from "./archive.ts"; +import { addressParam, HostData } from "./data.ts"; import { type FilesCaps, filesFacet } from "./files.ts"; import { appInfoFacet } from "./info.ts"; import { type AppLayout, createLayout, defaultDataDir, openDataDir, privateStore, workerDirectory } from "./layout.ts"; import { OWNER_PATH, OwnerLink, type OwnerPrincipal, ownerFacet, ownerTokenFile } from "./owner.ts"; import { Pairings, pairingFacet } from "./pairing.ts"; import { secretFile } from "./secrets.ts"; -import { sessionsFacet } from "./sessions.ts"; +import { createSessions } from "./sessions.ts"; /** The server's host kind: the kit's server host is `Server()`. */ export const Server = hostKind("server"); @@ -78,7 +83,7 @@ export type AppError = | { readonly kind: "workers"; readonly error: WorkersError } | { readonly kind: "plugins"; readonly error: PluginsError }; -/** What `startApp` takes. `{ kind: "owner" }` must be one of the app's principals (the kit's owner). */ +/** What `startServer` takes. `{ kind: "owner" }` must be one of the app's principals (the kit's owner). */ export type AppOptions

    = { /** The app's name: its default data directory is `~/.`, its archives carry it. */ readonly name: string; @@ -101,6 +106,13 @@ export type AppOptions

    = { readonly limits?: AppLimits; /** How installs run npm (slice 12's `InstallOptions`): `npm` on the `PATH` by default. */ readonly install?: InstallOptions; + /** + * The web surface (slice 11): the app's web entry (the base bundle's), its page template, the paths serving the + * page (`["/"]`), whether the page is public; the `web` facet kind. None: no web surface. + */ + readonly web?: AppWeb; + /** The terminal surface (slice 11): the app's terminal entry; the `tui` facet kind. None: no terminal clients. */ + readonly tui?: { readonly entry: string }; readonly onError: (error: AppError) => void; } & (OwnerPrincipal extends P ? unknown : { readonly "the app's principal type must have { kind: 'owner' }": never }); @@ -119,6 +131,9 @@ export type AppLimits = { readonly importEntries?: number; }; +/** The kit's web surface options (`webSurface`'s, its kind `web` and plugins the kit's). */ +export type AppWeb = Pick; + /** The plugins of a running app, for its "apply changes" control and its installs. */ export type AppPlugins = { /** The marks (`dirty`, `install`, `set`, `base`): what the app's button shows. */ @@ -136,18 +151,29 @@ export interface App

    { /** The workers components, by host kind name. */ readonly workers: ReadonlyMap>; readonly plugins: AppPlugins; + /** The web surface's base bundle hash, when the app has one. */ + readonly web: { readonly base: string } | undefined; + /** The terminal surface's base bundle hash, when the app has one. */ + readonly tui: { readonly base: string } | undefined; /** The data directory. */ readonly dataDir: string; /** The owner link of this start (single use, 10 minutes): `http://127.0.0.1:/_chord/owner#`. */ ownerLink(): string; /** The owner token file's path (a device session of the owner, mode 0600). */ readonly ownerTokenFile: string; - /** Builds the dirty plugins and rolls them out: the server, then every worker kind; never by itself. */ + /** Builds the dirty plugins and rolls them out: the server, every worker kind, then the clients; never by itself. */ applyChanges(context: Context): Promise; - /** Stops the worker at `address` and writes its archive (store and files) to `path`. */ - exportAddress(address: Address, path: string, context: Context): Promise; - /** Makes `address` (which must have no data yet) from the archive at `archive`; its next hold starts it. */ - importAddress(archive: string, address: Address, context: Context): Promise; + /** + * Writes the data the worker at `address` created (its store and finished files) as an archive at `path`, stopping + * nothing: a running worker packs its own store at one consistent point; without one the server packs it under the + * worker's lock (no worker starts meanwhile). + */ + exportData(address: Address, path: string, context: Context): Promise; + /** + * Makes the data of `address` (which must have none yet: nothing is overwritten) from the archive at `archive`, + * under the worker's lock; its next hold starts it. + */ + importData(archive: string, address: Address, context: Context): Promise; /** Closes the HTTP component, the workers, the host, the store, the watcher and the data directory's lock. */ close(context: Context): Promise; } @@ -167,12 +193,13 @@ export type KitStart = { * Starts the app's server (§1): resolves once its plugins are loaded and it listens. A failing plugin of the app's * root (or a problem discovery reports there) rejects with the plugins component's `PluginError`, everything closed. */ -export async function startApp

    (options: AppOptions

    , context: Context): Promise> { +export async function startServer

    (options: AppOptions

    , context: Context): Promise> { const { layout, cookieSecret, store, lock } = await openServer( resolve(options.dataDir ?? defaultDataDir(options.name)), context, ); const { onError, api, limits = {} } = options; + const sessions = createSessions

    (cookieSecret, { maxEphemeral: limits.ephemeralSessions }); const host = createHost

    ({ address: Server(), principal: options.principal.server, @@ -183,7 +210,7 @@ export async function startApp

    (options: AppOptions

    , cont host, { listen: options.listen ?? { host: "127.0.0.1", port: 0 }, - links: { path: "/chord" }, + links: { path: "/chord", opened: sessions.opened }, onError: (error) => onError({ kind: "http", error }), }, context, @@ -199,6 +226,7 @@ export async function startApp

    (options: AppOptions

    , cont ]), ), install: options.install, + publicPath: `${WEB_PREFIX}assets/`, onError: (error) => onError({ kind: "plugins", error }), }, context, @@ -214,7 +242,7 @@ export async function startApp

    (options: AppOptions

    , cont name: "chord.app.info", facet: appInfoFacet({ dataDir: layout.dataDir, name: options.name, version: api.versions[0] ?? "" }), }, - { name: "chord.app.sessions", facet: sessionsFacet(cookieSecret, { maxEphemeral: limits.ephemeralSessions }) }, + { name: "chord.app.sessions", facet: sessions.facet }, { name: "chord.app.owner", facet: ownerFacet(link) }, { name: "chord.app.pairing", @@ -223,9 +251,16 @@ export async function startApp

    (options: AppOptions

    , cont ), }, { name: "chord.app.files", facet: filesFacet(join(layout.server, "files"), options.files, limits.uploads) }, + { name: "chord.themes", facet: themesFacet() }, + { name: "chord.themes.choice", facet: themeChoiceFacet() }, ]; - const app = new RunningApp(host, http, workers, plugins, { store, lock }, layout, options); + const surfaces: Surface

    [] = []; + const app = new RunningApp(host, http, workers, plugins, surfaces, { store, lock }, layout, options); try { + for (const surface of await surfacesOf(host, options, plugins, context)) { + surfaces.push(surface); + core.push({ name: `chord.${surface.name}`, facet: surface.facet }); + } await plugins.load(host, { kind: "server", core, modules: modulesOf(api, options.kinds.server) }, context); await ownerTokenFile(layout.ownerToken, host, context); app.started(link.renew(), await http.listening(context)); @@ -236,6 +271,30 @@ export async function startApp

    (options: AppOptions

    , cont return app; } +/** A client surface of the app: its name (`web`, `tui`), its facet, its rollout step, its base bundle's hash. */ +type Surface

    = { + readonly name: "web" | "tui"; + readonly facet: Facet

    ; + step(name: string): RolloutStep; + readonly base: string; +}; + +/** The app's client surfaces, built (their base bundles) in order: the web's, the terminal's. */ +async function surfacesOf

    ( + host: Host

    , + options: AppOptions

    , + plugins: Plugins, + context: Context, +): Promise[]> { + const surfaces: Surface

    [] = []; + if (options.web !== undefined) + surfaces.push({ name: "web", ...(await webSurface(host, { ...options.web, plugins }, context)) }); + if (options.tui !== undefined) { + surfaces.push({ name: "tui", ...(await tuiSurface(host, { entry: options.tui.entry, plugins }, context)) }); + } + return surfaces; +} + /** How long a second server waits for the data directory's lock before it is refused. */ const SERVER_LOCK_MS = 1000; @@ -251,7 +310,7 @@ async function openServer( const dataDir = await openDataDir(path); const waited = AbortSignal.timeout(SERVER_LOCK_MS); const lock = await processLock(`chord-app:${dataDir}`, withAbortSignal(waited, context)).catch((error: unknown) => { - throw waited.aborted ? new Error(`startApp: another server runs on ${dataDir}`) : error; + throw waited.aborted ? new Error(`startServer: another server runs on ${dataDir}`) : error; }); try { const layout = await createLayout(dataDir); @@ -286,6 +345,8 @@ function workersOf

    ( context: Context, ): Workers { const facet = kind.facet ?? "worker"; + const serverKey = canonicalJson(options.principal.server as unknown as JsonValue, context); + const isServer = (principal: P): boolean => canonicalJson(principal as unknown as JsonValue, context) === serverKey; return createWorkers( host, // A host kind of the app: any params; its addresses come back from its route as the kit hands them on. @@ -295,7 +356,8 @@ function workersOf

    ( entry: kind.entry, ...(kind.execArgv === undefined ? {} : { execArgv: kind.execArgv }), ...(kind.env === undefined ? {} : { env: kind.env }), - authorize: kind.authorize, + // The server reaches its workers' kit objects (`HostData`) whatever the app admits. + authorize: (principal, address) => isServer(principal) || kind.authorize(principal, address), principal: options.principal.worker, start: async (address, ctx) => { const kit: KitStart = { @@ -323,10 +385,7 @@ function workerLock(name: string, dataDir: string, address: Address): string { return `${name}:${dataDir}:${address.key}`; } -/** An address `#quiesce` made quiet: its worker's lock, and whether it was failed before. */ -type Quiet = { readonly lock: ProcessLock; readonly failed: boolean }; - -/** How long an export or import waits for a stopped worker's lock before it stops the worker again. */ +/** How long an export or import waits for a worker's lock: past it a worker holds it (it runs or starts). */ const ADDRESS_LOCK_MS = 2000; /** The host of the owner link for the address listened on: loopback for a wildcard, an IPv6 address in brackets. */ @@ -344,6 +403,8 @@ class RunningApp

    implements App

    { readonly dataDir: string; readonly ownerTokenFile: string; readonly #component: Plugins; + /** The client surfaces, filled before the server's load. */ + readonly #surfaces: readonly Surface

    []; readonly #store: Store; readonly #lock: ProcessLock; readonly #options: AppOptions

    ; @@ -355,6 +416,7 @@ class RunningApp

    implements App

    { http: HttpComponent

    , workers: ReadonlyMap>, plugins: Plugins, + surfaces: readonly Surface

    [], server: { readonly store: Store; readonly lock: ProcessLock }, layout: { readonly dataDir: string; readonly plugins: string; readonly ownerToken: string }, options: AppOptions

    , @@ -363,6 +425,7 @@ class RunningApp

    implements App

    { this.http = http; this.workers = workers; this.#component = plugins; + this.#surfaces = surfaces; this.#store = server.store; this.#lock = server.lock; this.#options = options; @@ -384,72 +447,86 @@ class RunningApp

    implements App

    { return this.#link; } + get web(): { readonly base: string } | undefined { + return this.#surfaces.find(({ name }) => name === "web"); + } + + get tui(): { readonly base: string } | undefined { + return this.#surfaces.find(({ name }) => name === "tui"); + } + async applyChanges(context: Context): Promise { const plugins = this.#component; const built = await plugins.build(plugins.state.value.dirty, context); - const steps = [plugins.step("server", this.host), ...[...this.workers].map(([name, each]) => each.step(name))]; + const steps = [ + plugins.step("server", this.host), + ...[...this.workers].map(([name, each]) => each.step(name)), + ...this.#surfaces.map((surface) => surface.step(surface.name)), + ]; return plugins.rollout(steps, built, context); } - async exportAddress(address: Address, path: string, context: Context): Promise { - const quiet = await this.#quiesce(address, context); - try { - const manifest = this.#manifest(address); - await packAddress(workerDirectory(this.dataDir, address), manifest, resolve(path), context); - } finally { - await this.#resume(address, quiet, context); + async exportData(address: Address, path: string, context: Context): Promise { + const workers = this.#workersOf(address); + const manifest = this.#manifest(address); + const target = resolve(path); + // Running: the worker packs its own store. Else the lock, unless a worker starts first: then ask it. + for (let attempt = 1; ; attempt++) { + if (workers.state.value[address.key]?.status === "running") { + const data = await this.host.get(HostData, { address: addressParam(address) }, context); + try { + await data.exportTo({ path: target, manifest: JSON.stringify(manifest) }, context); + } finally { + data.release(); + } + return; + } + const lock = await this.#workerLock(address, context); + if (lock !== undefined) { + try { + await packData(workerDirectory(this.dataDir, address), manifest, target, context); + } finally { + await lock.release(); + } + return; + } + if (attempt === 5) throw new Error(`exportData: the worker at ${address.key} keeps starting; try again`); } } - async importAddress(archive: string, address: Address, context: Context): Promise { + async importData(archive: string, address: Address, context: Context): Promise { + this.#workersOf(address); const { name, limits = {} } = this.#options; const accept = (manifest: ArchiveManifest): void => { // A trust boundary: an archive from anywhere. - if (manifest?.format !== ARCHIVE_FORMAT) - throw new Error(`importAddress: not a format ${ARCHIVE_FORMAT} archive`); - if (manifest.app !== name) throw new Error(`importAddress: an archive of "${manifest.app}", not "${name}"`); + if (manifest?.format !== ARCHIVE_FORMAT) throw new Error(`importData: not a format ${ARCHIVE_FORMAT} archive`); + if (manifest.app !== name) throw new Error(`importData: an archive of "${manifest.app}", not "${name}"`); }; - const quiet = await this.#quiesce(address, context); + const lock = await this.#workerLock(address, context); + // A worker holds it: it runs, so the address has data, and nothing is overwritten. + if (lock === undefined) throw new Error(`importData: ${address.key} has data (its worker runs)`); try { const bounds = { bytes: limits.importBytes, entries: limits.importEntries }; - await unpackAddress(resolve(archive), workerDirectory(this.dataDir, address), accept, bounds, context); + await unpackData(resolve(archive), workerDirectory(this.dataDir, address), accept, bounds, context); } finally { - await this.#resume(address, quiet, context); + await lock.release(); } } - /** - * Takes the process lock of the worker at `address` (the one its worker takes at start) for an export or an import, - * stopping a worker that is not failed first: while it is held, a hold arriving starts a worker that waits for it, - * then opens the finished state. A worker started between the stop and the lock holds the lock first: it is - * stopped again (at most 5 times). A failed address stays failed: it starts nothing (its holds see its failure), - * and `#resume` leaves it so. - */ - async #quiesce(address: Address, context: Context): Promise { - const workers = this.#workersOf(address); - const failed = workers.state.value[address.key]?.status === "failed"; - for (let attempt = 1; ; attempt++) { - if (!failed) await workers.stop(address, context); - const waited = AbortSignal.timeout(ADDRESS_LOCK_MS); - try { - const lock = await processLock( - workerLock(this.#options.name, this.dataDir, address), - withAbortSignal(waited, context), - ); - return { lock, failed }; - } catch (error) { - if (!waited.aborted) throw error; - if (attempt === 5) throw new Error(`app: the worker at ${address.key} keeps starting; try again`); - } + /** The process lock of the worker at `address` (the one its worker takes at start), or `undefined` while held. */ + async #workerLock(address: Address, context: Context): Promise { + const waited = AbortSignal.timeout(ADDRESS_LOCK_MS); + try { + return await processLock( + workerLock(this.#options.name, this.dataDir, address), + withAbortSignal(waited, context), + ); + } catch (error) { + if (waited.aborted) return undefined; + throw error; } } - /** Releases the lock `#quiesce` took, and lets the address start again unless it was failed before. */ - async #resume(address: Address, quiet: Quiet, context: Context): Promise { - await quiet.lock.release(); - if (!quiet.failed) await this.#workersOf(address).retry(address, context); - } - close(context: Context): Promise { this.#closing ??= this.#close(context); return this.#closing; diff --git a/packages/chord/src/next-node/app/archive.ts b/packages/chord/src/next-node/app/archive.ts index 9177bcd76..a0b420f1d 100644 --- a/packages/chord/src/next-node/app/archive.ts +++ b/packages/chord/src/next-node/app/archive.ts @@ -1,14 +1,16 @@ -// Address archives (slice 13 §4, decision "The app kit, point 5" (export and import per address)): one tar of a -// worker's address, `manifest.json` (the app, its API version, the kit's format version), `store.archive` (the store's -// export archive, slice 6a, as JSONL) and `files/` (its files directory). Packing reads a stopped worker's directory; -// unpacking makes a new directory for an address that has none, refusing every path that would leave it. +// Data archives (slice 13 §4, decisions "The app kit, point 5" and "App kit naming and export"): one tar of the data a +// host created, `manifest.json` (the app, its API version, the kit's format version, the address), `store.archive` (the +// store's export archive, slice 6a, as JSONL: one consistent point, the export holding the commit queue while it reads) +// and `files/` (its finished files). A running worker packs its own open store; without one the server packs the +// directory under the worker's lock. Unpacking makes a new directory for an address that has none, refusing every path +// that would leave it. -import { createReadStream, createWriteStream } from "node:fs"; -import { mkdir, mkdtemp, readdir, rename, rm, stat } from "node:fs/promises"; +import { createReadStream, createWriteStream, type Dirent } from "node:fs"; +import { mkdir, mkdtemp, open, readdir, rename, rm, stat } from "node:fs/promises"; import { dirname, join, relative, sep } from "node:path"; import { pipeline } from "node:stream/promises"; import type { Context } from "../../context/index.ts"; -import { decodeJsonl, encodeJsonl, importStore, openStore, sqliteBackend } from "../../next/store/index.ts"; +import { decodeJsonl, encodeJsonl, importStore, openStore, type Store, sqliteBackend } from "../../next/store/index.ts"; import { openNodeSqliteDatabase } from "../../sqlite/node.ts"; import { privateDirectory, privateStore } from "./layout.ts"; import { readTar, type TarEntry, writeTar } from "./tar.ts"; @@ -28,33 +30,44 @@ export type ArchiveManifest = { const STORE = "store.sqlite"; /** - * Packs the host directory `dir` (its worker stopped: nothing writes it) into the archive at `path`: `manifest`, its - * store's export (an empty store's when it has none yet) and its `files/` (uploads under way left out). + * Packs the host directory `dir` of a host that does not run (its worker's lock held: nothing writes it) into the + * archive at `path`: `manifest`, its store's export and its files. */ -export async function packAddress( - dir: string, +export async function packData(dir: string, manifest: ArchiveManifest, path: string, context: Context): Promise { + if (!(await exists(dir))) throw new Error(`exportData: ${manifest.address.key} has no data`); + const store = await openStore(sqliteBackend(await openNodeSqliteDatabase(join(dir, STORE))), context); + try { + await packStore(store, join(dir, "files"), manifest, path, context); + } finally { + await store.close(context); + } +} + +/** + * Packs the open `store` and the files directory `files` into the archive at `path` (mode 0600, staged beside it and + * renamed): `manifest`, the store's export (one consistent point, written while the host runs on) and the finished + * files (uploads under way left out). + */ +export async function packStore( + store: Store, + files: string, manifest: ArchiveManifest, path: string, context: Context, ): Promise { - if (!(await exists(dir))) throw new Error(`exportAddress: ${manifest.address.key} has no data`); const staging = await mkdtemp(join(dirname(path), ".chord-export-")); try { const exported = join(staging, "store.archive"); - const store = await openStore(sqliteBackend(await openNodeSqliteDatabase(join(dir, STORE))), context); - try { - await pipeline(encodeJsonl(store.export(context)), createWriteStream(exported)); - } finally { - await store.close(context); - } + await pipeline(encodeJsonl(store.export(context)), createWriteStream(exported, { mode: 0o600 })); const text = new TextEncoder().encode(JSON.stringify(manifest)); - const entries: TarEntry[] = [ - { path: "manifest.json", size: text.length, data: chunksOf(text) }, - { path: "store.archive", size: (await stat(exported)).size, data: createReadStream(exported) }, - ...(await filesOf(join(dir, "files"))), - ]; + const size = (await stat(exported)).size; + async function* entries(): AsyncGenerator { + yield { path: "manifest.json", size: text.length, data: chunksOf(text) }; + yield { path: "store.archive", size, data: createReadStream(exported) }; + yield* filesOf(files); + } const written = join(staging, "archive.tar"); - await pipeline(writeTar(entries), createWriteStream(written, { mode: 0o600 })); + await pipeline(writeTar(entries()), createWriteStream(written, { mode: 0o600 })); await rename(written, path); } finally { await rm(staging, { recursive: true, force: true }); @@ -66,30 +79,52 @@ async function* chunksOf(bytes: Uint8Array): AsyncGenerator { yield bytes; } -/** The files under `root` as entries `files/` (hidden ones, uploads under way, left out). */ -async function filesOf(root: string): Promise { - const entries: TarEntry[] = []; +/** + * The files under `root` as entries `files/` (hidden ones, uploads under way, left out), each opened only + * when the archive comes to it (one descriptor at a time) and its size taken from the open file: a file deleted while a + * running worker is exported is skipped, or read whole from its open handle (finished files never change; Opus L7). + */ +async function* filesOf(root: string): AsyncGenerator { + for (const path of await pathsUnder(root)) { + const handle = await open(path, "r").catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return undefined; + throw error; + }); + if (handle === undefined) continue; + const info = await handle.stat(); + if (!info.isFile()) { + await handle.close(); + continue; + } + const inside = relative(root, path).split(sep).join("/"); + yield { + path: `files/${inside}`, + size: info.size, + data: handle.createReadStream({ start: 0, end: Math.max(0, info.size - 1) }), + }; + } +} + +/** The paths of the files under `root`, sorted, hidden ones left out (a directory gone meanwhile: none of it). */ +async function pathsUnder(root: string): Promise { + const paths: string[] = []; const walk = async (dir: string): Promise => { - let names: string[]; + let entries: Dirent[]; try { - names = (await readdir(dir)).sort(); + entries = (await readdir(dir, { withFileTypes: true })).sort((a, b) => (a.name < b.name ? -1 : 1)); } catch (error) { if ((error as NodeJS.ErrnoException).code === "ENOENT") return; throw error; } - for (const name of names) { - if (name.startsWith(".")) continue; - const path = join(dir, name); - const info = await stat(path); - if (info.isDirectory()) await walk(path); - else if (info.isFile()) { - const inside = relative(root, path).split(sep).join("/"); - entries.push({ path: `files/${inside}`, size: info.size, data: createReadStream(path) }); - } + for (const entry of entries) { + if (entry.name.startsWith(".")) continue; + const path = join(dir, entry.name); + if (entry.isDirectory()) await walk(path); + else if (entry.isFile()) paths.push(path); } }; await walk(root); - return entries; + return paths; } /** What an import takes at most: unpacked bytes (4 GiB by default) and entries (100,000 by default). */ @@ -100,14 +135,14 @@ export type ImportLimits = { readonly bytes?: number; readonly entries?: number * `accept` (a throw refuses it), the store made from `store.archive`, `files/` restored, within `limits`. Staged beside * `dir` (mode 0700, files 0600) and renamed into place at the end, so a refused or broken archive leaves nothing. */ -export async function unpackAddress( +export async function unpackData( archive: string, dir: string, accept: (manifest: ArchiveManifest) => void, limits: ImportLimits, context: Context, ): Promise { - if (await exists(dir)) throw new Error(`importAddress: ${dir} exists already (an address is imported once)`); + if (await exists(dir)) throw new Error(`importData: ${dir} exists already (an address is imported once)`); await privateDirectory(dirname(dir)); const staging = await mkdtemp(join(dirname(dir), ".chord-import-")); try { @@ -144,12 +179,12 @@ async function unpackEntries( let entries = 0; for await (const entry of readTar(createReadStream(archive))) { context.abortSignal?.throwIfAborted(); - if (++entries > maxEntries) throw new Error(`importAddress: the archive has more than ${maxEntries} entries`); + if (++entries > maxEntries) throw new Error(`importData: the archive has more than ${maxEntries} entries`); if (entry.kind === "directory") continue; bytes += entry.size; - if (bytes > maxBytes) throw new Error(`importAddress: the archive unpacks to more than ${maxBytes} bytes`); + if (bytes > maxBytes) throw new Error(`importData: the archive unpacks to more than ${maxBytes} bytes`); if (manifest === undefined && entry.path !== "manifest.json") { - throw new Error("importAddress: the archive does not start with manifest.json"); + throw new Error("importData: the archive does not start with manifest.json"); } if (entry.path === "manifest.json") { // A trust boundary: the archive's own description, checked by `accept`. @@ -164,11 +199,11 @@ async function unpackEntries( await mkdir(dirname(target), { recursive: true, mode: 0o700 }); await pipeline(entry.data, createWriteStream(target, { mode: 0o600 })); } else { - throw new Error(`importAddress: the archive holds ${entry.path}, which an address archive does not`); + throw new Error(`importData: the archive holds ${entry.path}, which an address archive does not`); } } if (manifest === undefined || storeArchive === undefined) { - throw new Error("importAddress: the archive has no manifest.json or store.archive"); + throw new Error("importData: the archive has no manifest.json or store.archive"); } return storeArchive; } @@ -179,7 +214,7 @@ async function collect(data: AsyncIterable, limit: number): Promise< let size = 0; for await (const chunk of data) { size += chunk.length; - if (size > limit) throw new Error("importAddress: manifest.json is too large"); + if (size > limit) throw new Error("importData: manifest.json is too large"); chunks.push(chunk); } return Buffer.concat(chunks); diff --git a/packages/chord/src/next-node/app/data.ts b/packages/chord/src/next-node/app/data.ts new file mode 100644 index 000000000..df18e6dce --- /dev/null +++ b/packages/chord/src/next-node/app/data.ts @@ -0,0 +1,52 @@ +// A host's data on its worker (decision "App kit naming and export"): the kit's object a running worker serves, through +// which the server exports the worker's data without stopping it. The worker packs its own open store (its export holds +// the commit queue while it reads: one consistent point) and its finished files into the archive at the path the server +// names. Only the server may ask (its principal); the worker's plugins never see it. + +import { BACKGROUND_CONTEXT } from "../../context/index.ts"; +import type { JsonValue } from "../../json.ts"; +import { type Address, method, obj, remote, str } from "../../next/contracts/index.ts"; +import { ChordError, type Principal } from "../../next/data/index.ts"; +import { canonicalJson } from "../../next/data/principal.ts"; +import { defineFacet, type Facet } from "../../next/host/index.ts"; +import type { Store } from "../../next/store/index.ts"; +import { type ArchiveManifest, packStore } from "./archive.ts"; + +/** The data of the worker at `address` (its JSON, as `addressParam` writes it): `exportTo` packs it at `path`. */ +export const HostData = remote("chord.app.data", { + version: 1, + params: { address: str() }, + // The address a worker has: the server names it (`addressParam`); only its kind and params place the object. + hostAddress: ({ address }) => JSON.parse(address) as Address, + variants: { main: { methods: { exportTo: method(obj({ path: str(), manifest: str() })) } } }, +}); + +/** `address` as `HostData`'s param. */ +export function addressParam(address: Address): string { + return JSON.stringify({ kind: address.kind, params: address.params, key: address.key }); +} + +/** + * The kit's worker facet serving `HostData` for the worker's open `store` and its files directory `files`, to `server` + * (the server's principal) only. + */ +export function hostDataFacet

    (store: Store, files: string, server: P): Facet

    { + const serverKey = canonicalJson(server as JsonValue, BACKGROUND_CONTEXT); + return defineFacet

    ((host) => + host.serve(HostData, (_params, principal) => { + if (canonicalJson(principal as JsonValue, BACKGROUND_CONTEXT) !== serverKey) { + throw new ChordError({ code: "forbidden", message: "chord.app.data: the server's only" }); + } + return { + variant: "main", + members: { + exportTo: async ({ path, manifest }, ctx) => { + // The server's own manifest for this address. + await packStore(store, files, JSON.parse(manifest) as ArchiveManifest, path, ctx); + return null; + }, + }, + }; + }), + ); +} diff --git a/packages/chord/src/next-node/app/ephemeral-sessions.ts b/packages/chord/src/next-node/app/ephemeral-sessions.ts new file mode 100644 index 000000000..a64f6404a --- /dev/null +++ b/packages/chord/src/next-node/app/ephemeral-sessions.ts @@ -0,0 +1,80 @@ +// Ephemeral sessions (slice 13 §3, carry-over to slice 11): a pick-a-name login's sessions, in memory only, at most +// `max` (past it the oldest ends), and each ending with its client's connection: once the last Chord link +// authenticated with it closed and none opened for a grace period (a page reload, a reconnect after a drop), or when +// none opened within that period of the login. + +import type { Principal } from "../../next/data/index.ts"; + +/** How long an ephemeral session lives without a link: a minute, past a reload and the reconnect loop's pauses. */ +export const EPHEMERAL_GRACE_MS = 60_000; + +/** One ephemeral session: its secret's hash, its principal, its open links, its end's timer while it has none. */ +type Session = { + readonly hash: string; + readonly principal: Principal; + links: number; + timer: ReturnType | undefined; +}; + +/** The server's ephemeral sessions. */ +export class EphemeralSessions { + readonly #max: number; + readonly #graceMs: number; + /** Insertion order is age order: the first entry is the oldest. */ + readonly #sessions = new Map(); + + constructor(max: number, graceMs = EPHEMERAL_GRACE_MS) { + this.#max = max; + this.#graceMs = graceMs; + } + + /** A new session `id` (its secret's hash `hash`) for `principal`; the oldest ends past the cap. */ + issue(id: string, hash: string, principal: Principal): void { + const session: Session = { hash, principal, links: 0, timer: undefined }; + this.#sessions.set(id, session); + this.#waitForLink(id, session); + for (const [oldest] of this.#sessions) { + if (this.#sessions.size <= this.#max) break; + this.revoke(oldest); + } + } + + /** Session `id`'s hash and principal, while it lives. */ + get(id: string): { readonly hash: string; readonly principal: Principal } | undefined { + return this.#sessions.get(id); + } + + /** Ends session `id`; whether it was one. */ + revoke(id: string): boolean { + const session = this.#sessions.get(id); + if (session === undefined) return false; + clearTimeout(session.timer); + this.#sessions.delete(id); + return true; + } + + /** A link authenticated with session `id` opened: what to call when it closes; `undefined` for no such session. */ + linkOpened(id: string): (() => void) | undefined { + const session = this.#sessions.get(id); + if (session === undefined) return undefined; + session.links++; + clearTimeout(session.timer); + session.timer = undefined; + let closed = false; + return () => { + if (closed) return; + closed = true; + session.links--; + if (session.links === 0 && this.#sessions.get(id) === session) this.#waitForLink(id, session); + }; + } + + /** Ends `session` unless a link opens within the grace period. */ + #waitForLink(id: string, session: Session): void { + session.timer = setTimeout(() => { + if (session.links === 0) this.revoke(id); + }, this.#graceMs); + // The server's process does not wait for it. + session.timer.unref?.(); + } +} diff --git a/packages/chord/src/next-node/app/index.ts b/packages/chord/src/next-node/app/index.ts index aff1f6f71..2eb6e1b4b 100644 --- a/packages/chord/src/next-node/app/index.ts +++ b/packages/chord/src/next-node/app/index.ts @@ -1,4 +1,4 @@ -// Chord v2's app kit (`@earendil-works/chord/next/app` once v2 has its entry point; slice 13): `startApp` on the server +// Chord v2's app kit (`@earendil-works/chord/next/app` once v2 has its entry point; slice 13): `startServer` on the server // and `startWorker` in a worker's entry, the start-data objects, sessions and the owner's sign-in, device pairing, the // files of each host, export and import per address, and the optional plugins (users, invites, ephemeral login). @@ -11,7 +11,7 @@ export { type AppPlugins, type AppWorkerKind, Server, - startApp, + startServer, } from "./app.ts"; export { ARCHIVE_FORMAT, type ArchiveManifest } from "./archive.ts"; export { type EphemeralUser, ephemeralLoginFacet } from "./ephemeral.ts"; diff --git a/packages/chord/src/next-node/app/layout.ts b/packages/chord/src/next-node/app/layout.ts index 063edd825..ad3b379fa 100644 --- a/packages/chord/src/next-node/app/layout.ts +++ b/packages/chord/src/next-node/app/layout.ts @@ -38,11 +38,11 @@ export async function openDataDir(path: string): Promise { const home = await realpath(homedir()); if (dataDir === home || dataDir === parse(dataDir).root) { const what = dataDir === home ? "the home directory" : "a file system's root"; - throw new Error(`startApp: the data directory ${dataDir} is ${what}; name a directory of the app's own`); + throw new Error(`startServer: the data directory ${dataDir} is ${what}; name a directory of the app's own`); } const { uid } = await stat(dataDir); if (process.getuid !== undefined && uid !== process.getuid()) { - throw new Error(`startApp: the data directory ${dataDir} belongs to another user (uid ${uid})`); + throw new Error(`startServer: the data directory ${dataDir} belongs to another user (uid ${uid})`); } return dataDir; } diff --git a/packages/chord/src/next-node/app/sessions.ts b/packages/chord/src/next-node/app/sessions.ts index f42a4e5a1..88ce03bba 100644 --- a/packages/chord/src/next-node/app/sessions.ts +++ b/packages/chord/src/next-node/app/sessions.ts @@ -3,8 +3,9 @@ // the secret's hash is kept. A browser carries its token in the kit's signed session cookie (`HttpOnly`, // `SameSite=Strict`), a command-line client in `Authorization: Bearer `. Browser and device sessions persist in // the server's store and expire after 400 days unused; ephemeral sessions (a pick-a-name login, a public route) live -// in memory only, at most `maxEphemeral` of them (past it the oldest ends), so a public login costs the same whatever -// came before, and none outlives the server's run. +// in memory only (`EphemeralSessions`), at most `maxEphemeral` of them (past it the oldest ends), so a public login +// costs the same whatever came before; each ends with its client's connection (its last link closed, a grace period +// passed: the links endpoint's `opened` hook counts them), and none outlives the server's run. import type { Context } from "../../context/index.ts"; import type { JsonValue } from "../../json.ts"; @@ -22,6 +23,7 @@ import { readCookie, readSignedCookie, } from "../http/index.ts"; +import { EphemeralSessions } from "./ephemeral-sessions.ts"; import { hashToken, matchesHash, newToken } from "./secrets.ts"; /** What a session is for: a browser (its cookie), a device (a command-line client's token), an ephemeral login. */ @@ -96,9 +98,6 @@ const SessionsDocument = defineState("chord.app.sessions", { checkpointWhen: (_value, _ops, { deltasSinceBase }) => deltasSinceBase >= 50, }); -/** An ephemeral session, in memory: its secret's hash and its principal. */ -type EphemeralSession = { readonly hash: string; readonly principal: Principal }; - /** How often a session's `lastUsed` is written at most: once a minute (not one commit per request). */ const TOUCH_MS = 60_000; @@ -108,17 +107,33 @@ const SESSION_IDLE_MS = 400 * 24 * 60 * 60_000; /** The sessions' limits: how many ephemeral sessions live at once (1,000 by default; past it the oldest ends). */ export type SessionLimits = { readonly maxEphemeral?: number }; +/** The kit's sessions: the facet for the server's core facets, and the links endpoint's `opened` hook. */ +export type KitSessions

    = { + readonly facet: Facet

    ; + /** A link opened with `request`'s credentials: an ephemeral session's links are counted (it ends with its last). */ + readonly opened: (request: HttpRequest) => (() => void) | undefined; +}; + /** - * The kit's facet serving `Sessions` (a host-lifetime object) and contributing its authenticator: the bearer token - * first, else the session cookie signed with `cookieSecret`. + * The kit's sessions: its facet serving `Sessions` (a host-lifetime object) and contributing its authenticator (the + * bearer token first, else the session cookie signed with `cookieSecret`), and the links hook ending an ephemeral + * session with its client's connection. */ -export function sessionsFacet

    (cookieSecret: string, limits: SessionLimits = {}): Facet

    { - const maxEphemeral = limits.maxEphemeral ?? 1000; +export function createSessions

    (cookieSecret: string, limits: SessionLimits = {}): KitSessions

    { + const ephemeral = new EphemeralSessions(limits.maxEphemeral ?? 1000); + const opened = (request: HttpRequest): (() => void) | undefined => { + const token = bearerToken(request) ?? readCookie(request, SESSION_COOKIE, cookieSecret); + const dot = token?.indexOf(".") ?? -1; + return dot > 0 ? ephemeral.linkOpened(token!.slice(0, dot)) : undefined; + }; + return { facet: sessionsFacet

    (cookieSecret, ephemeral), opened }; +} + +/** The facet of `createSessions`. */ +function sessionsFacet

    (cookieSecret: string, ephemeral: EphemeralSessions): Facet

    { return defineFacet

    ((host) => host.serve(Sessions, (_params, self) => { const document = self.state(SessionsDocument); - // Insertion order is age order: the first entry is the oldest. - const ephemeral = new Map(); const change = (fn: (sessions: Record) => void, context: Context) => // Not awaited by authentication: the request goes on; a write that fails is made again by a later request. void document.change((draft) => fn(draft.sessions), context).catch(() => {}); @@ -172,11 +187,7 @@ export function sessionsFacet

    (cookieSecret: string, limits: const secret = newToken(32); if (kind === "ephemeral") { // Principals are the app's: `issue`'s callers pass one of its type. - ephemeral.set(id, { hash: hashToken(secret), principal: principal as Principal }); - for (const [oldest] of ephemeral) { - if (ephemeral.size <= maxEphemeral) break; - ephemeral.delete(oldest); - } + ephemeral.issue(id, hashToken(secret), principal as Principal); } else { const now = Date.now(); const stored = JSON.stringify(principal); @@ -199,7 +210,7 @@ export function sessionsFacet

    (cookieSecret: string, limits: }, verify: async (token, ctx) => (authenticate(token, ctx) as JsonValue | undefined) ?? null, revoke: async (id, ctx) => { - if (ephemeral.delete(id)) return null; + if (ephemeral.revoke(id)) return null; await document.change((draft) => { delete draft.sessions[id]; }, ctx); diff --git a/packages/chord/src/next-node/app/worker.ts b/packages/chord/src/next-node/app/worker.ts index edb398434..d9addad45 100644 --- a/packages/chord/src/next-node/app/worker.ts +++ b/packages/chord/src/next-node/app/worker.ts @@ -12,11 +12,12 @@ import { openNodeSqliteDatabase } from "../../sqlite/node.ts"; import type { Watchable } from "../../watch.ts"; import { workerMain } from "../workers/index.ts"; import type { AppApi, AppKind, KitStart } from "./app.ts"; +import { hostDataFacet } from "./data.ts"; import { filesFacet } from "./files.ts"; import { workerInfoFacet } from "./info.ts"; import { privateDirectory, privateStore } from "./layout.ts"; -/** What `startWorker` takes: the app's API and kinds as the server's `startApp` has them, this process's modules. */ +/** What `startWorker` takes: the app's API and kinds as the server's `startServer` has them, this process's modules. */ export type WorkerOptions = { readonly api: AppApi; readonly kinds: Readonly>; @@ -32,7 +33,7 @@ export function startWorker

    (options: WorkerOptions): void { const packages = Object.assign({}, ...Object.values(options.kinds).map((kind) => kind.packages ?? {})); workerMain( async ({ address, start, parent, plugins, onClose }, context) => { - // The server's `startApp` sends `{ kit, app }`. + // The server's `startServer` sends `{ kit, app }`. const { kit, app } = start as { readonly kit: KitStart; readonly app: JsonValue }; await privateDirectory(kit.dir); await privateStore(join(kit.dir, "store.sqlite")); @@ -58,6 +59,10 @@ export function startWorker

    (options: WorkerOptions): void { const core = [ { name: "chord.app.worker-info", facet: workerInfoFacet

    (info) }, { name: "chord.app.files", facet: filesFacet

    (join(kit.dir, "files"), kit.files, kit.uploads) }, + { + name: "chord.app.data", + facet: hostDataFacet

    (store, join(kit.dir, "files"), kit.server as unknown as P), + }, ]; await plugins.load(host, { kind: kit.facet, core }, context); return options.busy === undefined ? {} : { busy: options.busy }; diff --git a/packages/chord/src/next-node/http/README.md b/packages/chord/src/next-node/http/README.md index 8d482c464..fbb51a58e 100644 --- a/packages/chord/src/next-node/http/README.md +++ b/packages/chord/src/next-node/http/README.md @@ -15,7 +15,8 @@ Exported as `@earendil-works/chord/next/http` once v2 has its entry point. // the app's settings are arguments, fixed for the component's life (changing one is a restart) const http = createHttp(server, { listen: { host: "127.0.0.1", port: 8788 }, // Chord's own listener; port 0 picks one; none: no listener - links: { path: "/chord", options: (request, principal) => ({ limits, maxPayload }) }, // or a value + links: { path: "/chord", options: (request, principal) => ({ limits, maxPayload }), // or a value + opened: (request, principal) => () => linkClosed() }, // called per accepted link; its return at its close closeTimeout: 30_000, onError: (error) => log(error), // HttpError }, ctx); @@ -50,6 +51,7 @@ self.contribute(Http, { framers: ["https://shell.example.com"], // who may frame the app's pages relays: [{ name: "radius", verify: (request) => verified(request) }], // { origin, framer? } | undefined csp: { "img-src": ["https://cdn.example.com"] }, + scripts: ["/_chord/web/"], // script-src: these paths of the origin (and the nonce), not all of it }); ``` @@ -136,7 +138,8 @@ await http.close(ctx); // before host.close - **Default headers** on every answer, the component's own 403, 404 and 503 included: `Cache-Control: no-store`, `X-Content-Type-Options: nosniff`, `Referrer-Policy: no-referrer`, `X-Frame-Options: DENY` (without framers), a CSP (`default-src 'none'`, scripts `'self'` and the response's own nonce, `connect-src` self and its WebSocket, `base-uri - 'none'`, `form-action 'self'`, `frame-ancestors` the framers or `'none'`, plus contributed sources). A response's own + 'none'`, `form-action 'self'`, `frame-ancestors` the framers or `'none'`, plus contributed sources; with contributed + `scripts` paths, `script-src` is those paths of the request's origin and the nonce instead of `'self'`). A response's own headers replace same-named ones (a list sends one line per value: several `Set-Cookie`); a header with a CR, LF or NUL, or a value that is no JSON, is a 500 reported `response_failed`. - **Sockets:** after admission, the origin checks and the authenticators (no principal and not public: 401 before diff --git a/packages/chord/src/next-node/http/headers.ts b/packages/chord/src/next-node/http/headers.ts index 4358e8140..cd79f088a 100644 --- a/packages/chord/src/next-node/http/headers.ts +++ b/packages/chord/src/next-node/http/headers.ts @@ -119,7 +119,7 @@ function csp(admitted: Admitted | undefined, router: Router | undefined): readon if (admitted?.framer !== undefined) framers.push(admitted.framer); const directives = new Map([ ["default-src", ["'none'"]], - ["script-src", ["'self'", NONCE]], + ["script-src", [...scriptSources(admitted, router), NONCE]], ["connect-src", ["'self'", ...(admitted === undefined ? [] : [admitted.origin.replace(/^http/, "ws")])]], ["base-uri", ["'none'"]], ["form-action", ["'self'"]], @@ -138,5 +138,12 @@ function csp(admitted: Admitted | undefined, router: Router | undefined): readon return [`${text.slice(0, at)}'nonce-`, `'${text.slice(at + NONCE.length)}`]; } +/** Where scripts may come from: the contributed script paths of the request's origin, else the whole origin. */ +function scriptSources(admitted: Admitted | undefined, router: Router | undefined): string[] { + const paths = router?.scripts ?? []; + if (admitted === undefined || paths.length === 0) return ["'self'"]; + return paths.map((path) => `${admitted.origin}${path}`); +} + /** Where the nonce goes in the CSP's text while it is made. */ const NONCE = "\0nonce\0"; diff --git a/packages/chord/src/next-node/http/http.ts b/packages/chord/src/next-node/http/http.ts index 4ebb25abe..e21271529 100644 --- a/packages/chord/src/next-node/http/http.ts +++ b/packages/chord/src/next-node/http/http.ts @@ -46,6 +46,11 @@ export type HttpOptions = { export type LinksOptions = { readonly path: string; readonly options?: LinkOptions | ((request: HttpRequest, principal: Principal) => LinkOptions); + /** + * Called when a link is accepted, with its request and principal; what it returns is called when that link closes + * (the app tracking its clients' links: the kit ends an ephemeral session with its last link). + */ + readonly opened?: (request: HttpRequest, principal: Principal) => (() => void) | undefined; }; /** What the component reports. */ @@ -59,7 +64,7 @@ export type HttpError = | { readonly kind: "hook_failed"; readonly hook: HookName; readonly plugin?: string; readonly error: unknown }; /** The hooks the component calls. */ -export type HookName = "authenticate" | "relay" | "links.options"; +export type HookName = "authenticate" | "relay" | "links.options" | "links.opened"; /** The running component. */ export interface HttpComponent

    { @@ -437,7 +442,15 @@ class Component

    implements HttpComponent

    , Handler { return refuse(stream, 403); } const admitted = { allowed, relay, principal, ...options }; - return this.#upgrades.link(message, stream, head, admitted, this.#host, this.#context); + const opened = (): (() => void) | undefined => { + try { + return links.opened?.(webRequest, principal); + } catch (error) { + this.#hookFailed("links.opened", undefined, error); + return undefined; + } + }; + return this.#upgrades.link(message, stream, head, admitted, this.#host, this.#context, opened); } const match = router.socket(webRequest.path); if (match === "bad_path") return refuse(stream, 400); diff --git a/packages/chord/src/next-node/http/router.ts b/packages/chord/src/next-node/http/router.ts index 1aa1debac..846fb3b80 100644 --- a/packages/chord/src/next-node/http/router.ts +++ b/packages/chord/src/next-node/http/router.ts @@ -80,6 +80,12 @@ export type HttpContribution = { readonly relays?: readonly HttpRelay[]; /** Extra CSP sources by directive (`{ "img-src": ["https://cdn.example.com"] }`). */ readonly csp?: Readonly>; + /** + * The paths scripts load from (`/_chord/web/`): with any, `script-src` is these paths of the request's origin and the + * response's nonce instead of the whole origin, so no other same-origin response (an upload served as JavaScript) + * runs as a script. + */ + readonly scripts?: readonly string[]; }; /** The router a request uses: as it was when the request started. */ @@ -96,6 +102,8 @@ export type Router = { readonly relays: readonly HttpRelay[]; /** Extra CSP sources by directive. */ readonly csp: ReadonlyMap; + /** The paths scripts load from; none: the whole origin. */ + readonly scripts: readonly string[]; /** The socket handlers it serves: a socket handed out by one no longer here is closed. */ readonly socketHandlers: ReadonlySet; }; @@ -147,11 +155,13 @@ export function composeRouter(table: RegistryTable, { report } const framers = new Set(); const relays: HttpRelay[] = []; const csp = new Map(); + const scripts: string[] = []; const socketHandlers = new Set(); for (const { plugin, value } of table) { for (const host of value.hosts ?? []) hosts.add(host.toLowerCase()); for (const framer of value.framers ?? []) framers.add(framer); relays.push(...(value.relays ?? [])); + scripts.push(...(value.scripts ?? [])); for (const [directive, sources] of Object.entries(value.csp ?? {})) { csp.set(directive, [...(csp.get(directive) ?? []), ...sources]); } @@ -215,6 +225,7 @@ export function composeRouter(table: RegistryTable, { report } framers, relays, csp, + scripts, socketHandlers, }; } diff --git a/packages/chord/src/next-node/http/upgrades.ts b/packages/chord/src/next-node/http/upgrades.ts index 820ba83fa..bc10751c2 100644 --- a/packages/chord/src/next-node/http/upgrades.ts +++ b/packages/chord/src/next-node/http/upgrades.ts @@ -48,7 +48,8 @@ export class Upgrades { /** * Upgrades `message` to the links endpoint: `host.accept` over the socket (its output capped) as the admitted - * principal, with `options`' limits; `context` is the parent of the work on the link. + * principal, with `options`' limits; `context` is the parent of the work on the link. `opened` runs once the link + * is accepted; what it returns runs when its socket closes. */ link

    ( message: IncomingMessage, @@ -57,6 +58,7 @@ export class Upgrades { admission: Admission

    & LinkOptions, host: Host

    , context: Context, + opened: () => (() => void) | undefined, ): void { const { principal, limits } = admission; upgrade(message, stream, head, admission.maxPayload ?? DEFAULT_LINK_PAYLOAD, (socket) => { @@ -69,6 +71,8 @@ export class Upgrades { seal: () => link.close(), }); host.accept(link, { principal, ...(limits === undefined ? {} : { limits }) }, context); + const closed = opened(); + if (closed !== undefined) socket.once("close", closed); }); } diff --git a/packages/chord/src/next-node/plugins/README.md b/packages/chord/src/next-node/plugins/README.md index 1a6965394..a79421bc1 100644 --- a/packages/chord/src/next-node/plugins/README.md +++ b/packages/chord/src/next-node/plugins/README.md @@ -60,7 +60,9 @@ await plugins.rollout([plugins.step("server", server), workers.step("workers")], it; an ordinary library names files as it likes) is refused naming the chain. A plugin's root may lie under a `node_modules` (an app installed from npm with its plugins): only the path below a plugin's directory counts. A copied file imports host packages and its own plugin's files only, no builtins. A single-file plugin imports host packages only. External: host packages, `@earendil-works/chord/next/plugins`, `chord.external` - packages and Node builtins (Node builds; a browser build refuses both). Browser builds collect CSS into one stylesheet (`css`) and + packages and Node builtins (Node builds; a browser build refuses both). Browser builds collect CSS into one stylesheet (`css`; a `*.module.css` file scoped to its + plugin and file: its classes renamed `chord___`, its import esbuild's mapping, `composes` and + escaped names included, `css-modules.ts`; a stylesheet's `url()` is an asset, an `@import`ed file plain CSS) and assets (`import url from "./icon.svg?url"`, `.wasm`) into content-hashed files (`assets`) under `publicPath`. Refused, with the file and line: `import.meta`, `__dirname`, `__filename` (each use, found through the script's source map), a literal dynamic `import("x")` and top-level `await` (pointing to `pluginFile`; a computed `import(x)` @@ -166,5 +168,6 @@ dependency pinned to a commit, per-kind contract files, a required root), `51-si per-kind files, roots, and input capture), `evaluate.ts` (evaluation, the `require` a script gets, `pluginFile`), `stack.ts` (the source maps of evaluated scripts while their facets live, `mapStack`), `install.ts` (dependencies, git folder packages, plugins from git and npm), `watcher.ts` (the watcher and its marks, a JSON watchable), -`runtime.ts` (per-host loaded facets, complete-set loads and reloads, required roots, no esbuild), `plugins.ts` (the +`css-modules.ts` (CSS modules scoped per plugin and file), `runtime.ts` (evaluated scripts as complete sets for the +portable `LoadedFacets` of `next/surfaces`, required roots, no esbuild), `plugins.ts` (the component: kinds, current sets, `build`, `load`, `reload`, `step`, `rollout`, installs), `testing.ts` (`testPlugin`). diff --git a/packages/chord/src/next-node/plugins/build.ts b/packages/chord/src/next-node/plugins/build.ts index dbcb319e2..adffb2e8a 100644 --- a/packages/chord/src/next-node/plugins/build.ts +++ b/packages/chord/src/next-node/plugins/build.ts @@ -10,8 +10,16 @@ import { createHash } from "node:crypto"; import { readFile, realpath } from "node:fs/promises"; import { isBuiltin } from "node:module"; import { basename, dirname, extname, isAbsolute, join, relative, sep } from "node:path"; -import { type BuildFailure, build, type Plugin as EsbuildPlugin, type Loader, type Message } from "esbuild"; +import { + type BuildFailure, + build, + type Plugin as EsbuildPlugin, + type Loader, + type Message, + type OnLoadResult, +} from "esbuild"; import type { Context } from "../../context/index.ts"; +import { SCOPED_CSS, scopeCssModule } from "./css-modules.ts"; import { discoverPackage, type Plugin } from "./discover.ts"; import { PLUGINS_SPECIFIER, packageName } from "./evaluate.ts"; import { inlineSourceMap } from "./stack.ts"; @@ -255,6 +263,8 @@ class ImportRules { readonly #hostPackages: ReadonlySet; readonly #inputs = new Map(); readonly #copied = new Set(); + /** The scoped stylesheet of every CSS module read, by its file. */ + readonly #scoped = new Map(); /** The first importer of every file resolved, by file. */ readonly #importers = new Map(); /** The API versions imported, each with its first importer. */ @@ -278,6 +288,9 @@ class ImportRules { name: "chord-plugin-imports", setup: (builder) => { builder.onResolve({ filter: /.*/ }, async (args) => { + if (args.path.startsWith(`${SCOPED_CSS}:`)) { + return { path: args.path.slice(SCOPED_CSS.length + 1), namespace: SCOPED_CSS }; + } // The plugin's files are resolved here, the entry too: esbuild then gives them no Node module type // (`"type": "module"`), whose interop would read a whole host package for a default import instead // of honouring the `__esModule` an ESM namespace is wrapped with (§3). @@ -326,18 +339,34 @@ class ImportRules { return { path: found.path, pluginData: resolved.asset ? ASSET : undefined }; }); builder.onLoad({ filter: /.*/ }, async (args) => { + if (args.namespace === SCOPED_CSS) { + // Its `url()` and `@import` resolve relative to the module file. + return { contents: this.#scoped.get(args.path) ?? "", loader: "css", resolveDir: dirname(args.path) }; + } if (args.namespace !== "file") return undefined; const asset = args.pluginData === ASSET; const loader = asset ? "file" : LOADERS[extname(args.path)]; if (loader === undefined || this.#place(args.path).kind === "package") return undefined; const contents = await readFile(args.path); this.#inputs.set(args.path, createHash("sha256").update(contents).digest("hex")); + if (!asset && args.path.endsWith(".module.css")) return this.#cssModule(args.path, contents); return { contents, loader, resolveDir: dirname(args.path) }; }); }, }; } + /** + * The CSS module `file` with `contents` as the facet imports it: a module whose default export is the class mapping, + * importing the scoped stylesheet (`scopeCssModule`), which the build collects with the facet's other CSS. + */ + async #cssModule(file: string, contents: Uint8Array): Promise { + const scoped = await scopeCssModule(this.#own.name, this.#own.dir, file, new TextDecoder().decode(contents)); + this.#scoped.set(file, scoped.css); + const module = `import ${JSON.stringify(`${SCOPED_CSS}:${file}`)};export default ${JSON.stringify(scoped.classes)};`; + return { contents: module, loader: "js", resolveDir: dirname(file) }; + } + /** Every plugin file read, sorted by path. */ inputs(): BuildInput[] { return [...this.#inputs].map(([file, hash]) => ({ file, hash })).sort((a, b) => (a.file < b.file ? -1 : 1)); @@ -412,7 +441,9 @@ class ImportRules { return { kind: "error", text: `"${specifier}": assets by URL are for browser builds (${PLUGIN_FILE_HINT})` }; } if (from.kind === "package") return { kind: "package" }; - if (specifier.startsWith(".") || isAbsolute(specifier)) return { kind: "path", path: specifier, asset: false }; + // A stylesheet's `url()` names an asset of the plugin (served under `publicPath`). + const asset = kind === "url-token" && browser; + if (specifier.startsWith(".") || isAbsolute(specifier)) return { kind: "path", path: specifier, asset }; const name = packageName(specifier); if (this.#own.external.includes(name)) { if (!browser) return { kind: "external" }; diff --git a/packages/chord/src/next-node/plugins/css-modules.ts b/packages/chord/src/next-node/plugins/css-modules.ts new file mode 100644 index 000000000..a5670cdaa --- /dev/null +++ b/packages/chord/src/next-node/plugins/css-modules.ts @@ -0,0 +1,65 @@ +// CSS modules in plugin builds (slice 11 §4b, decision "Looks and themes"; fix round 1, Opus L1): a plugin's +// `*.module.css` has its class names made unique to the file, the import giving the mapping, so twenty plugins' +// `.card` never collide. esbuild's `local-css` loader names classes uniquely within one build only (and a minified +// build names them `.a`, `.b`), while each facet is a build of its own; so each CSS module is scoped by a small build of +// its own whose names start with a prefix of the plugin and the file, and the facet's build gets the scoped stylesheet +// and the mapping. The mapping is esbuild's own (`composes` and escaped names as esbuild gives them); `url()` and +// `@import` stay as written and resolve in the facet's build, relative to the module (an `@import`ed file is plain CSS, +// its classes not scoped). + +import { createHash } from "node:crypto"; +import { basename, relative } from "node:path"; +import { Script } from "node:vm"; +import { build } from "esbuild"; + +/** A CSS module scoped: its stylesheet with unique class names, and the mapping its import gives. */ +export type ScopedCss = { readonly css: string; readonly classes: Readonly> }; + +/** The namespace of a scoped stylesheet the facet's build imports. */ +export const SCOPED_CSS = "chord-css-module"; + +/** + * Scopes the CSS module `file` of the plugin `plugin` (its directory `dir`) with `contents`: every class renamed + * `chord___`. Throws esbuild's failure (a syntax error) for the build to report. + */ +export async function scopeCssModule(plugin: string, dir: string, file: string, contents: string): Promise { + const scope = createHash("sha256") + .update(`${plugin}\0${relative(dir, file)}`) + .digest("hex") + .slice(0, 8); + // esbuild names a local class `_`: the file name carries the prefix. + const name = `chord_${scope}_${basename(file, ".module.css").replace(/[^\w-]/g, "_")}`; + const result = await build({ + stdin: { contents: `export { default } from "scoped:${name}.module.css";`, loader: "js" }, + bundle: true, + write: false, + outdir: "/", + format: "cjs", + minifyWhitespace: true, + logLevel: "silent", + plugins: [ + { + name: "chord-css-module", + setup: (builder) => { + builder.onResolve({ filter: /^scoped:/ }, (args) => ({ + path: `/${args.path.slice(7)}`, + namespace: "scoped", + })); + // Kept as written: the facet's build resolves them relative to the module file. + builder.onResolve({ filter: /.*/ }, (args) => + args.kind === "url-token" || args.kind === "import-rule" + ? { path: args.path, external: true } + : undefined, + ); + builder.onLoad({ filter: /.*/, namespace: "scoped" }, () => ({ contents, loader: "local-css" })); + }, + }, + ], + }); + const css = result.outputFiles.find((output) => output.path.endsWith(".css"))?.text ?? ""; + const js = result.outputFiles.find((output) => output.path.endsWith(".js"))?.text ?? ""; + // esbuild's mapping module (an object of strings it wrote), run alone to read the mapping. + const module = { exports: {} as { default?: Record } }; + new Script(js).runInNewContext({ module, exports: module.exports }); + return { css, classes: { ...module.exports.default } }; +} diff --git a/packages/chord/src/next-node/plugins/evaluate.ts b/packages/chord/src/next-node/plugins/evaluate.ts index 5641fa251..50aaf4bb1 100644 --- a/packages/chord/src/next-node/plugins/evaluate.ts +++ b/packages/chord/src/next-node/plugins/evaluate.ts @@ -9,6 +9,7 @@ import { basename, isAbsolute, join, relative, resolve } from "node:path"; import { Script } from "node:vm"; import type { Principal } from "../../next/data/index.ts"; import type { Facet } from "../../next/host/index.ts"; +import { interop } from "../../next/surfaces/registered.ts"; import type { BuiltFacet } from "./build.ts"; import type { Plugin } from "./discover.ts"; import { registerScript } from "./stack.ts"; @@ -145,22 +146,3 @@ function fileOf(plugin: Plugin, path: string): string { } return file; } - -/** ESM namespaces wrapped for a CommonJS script, by namespace: one wrapper per namespace, held as long as it is. */ -const wrapped = new WeakMap(); - -/** - * A host package as a CommonJS script imports it: an ESM namespace wrapped once with `__esModule` (so esbuild's interop - * reads its `default` for a default import, its names for named ones), any other value as it is. - */ -function interop(value: unknown): unknown { - if (typeof value !== "object" || value === null || Object.prototype.toString.call(value) !== "[object Module]") { - return value; - } - let made = wrapped.get(value); - if (made === undefined) { - made = { ...value, __esModule: true }; - wrapped.set(value, made); - } - return made; -} diff --git a/packages/chord/src/next-node/plugins/runtime.ts b/packages/chord/src/next-node/plugins/runtime.ts index 85b8c3a3a..8b1751172 100644 --- a/packages/chord/src/next-node/plugins/runtime.ts +++ b/packages/chord/src/next-node/plugins/runtime.ts @@ -7,16 +7,9 @@ import { setFlagsFromString } from "node:v8"; import type { Context } from "../../context/index.ts"; -import { isChordError, type Principal } from "../../next/data/index.ts"; -import type { - CommittedReload, - Facet, - Host, - LoadReport, - PluginFacet, - PluginFailure, - ReloadReport, -} from "../../next/host/index.ts"; +import type { Principal } from "../../next/data/index.ts"; +import type { Host, LoadReport, PluginFacet, PluginFailure, ReloadReport } from "../../next/host/index.ts"; +import { LoadedFacets, type Prepared, type SetEntry } from "../../next/surfaces/loaded.ts"; import type { BuiltFacet } from "./build.ts"; import type { DiscoveryProblem, Plugin } from "./discover.ts"; import { evaluateFacet, type HostPackages } from "./evaluate.ts"; @@ -103,57 +96,19 @@ export function createPluginRuntime(): PluginRuntime { /** Whether this process turned V8's compilation cache off already. */ let cacheOff = false; -/** A plugin's facet as a host loaded it, with its script's digest. */ -type LoadedFacet = { readonly facet: Facet; readonly digest: string }; - -/** - * A host's plugins as the runtime loaded them: its core facets, its modules, each plugin's loaded facet, and its - * reloads, one after another (each computes its complete set once the previous one adopted its result). - */ -type HostPlugins = { - readonly core: readonly PluginFacet[]; - readonly modules: HostPackages; - readonly loaded: Map; - reloading: Promise; -}; - -/** A plugin's facet passed to a host, or why it was not: a build or evaluation failure. */ -type Prepared = - | { readonly kind: "facet"; readonly facet: Facet; readonly digest: string } - | { readonly kind: "failed"; readonly error: unknown }; - class Runtime implements PluginRuntime { - /** Each loaded host's plugins, held as long as the host is. */ - readonly #hosts = new WeakMap(); + /** Each loaded host's plugins and modules, held as long as the host is. */ + readonly #hosts = new WeakMap(); async load

    (host: Host

    , options: RuntimeLoad

    , context: Context): Promise { - const { modules } = options; - const prepared = options.scripts.map( - ({ plugin, built }) => [plugin.name, prepare(plugin, built, modules)] as const, - ); - const failed: PluginFailure[] = []; - const facets: PluginFacet

    [] = [...options.core]; - for (const [plugin, each] of prepared) { - if (each.kind === "failed") failed.push({ plugin, error: each.error }); - // Evaluated for this host's principal type: the facets are the app's (`Facet

    ` is invariant in `P`). - else facets.push({ name: plugin, facet: each.facet as unknown as Facet

    }); - } - const report = await host.load(facets, context); - const left = new Set(report.failed.map(({ plugin }) => plugin)); - const loaded = new Map(); - for (const [plugin, each] of prepared) { - if (each.kind === "facet" && !left.has(plugin)) loaded.set(plugin, each); - } - // The host's own facets, for its later reloads. - const core = options.core as unknown as readonly PluginFacet[]; - this.#hosts.set(host, { core, modules, loaded, reloading: Promise.resolve() }); - const all = [...failed, ...report.failed]; - const required = new Set( - options.scripts.filter(({ plugin }) => plugin.required).map(({ plugin }) => plugin.name), - ); - const fatal = all.filter(({ plugin }) => required.has(plugin)); - if (fatal.length > 0) throw new PluginError(fatal, all); - return { failed: all }; + const { modules, scripts } = options; + const entries = scripts.map((script) => entryOf(script, modules)); + const { loaded, report } = await LoadedFacets.load(host, options.core, entries, context); + this.#hosts.set(host, { loaded, modules }); + const required = new Set(scripts.filter(({ plugin }) => plugin.required).map(({ plugin }) => plugin.name)); + const fatal = report.failed.filter(({ plugin }) => required.has(plugin)); + if (fatal.length > 0) throw new PluginError(fatal, report.failed); + return report; } reload

    ( @@ -163,60 +118,18 @@ class Runtime implements PluginRuntime { ): Promise { const state = this.#hosts.get(host); if (state === undefined) throw new Error("plugins.reload: the host was not loaded by this runtime"); - // One after another per host: a reload computing its set from a loaded map an earlier one is about to change - // would pass that plugin's old facet and roll it back. - const reload = state.reloading.then(() => this.#reload(host, state, scripts, context)); - // The tail settles to nothing: a report kept as its value would keep a rejected facet's closure (its error). - state.reloading = reload.then( - () => {}, - () => {}, - ); - return reload; + const entries = scripts.map((script) => entryOf(script, state.modules)); + return state.loaded.reload(host, entries, context); } +} - /** One reload of `host` with the complete set `scripts`. */ - async #reload

    ( - host: Host

    , - state: HostPlugins, - scripts: readonly PluginScript[], - context: Context, - ): Promise { - const failed: PluginFailure[] = []; - const next = new Map(); - for (const { plugin, built } of scripts) { - const { name } = plugin; - const loaded = state.loaded.get(name); - if (built.kind === "script" && built.digest === loaded?.digest) { - next.set(name, loaded); - continue; - } - const prepared = prepare(plugin, built, state.modules); - if (prepared.kind === "facet") { - next.set(name, prepared); - continue; - } - failed.push({ plugin: name, error: prepared.error }); - if (loaded !== undefined) next.set(name, loaded); - } - const facets = [...state.core, ...[...next].map(([name, { facet }]) => ({ name, facet }))]; - // The host takes facets of its own principal type: the runtime loaded them for it. The plugins whose build or - // evaluation failed are the host's to report, and its closure rule's to consider. - let report: ReloadReport; - try { - report = await host.reload(facets as unknown as readonly PluginFacet

    [], context, { failed }); - } catch (error) { - // Aborted after its commit: the commit stands, and is what the host runs now. - const committed = committedOf(error); - if (committed !== undefined) adopt(state, next, committed.failed); - throw error; - } - adopt( - state, - next, - report.failed.map(({ plugin }) => plugin), - ); - return report; - } +/** `script` as one plugin of a complete set: its digest, and its evaluation with `modules` when it is new. */ +function entryOf({ plugin, built }: PluginScript, modules: HostPackages): SetEntry { + return { + name: plugin.name, + digest: built.kind === "script" ? built.digest : undefined, + prepare: () => prepare(plugin, built, modules), + }; } /** `built` evaluated for `plugin` with `modules`, or why not: its build failed, or its evaluation threw. */ @@ -232,25 +145,3 @@ function prepare(plugin: Plugin, built: BuiltFacet, modules: HostPackages): Prep return { kind: "failed", error }; } } - -/** - * Adopts as `state`'s loaded facets the set `next` a reload committed: a plugin the host kept old (`failed`) stays loaded - * as it was, an added one it left out is not loaded. - */ -function adopt(state: HostPlugins, next: Map, failed: readonly string[]): void { - for (const plugin of failed) { - const loaded = state.loaded.get(plugin); - if (loaded === undefined) next.delete(plugin); - else next.set(plugin, loaded); - } - state.loaded.clear(); - for (const [name, each] of next) state.loaded.set(name, each); -} - -/** What the commit of a reload that rejected applied: an `aborted` error after its commit says (`details.committed`). */ -function committedOf(error: unknown): CommittedReload | undefined { - if (!isChordError(error, "aborted")) return undefined; - const details = error.details as { readonly committed?: CommittedReload } | null | undefined; - // The kernel's own error: its `committed` is a `CommittedReload`. - return typeof details === "object" && details !== null ? details.committed : undefined; -} diff --git a/packages/chord/src/next-node/surfaces/README.md b/packages/chord/src/next-node/surfaces/README.md new file mode 100644 index 000000000..3859e50fa --- /dev/null +++ b/packages/chord/src/next-node/surfaces/README.md @@ -0,0 +1,40 @@ +# surfaces (server side) + +The web and terminal surfaces' server side (slice 11 §1, §4; decisions "Slice 11, P3" and "P5"): +`@earendil-works/chord/next/web/server` once v2 has its entry point. The app kit (`startServer({ web, tui })`) creates +them; by hand: + +```ts +const web = await webSurface(server, { plugins, entry: "./src/web/main.ts", page: (request, assets) => shell(request.nonce, assets), paths: ["/"], public: false }, ctx); +const tui = await tuiSurface(server, { plugins, entry: "./src/tui/main.ts" }, ctx); +// web.facet, tui.facet among the server's core facets; web.step("web"), tui.step("tui") after the server's in a rollout +``` + +- **Base bundles** (`bundle.ts`): built at start by esbuild from the entry (its imports inlined once: the app's API for + the kind, Chord, React), named by the hash of all its outputs. A page's: an IIFE `base..js`, its CSS + `base..css`, assets under `assets/`. A terminal's: one CommonJS script the sandbox evaluates from memory with its own + `require` (the allowed builtins only). A broken base fails the start. +- **Published scripts** (`published.ts`, `current.ts`): the kind's current set (`plugins.scripts(kind)` at start, + `plugins.current(kind)` at each rollout step) as registered scripts `plugin/..js`, their + CSS beside them (each script with its inline source map), plugin assets under `assets/` (the kit's + `publicPath`), and the combined file `plugins..js` (without source maps: a browser reads one per file); + a plugin whose new build failed keeps its last script, CSS and assets (reported, stays dirty); the files of the last + four sets stay served. Terminal plugins have no assets (a Node build refuses them). The set goes to a local object (`chord.web.current`, a writable) the scripts object + derives from; a rollout step publishes without waiting for clients. +- **Files** (`served.ts`): `Cache-Control: private, max-age=31536000, immutable` (the baseline's `no-store` overridden + for these routes only). +- **A page's load:** the page, two scripts (the combined plugins file, the base bundle) and one stylesheet per plugin + that has CSS (beside the base bundle's own). +- **Web routes** (`web.ts`): `GET /_chord/web/:file`, `plugin/:file`, `assets/:file` (signed in unless `public`), `GET + /_chord/web/base` (public, `no-store`: an outdated page asks it), the page at `paths` (`page: true`, signed in unless + `public`). CSP additions: `style-src 'self'`, `img-src 'self' data:`, `font-src 'self'`. Scripts only from `/_chord/web/` (the HTTP component's `scripts`: `script-src` is that path of the request's origin + and the nonce, so no other same-origin response runs as a script). `PageAssets` (`bootstrap`, + `script`, `plugins`, `styles`) and `pageScripts(assets, nonce)` for the app's template. +- **Terminal routes** (`tui.ts`): `GET /_chord/tui/start` (`TuiStart`: base hash, script, combined; signed in), `GET + /_chord/tui/base` (public), the files (signed in). + +Examples: `56-web` (a page, two web plugins, an edit applied in place), `57-tui` (a sandboxed terminal client), +`58-themes` (a theme plugin restyling a page and a terminal live), `59-chat` (a chat: one worker per channel, uploads +through the kit's `Files`, a page and a terminal live in one room, pick-a-name login). + +Tests: `test/next-node/surfaces/`. diff --git a/packages/chord/src/next-node/surfaces/bundle.ts b/packages/chord/src/next-node/surfaces/bundle.ts new file mode 100644 index 000000000..c10a56387 --- /dev/null +++ b/packages/chord/src/next-node/surfaces/bundle.ts @@ -0,0 +1,71 @@ +// Base bundles (slice 11 §1, decision "Slice 11, P3"): a client kind's base code built on the server at start by +// esbuild from the app's entry (its start code, its API with the kind's module, Chord, React: inlined once), named by +// the hash of everything it is made of, served forever. A page's is a browser script with its stylesheet and assets; +// a terminal's one CommonJS script the sandbox evaluates with its own `require` (its allowed builtins only). + +import { createHash } from "node:crypto"; +import { basename, extname } from "node:path"; +import { build } from "esbuild"; +import { CSS, JAVASCRIPT, mimeOf, type ServedFile } from "./served.ts"; + +/** + * A built base bundle: its hash, its script's and stylesheet's paths (under the surface's prefix), and every file by + * path. + */ +export type BaseBundle = { + readonly hash: string; + readonly script: string; + readonly style: string | undefined; + readonly files: ReadonlyMap; +}; + +/** Where a base bundle runs: a page (a script, its CSS and assets), or a terminal's sandbox (one CommonJS script). */ +export type BasePlatform = "browser" | "sandbox"; + +/** + * Builds the base bundle of `entry` for `platform`, its files under `prefix` (`/_chord/web/`): the script + * `base..js`, the stylesheet `base..css` when it imports CSS, assets under `assets/` by their own hashes. + * Throws esbuild's failure: the server does not start with a broken base. + */ +export async function buildBase(entry: string, platform: BasePlatform, prefix: string): Promise { + const browser = platform === "browser"; + const result = await build({ + entryPoints: [entry], + outdir: "/", + entryNames: "base", + assetNames: "assets/[name]-[hash]", + publicPath: prefix, + write: false, + bundle: true, + format: browser ? "iife" : "cjs", + platform: browser ? "browser" : "node", + target: browser ? "es2022" : "node22", + minify: true, + legalComments: "none", + // Ignore any tsconfig.json: TypeScript is stripped, JSX compiled with the automatic React runtime. + tsconfigRaw: "{}", + jsx: "automatic", + define: { "process.env.NODE_ENV": '"production"' }, + logLevel: "silent", + }); + const hash = createHash("sha256"); + for (const output of [...result.outputFiles].sort((a, b) => (a.path < b.path ? -1 : 1))) { + hash.update(output.path).update(output.contents); + } + const digest = hash.digest("hex").slice(0, 16); + const files = new Map(); + let script = ""; + let style: string | undefined; + for (const output of result.outputFiles) { + const extension = extname(output.path); + if (basename(output.path) === `base${extension}`) { + const path = `${prefix}base.${digest}${extension}`; + if (extension === ".js") script = path; + else style = path; + files.set(path, { mime: extension === ".js" ? JAVASCRIPT : CSS, bytes: output.contents }); + } else { + files.set(`${prefix}assets/${basename(output.path)}`, { mime: mimeOf(output.path), bytes: output.contents }); + } + } + return { hash: digest, script, style, files }; +} diff --git a/packages/chord/src/next-node/surfaces/current.ts b/packages/chord/src/next-node/surfaces/current.ts new file mode 100644 index 000000000..0814c698b --- /dev/null +++ b/packages/chord/src/next-node/surfaces/current.ts @@ -0,0 +1,70 @@ +// The scripts objects a surface serves (slice 11 §1, decision "Slice 11, P5"): on the server, one local object per +// surface holds the published set (a writable, set by each publish), and the surface's scripts object (`WebScripts`, +// `TuiScripts`), made per principal, shows it. A rollout publishes without waiting for clients: each client's hold gets +// one frame, and a reconnect's snapshot brings the set it missed. + +import type { Context } from "../../context/index.ts"; +import { local, method } from "../../next/contracts/index.ts"; +import type { Principal } from "../../next/data/index.ts"; +import { defineFacet, type Facet, type Host } from "../../next/host/index.ts"; +import { ScriptSet, type ScriptsContract, TuiScripts, WebScripts } from "../../next/surfaces/index.ts"; + +/** A surface: the web (pages) or the terminal. */ +export type SurfaceName = "web" | "tui"; + +/** The local object holding a surface's published set, by surface. */ +const CURRENT = { + web: currentContract("chord.web.current"), + tui: currentContract("chord.tui.current"), +}; + +/** The scripts object each surface serves. */ +const SCRIPTS: Readonly> = { web: WebScripts, tui: TuiScripts }; + +/** The contract of a surface's published set: its `set` and `publish`. */ +function currentContract(name: N) { + return local(name, { version: 1, params: {}, members: { set: ScriptSet, publish: method(ScriptSet) } }); +} + +/** + * The surface's facet serving its published set (starting at `published.set`, the set taken before the load) and its + * scripts object for every principal that may hold it. + */ +export function scriptsFacet

    ( + surface: SurfaceName, + published: { readonly set: ScriptSet }, +): Facet

    { + const Current = CURRENT[surface]; + return defineFacet

    ((host) => { + host.serve(Current, (_params, self) => { + // The latest set: an attempt after a failure starts at what was published last. + const set = self.writable(published.set); + return { + set: self.derive([set], ([value]) => value), + publish: async (next) => { + set.set(next); + return null; + }, + }; + }); + host.serve(SCRIPTS[surface], (_params, _principal, self) => { + const current = self.get(Current, {}); + return { variant: "main", members: { set: self.derive([current.set], ([value]) => value) } }; + }); + }); +} + +/** Publishes `set` as `surface`'s current set on `host` (loaded with `scriptsFacet`). */ +export async function publishTo

    ( + host: Host

    , + surface: SurfaceName, + set: ScriptSet, + context: Context, +): Promise { + const current = await host.get(CURRENT[surface], {}, context); + try { + await current.publish(set, context); + } finally { + current.release(); + } +} diff --git a/packages/chord/src/next-node/surfaces/index.ts b/packages/chord/src/next-node/surfaces/index.ts new file mode 100644 index 000000000..186bce171 --- /dev/null +++ b/packages/chord/src/next-node/surfaces/index.ts @@ -0,0 +1,14 @@ +// Chord v2's client surfaces, the server side (`@earendil-works/chord/next/web/server` once v2 has its entry point; +// slice 11 §1, §4): the web surface (base bundle, plugin scripts, `WebScripts`, the page) and the terminal surface. + +export type { BaseBundle } from "./bundle.ts"; +export type { ServedFile } from "./served.ts"; +export { TUI_PREFIX, type TuiStart, type TuiSurface, type TuiSurfaceOptions, tuiSurface } from "./tui.ts"; +export { + type PageAssets, + pageScripts, + WEB_PREFIX, + type WebSurface, + type WebSurfaceOptions, + webSurface, +} from "./web.ts"; diff --git a/packages/chord/src/next-node/surfaces/published.ts b/packages/chord/src/next-node/surfaces/published.ts new file mode 100644 index 000000000..c868e8b40 --- /dev/null +++ b/packages/chord/src/next-node/surfaces/published.ts @@ -0,0 +1,159 @@ +// A client kind's published scripts (slice 11 §1, decisions "Slice 11, P3" and "P5"): what the server serves of the +// kind's plugins, as files under the surface's prefix, and the set its scripts object shows. Each plugin's facet is a +// script that registers itself (`plugin/..js`, its stylesheet beside it), the whole set one +// combined file named by its hash (`plugins..js`), assets under `assets/`; all named by their +// content, served forever. A rollout shipping the kind publishes a new set (a plugin whose new build failed keeps its +// last script) without waiting for clients; the files of the last few sets stay served for clients that still load +// them. + +import { createHash } from "node:crypto"; +import type { Context } from "../../context/index.ts"; +import type { Principal } from "../../next/data/index.ts"; +import type { Host, PluginFailure } from "../../next/host/index.ts"; +import { registeredSource, type ScriptSet } from "../../next/surfaces/index.ts"; +import type { Built, PluginScript, Plugins, RolloutStep } from "../plugins/index.ts"; +import { publishTo, type SurfaceName } from "./current.ts"; +import { CSS, JAVASCRIPT, mimeOf, type ServedFile } from "./served.ts"; + +/** How many sets' files stay served: the current one and the few before it, for clients loading them meanwhile. */ +const KEPT_SETS = 4; + +/** One plugin's published script: its digest, its registered text, its stylesheet. */ +/** + * One plugin's published script: its digest, its registered text, its stylesheet and assets (kept with the script: a + * plugin whose next build fails keeps all three). + */ +type Published = { + readonly digest: string; + readonly text: string; + readonly css: string | undefined; + readonly assets: readonly { readonly name: string; readonly bytes: Uint8Array }[]; +}; + +/** A script's inline source map comment. */ +const SOURCE_MAP = /\n\/\/# sourceMappingURL=data:[^\n]*/g; + +/** A client kind's published scripts. */ +export class PublishedScripts

    { + readonly #surface: SurfaceName; + readonly #kind: string; + readonly #prefix: string; + readonly #base: string; + readonly #plugins: Plugins; + readonly #host: Host

    ; + /** The published script of each plugin now. */ + readonly #current = new Map(); + /** The files of the last `KEPT_SETS` sets, newest last, each by path. */ + readonly #sets: ReadonlyMap[] = []; + #set: ScriptSet | undefined; + + constructor(surface: SurfaceName, kind: string, prefix: string, base: string, plugins: Plugins, host: Host

    ) { + this.#surface = surface; + this.#kind = kind; + this.#prefix = prefix; + this.#base = base; + this.#plugins = plugins; + this.#host = host; + } + + /** The set published now (after `start`). */ + get set(): ScriptSet { + return this.#set!; + } + + /** + * Takes the kind's current set as the first one (its plugins' facets of the kind built on first use), before the + * server's load (the scripts object starts with it). The plugins whose build failed. + */ + async start(context: Context): Promise { + return this.#publish(await this.#plugins.scripts(this.#kind, context)); + } + + /** A published file by its path, from the last few sets. */ + file(path: string): ServedFile | undefined { + for (let index = this.#sets.length - 1; index >= 0; index--) { + const found = this.#sets[index]!.get(path); + if (found !== undefined) return found; + } + return undefined; + } + + /** + * The rollout step publishing the kind's current set as the rollout applied its build: one "host" named `name`, its + * report the plugins whose new build failed (they keep their last script, and stay dirty). Clients are not waited for. + */ + step(name: string): RolloutStep { + return { + name, + kinds: [this.#kind], + reload: async (_built: Built, context: Context) => { + const before = new Map([...this.#current].map(([plugin, { digest }]) => [plugin, digest])); + const failed = this.#publish(this.#plugins.current(this.#kind) ?? []); + await publishTo(this.#host, this.#surface, this.set, context); + const after = [...this.#current]; + const plugins = { + added: after.filter(([plugin]) => !before.has(plugin)).map(([plugin]) => plugin), + changed: after + .filter(([plugin, { digest }]) => before.has(plugin) && before.get(plugin) !== digest) + .map(([plugin]) => plugin), + removed: [...before.keys()].filter((plugin) => !this.#current.has(plugin)), + }; + return { hosts: [{ host: name, report: { plugins, failed, objects: [] } }] }; + }, + }; + } + + /** + * Takes `scripts` (the kind's complete set) as the current one: a failed build keeps its plugin's last script; its + * files kept with the last few sets'. The plugins whose build failed. + */ + #publish(scripts: readonly PluginScript[]): readonly PluginFailure[] { + const failed: PluginFailure[] = []; + const next = new Map(); + for (const { plugin, built } of scripts) { + if (built.kind === "script") { + const text = registeredSource(plugin.name, built.digest, built.source); + next.set(plugin.name, { digest: built.digest, text, css: built.css?.text, assets: built.assets }); + continue; + } + failed.push({ plugin: plugin.name, error: new Error(built.errors.join("\n")) }); + const kept = this.#current.get(plugin.name); + if (kept !== undefined) next.set(plugin.name, kept); + } + this.#current.clear(); + for (const [name, each] of [...next].sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0))) + this.#current.set(name, each); + const { set, files } = this.#files(); + this.#sets.push(files); + if (this.#sets.length > KEPT_SETS) this.#sets.shift(); + this.#set = set; + return failed; + } + + /** + * The set of the current scripts and its files: each script (its source map inline), stylesheet and asset, and the + * combined file (without source maps: a browser reads only one per file, so they would be bytes for nothing). + */ + #files(): { set: ScriptSet; files: Map } { + const files = new Map(); + const entries: ScriptSet["plugins"][number][] = []; + const combined: string[] = []; + for (const [name, { digest, text, css, assets }] of this.#current) { + // A plugin name (`@scope/name`) as one path segment, its slash a tilde: the set's URLs are looked up as they + // are, never parsed back. + const stem = `${this.#prefix}plugin/${name.replace("/", "~")}.${digest}`; + files.set(`${stem}.js`, { mime: JAVASCRIPT, bytes: new TextEncoder().encode(text) }); + if (css !== undefined) files.set(`${stem}.css`, { mime: CSS, bytes: new TextEncoder().encode(css) }); + entries.push({ name, digest, url: `${stem}.js`, ...(css === undefined ? {} : { css: `${stem}.css` }) }); + combined.push(text.replace(SOURCE_MAP, "")); + for (const asset of assets) { + files.set(`${this.#prefix}assets/${asset.name}`, { mime: mimeOf(asset.name), bytes: asset.bytes }); + } + } + const body = combined.join(""); + const hash = createHash("sha256").update(body).digest("hex").slice(0, 16); + const url = `${this.#prefix}plugins.${hash}.js`; + files.set(url, { mime: JAVASCRIPT, bytes: new TextEncoder().encode(body) }); + return { set: { base: this.#base, combined: url, plugins: entries }, files }; + } +} diff --git a/packages/chord/src/next-node/surfaces/served.ts b/packages/chord/src/next-node/surfaces/served.ts new file mode 100644 index 000000000..142b1b8b3 --- /dev/null +++ b/packages/chord/src/next-node/surfaces/served.ts @@ -0,0 +1,50 @@ +// The files the surfaces serve (slice 11 §1, §4): base bundles, plugin scripts, stylesheets and assets, each named by +// its content and so cached by clients forever; their types. + +import { extname } from "node:path"; +import type { HttpResponse } from "../http/index.ts"; + +/** A file the server serves: its type and bytes. */ +export type ServedFile = { readonly mime: string; readonly bytes: Uint8Array }; + +/** A served file's response, cached by the browser forever (its path names its content); 404 for none. */ +export function served(found: ServedFile | undefined): HttpResponse { + if (found === undefined) return { kind: "error", status: 404 }; + const { mime, bytes } = found; + return { + kind: "bytes", + mime, + size: bytes.length, + read: async (offset, length) => bytes.subarray(offset, offset + length), + // Overrides the baseline's `no-store` for these files only. + headers: { "Cache-Control": "private, max-age=31536000, immutable" }, + }; +} + +/** The type of a script. */ +export const JAVASCRIPT = "text/javascript; charset=utf-8"; + +/** The type of a stylesheet. */ +export const CSS = "text/css; charset=utf-8"; + +/** The type of a built asset, by extension. */ +export function mimeOf(path: string): string { + return ASSET_TYPES[extname(path).slice(1).toLowerCase()] ?? "application/octet-stream"; +} + +/** The types of the assets builds collect, by extension. */ +const ASSET_TYPES: Readonly> = { + svg: "image/svg+xml", + png: "image/png", + jpg: "image/jpeg", + jpeg: "image/jpeg", + gif: "image/gif", + webp: "image/webp", + woff: "font/woff", + woff2: "font/woff2", + ttf: "font/ttf", + wasm: "application/wasm", + json: "application/json", + css: CSS, + js: JAVASCRIPT, +}; diff --git a/packages/chord/src/next-node/surfaces/tui.ts b/packages/chord/src/next-node/surfaces/tui.ts new file mode 100644 index 000000000..ddb309c4f --- /dev/null +++ b/packages/chord/src/next-node/surfaces/tui.ts @@ -0,0 +1,67 @@ +// The terminal surface's server side (slice 11 §4, decisions "Slice 11, P3" and "P8"): the terminal kind's base bundle +// (one ES module the sandbox evaluates) built from the app's terminal entry at start, the kind's plugin scripts +// published as registered scripts and one combined file, `TuiScripts` showing the current set, all served to the +// launcher, which signs in with a token. Routes and objects are `Http` contributions of the surface's facet. + +import type { Context } from "../../context/index.ts"; +import type { Principal } from "../../next/data/index.ts"; +import { defineFacet, type Facet, type Host } from "../../next/host/index.ts"; +import { Http, type HttpRequest, type HttpResponse } from "../http/index.ts"; +import type { Plugins, RolloutStep } from "../plugins/index.ts"; +import { buildBase } from "./bundle.ts"; +import { scriptsFacet } from "./current.ts"; +import { PublishedScripts } from "./published.ts"; +import { type ServedFile, served } from "./served.ts"; + +/** What `tuiSurface` takes. */ +export type TuiSurfaceOptions = { + /** The terminal facet kind: `tui` by default. */ + readonly kind?: string; + readonly plugins: Plugins; + /** The app's terminal start code (it calls `startTerminalRuntime`): the base bundle's entry. */ + readonly entry: string; +}; + +/** What a launcher downloads at its start (`GET /_chord/tui/start`): the base bundle and the combined plugin file. */ +export type TuiStart = { readonly base: string; readonly script: string; readonly combined: string }; + +/** The terminal surface: its facet (for the server's core facets), its rollout step, its base bundle's hash. */ +export type TuiSurface

    = { + readonly facet: Facet

    ; + step(name: string): RolloutStep; + readonly base: string; +}; + +/** Where the terminal surface's files live. */ +export const TUI_PREFIX = "/_chord/tui/"; + +/** Builds the terminal's base bundle and takes the kind's current set, then returns the surface. */ +export async function tuiSurface

    ( + server: Host

    , + options: TuiSurfaceOptions, + context: Context, +): Promise> { + const base = await buildBase(options.entry, "sandbox", TUI_PREFIX); + const published = new PublishedScripts("tui", options.kind ?? "tui", TUI_PREFIX, base.hash, options.plugins, server); + await published.start(context); + const file = (path: string): ServedFile | undefined => base.files.get(path) ?? published.file(path); + const serve = async (request: HttpRequest): Promise => served(file(request.path)); + const facet = defineFacet

    ((host, self) => { + scriptsFacet

    ("tui", published).setup(host, self); + self.contribute(Http, { + routes: { + [`GET ${TUI_PREFIX}start`]: async () => { + const start: TuiStart = { base: base.hash, script: base.script, combined: published.set.combined }; + return { kind: "json", value: start, headers: { "Cache-Control": "no-store" } }; + }, + [`GET ${TUI_PREFIX}base`]: { + public: true, + handler: async () => ({ kind: "json", value: base.hash, headers: { "Cache-Control": "no-store" } }), + }, + [`GET ${TUI_PREFIX}:file`]: serve, + [`GET ${TUI_PREFIX}plugin/:file`]: serve, + }, + }); + }); + return { facet, step: (name) => published.step(name), base: base.hash }; +} diff --git a/packages/chord/src/next-node/surfaces/web.ts b/packages/chord/src/next-node/surfaces/web.ts new file mode 100644 index 000000000..87cb8dca1 --- /dev/null +++ b/packages/chord/src/next-node/surfaces/web.ts @@ -0,0 +1,137 @@ +// The web surface's server side (slice 11 §1, decisions "Slice 11's points" and "P3" to "P5"): the base bundle built +// from the app's entry at start, the plugins' web scripts published as registered scripts and one combined file, +// `WebScripts` showing the current set, and the page through the app's own template; everything as `Http` +// contributions of the surface's facet, which the app puts among the server's core facets. The app keeps its HTML, +// CSS, components and themes. + +import type { Context } from "../../context/index.ts"; +import type { Principal } from "../../next/data/index.ts"; +import { defineFacet, type Facet, type Host } from "../../next/host/index.ts"; +import { REGISTER, type ScriptSet } from "../../next/surfaces/index.ts"; +import { Http, type HttpRequest, type HttpResponse, type HttpRouteOptions } from "../http/index.ts"; +import type { Plugins, RolloutStep } from "../plugins/index.ts"; +import { type BaseBundle, buildBase } from "./bundle.ts"; +import { scriptsFacet } from "./current.ts"; +import { PublishedScripts } from "./published.ts"; +import { type ServedFile, served } from "./served.ts"; + +/** What `webSurface` takes. */ +export type WebSurfaceOptions = { + /** The web facet kind: `web` by default. */ + readonly kind?: string; + readonly plugins: Plugins; + /** The app's web start code (it calls `startPage`): the base bundle's entry. */ + readonly entry: string; + /** The page's HTML: the app's template, given the request (its `nonce`) and the assets to include. */ + readonly page: (request: HttpRequest, assets: PageAssets) => string | Promise; + /** The paths serving the page: `["/"]` by default (patterns, `/*` for a page with its own routing). */ + readonly paths?: readonly string[]; + /** Whether the page and its scripts are served without a principal (a public app); signed in only by default. */ + readonly public?: boolean; +}; + +/** + * What a page includes, in this order: `bootstrap` as an inline script with the response's nonce, then the scripts + * `plugins` and `script` (deferred, in that order), and the stylesheets `styles` (the base bundle's first, then each + * plugin's). `pageScripts` writes the script tags. + */ +export type PageAssets = { + readonly bootstrap: string; + readonly script: string; + readonly plugins: string; + readonly styles: readonly string[]; +}; + +/** The web surface: its facet (for the server's core facets), its rollout step, its base bundle's hash. */ +export type WebSurface

    = { + readonly facet: Facet

    ; + /** The rollout step publishing the web kind's current set (after the server's step). */ + step(name: string): RolloutStep; + readonly base: string; +}; + +/** Where the web surface's files live. */ +export const WEB_PREFIX = "/_chord/web/"; + +/** + * Builds the base bundle and takes the web kind's current set (its facets built on first use), then returns the + * surface. Rejects with esbuild's failure when the base bundle does not build. + */ +export async function webSurface

    ( + server: Host

    , + options: WebSurfaceOptions, + context: Context, +): Promise> { + const base = await buildBase(options.entry, "browser", WEB_PREFIX); + const published = new PublishedScripts("web", options.kind ?? "web", WEB_PREFIX, base.hash, options.plugins, server); + await published.start(context); + const file = (path: string): ServedFile | undefined => base.files.get(path) ?? published.file(path); + const facet = defineFacet

    ((host, self) => { + scriptsFacet

    ("web", published).setup(host, self); + self.contribute(Http, { + routes: webRoutes(options, base.hash, file, () => pageAssets(base, published)), + csp: { "style-src": ["'self'"], "img-src": ["'self'", "data:"], "font-src": ["'self'"] }, + // Scripts from the surface's files only (and the page's nonce): no other same-origin response runs (Opus L8). + scripts: [WEB_PREFIX], + }); + }); + return { facet, step: (name) => published.step(name), base: base.hash }; +} + +/** The surface's routes: its files (served forever), its current base hash (public), and the page. */ +function webRoutes( + options: WebSurfaceOptions, + base: string, + file: (path: string) => ServedFile | undefined, + assets: () => PageAssets, +): Record { + const access = options.public === true ? { public: true as const } : {}; + const serve = async (request: HttpRequest): Promise => served(file(request.path)); + const routes: Record = { + [`GET ${WEB_PREFIX}:file`]: { ...access, handler: serve }, + [`GET ${WEB_PREFIX}plugin/:file`]: { ...access, handler: serve }, + [`GET ${WEB_PREFIX}assets/:file`]: { ...access, handler: serve }, + // Asked by a page before it reconnects: another base means its code is outdated. Public: an outdated page whose + // session ended still learns it should reload. + [`GET ${WEB_PREFIX}base`]: { + public: true, + handler: async () => ({ kind: "json", value: base, headers: { "Cache-Control": "no-store" } }), + }, + }; + for (const path of options.paths ?? ["/"]) { + routes[`GET ${path}`] = { + ...access, + page: true, + handler: async (request) => ({ kind: "html", body: await options.page(request, assets()) }), + }; + } + return routes; +} + +/** What the page includes now: the bootstrap, the base script, the current combined file, the stylesheets. */ +function pageAssets(base: BaseBundle, published: { readonly set: ScriptSet }): PageAssets { + const { set } = published; + const styles = [ + ...(base.style === undefined ? [] : [base.style]), + ...set.plugins.flatMap(({ css }) => (css === undefined ? [] : [css])), + ]; + return { bootstrap: bootstrap(base.hash), script: base.script, plugins: set.combined, styles }; +} + +/** + * The page's inline bootstrap: the base hash for `startPage`, and the `chordFacet` the combined file's scripts call + * before the base bundle runs (they queue; `startPage` takes them). + */ +function bootstrap(hash: string): string { + return `globalThis.chordPage={base:${JSON.stringify(hash)},queued:[]};globalThis.${REGISTER}=function(p,d,f){chordPage.queued.push([p,d,f])};`; +} + +/** The page's script tags for `assets`: the inline bootstrap with `nonce`, then the deferred scripts; the stylesheets. */ +export function pageScripts(assets: PageAssets, nonce: string): string { + return [ + ...assets.styles.map((href) => ``), + ``, + ``, + ``, + ].join("\n"); +} diff --git a/packages/chord/src/next-node/tui/README.md b/packages/chord/src/next-node/tui/README.md new file mode 100644 index 000000000..79803bf19 --- /dev/null +++ b/packages/chord/src/next-node/tui/README.md @@ -0,0 +1,75 @@ +# tui (launcher) + +The trusted side of a terminal client (slice 11 §4, decision "Slice 11, P8"; p2's `tui-client.ts`, `tui-sandbox.ts`, +`tui-bootstrap.mjs`, `tui-filter.ts`): `@earendil-works/chord/next/tui/launcher` once v2 has its entry point. Needs Node +26.5.1 or later (the Node its sandbox was checked on); `requireSandboxNode` refuses older ones with a clear error. + +```ts +const terminal = await startTerminal({ + url: "http://127.0.0.1:8788", + credentials: { kind: "file", path: "~/.acme/owner-token" }, // or { kind: "token", token }, { kind: "pairing", label, save? } + mode: "dark", colors: "truecolor", // the terminal's background and colours +}); +process.exitCode = await terminal.exited; +``` + +- **Credentials** (`credentials.ts`): the launcher keeps the token; pairing asks for a code, shows it, polls every 2 s + until the owner approves it, and may save the token (created, mode 0600). +- **The sandbox** (`sandbox.ts`): a Node child process of the launcher's binary, in two layers: + - the operating system's: stdin, stdout and stderr pipes to the launcher (no descriptor of the child is a terminal), + an empty environment (but `LD_LIBRARY_PATH` when the launcher has one: it needed it to run the same binary), a + private directory holding only the bootstrap; `--permission` allowing only reading the bootstrap's file (no other + file, no network, child processes, workers, addons, inspector or WASI), `--no-experimental-sqlite` (SQLite opens + files past the permission model), a 96 MiB heap. The bootstrap is a file, not `-e`, which puts every builtin on + `globalThis`; + - JavaScript's, a second layer: only the bootstrap evaluates downloaded code (the base bundle, at most 8 MiB, a + CommonJS script; the plugins' registered scripts through the bridge's `evaluate`), with `vm.Script` (options of no + prototype, no dynamic import callback: `import()` throws) and a `require` (and `process.getBuiltinModule`) + answering a short list of builtins that compute in memory (`SANDBOX_BUILTINS`, looked up in an object of no + prototype); `process` is a frozen plain object with `SANDBOX_PROCESS` only; the bootstrap keeps its own references + to what it uses, and Node's `process` has its listener calls as own properties (changing the shared + `EventEmitter.prototype` catches nothing: Opus E2); every prototype of Node's `process`, `Object.prototype` + included, takes no new property and keeps its own ones (a getter for a property Node reads on its process, such as + `noDeprecation`, would get the real one: Opus E7); + - messages both ways as length-prefixed binary frames (`pipe.ts`: a header as JSON, the text or bytes as body), each + at most 17 MiB; the launcher checks every message and ends a sandbox sending a malformed one or not reading; + - known gaps, each needing the JavaScript layer broken first unless noted: the child may signal the user's processes + (Node has no permission for signals; the launcher ignores SIGUSR1, which would open its inspector: Opus E8), read + the machine's details (`os`), and learn whether any path exists (`require.resolve` fails one way for a file that + exists, another for one that does not: Opus E13, a Node gap); memory outside its heap has no limit (only the child + dies). +- **The launcher** (`launcher.ts`): downloads `/_chord/tui/start`, the base bundle and the combined file with the token + (each body read up to 8 MiB, no further); gives the sandbox input (raw mode on a TTY) and resizes; writes its frames + through the output filter; opens and relays its link sockets to `/chord` (the token in the upgrade's header; at most + four at once); answers its fetches of terminal files only (`terminalFile`: the path resolved first, then + `/_chord/tui/` or `/_chord/tui/plugin/` of the server's origin); starts a new sandbox only when the server's + base bundle is another than the running one; prints the server's text (a pairing code, which must be a plain code) + through the filter; at its end resets the terminal's modes (sync, mouse, focus, paste, cursor, alternate screen, + colours, scroll region, autowrap), also when its process exits without that end; answers at most eight requests of + the sandbox at once (more are refused at once); ignores SIGUSR1 while it runs (no inspector opened by a signal). + Nothing the sandbox sends ends the launcher: a send on a socket not yet open is dropped, a socket with 16 MiB waiting + is closed, a message it fails on ends the client. +- **The output filter** (`filter.ts`): text and drawing sequences pass (cursor moves, erasing, SGR colours, the alternate + screen, bracketed paste, sync, mouse reporting); OSC (the clipboard among them), DCS and other control strings, + other modes and ESC sequences are dropped; an unterminated sequence ends with its frame; frames above 64 KiB are + dropped (the sandbox splits a large write into frames between sequences: `next-tui/frames.ts`). +- **Actions** (`actions.ts`), each confirmed on the launcher's prompt (the bottom lines, black on white after a reset + of all attributes, whatever the sandbox left; `y`): one at a time (a second request is refused at once); from the + request to the action's end the sandbox's frames are held (1 MiB at most); while the prompt is open its keys go to + it, and those in its first 300 ms are ignored (typed for the sandbox); a synchronized update the sandbox began is + closed and the prompt drawn last (again after a resize); its lines are wrapped to the terminal's width, never cut (a + prompt taller than the terminal is refused). Afterwards the held frames are written and the sandbox asked to redraw. + `copy` shows the text (five rows, then how many more characters) and refuses control and bidirectional characters (a + newline and a tab aside), then the launcher writes OSC 52. `open` takes http and https URLs without user info, names + the host first, and opens with `open`, `xdg-open` or `rundll32 url.dll,FileProtocolHandler` (as the coding agent), + never through a shell. `attach` refuses a path with control, format or unusual whitespace characters (on Windows + also UNC, device and `//host` paths and reserved device names), shows the path resolved by its text (its name, then + its directory) without looking at the machine, after a yes resolves its links and asks again when they lead + elsewhere, then opens it without blocking (no wait on a FIFO) and reads it only when it is the file confirmed (its + device and inode as when the last prompt was shown: no parent directory swapped for a link meanwhile) and a regular + file of at most 16 MiB, at the size its open handle had; one refusal for anything else. While a prompt is open, + Ctrl-C ends the client. +- **The way out:** three Ctrl-C within a second always end the client (exit code 130, the terminal reset), whatever the + sandbox does; each still reaches the sandbox, so an app that handles Ctrl-C itself works as before. + +Tests: `test/next-node/tui/` (the launcher and sandbox ones under `CHORD_TUI_NODE` or `node26`; skipped without). diff --git a/packages/chord/src/next-node/tui/actions.ts b/packages/chord/src/next-node/tui/actions.ts new file mode 100644 index 000000000..491a8f10d --- /dev/null +++ b/packages/chord/src/next-node/tui/actions.ts @@ -0,0 +1,212 @@ +// The launcher's actions (slice 11 §4; fix rounds 1 to 4: Fable's H1, H2, M1, L4, L6 and Opus's M1, M3, L2, L3, E5, +// E6, E9, E12, R1): what the sandbox may ask of the machine, each confirmed by the user on the launcher's own prompt +// before it touches it. Every line the prompt shows is wrapped to the terminal's width, never cut. Copying writes the +// text to the terminal's clipboard (OSC 52, which the output filter refuses to sandboxed code) and shows it first (up +// to five rows, then how many more characters); text with control characters (a newline and a tab aside) or +// bidirectional controls is refused. Opening a URL (http or https, no user info) names its host first and runs the +// platform's opener without a shell (as the coding agent's `open-browser.ts`). Attaching refuses a path with control, +// format or unusual whitespace characters (and on Windows UNC and device paths), shows the path resolved by its text +// (its name on its own line) without looking at the machine, after the user's yes resolves its links and asks again +// when they lead elsewhere, and reads the very file confirmed (its device and inode), a regular file of at most 16 MiB, +// exactly at the size its open handle has. + +import { spawn } from "node:child_process"; +import { constants } from "node:fs"; +import { open, realpath, stat } from "node:fs/promises"; +import { homedir } from "node:os"; +import { posix, win32 } from "node:path"; +import type { BridgeRequest } from "../../next-tui/bridge.ts"; + +/** + * The launcher's prompt on the real terminal, `columns` wide: `ask` shows `lines` (the question last) and resolves + * with the user's answer, yes or no; `write` writes to the terminal past the output filter (the launcher's own + * sequences). + */ +export type PromptTerminal = { + readonly columns: number; + ask(lines: readonly string[]): Promise; + write(data: string): void; +}; + +/** The prompt's rows for `lines` at `columns` wide: control characters removed, each line wrapped, never cut. */ +export function promptLines(lines: readonly string[], columns: number): string[] { + return lines.flatMap((line) => wrap(line, promptWidth(columns))); +} + +/** The text width of a prompt row (one column of padding each side). */ +function promptWidth(columns: number): number { + return Math.max(10, columns - 2); +} + +/** + * `line` with control characters removed, in rows of at most `width` columns, counting every character but printable + * ASCII as two (wide characters never push text past the edge, where the next row would cover it). + */ +export function wrap(line: string, width: number): string[] { + const rows: string[] = []; + let row = ""; + let used = 0; + for (const character of visible(line)) { + const cells = character >= " " && character <= "~" ? 1 : 2; + if (used + cells > width) { + rows.push(row); + row = ""; + used = 0; + } + row += character; + used += cells; + } + rows.push(row); + return rows; +} + +/** How many rows of a copied text the prompt shows before "and N more characters". */ +const SHOWN_ROWS = 5; + +/** + * Does the sandbox's `request` (`copy`, `open`) once the user confirmed it: whether it was done. Throws (the request + * refused) for text it will not copy and URLs it will not open. `opener` opens a URL (the platform's by default). + */ +export async function act( + request: Extract, + terminal: PromptTerminal, + opener: (url: string) => Promise = platformOpener, +): Promise { + if (request.kind === "copy") { + const { text } = request; + // Control characters (but newlines and tabs) and bidirectional controls could hide what a paste runs. + if (/[\u0000-\u0008\u000b-\u001f\u007f-\u009f\u202a-\u202e\u2066-\u2069]/.test(text)) { + throw new Error("copy: the text holds control characters"); + } + const lines = text.split("\n"); + // Wrapped, never cut: a long line's end is where a pasted command would hide (Opus E6). + const rows = lines.flatMap((line) => wrap(line, promptWidth(terminal.columns) - 2)); + const hidden = rows.slice(SHOWN_ROWS).reduce((sum, row) => sum + row.length, 0); + const question = `Copy these ${text.length} characters (${lines.length} line${lines.length === 1 ? "" : "s"})? [y/N]`; + const shown = [ + ...rows.slice(0, SHOWN_ROWS).map((row) => ` ${row}`), + ...(hidden > 0 ? [` … and ${hidden} more characters`] : []), + ]; + if (!(await terminal.ask([...shown, question]))) return false; + terminal.write(`\x1b]52;c;${Buffer.from(text).toString("base64")}\x07`); + return true; + } + const url = openable(request.url); + if (url === undefined) throw new Error("open: only http and https URLs without user info open"); + if (!(await terminal.ask([` ${url.href}`, `Open ${url.host} in the browser? [y/N]`]))) return false; + await opener(url.href); + return true; +} + +/** The largest file the sandbox may be given: 16 MiB. */ +export const MAX_ATTACHMENT = 16 * 1024 * 1024; + +/** + * Reads the file at `path` (as the sandbox gave it; `~/` the home directory) for the sandbox once the user confirmed + * it: its name and bytes, or `null` when declined. Refused at once: a path with control, format or unusual whitespace + * characters, and on Windows a UNC, device or `//host` path or a reserved device name. Nothing on the machine is looked + * at before the user's yes (Opus E9: on Windows a lookup of `\\host\share` alone sends the user's credentials): the + * prompt shows the path resolved by its text (`~`, `.` and `..`); after the yes its links are resolved, and a path + * that leads elsewhere is asked about again, showing where; what is opened must be the file confirmed (its device and + * inode as when the last prompt was shown). Then it is opened without blocking (a FIFO waits for no + * writer: Opus E12) and read only when it is a regular file of at most `MAX_ATTACHMENT` bytes; one refusal for anything + * else (no hint what is there). + */ +export async function attach( + path: string, + terminal: PromptTerminal, + platform: NodeJS.Platform = process.platform, +): Promise<{ readonly name: string; readonly bytes: Uint8Array } | null> { + if (!attachable(path, platform)) throw cannotAttach(); + const paths = platform === "win32" ? win32 : posix; + const home = /^~(?:$|[\\/])/.test(path) ? paths.join(homedir(), path.slice(1)) : path; + const named = paths.resolve(home); + if (!(await terminal.ask(fileLines(named, paths, "Attach this file? [y/N]")))) return null; + const file = await realpath(named).catch(() => undefined); + if (file === undefined) throw cannotAttach(); + // The file the user confirms, by identity: what is read must be it (Opus R1: a parent directory swapped for a link + // after the yes would lead elsewhere, past `O_NOFOLLOW`, which guards the last step only). + const shown = await stat(file, { bigint: true }).catch(() => undefined); + if (shown === undefined) throw cannotAttach(); + if (file !== named && !(await terminal.ask(fileLines(file, paths, "It links here. Attach this file? [y/N]")))) { + return null; + } + // No waiting on a FIFO or device, and no link swapped in at the last step since the user saw the path. + const flags = constants.O_RDONLY | (constants.O_NONBLOCK ?? 0) | (constants.O_NOFOLLOW ?? 0); + const handle = await open(file, flags).catch(() => undefined); + if (handle === undefined) throw cannotAttach(); + try { + const info = await handle.stat({ bigint: true }); + if (info.dev !== shown.dev || info.ino !== shown.ino) throw cannotAttach(); + if (!info.isFile() || info.size > BigInt(MAX_ATTACHMENT)) throw cannotAttach(); + // Exactly its size now: a file growing meanwhile is read up to what was checked. + const bytes = new Uint8Array(Number(info.size)); + let read = 0; + while (read < bytes.length) { + const { bytesRead } = await handle.read(bytes, read, bytes.length - read, read); + if (bytesRead === 0) break; + read += bytesRead; + } + return { name: paths.basename(file), bytes: bytes.subarray(0, read) }; + } finally { + await handle.close(); + } +} + +/** + * Whether `path` may be shown and, once confirmed, opened on `platform`: no character that hides or moves what the + * prompt shows (controls, format characters such as bidirectional and zero-width ones, whitespace but single spaces + * between words: Opus E5), and on Windows no UNC, device or `//host` path and no reserved device name (Opus E9). + */ +function attachable(path: string, platform: NodeJS.Platform): boolean { + if (/[\p{Cc}\p{Cf}\p{Zl}\p{Zp}]|[^\S ]| {2}|^ | $/u.test(path)) return false; + if (platform !== "win32") return true; + if (/^[\\/]{2}/.test(path)) return false; + return !path + .split(/[\\/]/) + .some((segment) => /^(?:con|prn|aux|nul|com\d|lpt\d|conin\$|conout\$)(?:\..*)?$/i.test(segment)); +} + +/** The prompt's lines for the file at `file`: its name, its directory, then `question`. */ +function fileLines(file: string, paths: typeof posix, question: string): string[] { + return [`File: ${paths.basename(file)}`, `In: ${paths.dirname(file)}`, question]; +} + +function cannotAttach(): Error { + return new Error("attach: that file cannot be attached"); +} + +/** `text` with control and format characters (bidirectional, zero width) removed: sandboxed text in the prompt. */ +function visible(text: string): string { + return text.replace(/[\p{Cc}\p{Cf}]/gu, ""); +} + +/** `url` when it is an http or https URL without user info; else `undefined`. */ +function openable(url: string): URL | undefined { + try { + const value = new URL(url); + const web = value.protocol === "http:" || value.protocol === "https:"; + return web && value.username === "" && value.password === "" ? value : undefined; + } catch { + return undefined; + } +} + +/** + * The command and arguments opening `url` on `platform`, as the coding agent opens a browser (`open-browser.ts`): never + * a shell (on Windows not `cmd /c start`, whose parsing runs what follows a `&`): `rundll32 + * url.dll,FileProtocolHandler ` on Windows, `open` on macOS, `xdg-open` elsewhere, the URL one argument. + */ +export function openerCommand(url: string, platform: NodeJS.Platform = process.platform): [string, string[]] { + if (platform === "win32") return ["rundll32", ["url.dll,FileProtocolHandler", url]]; + return [platform === "darwin" ? "open" : "xdg-open", [url]]; +} + +/** Opens `url` with the platform's opener (`openerCommand`), detached, best effort: a missing opener is no crash. */ +async function platformOpener(url: string): Promise { + const [command, args] = openerCommand(url); + // The root as its directory: no opener needs the launcher's (a WSL path too long for explorer.exe, pi #11592). + spawn(command, args, { stdio: "ignore", detached: true, cwd: "/", shell: false }) + .on("error", () => {}) + .unref(); +} diff --git a/packages/chord/src/next-node/tui/credentials.ts b/packages/chord/src/next-node/tui/credentials.ts new file mode 100644 index 000000000..35fc0b970 --- /dev/null +++ b/packages/chord/src/next-node/tui/credentials.ts @@ -0,0 +1,71 @@ +// The launcher's credentials (slice 11 §4, decision "The app kit, point 3" (c)): a device token the launcher keeps, +// never the sandbox. It is given, read from a token file (the owner token file on the server's machine, slice 13), or +// got by device pairing: the launcher asks for a code, shows it, and polls until the owner approved it in a browser; +// a paired token may be saved to a file (mode 0600) for the next start. + +import { readFile } from "node:fs/promises"; +import { writeSecret } from "../app/secrets.ts"; + +/** Where the launcher's token comes from. */ +export type TerminalCredentials = + | { readonly kind: "token"; readonly token: string } + | { readonly kind: "file"; readonly path: string } + /** Pairing with the server, the device named `label`; the token saved to `save` (created, mode 0600) when given. */ + | { readonly kind: "pairing"; readonly label: string; readonly save?: string }; + +/** How often a pairing is polled. */ +const POLL_MS = 2000; + +/** The launcher's token for the server at `url`; pairing shows its code through `say`. */ +export async function tokenOf( + credentials: TerminalCredentials, + url: URL, + say: (line: string) => void, + signal: AbortSignal, +): Promise { + switch (credentials.kind) { + case "token": + return credentials.token; + case "file": + return (await readFile(credentials.path, "utf8")).trim(); + case "pairing": + return pair(credentials, url, say, signal); + } +} + +/** Pairs this device: a code shown, the approval polled for, the token returned (and saved when asked). */ +async function pair( + credentials: Extract, + url: URL, + say: (line: string) => void, + signal: AbortSignal, +): Promise { + const asked = await post(new URL("/_chord/pair", url), { label: credentials.label }, signal); + if (!asked.ok) throw new Error(`terminal: the server refused to pair (${asked.status})`); + const { code, poll } = (await asked.json()) as { readonly code?: unknown; readonly poll?: unknown }; + // A trust boundary: the server's answer is shown on the terminal and sent back. + if (typeof code !== "string" || !/^[A-Za-z0-9-]{4,32}$/.test(code) || typeof poll !== "string") { + throw new Error("terminal: the server's pairing answer is malformed"); + } + say(`Pairing code ${code}: approve it at ${new URL(`/_chord/pair/${code}`, url)} (signed in as the owner).`); + for (;;) { + await new Promise((resolve) => setTimeout(resolve, POLL_MS)); + signal.throwIfAborted(); + const polled = await post(new URL("/_chord/pair/poll", url), { poll }, signal); + if (!polled.ok) throw new Error("terminal: the pairing ended without approval"); + const answer = (await polled.json()) as { readonly status: string; readonly token?: string }; + if (answer.status !== "approved" || answer.token === undefined) continue; + if (credentials.save !== undefined) await writeSecret(credentials.save, answer.token, "create"); + return answer.token; + } +} + +/** A JSON post. */ +function post(url: URL, body: unknown, signal: AbortSignal): Promise { + return fetch(url, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify(body), + signal, + }); +} diff --git a/packages/chord/src/next-node/tui/filter.ts b/packages/chord/src/next-node/tui/filter.ts new file mode 100644 index 000000000..be11dcb19 --- /dev/null +++ b/packages/chord/src/next-node/tui/filter.ts @@ -0,0 +1,75 @@ +// The terminal's output filter (slice 11 §4; p2's `tui-filter.ts`): the only path from the sandbox to the real +// terminal. A frame's text and the escape sequences a terminal UI needs to draw pass (cursor moves, erasing, colours, +// the alternate screen, bracketed paste, synchronized output, mouse reporting); every other sequence is dropped, so +// sandboxed code cannot set the clipboard (OSC 52), retitle or reconfigure the terminal, or smuggle a sequence across +// frames: an unterminated one ends with its frame. + +/** The longest frame the sandbox may write. */ +export const MAX_FRAME = 64 * 1024; + +const ESC = "\x1b"; + +/** DEC private modes output may set: cursor visibility, bracketed paste, sync, alternate screen, autowrap, mouse. */ +const MODES: ReadonlySet = new Set(["25", "2004", "2026", "1049", "7", "1000", "1002", "1006", "1004"]); + +/** `output` with only the allowed sequences; throws for a frame over `MAX_FRAME`. */ +export function filterOutput(output: string): string { + if (output.length > MAX_FRAME || Buffer.byteLength(output) > MAX_FRAME) { + throw new RangeError("terminal: a frame over 64 KiB"); + } + let kept = ""; + for (let index = 0; index < output.length; index++) { + const code = output.charCodeAt(index); + if (code === 27 || (code >= 0x80 && code <= 0x9f)) { + const [next, sequence] = escapeAt(output, index); + index = next; + kept += sequence; + continue; + } + if (code === 9 || code === 10 || code === 13 || (code >= 0x20 && code !== 0x7f)) kept += output[index]; + } + return kept; +} + +/** The escape at `index`: the index of its last character, and the sequence kept (`""` when dropped). */ +function escapeAt(output: string, start: number): [number, string] { + let index = start; + const code = output.charCodeAt(index); + const introducer = code === 27 ? (output[++index] ?? "") : output[index]!; + if (introducer === "[" || code === 0x9b) { + const body = index + 1; + while ( + index + 1 < output.length && + output.charCodeAt(index + 1) >= 0x20 && + output.charCodeAt(index + 1) <= 0x3f + ) { + index++; + } + const final = output[++index] ?? ""; + const text = output.slice(body, index); + return [index, code === 27 && final !== "" && allowedCsi(text, final) ? `${ESC}[${text}${final}` : ""]; + } + if ((introducer !== "" && "]PX^_".includes(introducer)) || [0x90, 0x98, 0x9d, 0x9e, 0x9f].includes(code)) { + // A control string (OSC, DCS, ...): dropped up to its terminator, or the end of the frame. + while (++index < output.length) { + if (output.charCodeAt(index) === 0x9c || output.charCodeAt(index) === 7) break; + if (output[index] === ESC && output[index + 1] === "\\") return [index + 1, ""]; + } + return [index, ""]; + } + // Any other ESC sequence with its intermediates (charset selection, ...): dropped. + while (index < output.length && output.charCodeAt(index) >= 0x20 && output.charCodeAt(index) <= 0x2f) index++; + return [index, ""]; +} + +/** Whether the CSI sequence with parameters `body` and final `final` may reach the terminal. */ +function allowedCsi(body: string, final: string): boolean { + if (body.length > 128) return false; + if (final === "h" || final === "l") return body.startsWith("?") && MODES.has(body.slice(1)); + if (final === "m") return /^[\d;:]*$/.test(body) && !/\d{5}/.test(body); + if (!/^[\d;]*$/.test(body) || /\d{5}/.test(body)) return false; + if ("ABCDEFGHfd".includes(final)) return body.split(";").length <= ("Hf".includes(final) ? 2 : 1); + if (final === "J" || final === "K") return /^[012]?$/.test(body); + if (final === "r") return /^(?:\d{1,4};\d{1,4})?$/.test(body); + return (final === "s" || final === "u") && body === ""; +} diff --git a/packages/chord/src/next-node/tui/index.ts b/packages/chord/src/next-node/tui/index.ts new file mode 100644 index 000000000..0d0f52e82 --- /dev/null +++ b/packages/chord/src/next-node/tui/index.ts @@ -0,0 +1,8 @@ +// Chord v2's terminal launcher (`@earendil-works/chord/next/tui/launcher` once v2 has its entry point; slice 11 §4): +// the trusted Node side of a terminal client, running the app's terminal code in the sandbox. + +export type { PromptTerminal } from "./actions.ts"; +export type { TerminalCredentials } from "./credentials.ts"; +export { filterOutput } from "./filter.ts"; +export { type RunningTerminal, startTerminal, type TerminalOptions, terminalFile } from "./launcher.ts"; +export { requireSandboxNode } from "./sandbox.ts"; diff --git a/packages/chord/src/next-node/tui/launcher.ts b/packages/chord/src/next-node/tui/launcher.ts new file mode 100644 index 000000000..66a5b605b --- /dev/null +++ b/packages/chord/src/next-node/tui/launcher.ts @@ -0,0 +1,466 @@ +// `startTerminal` (slice 11 §4, decision "Slice 11, P8"): the trusted launcher of a terminal client. It signs in with +// its token (given, from a file, or by pairing), downloads the terminal kind's base bundle and plugin scripts, and +// runs them in the sandbox, always. It keeps the credentials, the real terminal and the user's choices: the sandbox +// gets input and resizes, its frames pass the output filter, its Chord link is a socket the launcher opens and relays, +// its plugin script fetches go only to the terminal surface's files, and its actions on the machine (copy, open a URL, +// attach a file) are confirmed by the launcher's own prompt. A new base bundle on the server starts a new sandbox. +// Nothing the sandbox sends can end the launcher (fix round 2, Opus E3): its messages are checked (`pipe.ts`), a send +// on a socket not yet open is dropped, every buffer and the requests in flight are bounded, a signal cannot open the +// launcher's inspector, and however the launcher's process ends the terminal is given back (fix round 3, Opus E8, +// E10, E11). During a prompt Ctrl-C ends the client: its keys are the launcher's; and three Ctrl-C within a second +// always end it (Opus R2). + +import type { Readable, Writable } from "node:stream"; +import { WebSocket } from "ws"; +import type { ColorDepth } from "../../next/surfaces/index.ts"; +import type { BridgeRequest, SandboxMessage } from "../../next-tui/bridge.ts"; +import type { TuiStart } from "../surfaces/tui.ts"; +import { act, attach, type PromptTerminal, promptLines } from "./actions.ts"; +import { type TerminalCredentials, tokenOf } from "./credentials.ts"; +import { filterOutput } from "./filter.ts"; +import { MAX_BUNDLE, requireSandboxNode, Sandbox } from "./sandbox.ts"; + +/** What `startTerminal` takes. */ +export type TerminalOptions = { + /** The app's server (`http://127.0.0.1:8788`). */ + readonly url: string; + readonly credentials: TerminalCredentials; + /** The terminal's input and output: the process's by default (raw mode while running, when a TTY). */ + readonly input?: Readable & { setRawMode?(raw: boolean): unknown; readonly isTTY?: boolean }; + readonly output?: Writable & { readonly columns?: number; readonly rows?: number }; + /** How many colours the terminal shows: true colour when `COLORTERM` says so, else 256. */ + readonly colors?: ColorDepth; + /** The terminal's background, choosing between the user's dark and light theme: dark by default. */ + readonly mode?: "dark" | "light"; + /** Opens a URL the user confirmed: the platform's opener by default. */ + readonly opener?: (url: string) => Promise; +}; + +/** A running terminal client: its exit code once it ends, and its stop. */ +export type RunningTerminal = { readonly exited: Promise; stop(): Promise }; + +/** The terminal surface's files (the server's `TUI_PREFIX`). */ +const PREFIX = "/_chord/tui/"; + +/** + * `path` resolved on the server `server` when it names one of the terminal surface's files (dot segments and encodings + * resolved first: Opus M2, Fable L1); throws otherwise: the sandbox reaches nothing else of the server through the + * launcher's token. + */ +export function terminalFile(path: unknown, server: URL): string { + const target = typeof path === "string" ? new URL(path, server) : undefined; + if ( + target === undefined || + target.origin !== server.origin || + target.search !== "" || + !TERMINAL_FILE.test(target.pathname) + ) { + throw new Error("terminal: no terminal file"); + } + return target.pathname; +} + +/** A terminal surface file's path, as the sandbox may ask for it. */ +const TERMINAL_FILE = /^\/_chord\/tui\/(?:plugin\/)?[\w.~@-]+$/; + +/** The most frames held while a prompt is open: 1 MiB (characters); later ones are dropped. */ +const MAX_HELD = 1024 * 1024; + +/** The most link sockets the sandbox may have open at once. */ +const MAX_SOCKETS = 4; + +/** The most bytes a link socket may have waiting to go out; a sandbox sending more loses the socket. */ +const MAX_SOCKET_BUFFER = 16 * 1024 * 1024; + +/** How long after a prompt appears its keys are ignored: a key typed for the sandbox does not answer it (Opus E4). */ +const PROMPT_GUARD_MS = 300; + +/** + * The terminal modes the sandbox may have set, back to the shell's: sync, mouse, focus, paste, cursor, the scroll region + * (the cursor kept where it was) and autowrap (Opus E11), screen, colours. + */ +const RESET_MODES = + "\x1b[?2026l\x1b[?1000l\x1b[?1002l\x1b[?1006l\x1b[?1004l\x1b[?2004l\x1b[?25h\x1b7\x1b[r\x1b8\x1b[?7h\x1b[?1049l\x1b[0m"; + +/** The most requests of one sandbox the launcher answers at once; more are refused at once (Opus E10). */ +const MAX_REQUESTS = 8; + +/** + * Ctrl-C: during a prompt it ends the client, so a sandbox asking again and again can always be left (Opus E11); and + * three within a second always end it, whatever the sandbox does with them (Opus R2). + */ +const INTERRUPT = "\x03"; + +/** How many Ctrl-C within `INTERRUPT_WINDOW_MS` end the client however the sandbox takes them. */ +const INTERRUPTS_TO_END = 3; +const INTERRUPT_WINDOW_MS = 1000; + +/** Starts a terminal client; rejects when it cannot sign in or download, or Node is too old for the sandbox. */ +export async function startTerminal(options: TerminalOptions): Promise { + requireSandboxNode(); + const output = options.output ?? process.stdout; + const url = new URL(options.url); + const abort = new AbortController(); + // What the launcher prints of the server's (the pairing's code) passes the output filter too. + const token = await tokenOf( + options.credentials, + url, + (line) => output.write(`${filterOutput(line)}\r\n`), + abort.signal, + ); + const launcher = new Launcher(url, token, options); + await launcher.start(); + return { exited: launcher.exited, stop: () => launcher.finish(0) }; +} + +/** One terminal client's launcher. */ +class Launcher { + readonly #url: URL; + readonly #headers: Readonly>; + readonly #options: TerminalOptions; + readonly #input: NonNullable; + readonly #output: NonNullable; + readonly #sockets = new Map(); + #sandbox: Sandbox | undefined; + /** The running base bundle's hash: an `outdated` sandbox restarts only for another. */ + #base = ""; + /** + * Whether an action is being confirmed (one at a time), from its request to its end: meanwhile the sandbox's frames + * are held (`#held`, at most `MAX_HELD` bytes), so nothing draws over the prompt, even before it appears. + */ + #confirming = false; + /** The running sandbox's requests being answered (at most `MAX_REQUESTS`). */ + #requests = 0; + /** The prompt open: its lines, its answer, and when its keys start to count. Keys go to it, not the sandbox. */ + #prompt: + | { readonly lines: readonly string[]; readonly answer: (yes: boolean) => void; readonly ready: number } + | undefined; + #held: string[] = []; + #heldBytes = 0; + #finished = false; + #exit!: (code: number) => void; + readonly exited = new Promise((resolve) => { + this.#exit = resolve; + }); + readonly #onInput = (data: Buffer | string): void => { + const text = data.toString(); + if (this.#interrupted(text)) { + void this.finish(130); + return; + } + if (this.#prompt === undefined) this.#sandbox?.post({ kind: "input", data: text }); + else if (text.includes(INTERRUPT)) void this.finish(130); + else if (Date.now() >= this.#prompt.ready) this.#answered(text === "y" || text === "Y"); + }; + readonly #onResize = (): void => { + if (this.#prompt !== undefined) this.#drawPrompt(); + this.#sandbox?.post({ kind: "resize", columns: this.#columns, rows: this.#rows }); + }; + /** When the last Ctrl-C keys came: three within a second end the client. */ + #interrupts: number[] = []; + /** The terminal given back if the launcher's process ends without `finish` (a crash). */ + readonly #onExit = (): void => { + this.#output.write(RESET_MODES); + if (this.#input.isTTY === true) this.#input.setRawMode?.(false); + }; + + constructor(url: URL, token: string, options: TerminalOptions) { + this.#url = url; + this.#headers = { authorization: `Bearer ${token}` }; + this.#options = options; + this.#input = options.input ?? process.stdin; + this.#output = options.output ?? process.stdout; + } + + /** Whether `text` brings the third Ctrl-C within a second (each still goes to the sandbox otherwise). */ + #interrupted(text: string): boolean { + const now = Date.now(); + for (const character of text) if (character === INTERRUPT) this.#interrupts.push(now); + this.#interrupts = this.#interrupts.filter((at) => now - at < INTERRUPT_WINDOW_MS); + return this.#interrupts.length >= INTERRUPTS_TO_END; + } + + // A pseudo-terminal may report 0 (Opus E4). + get #columns(): number { + return this.#output.columns || 80; + } + + get #rows(): number { + return this.#output.rows || 24; + } + + /** Downloads the base bundle and the plugins, starts the sandbox and takes the terminal. */ + async start(): Promise { + await this.#startSandbox(); + if (this.#input.isTTY === true) this.#input.setRawMode?.(true); + this.#input.on("data", this.#onInput); + this.#output.on("resize", this.#onResize); + process.once("exit", this.#onExit); + // Node opens its inspector on SIGUSR1 unless the signal has a listener: a sandbox may signal its launcher, and + // an inspector would give any local process the launcher's token and files (Opus E8). Windows has no SIGUSR1. + if (process.platform !== "win32") process.on("SIGUSR1", ignoreSignal); + } + + /** Ends the client with `code`: the terminal given back, the sockets closed, the sandbox ended. */ + async finish(code: number): Promise { + if (this.#finished) return; + this.#finished = true; + this.#input.off("data", this.#onInput); + this.#output.off("resize", this.#onResize); + process.off("exit", this.#onExit); + if (process.platform !== "win32") process.off("SIGUSR1", ignoreSignal); + if (this.#input.isTTY === true) this.#input.setRawMode?.(false); + this.#prompt?.answer(false); + this.#prompt = undefined; + this.#output.write(RESET_MODES); + this.#closeSockets(); + await this.#sandbox?.terminate(); + this.#exit(code); + } + + /** A new sandbox with what the server serves now. */ + async #startSandbox(): Promise { + const start = JSON.parse(await this.#get(`${PREFIX}start`)) as TuiStart; + const [source, combined] = await Promise.all([this.#get(start.script), this.#get(start.combined)]); + this.#base = start.base; + const colors = + this.#options.colors ?? (/truecolor|24bit/.test(process.env.COLORTERM ?? "") ? "truecolor" : "256"); + const bridge = { + base: start.base, + combined, + columns: this.#columns, + rows: this.#rows, + colors, + mode: this.#options.mode ?? "dark", + } as const; + this.#requests = 0; + this.#sandbox = await Sandbox.start( + source, + bridge, + (message) => this.#message(message), + () => void this.finish(1), + ); + } + + /** One message of the sandbox; one the launcher fails on ends the client, never the launcher's process. */ + #message(message: SandboxMessage): void { + try { + this.#handle(message); + } catch { + void this.finish(1); + } + } + + #handle(message: SandboxMessage): void { + switch (message.kind) { + case "write": + this.#write(message.data); + return; + case "socket_open": + this.#openSocket(message.socket); + return; + case "socket_send": + this.#send(message.socket, message.bytes); + return; + case "socket_close": + this.#sockets.get(message.socket)?.close(); + return; + case "request": + void this.#answer(message.id, message.request); + return; + case "outdated": + void this.#outdated(); + return; + case "failed": + this.#output.write("terminal: the terminal client failed to start\n"); + void this.finish(1); + return; + case "exit": + void this.finish(message.code); + } + } + + /** Writes the sandbox's frame through the output filter (held while a prompt is open); one over 64 KiB is dropped. */ + #write(data: string): void { + if (this.#confirming) { + if (typeof data === "string" && this.#heldBytes + data.length <= MAX_HELD) { + this.#held.push(data); + this.#heldBytes += data.length; + } + return; + } + try { + this.#output.write(filterOutput(data)); + } catch { + // Over 64 KiB (or no text): dropped, as an escape it could hide would be. + } + } + + /** Opens socket `socket` to the links endpoint with the launcher's token, relaying it to the sandbox. */ + #openSocket(socket: number): void { + // A few at a time: the connection loop needs one; a runaway sandbox gets no more. + if (this.#sockets.size >= MAX_SOCKETS) { + this.#sandbox?.post({ kind: "socket_closed", socket }); + return; + } + const target = new URL("/chord", this.#url); + target.protocol = target.protocol === "https:" ? "wss:" : "ws:"; + const ws = new WebSocket(target, { headers: this.#headers }); + this.#sockets.set(socket, ws); + ws.on("open", () => this.#sandbox?.post({ kind: "socket_opened", socket })); + ws.on("message", (data: Buffer) => { + this.#sandbox?.post({ kind: "socket_message", socket, bytes: new Uint8Array(data) }); + }); + const closed = (): void => { + if (this.#sockets.get(socket) !== ws) return; + this.#sockets.delete(socket); + this.#sandbox?.post({ kind: "socket_closed", socket }); + }; + ws.on("close", closed); + ws.on("error", closed); + } + + /** + * Sends `bytes` on socket `socket` once it is open (before, the sandbox's link holds its messages; a send then is + * dropped: Opus E3); a socket with more than 16 MiB waiting is closed. + */ + #send(socket: number, bytes: Uint8Array): void { + const ws = this.#sockets.get(socket); + if (ws?.readyState !== WebSocket.OPEN) return; + if (ws.bufferedAmount + bytes.length > MAX_SOCKET_BUFFER) ws.terminate(); + else ws.send(bytes); + } + + /** Answers the sandbox's request `id`. */ + async #answer(id: number, request: BridgeRequest): Promise { + // The sandbox that asked: a reply never goes to the one a restart started meanwhile. + const sandbox = this.#sandbox; + if (sandbox === undefined) return; + if (this.#requests >= MAX_REQUESTS) { + sandbox.post({ kind: "refused", id, message: "terminal: too many requests at once" }); + return; + } + this.#requests++; + try { + let value: unknown; + if (request.kind === "fetch") value = await this.#get(terminalFile(request.path, this.#url)); + else if (request.kind === "base") value = JSON.parse(await this.#get(`${PREFIX}base`)); + else if (request.kind === "attach") { + const file = await this.#confirm(() => attach(request.path, this.#prompter())); + if (file !== null) { + sandbox.post({ kind: "attached", id, name: file.name, bytes: file.bytes }); + return; + } + value = null; + } else value = await this.#confirm(() => act(request, this.#prompter(), this.#options.opener)); + sandbox.post({ kind: "reply", id, value }); + } catch (error) { + sandbox.post({ kind: "refused", id, message: error instanceof Error ? error.message : String(error) }); + } finally { + if (this.#sandbox === sandbox) this.#requests--; + } + } + + /** + * Runs the action `run` (it asks the user), one at a time: a second is refused while one runs. Afterwards the held + * frames are written and the sandbox asked to redraw. + */ + async #confirm(run: () => Promise): Promise { + if (this.#confirming) throw new Error("terminal: a prompt is open"); + this.#confirming = true; + try { + return await run(); + } finally { + this.#confirming = false; + const held = this.#held; + this.#held = []; + this.#heldBytes = 0; + for (const frame of held) this.#write(frame); + this.#sandbox?.post({ kind: "resize", columns: this.#columns, rows: this.#rows }); + } + } + + /** The launcher's own prompt on the real terminal, for the action being confirmed: its key not the sandbox's. */ + #prompter(): PromptTerminal { + return { + columns: this.#columns, + ask: (lines) => { + // Shown whole or not at all: a prompt is never cut. + if (promptLines(lines, this.#columns).length >= this.#rows) { + return Promise.reject(new Error("terminal: the prompt does not fit the terminal")); + } + return new Promise((answer) => { + this.#prompt = { lines, answer, ready: Date.now() + PROMPT_GUARD_MS }; + this.#drawPrompt(); + }); + }, + write: (data) => this.#output.write(data), + }; + } + + /** + * Draws the open prompt on the bottom lines after closing a synchronized update the sandbox began: its attributes + * reset and its colours explicit, black on white, so no state the sandbox left (concealed, black on black: Opus E4) + * applies to it. + */ + #drawPrompt(): void { + const lines = promptLines(this.#prompt!.lines, this.#columns); + const top = Math.max(1, this.#rows - lines.length + 1); + const drawn = lines.map((line, index) => `\x1b[${top + index};1H\x1b[0;30;47m\x1b[2K ${line} `); + this.#output.write(`\x1b[?2026l\x1b[s${drawn.join("")}\x1b[0m`); + } + + /** The prompt answered: cleared, the cursor back where the sandbox had it. */ + #answered(yes: boolean): void { + const { lines, answer } = this.#prompt!; + this.#prompt = undefined; + const rows = promptLines(lines, this.#columns); + const top = Math.max(1, this.#rows - rows.length + 1); + const cleared = rows.map((_row, index) => `\x1b[${top + index};1H\x1b[0m\x1b[2K`).join(""); + this.#output.write(`${cleared}\x1b[u`); + answer(yes); + } + + /** The sandbox says its code is outdated: a new sandbox only when the server's base bundle is another. */ + async #outdated(): Promise { + const base = JSON.parse(await this.#get(`${PREFIX}base`).catch(() => "null")) as unknown; + if (typeof base === "string" && base !== this.#base) await this.#restart(); + } + + /** The server has a new base bundle: a new sandbox with it, the old one ended. */ + async #restart(): Promise { + this.#closeSockets(); + await this.#sandbox?.terminate(); + this.#sandbox = undefined; + try { + await this.#startSandbox(); + this.#onResize(); + } catch { + await this.finish(1); + } + } + + #closeSockets(): void { + for (const ws of this.#sockets.values()) ws.close(); + this.#sockets.clear(); + } + + /** The text at `path` on the server, signed in; at most 8 MiB, read no further. */ + async #get(path: string): Promise { + const response = await fetch(new URL(path, this.#url), { headers: this.#headers }); + if (!response.ok || response.body === null) throw new Error(`terminal: GET ${path} answered ${response.status}`); + if (Number(response.headers.get("content-length") ?? 0) > MAX_BUNDLE) + throw new Error(`terminal: ${path} exceeds 8 MiB`); + const chunks: Uint8Array[] = []; + let size = 0; + for await (const chunk of response.body) { + size += chunk.length; + if (size > MAX_BUNDLE) throw new Error(`terminal: ${path} exceeds 8 MiB`); + chunks.push(chunk); + } + return new TextDecoder().decode(Buffer.concat(chunks)); + } +} + +/** The launcher's SIGUSR1 listener: the signal does nothing. */ +function ignoreSignal(): void {} diff --git a/packages/chord/src/next-node/tui/pipe.ts b/packages/chord/src/next-node/tui/pipe.ts new file mode 100644 index 000000000..a775303a5 --- /dev/null +++ b/packages/chord/src/next-node/tui/pipe.ts @@ -0,0 +1,132 @@ +// The launcher's end of the sandbox's pipes (slice 11 §4; fix round 2: the sandbox a child process): messages as +// plain length-prefixed binary frames, not Node's IPC. A frame is a 4-byte big-endian length, then the payload: a +// 4-byte big-endian header length, the header (the message as JSON, its body field left out), and the body (the raw +// bytes of `bytes`, or the UTF-8 text of `data`). The sandbox's bootstrap has the same codec (`sandbox.ts`). What the +// sandbox sends is untrusted: each frame is bounded, and each message checked here before the launcher acts on it. + +import type { BridgeRequest, LauncherMessage, SandboxMessage } from "../../next-tui/bridge.ts"; + +/** The largest frame either side takes: 17 MiB (an attached file of 16 MiB and its header). */ +export const MAX_PIPE_FRAME = 17 * 1024 * 1024; + +/** The largest header of a frame from the sandbox: 1 MiB (a request's text or URL; frames carry their text as body). */ +const MAX_HEADER = 1024 * 1024; + +/** `message` as one frame for the sandbox. */ +export function launcherFrame(message: LauncherMessage): Buffer { + if (message.kind === "input") return frame({ kind: "input" }, Buffer.from(message.data)); + if (message.kind === "socket_message") return frame({ kind: message.kind, socket: message.socket }, message.bytes); + if (message.kind === "attached") + return frame({ kind: message.kind, id: message.id, name: message.name }, message.bytes); + return frame(message, undefined); +} + +/** The first frame: the bridge's start and the base bundle's text as body. */ +export function startFrame(start: unknown, source: string): Buffer { + return frame({ kind: "start", start }, Buffer.from(source)); +} + +/** A frame of `header` and `body`. */ +function frame(header: object, body: Uint8Array | undefined): Buffer { + const json = Buffer.from(JSON.stringify(header)); + const prefix = Buffer.allocUnsafe(8); + prefix.writeUInt32BE(4 + json.length + (body?.length ?? 0), 0); + prefix.writeUInt32BE(json.length, 4); + return Buffer.concat(body === undefined ? [prefix, json] : [prefix, json, body]); +} + +/** + * Cuts the sandbox's output into messages as chunks arrive; throws (the sandbox is ended) on a frame over + * `MAX_PIPE_FRAME`, a header over 1 MiB, or a message of no known shape. Each byte is copied in once, however small the + * chunks the sandbox writes. + */ +export class FrameReader { + #buffer = Buffer.alloc(64 * 1024); + #start = 0; + #end = 0; + + /** The messages complete with `chunk`. */ + push(chunk: Buffer): SandboxMessage[] { + this.#append(chunk); + const messages: SandboxMessage[] = []; + while (this.#end - this.#start >= 4) { + const length = this.#buffer.readUInt32BE(this.#start); + if (length < 4 || length > MAX_PIPE_FRAME) throw new Error("terminal: the sandbox sent a malformed frame"); + if (this.#end - this.#start < 4 + length) break; + // The message copies what it keeps: the buffer is reused. + messages.push(sandboxMessage(this.#buffer.subarray(this.#start + 4, this.#start + 4 + length))); + this.#start += 4 + length; + } + if (this.#start === this.#end) this.#start = this.#end = 0; + return messages; + } + + /** Copies `chunk` in after the unread bytes, moving them to the front or growing the buffer when it is full. */ + #append(chunk: Buffer): void { + const unread = this.#end - this.#start; + if (this.#end + chunk.length > this.#buffer.length) { + const needed = unread + chunk.length; + const target = + needed <= this.#buffer.length + ? this.#buffer + : Buffer.alloc(Math.max(needed, Math.min(this.#buffer.length * 2, MAX_PIPE_FRAME + 4))); + this.#buffer.copy(target, 0, this.#start, this.#end); + this.#buffer = target; + this.#start = 0; + this.#end = unread; + } + chunk.copy(this.#buffer, this.#end); + this.#end += chunk.length; + } +} + +/** + * The message in `payload` (a frame from the sandbox, without its length); throws for anything else. Trust boundary: + * the sandbox's pipe. + */ +function sandboxMessage(payload: Buffer): SandboxMessage { + const headerLength = payload.readUInt32BE(0); + if (headerLength > MAX_HEADER || 4 + headerLength > payload.length) throw malformed(); + const header = JSON.parse(payload.toString("utf8", 4, 4 + headerLength)) as Record | null; + const body = payload.subarray(4 + headerLength); + if (typeof header !== "object" || header === null) throw malformed(); + const socket = header.socket; + switch (header.kind) { + case "write": + return { kind: "write", data: body.toString("utf8") }; + case "socket_send": + if (!Number.isSafeInteger(socket)) throw malformed(); + // A copy: the payload's memory is the reader's. + return { kind: "socket_send", socket: socket as number, bytes: new Uint8Array(body) }; + case "socket_open": + case "socket_close": + if (!Number.isSafeInteger(socket)) throw malformed(); + return { kind: header.kind, socket: socket as number }; + case "request": + if (!Number.isSafeInteger(header.id)) throw malformed(); + return { kind: "request", id: header.id as number, request: bridgeRequest(header.request) }; + case "outdated": + return { kind: "outdated" }; + case "failed": + return { kind: "failed", message: String(header.message) }; + case "exit": + return { kind: "exit", code: Number.isSafeInteger(header.code) ? (header.code as number) : 1 }; + default: + throw malformed(); + } +} + +/** The request `value` names; throws for anything else. */ +function bridgeRequest(value: unknown): BridgeRequest { + const request = (typeof value === "object" && value !== null ? value : {}) as Record; + if (request.kind === "base") return { kind: "base" }; + if (request.kind === "fetch" && typeof request.path === "string") return { kind: "fetch", path: request.path }; + if (request.kind === "copy" && typeof request.text === "string") return { kind: "copy", text: request.text }; + if (request.kind === "open" && typeof request.url === "string") return { kind: "open", url: request.url }; + if (request.kind === "attach" && typeof request.path === "string") return { kind: "attach", path: request.path }; + throw malformed(); +} + +function malformed(): Error { + return new Error("terminal: the sandbox sent a malformed message"); +} diff --git a/packages/chord/src/next-node/tui/sandbox.ts b/packages/chord/src/next-node/tui/sandbox.ts new file mode 100644 index 000000000..c9b6384d9 --- /dev/null +++ b/packages/chord/src/next-node/tui/sandbox.ts @@ -0,0 +1,450 @@ +// The terminal's sandbox (slice 11 §4, decision "Slice 11, P8"; fix round 2: a child process, as the owner decided +// after the worker thread was escaped through its shared realm): a Node child process of the launcher's binary. +// - The operating system's side: its stdin, stdout and stderr are pipes to the launcher (never the terminal: no +// descriptor of the child is a TTY), its environment empty, its directory a private one holding only the bootstrap; +// Node's permission model with nothing allowed but reading the bootstrap (no other file read, no write, no network, +// no child processes, no workers, no addons, no inspector, no WASI) and `node:sqlite` removed +// (`--no-experimental-sqlite`: SQLite opens files past the permission model). An escape from the JavaScript layer +// reaches no terminal, no file, no socket, no launcher memory or token. +// - The JavaScript side, a second layer: downloaded code (the base bundle, a CommonJS script; the plugins' registered +// scripts) is only evaluated by the bootstrap (`vm.Script` with no dynamic import callback, so `import()` throws), +// with a `require` answering the allowed builtins only (`SANDBOX_BUILTINS`); `process` is a substitute with +// `SANDBOX_PROCESS` only. The bootstrap keeps its own references to what it uses before any downloaded code runs. +// Messages go both ways as length-prefixed binary frames (`pipe.ts`). Known gaps, each needing the JavaScript layer +// broken first unless noted: the child may signal processes of the same user (Node has no permission for signals; the +// launcher ignores SIGUSR1, which would open its inspector), read machine details (`os`), learn whether any path exists +// (`require.resolve` fails one way for a file that exists, another for one that does not: Opus E13), and allocate +// memory outside its heap (no limit without a shell's `ulimit`; only the child dies). Needs Node 26.5.1 or later. + +import { type ChildProcessWithoutNullStreams, spawn } from "node:child_process"; +import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { BridgeStart, LauncherMessage, SandboxMessage } from "../../next-tui/bridge.ts"; +import { FrameReader, launcherFrame, MAX_PIPE_FRAME, startFrame } from "./pipe.ts"; + +/** The largest base bundle a sandbox runs: 8 MiB. */ +export const MAX_BUNDLE = 8 * 1024 * 1024; + +/** + * The builtins code in the sandbox may `require`, each computation in memory without I/O, descriptors or machine + * details: bytes and text (`buffer`, `string_decoder`, `punycode`, `querystring`), events and streams in memory + * (`events`, `stream` and its submodules), formatting and type checks (`util`, `util/types`), URL and path strings + * (`url`, `path`), timers, assertions, hashing and randomness (`crypto`: no keys from files, no network) and compression + * in memory (`zlib`). Not `fs`, `tty`, `net`, `dgram`, `dns`, `http`, `https`, `http2`, `tls`, `child_process`, + * `cluster`, `worker_threads`, `sqlite`, `inspector`, `vm`, `v8`, `os`, `module`, `process`, `perf_hooks`, `wasi`, + * `repl`, `readline`, `trace_events` or anything else. + */ +export const SANDBOX_BUILTINS: readonly string[] = [ + "assert", + "assert/strict", + "buffer", + "crypto", + "events", + "path", + "path/posix", + "path/win32", + "punycode", + "querystring", + "stream", + "stream/consumers", + "stream/promises", + "stream/web", + "string_decoder", + "timers", + "timers/promises", + "url", + "util", + "util/types", + "zlib", +]; + +/** + * The sandbox's `process`: a frozen plain object in place of Node's (no event emitter, so no prototype shared with + * Node's internals leads back to the real one). Its properties, each harmless: which Node it is (`arch`, `platform`, + * `version`, `versions`, `release`), time (`nextTick`, `hrtime`, `uptime`), ending the sandbox (`exit`), the permission + * checks (`permission`), the allowed builtins (`getBuiltinModule`), fixed values (`env` empty, `argv`, `argv0`, + * `execArgv`, `execPath`, `title`, `cwd` `/`), `stdout` (to the terminal through the bridge) and `stderr` (dropped), + * and listener calls that keep nothing (`on`, `once`, `off`, `addListener`, `removeListener`, `removeAllListeners`, + * `emit`, `emitWarning`, `listenerCount`, `listeners`). Not `binding`, `_linkedBinding`, `dlopen`, `_rawDebug`, `kill`, + * `execve`, `chdir`, `abort`, `reallyExit`, `stdin`, `openStdin`, `report`, `pid`, `ppid`, `config`, the user and group + * calls, `loadEnvFile`, `moduleLoadList`, resource and handle listings or any other. + */ +export const SANDBOX_PROCESS: readonly string[] = [ + "addListener", + "arch", + "argv", + "argv0", + "cwd", + "emit", + "emitWarning", + "env", + "execArgv", + "execPath", + "exit", + "getBuiltinModule", + "hrtime", + "listenerCount", + "listeners", + "nextTick", + "off", + "on", + "once", + "permission", + "platform", + "release", + "removeAllListeners", + "removeListener", + "stderr", + "stdout", + "title", + "uptime", + "version", + "versions", +]; + +/** + * The child's Node flags for its script `script`: the permission model allowing only reading that file, no SQLite, a + * 96 MiB heap. Exported for the tests of the operating system's layer alone. + */ +export function sandboxFlags(script: string): string[] { + return ["--permission", `--allow-fs-read=${script}`, "--no-experimental-sqlite", "--max-old-space-size=96", script]; +} + +/** The bootstrap's file in the sandbox's directory: a script, not `-e`, which puts every builtin on `globalThis`. */ +export const BOOTSTRAP_FILE = "bootstrap.cjs"; + +/** The most bytes waiting for the sandbox to read them before it counts as stuck: 64 MiB. */ +const MAX_UNREAD = 64 * 1024 * 1024; + +/** Throws unless this Node is 26.5.1 or later, the Node the sandbox's permissions were checked on. */ +export function requireSandboxNode(version: string = process.versions.node): void { + const [major = 0, minor = 0, patch = 0] = version.split(".").map(Number); + if (major > 26 || (major === 26 && (minor > 5 || (minor === 5 && patch >= 1)))) return; + throw new Error(`the terminal client needs Node 26.5.1 or later for its sandbox (this is Node ${version})`); +} + +/** + * The child's environment: empty, but for `LD_LIBRARY_PATH` when the launcher has one: the launcher needed it to run + * this same binary (a Node built against libraries outside the system's paths), and the server cannot influence it. + */ +export function sandboxEnvironment(): Record { + const libraries = process.env.LD_LIBRARY_PATH; + return libraries === undefined ? {} : { LD_LIBRARY_PATH: libraries }; +} + +/** A running sandbox: messages both ways, and its end. */ +export class Sandbox { + readonly #child: ChildProcessWithoutNullStreams; + readonly #directory: string; + readonly #onMessage: (message: SandboxMessage) => void; + readonly #onEnd: (reason: string) => void; + readonly #reader = new FrameReader(); + readonly #exited: Promise; + #ended = false; + /** Whether the child has sent anything: its bootstrap was read, so the file goes. */ + #running = false; + + /** + * Starts a sandbox running the base bundle `source` (a CommonJS script) with `start` on its bridge; `onMessage` gets + * what it sends, `onEnd` its end (an exit, a crash, its memory limit, a malformed frame), once. + */ + static async start( + source: string, + start: BridgeStart, + onMessage: (message: SandboxMessage) => void, + onEnd: (reason: string) => void, + ): Promise { + requireSandboxNode(); + if (Buffer.byteLength(source) > MAX_BUNDLE) throw new Error("terminal: the base bundle exceeds 8 MiB"); + const directory = await mkdtemp(join(tmpdir(), "chord-tui-")); + await writeFile(join(directory, BOOTSTRAP_FILE), BOOTSTRAP); + const sandbox = new Sandbox(directory, onMessage, onEnd); + sandbox.#write(startFrame(start, source)); + return sandbox; + } + + private constructor( + directory: string, + onMessage: (message: SandboxMessage) => void, + onEnd: (reason: string) => void, + ) { + this.#directory = directory; + this.#onMessage = onMessage; + this.#onEnd = onEnd; + this.#child = spawn(process.execPath, sandboxFlags(join(directory, BOOTSTRAP_FILE)), { + stdio: ["pipe", "pipe", "pipe"], + env: sandboxEnvironment(), + cwd: directory, + windowsHide: true, + }); + this.#exited = new Promise((resolve) => this.#child.once("close", () => resolve())); + this.#child.stdout.on("data", (chunk: Buffer) => this.#read(chunk)); + // Dropped: the sandbox writes only through the bridge, and the terminal is the launcher's. + this.#child.stderr.resume(); + // A write after the child ended fails here; its end is reported by `close`. + this.#child.stdin.on("error", () => {}); + this.#child.on("error", (error) => this.#end(`the terminal client failed: ${error.message}`)); + this.#child.on("close", () => { + this.#end("the terminal client exited"); + void rm(this.#directory, { recursive: true, force: true }); + }); + } + + /** Sends `message` to the sandbox; a sandbox not reading what it was sent is ended. */ + post(message: LauncherMessage): void { + if (!this.#ended) this.#write(launcherFrame(message)); + } + + /** Ends the sandbox and waits for its exit. */ + async terminate(): Promise { + this.#ended = true; + this.#child.kill("SIGKILL"); + await this.#exited; + } + + #write(frame: Buffer): void { + if (this.#child.stdin.writableLength + frame.length > MAX_UNREAD + MAX_PIPE_FRAME) { + this.#fail("the terminal client stopped reading"); + return; + } + this.#child.stdin.write(frame); + } + + /** The sandbox's output: its messages, each passed on; a malformed one ends the sandbox. */ + #read(chunk: Buffer): void { + if (this.#ended) return; + if (!this.#running) { + this.#running = true; + // Read once at the start; a launcher killed later leaves at most an empty directory. + void rm(join(this.#directory, BOOTSTRAP_FILE), { force: true }); + } + let messages: SandboxMessage[]; + try { + messages = this.#reader.push(chunk); + } catch (error) { + this.#fail((error as Error).message); + return; + } + for (const message of messages) if (!this.#ended) this.#onMessage(message); + } + + #fail(reason: string): void { + this.#child.kill("SIGKILL"); + this.#end(reason); + } + + #end(reason: string): void { + if (this.#ended) return; + this.#ended = true; + this.#onEnd(reason); + } +} + +/** + * The trusted bootstrap (strict CommonJS, the child's main script). Before any downloaded code runs it keeps its own + * references to what it uses (so code that changes shared prototypes later cannot steer it), closes the file system's + * functions, shadows the real `process`'s listener calls, puts the substitute `process` in place, removes the module + * globals and leaves the bridge on `globalThis`; then it waits for the launcher's start frame and evaluates the base + * bundle through its own `require`. Frames as in `pipe.ts`. + */ +const BOOTSTRAP = `"use strict"; +const real = process; +const getBuiltin = real.getBuiltinModule; +const apply = Reflect.apply; +const uncurry = Function.prototype.call.bind(Function.prototype.bind, Function.prototype.call); +const parse = JSON.parse; +const stringify = JSON.stringify; +const freeze = Object.freeze; +const define = Object.defineProperty; +const BufferFrom = Buffer.from; +const BufferAlloc = Buffer.allocUnsafe; +const BufferConcat = Buffer.concat; +const readUInt32BE = uncurry(Buffer.prototype.readUInt32BE); +const writeUInt32BE = uncurry(Buffer.prototype.writeUInt32BE); +const bufferText = uncurry(Buffer.prototype.toString); +const subarray = uncurry(Buffer.prototype.subarray); +const copyBytes = uncurry(Buffer.prototype.copy); +const isUint8Array = (value) => value instanceof Uint8Array; +const builtin0 = (id) => apply(getBuiltin, real, [id]); +const Script = builtin0("node:vm").Script; +const runInThisContext = uncurry(Script.prototype.runInThisContext); +const nextTick = real.nextTick; +const hrtime = real.hrtime; +const hrtimeBigint = real.hrtime.bigint; +const uptime = real.uptime; +const exit = real.exit; +const input = real.stdin; +const output = real.stdout; +const writeOutput = uncurry(output.write); +const onInput = uncurry(input.on); +const MAX_FRAME = ${MAX_PIPE_FRAME}; +const deny = () => { + throw new Error("unavailable in the terminal's sandbox"); +}; + +// The file system's functions closed: the permission model guards paths, not the pipes' descriptors. +for (const facade of [builtin0("node:fs"), builtin0("node:fs/promises")]) { + for (const name of Object.keys(facade)) { + if (typeof facade[name] === "function") define(facade, name, { value: deny, writable: false, configurable: false, enumerable: true }); + } +} +const v8 = builtin0("node:v8"); +for (const name of ["writeHeapSnapshot", "getHeapSnapshot", "queryObjects", "setFlagsFromString", "takeCoverage", "startCpuProfile", "startHeapProfile"]) { + if (typeof v8[name] === "function") v8[name] = deny; +} +builtin0("node:module").syncBuiltinESMExports(); +// Node emits on its own process (warnings among them): its listener calls are its own properties, not the shared +// prototype's, which downloaded code may change. +const emitter = builtin0("node:events").EventEmitter.prototype; +for (const name of Object.getOwnPropertyNames(emitter)) { + if (name !== "constructor" && typeof emitter[name] === "function") define(real, name, { value: emitter[name], writable: false, configurable: false }); +} +// Node reads properties its process does not have (noDeprecation, throwDeprecation, traceDeprecation, +// traceProcessWarnings, channel, connected, …): such a read goes up the prototype chain, where a getter would get the +// real process as its this (Opus E7: Buffer(1) warns, and the warning reads process.noDeprecation). Every prototype of +// the chain, Object.prototype included, takes no new property and keeps its own ones, so no getter and no other +// prototype gets in. Not frozen: Node assigns over prototype fields (\`this._events ??= {}\`). +for (let prototype = Object.getPrototypeOf(real); prototype !== null; prototype = Object.getPrototypeOf(prototype)) { + for (const key of Reflect.ownKeys(prototype)) define(prototype, key, { configurable: false }); + Object.preventExtensions(prototype); +} + +// The allowed builtins, by name with or without "node:"; looked up in an object of no prototype. +const allowed = Object.create(null); +for (const name of ${JSON.stringify(SANDBOX_BUILTINS)}) allowed[name] = true; +const builtin = (id) => { + if (typeof id !== "string") throw new Error("unavailable in the terminal's sandbox"); + const name = id.length > 5 && id[0] === "n" && id[1] === "o" && id[2] === "d" && id[3] === "e" && id[4] === ":" ? id.slice(5) : id; + if (typeof name !== "string" || allowed[name] !== true) throw new Error(id + ": unavailable in the terminal's sandbox"); + return builtin0("node:" + name); +}; + +// Frames out: the message's header as JSON, its write text or socket bytes as the body. +const send = (message) => { + const kind = message.kind; + let header; + let body; + if (kind === "write") { + const data = String(message.data); + // Refused before it is copied: a huge string costs no buffer of its size. + if (data.length > MAX_FRAME) throw new RangeError("the terminal's bridge takes frames of at most 17 MiB"); + header = { kind }; + body = BufferFrom(data); + } else if (kind === "socket_send") { + if (!isUint8Array(message.bytes)) throw new TypeError("socket_send: bytes expected"); + header = { kind, socket: message.socket }; + body = message.bytes; + } else { + header = message; + } + const json = BufferFrom(stringify(header)); + const length = 4 + json.length + (body === undefined ? 0 : body.length); + if (length > MAX_FRAME) throw new RangeError("the terminal's bridge takes frames of at most 17 MiB"); + const prefix = BufferAlloc(8); + writeUInt32BE(prefix, length, 0); + writeUInt32BE(prefix, json.length, 4); + writeOutput(output, BufferConcat(body === undefined ? [prefix, json] : [prefix, json, body])); +}; + +// The substitute process. +const stdout = freeze({ isTTY: false, write: (data) => (send({ kind: "write", data: String(data) }), true) }); +const listening = (name) => [name, function () { return fake; }]; +const fake = freeze({ + arch: real.arch, + platform: real.platform, + version: real.version, + versions: freeze({ ...real.versions }), + release: freeze({ ...real.release }), + nextTick: (...args) => apply(nextTick, real, args), + hrtime: Object.assign((...args) => apply(hrtime, real, args), { bigint: () => apply(hrtimeBigint, real, []) }), + uptime: () => apply(uptime, real, []), + exit: (code) => apply(exit, real, [code]), + permission: real.permission, + env: freeze({}), + argv: freeze(["node"]), + argv0: "node", + execArgv: freeze([]), + execPath: "node", + title: "node", + cwd: () => "/", + getBuiltinModule: builtin, + stdout, + stderr: freeze({ isTTY: false, write: () => true }), + ...Object.fromEntries(["on", "once", "off", "addListener", "removeListener", "removeAllListeners"].map(listening)), + emit: () => false, + emitWarning: () => {}, + listenerCount: () => 0, + listeners: () => [], +}); +define(globalThis, "process", { value: fake, writable: false, configurable: false, enumerable: false }); +for (const name of ["BroadcastChannel", "localStorage", "sessionStorage"]) delete globalThis[name]; + +// Downloaded code runs only through here, with options of no prototype (nothing read from Object.prototype). +const evaluate = (text, filename) => { + const options = Object.create(null); + options.filename = String(filename); + return runInThisContext(new Script(String(text), options)); +}; + +// Frames in: the first is the start (the bridge's start and the base bundle), the rest the launcher's messages. +let listener; +let started = false; +// Unread bytes: one buffer, grown by doubling, each byte copied in once. +let pending = BufferAlloc(64 * 1024); +let start = 0; +let end = 0; +const received = (payload) => { + const headerLength = readUInt32BE(payload, 0); + const header = parse(bufferText(payload, "utf8", 4, 4 + headerLength)); + const body = subarray(payload, 4 + headerLength); + if (!started) { + started = true; + run(header.start, bufferText(body, "utf8")); + return; + } + if (header.kind === "input") header.data = bufferText(body, "utf8"); + else if (header.kind === "socket_message" || header.kind === "attached") header.bytes = new Uint8Array(body); + if (listener !== undefined) listener(header); +}; +onInput(input, "data", (chunk) => { + if (end + chunk.length > pending.length) { + const unread = end - start; + const target = unread + chunk.length <= pending.length ? pending : BufferAlloc(Math.max(unread + chunk.length, pending.length * 2)); + copyBytes(pending, target, 0, start, end); + pending = target; + start = 0; + end = unread; + } + copyBytes(chunk, pending, end); + end += chunk.length; + while (end - start >= 4) { + const length = readUInt32BE(pending, start); + if (length > MAX_FRAME) apply(exit, real, [70]); + if (end - start < 4 + length) break; + // A copy: the buffer is reused, and what the frame carries is kept by the sandbox. + const payload = BufferAlloc(length); + copyBytes(pending, payload, 0, start + 4, start + 4 + length); + start += 4 + length; + received(payload); + } + if (start === end) start = end = 0; +}); +onInput(input, "end", () => apply(exit, real, [0])); +const run = (start, source) => { + globalThis.chordBridge = freeze({ + start: freeze(start), + send, + onMessage: (next) => { + listener = next; + }, + evaluate, + }); + try { + const base = { exports: {} }; + evaluate("(function (require, module, exports) {" + source + "\\n})", "chord-base.js")(builtin, base, base.exports); + } catch (error) { + send({ kind: "failed", message: String(error && error.message) }); + } +}; +`; diff --git a/packages/chord/src/next-react/README.md b/packages/chord/src/next-react/README.md new file mode 100644 index 000000000..11ee4c3af --- /dev/null +++ b/packages/chord/src/next-react/README.md @@ -0,0 +1,14 @@ +# react + +React bindings (slice 11 §3, decision "Slice 11, P7"): `@earendil-works/chord/next/react` once v2 has its entry point; +optional (React is an optional peer), non-React apps never load it. No components, no styling. + +- `HostProvider({ host })`, `useHost()`. +- `useHold(Type, params)`: a reference held while the component is mounted, released on unmount, held again when its + params change (compared by value); `undefined` on the first render. +- `useWatch(watchable)`: its current value (`undefined` while it has none), re-rendering on change + (`useSyncExternalStore` over `watchStore`). +- `useRegistry(R)`: the registry's composed value; `useRefState(ref)`: the reference's state. +- A component unmounted by a reload has not left (design §21): its releases are plain releases. + +Tests: `test/next-node/react/` (`stores.test.ts` in Node, `hooks.test.ts` in a real browser). diff --git a/packages/chord/src/next-react/hooks.ts b/packages/chord/src/next-react/hooks.ts new file mode 100644 index 000000000..a3731a920 --- /dev/null +++ b/packages/chord/src/next-react/hooks.ts @@ -0,0 +1,79 @@ +// React bindings (slice 11 §3, decision "Slice 11, P7"; p2's `useObject`, `useReadable`, `useRegistry`): a host in +// context, references held for a component's life, and watchables read with `useSyncExternalStore`. A component +// unmounted by a reload has not left (design §21): its releases are plain releases, presence is the app's. No +// components, no styling. + +import { + createContext, + createElement, + type ReactNode, + useContext, + useEffect, + useMemo, + useState, + useSyncExternalStore, +} from "react"; +import { BACKGROUND_CONTEXT, withAbortSignal } from "../context/index.ts"; +import type { Contract, ParamsOf } from "../next/contracts/index.ts"; +import type { Principal } from "../next/data/index.ts"; +import { canonicalJson } from "../next/data/principal.ts"; +import type { Held, Host, RefState } from "../next/host/index.ts"; +import type { Registry } from "../next/registries/index.ts"; +import type { NativeWatchable, Watchable } from "../watch.ts"; +import { watchStore } from "./stores.ts"; + +/** The host components use, given by `HostProvider`. */ +const HostContext = createContext | undefined>(undefined); + +/** Gives `host` to the components below. */ +export function HostProvider

    (props: { readonly host: Host

    ; readonly children?: ReactNode }) { + // Components hold for the app's principal type: the context keeps it as the host was given. + return createElement(HostContext.Provider, { value: props.host as unknown as Host }, props.children); +} + +/** The host `HostProvider` gives; throws outside one. */ +export function useHost

    (): Host

    { + const host = useContext(HostContext); + if (host === undefined) throw new Error("useHost: no HostProvider above this component"); + return host as unknown as Host

    ; +} + +/** + * A reference to `Type` with `params`, held while the component is mounted and released when it unmounts or its params + * change (compared by value); `undefined` before its first hold (the first render). + */ +export function useHold(Type: C, params: ParamsOf): Held | undefined { + const host = useHost(); + const key = canonicalJson(params as never, BACKGROUND_CONTEXT); + const [held, setHeld] = useState<{ readonly key: string; readonly ref: Held } | undefined>(undefined); + // biome-ignore lint/correctness/useExhaustiveDependencies: `key` is `params` by value: an inline object must not re-hold. + useEffect(() => { + const ref = host.hold(Type, params, BACKGROUND_CONTEXT); + setHeld({ key, ref }); + return () => ref.release(); + }, [host, Type, key]); + return held !== undefined && held.key === key ? held.ref : undefined; +} + +/** The current value of `watchable` (a state member, a derive), re-rendering on change; `undefined` while it has none. */ +export function useWatch(watchable: Watchable | NativeWatchable | undefined): T | undefined { + const store = useMemo(() => watchStore(watchable), [watchable]); + return useSyncExternalStore(store.subscribe, store.snapshot, store.snapshot); +} + +/** Registry `R`'s composed value on the component's host, re-rendering on change; `undefined` until it is usable. */ +export function useRegistry(R: Registry): V | undefined { + const host = useHost(); + const [value, setValue] = useState | undefined>(undefined); + useEffect(() => { + const abort = new AbortController(); + host.registry(R, withAbortSignal(abort.signal, BACKGROUND_CONTEXT)).then(setValue, () => {}); + return () => abort.abort(); + }, [host, R]); + return useWatch(value); +} + +/** The state of `ref` (loading, live, failed, ended), re-rendering on change; `undefined` without a reference. */ +export function useRefState(ref: { readonly state: NativeWatchable } | undefined): RefState | undefined { + return useWatch(ref?.state); +} diff --git a/packages/chord/src/next-react/index.ts b/packages/chord/src/next-react/index.ts new file mode 100644 index 000000000..03a78df79 --- /dev/null +++ b/packages/chord/src/next-react/index.ts @@ -0,0 +1,5 @@ +// Chord v2's React bindings (`@earendil-works/chord/next/react` once v2 has its entry point; slice 11 §3): optional, +// non-React apps never load them. + +export { HostProvider, useHold, useHost, useRefState, useRegistry, useWatch } from "./hooks.ts"; +export { type ExternalStore, watchStore } from "./stores.ts"; diff --git a/packages/chord/src/next-react/stores.ts b/packages/chord/src/next-react/stores.ts new file mode 100644 index 000000000..4b9e47e09 --- /dev/null +++ b/packages/chord/src/next-react/stores.ts @@ -0,0 +1,45 @@ +// External stores over Chord's watchables (slice 11 §3): what React's `useSyncExternalStore` reads. A store's snapshot +// is the watchable's current value (`undefined` while it has none: a reference not bound yet, ended, or a member its +// variant lacks; nothing throws during a render); subscribing starts a watch whose every call notifies React, and the +// unsubscribe ends it. + +import { BACKGROUND_CONTEXT, withAbortSignal } from "../context/index.ts"; +import type { NativeWatchable, Watchable } from "../watch.ts"; + +/** What `useSyncExternalStore` takes: subscribe (returning its unsubscribe) and the current snapshot. */ +export type ExternalStore = { + readonly subscribe: (changed: () => void) => () => void; + readonly snapshot: () => T | undefined; +}; + +/** The store of `source` (a state member, a derive, a registry value, `ref.state`); none for `undefined`. */ +export function watchStore(source: Watchable | NativeWatchable | undefined): ExternalStore { + return { + subscribe: (changed) => { + if (source === undefined) return () => {}; + const abort = new AbortController(); + const context = withAbortSignal(abort.signal, BACKGROUND_CONTEXT); + // A watch that cannot start (its source ended, the abort came first) only means no more changes. + source.watch(context).then( + (watch) => { + (watch.start as (listener: () => Promise) => void)(async () => { + changed(); + }); + // A watch starts at the value it acquired, which may be newer than the snapshot React read (a + // reference's members bind after the first render): its calls bring only later changes. + changed(); + }, + () => {}, + ); + return () => abort.abort(); + }, + snapshot: () => { + if (source === undefined) return undefined; + try { + return source.value; + } catch { + return undefined; + } + }, + }; +} diff --git a/packages/chord/src/next-tui/README.md b/packages/chord/src/next-tui/README.md new file mode 100644 index 000000000..31a5f5f8f --- /dev/null +++ b/packages/chord/src/next-tui/README.md @@ -0,0 +1,29 @@ +# tui (sandbox side) + +The terminal client's code inside the sandbox (slice 11 §4, decisions "Slice 11, P8" and "P9"): +`@earendil-works/chord/next/tui` once v2 has its entry point. The launcher is `next-node/tui`, the server side +`next-node/surfaces` (`tuiSurface`). + +```ts +// the app's terminal entry (its base bundle) +startTerminalRuntime

    ({ + modules: { "@acme/app/v3": api, "@acme/app/v3/tui": apiTui }, + core: [{ name: "@acme/tui", facet: tuiCore }], + render: ({ host, terminal, theme, copy, open, attach, exit }) => { + const scope = new TuiScope(host, () => redraw()); // hold, watch, registry; dispose + terminal.onInput((data) => …); terminal.write(frame); + theme.fg("accent", "Notes"); // colours by role + }, +}); +``` + +- `bridge.ts`: the messages between the sandbox and the launcher (input, resizes, relayed link sockets, requests with + replies: a plugin script, the server's base, `copy`, `open`; `outdated`, `exit`, `failed`). The bootstrap leaves the + bridge on `globalThis.chordBridge`; the runtime takes it off before any plugin runs. +- `channel.ts`: the sandbox's side: requests, `RelayLink` (a Chord link whose bytes the launcher relays), input and + resizes. `terminal.ts`: the bridged terminal (size, write, input, resize). `theme.ts`: the theme lookup (`fg`, `bg`, + `onChange`). `runtime.ts`: the client (`startClient` with `TuiScripts`; scripts run with `vm`; no wakes). +- `scope.ts`: `TuiScope(host, changed)`: `hold`, `watch` (a reader, `undefined` while no value), `registry`, `dispose` + (no `changed` after it). References wait and rebind by themselves; no terminal library dependency. + +Tests: `test/next/tui/` and, with the launcher and sandbox, `test/next-node/tui/`. diff --git a/packages/chord/src/next-tui/bridge.ts b/packages/chord/src/next-tui/bridge.ts new file mode 100644 index 000000000..8de935383 --- /dev/null +++ b/packages/chord/src/next-tui/bridge.ts @@ -0,0 +1,69 @@ +// The terminal's bridge (slice 11 §4, decision "Slice 11, P8"): the only way between the sandbox, where the app's +// terminal code and the plugins run, and the trusted launcher outside it. The launcher sends input, resizes and the +// relayed bytes of the Chord link; the sandbox sends frames for the terminal, opens and feeds link sockets through the +// launcher, asks for plugin scripts and the server's base hash, and asks for actions on the machine (copy, open a URL, +// attach a file the user names), which the launcher confirms with the user. Messages are plain data, framed over the +// sandbox's pipes (`next-node/tui/pipe.ts`); requests carry an id their reply has. + +/** What the launcher tells the sandbox at its start. */ +export type BridgeStart = { + /** The base bundle's hash (the sandbox runs it). */ + readonly base: string; + /** The text of the combined plugin scripts at start. */ + readonly combined: string; + readonly columns: number; + readonly rows: number; + /** How many colours the terminal shows. */ + readonly colors: "truecolor" | "256"; + /** The terminal's background: which of the user's two themes applies. */ + readonly mode: "dark" | "light"; +}; + +/** A request of the sandbox the launcher answers. */ +export type BridgeRequest = + | { readonly kind: "fetch"; readonly path: string } + | { readonly kind: "base" } + | { readonly kind: "copy"; readonly text: string } + | { readonly kind: "open"; readonly url: string } + /** A file of the machine the user names (`path`): its name and bytes once confirmed, else `null`. */ + | { readonly kind: "attach"; readonly path: string }; + +/** What the sandbox sends. */ +export type SandboxMessage = + | { readonly kind: "write"; readonly data: string } + | { readonly kind: "socket_open"; readonly socket: number } + | { readonly kind: "socket_send"; readonly socket: number; readonly bytes: Uint8Array } + | { readonly kind: "socket_close"; readonly socket: number } + | { readonly kind: "request"; readonly id: number; readonly request: BridgeRequest } + /** The sandbox's code is outdated (a new base bundle): the launcher starts a new one. */ + | { readonly kind: "outdated" } + /** The base bundle failed to start (its message; the launcher shows none of the sandbox's own output). */ + | { readonly kind: "failed"; readonly message: string } + | { readonly kind: "exit"; readonly code: number }; + +/** What the launcher sends. */ +export type LauncherMessage = + | { readonly kind: "input"; readonly data: string } + | { readonly kind: "resize"; readonly columns: number; readonly rows: number } + | { readonly kind: "socket_opened"; readonly socket: number } + | { readonly kind: "socket_message"; readonly socket: number; readonly bytes: Uint8Array } + | { readonly kind: "socket_closed"; readonly socket: number } + | { readonly kind: "reply"; readonly id: number; readonly value: unknown } + /** An attach request's answer: the file the user confirmed. */ + | { readonly kind: "attached"; readonly id: number; readonly name: string; readonly bytes: Uint8Array } + | { readonly kind: "refused"; readonly id: number; readonly message: string }; + +/** The bridge as the sandbox's code gets it from the bootstrap (once: it is taken off `globalThis`). */ +export type Bridge = { + readonly start: BridgeStart; + send(message: SandboxMessage): void; + onMessage(listener: (message: LauncherMessage) => void): void; + /** + * Runs `source` (registered plugin scripts) in the sandbox's context, as the bootstrap ran the base bundle: with no + * dynamic import callback (`import()` throws). + */ + evaluate(source: string, filename: string): void; +}; + +/** The global the sandbox's bootstrap leaves the bridge on, for the base bundle's runtime to take. */ +export const BRIDGE = "chordBridge"; diff --git a/packages/chord/src/next-tui/channel.ts b/packages/chord/src/next-tui/channel.ts new file mode 100644 index 000000000..d7ed7755c --- /dev/null +++ b/packages/chord/src/next-tui/channel.ts @@ -0,0 +1,163 @@ +// The sandbox's side of the bridge (slice 11 §4): one dispatcher of the launcher's messages (input, resizes, link +// sockets, replies), requests with their replies, and Chord links whose bytes the launcher relays to the server. + +import { LinkEnd } from "../next/link/link.ts"; +import type { Bridge, BridgeRequest, BridgeStart, LauncherMessage } from "./bridge.ts"; +import { splitFrames } from "./frames.ts"; + +/** What one relayed socket reports: it opened, a frame arrived. */ +export type RelayEvents = { opened(): void; received(): void }; + +/** The sandbox's bridge: requests, relayed sockets, and the terminal's input and resizes. */ +export class SandboxChannel { + readonly #bridge: Bridge; + readonly #replies = new Map(); + readonly #sockets = new Map(); + readonly #input = new Set<(data: string) => void>(); + readonly #resize = new Set<(columns: number, rows: number) => void>(); + #next = 0; + + constructor(bridge: Bridge) { + this.#bridge = bridge; + bridge.onMessage((message) => this.#dispatch(message)); + } + + /** What the launcher told at the start. */ + get start(): BridgeStart { + return this.#bridge.start; + } + + /** Asks the launcher; rejects with its refusal. */ + request(request: BridgeRequest): Promise { + const id = ++this.#next; + return new Promise((resolve, reject) => { + this.#replies.set(id, { resolve, reject }); + this.#bridge.send({ kind: "request", id, request }); + }); + } + + /** Runs the registered scripts `text` (named `filename` in stacks) through the sandbox's own evaluation. */ + evaluate(text: string, filename: string): void { + this.#bridge.evaluate(text, filename); + } + + /** Writes `data` to the terminal, in frames the launcher takes (`splitFrames`). */ + write(data: string): void { + for (const frame of splitFrames(data)) this.#bridge.send({ kind: "write", data: frame }); + } + + /** Tells the launcher this sandbox's code is outdated. */ + outdated(): void { + this.#bridge.send({ kind: "outdated" }); + } + + /** Ends the terminal with `code`. */ + exit(code: number): void { + this.#bridge.send({ kind: "exit", code }); + } + + /** Calls `listener` with each input (keys, pastes) until the returned removal. */ + onInput(listener: (data: string) => void): () => void { + this.#input.add(listener); + return () => this.#input.delete(listener); + } + + /** Calls `listener` at each resize until the returned removal. */ + onResize(listener: (columns: number, rows: number) => void): () => void { + this.#resize.add(listener); + return () => this.#resize.delete(listener); + } + + /** Opens a socket to the server's links endpoint through the launcher: its link, reporting to `events`. */ + openSocket(events: RelayEvents): RelayLink { + const socket = ++this.#next; + const link = new RelayLink(socket, events, this.#bridge, () => this.#sockets.delete(socket)); + this.#sockets.set(socket, link); + this.#bridge.send({ kind: "socket_open", socket }); + return link; + } + + /** One message of the launcher, to whom it is for. */ + #dispatch(message: LauncherMessage): void { + switch (message.kind) { + case "input": + for (const listener of this.#input) listener(message.data); + return; + case "resize": + for (const listener of this.#resize) listener(message.columns, message.rows); + return; + case "socket_opened": + this.#sockets.get(message.socket)?.opened(); + return; + case "socket_message": + this.#sockets.get(message.socket)?.received(message.bytes); + return; + case "socket_closed": + this.#sockets.get(message.socket)?.dropped(); + return; + case "reply": + case "attached": + case "refused": + this.#settle(message); + } + } + + /** Settles the request a reply or refusal answers. */ + #settle(message: Extract): void { + const waiting = this.#replies.get(message.id); + this.#replies.delete(message.id); + if (message.kind === "reply") waiting?.resolve(message.value); + else if (message.kind === "attached") waiting?.resolve({ name: message.name, bytes: message.bytes }); + else waiting?.reject(new Error(message.message)); + } +} + +/** A Chord link over a socket the launcher holds: bytes relayed both ways, buffered until the socket opened. */ +export class RelayLink extends LinkEnd { + readonly #socket: number; + readonly #events: RelayEvents; + readonly #bridge: Bridge; + readonly #forget: () => void; + /** Messages sent while the socket connects; `undefined` once open. */ + #connecting: Uint8Array[] | undefined = []; + + constructor(socket: number, events: RelayEvents, bridge: Bridge, forget: () => void) { + super(); + this.#socket = socket; + this.#events = events; + this.#bridge = bridge; + this.#forget = forget; + } + + send(message: Uint8Array): void { + if (this.closed) return; + if (this.#connecting !== undefined) this.#connecting.push(message); + else this.#bridge.send({ kind: "socket_send", socket: this.#socket, bytes: message }); + } + + close(): void { + if (this.closed) return; + this.#bridge.send({ kind: "socket_close", socket: this.#socket }); + this.dropped(); + } + + /** The launcher's socket opened: what was sent meanwhile goes out. */ + opened(): void { + const queued = this.#connecting ?? []; + this.#connecting = undefined; + this.#events.opened(); + for (const message of queued) this.send(message); + } + + /** A frame arrived. */ + received(bytes: Uint8Array): void { + this.#events.received(); + this.deliver(bytes); + } + + /** The socket closed (either side): the link ends. */ + dropped(): void { + this.#forget(); + this.ended(); + } +} diff --git a/packages/chord/src/next-tui/frames.ts b/packages/chord/src/next-tui/frames.ts new file mode 100644 index 000000000..29f3d5d91 --- /dev/null +++ b/packages/chord/src/next-tui/frames.ts @@ -0,0 +1,51 @@ +// Frames for the terminal (slice 11 §4; fix round 1, Fable L9): the launcher drops a frame over 64 KiB whole (an escape +// it could hide is never written in parts), so the sandbox splits a large write into frames, each cut only between +// escape sequences and never inside a surrogate pair: a first paint of a large true-colour screen arrives whole. + +/** The most UTF-16 units of one frame: at most 3 UTF-8 bytes each, so under the launcher's 64 KiB. */ +export const FRAME_UNITS = 16 * 1024; + +/** `data` as frames of at most `FRAME_UNITS` units, cut outside escape sequences (one longer is its own frame). */ +export function splitFrames(data: string): string[] { + if (data.length <= FRAME_UNITS) return [data]; + const frames: string[] = []; + let start = 0; + // The last place a frame may end: after a character or a whole sequence, never inside one. + let safe = 0; + let index = 0; + while (index < data.length) { + const end = data.charCodeAt(index) === 27 ? sequenceEnd(data, index) : unitEnd(data, index); + if (end - start > FRAME_UNITS && safe > start) { + frames.push(data.slice(start, safe)); + start = safe; + } + index = end; + safe = end; + } + frames.push(data.slice(start)); + return frames; +} + +/** The index after the character at `index`, a surrogate pair kept together. */ +function unitEnd(data: string, index: number): number { + const code = data.charCodeAt(index); + return code >= 0xd800 && code <= 0xdbff && index + 1 < data.length ? index + 2 : index + 1; +} + +/** The index after the escape sequence at `index` (CSI to its final; OSC, DCS and the like to their terminator). */ +function sequenceEnd(data: string, index: number): number { + const introducer = data[index + 1]; + if (introducer === "[") { + let at = index + 2; + while (at < data.length && data.charCodeAt(at) >= 0x20 && data.charCodeAt(at) <= 0x3f) at++; + return Math.min(at + 1, data.length); + } + if (introducer !== undefined && "]PX^_".includes(introducer)) { + for (let at = index + 2; at < data.length; at++) { + if (data.charCodeAt(at) === 7) return at + 1; + if (data[at] === "\x1b" && data[at + 1] === "\\") return at + 2; + } + return data.length; + } + return Math.min(index + 2, data.length); +} diff --git a/packages/chord/src/next-tui/index.ts b/packages/chord/src/next-tui/index.ts new file mode 100644 index 000000000..fd5db3903 --- /dev/null +++ b/packages/chord/src/next-tui/index.ts @@ -0,0 +1,14 @@ +// Chord v2's terminal surface, the sandbox side (`@earendil-works/chord/next/tui` once v2 has its entry point; slice +// 11 §4): `startTerminalRuntime` in the app's terminal entry, the bridged terminal, the theme lookup and `TuiScope`. +// The launcher and the server side are `next/tui/launcher` and `next/web/server`'s `tuiSurface`. + +export type { Bridge, BridgeRequest, BridgeStart, LauncherMessage, SandboxMessage } from "./bridge.ts"; +export { + type AttachedFile, + startTerminalRuntime, + type TerminalRuntime, + type TerminalRuntimeOptions, +} from "./runtime.ts"; +export { TuiScope } from "./scope.ts"; +export { BridgeTerminal } from "./terminal.ts"; +export { TerminalTheme } from "./theme.ts"; diff --git a/packages/chord/src/next-tui/runtime.ts b/packages/chord/src/next-tui/runtime.ts new file mode 100644 index 000000000..d89e34f7c --- /dev/null +++ b/packages/chord/src/next-tui/runtime.ts @@ -0,0 +1,155 @@ +// `startTerminalRuntime` (slice 11 §4, decisions "Slice 11, P8" and "P9"): the terminal client inside the sandbox, in +// the app's terminal entry (the base bundle the launcher downloaded). A client host like a page's: the plugin scripts +// the launcher brought, loaded with the app's core facets and the kit's themes; the link to the server relayed by the +// launcher, kept by the same connection loop; `TuiScripts` followed to apply new plugin scripts in place; a new base +// bundle hands over to the launcher, which starts the new one. The app gets the host, the bridged terminal, the theme +// lookup and the launcher's actions. + +import { BACKGROUND_CONTEXT } from "../context/index.ts"; +import type { Principal } from "../next/data/index.ts"; +import type { PluginFacet } from "../next/host/index.ts"; +import { + type Client, + type ClientError, + type ClientModules, + type ConnectionState, + FALLBACK_THEMES, + type FacetFunction, + followActiveTheme, + REGISTER, + type RegisteredScript, + startClient, + TuiScripts, + themesFacet, +} from "../next/surfaces/index.ts"; +import { BRIDGE, type Bridge } from "./bridge.ts"; +import { SandboxChannel } from "./channel.ts"; +import { BridgeTerminal } from "./terminal.ts"; +import { TerminalTheme } from "./theme.ts"; + +/** What `startTerminalRuntime` takes. */ +export type TerminalRuntimeOptions

    = { + /** The modules the terminal plugins import, by specifier: the app's API modules for the kind, its components. */ + readonly modules: ClientModules; + /** The app's own terminal facets. */ + readonly core?: readonly PluginFacet

    []; + readonly canRetry?: () => Promise; + readonly onConnection?: (state: ConnectionState) => void; + /** Draws the app, once its plugins are loaded. */ + readonly render: (runtime: TerminalRuntime

    ) => void; + /** Plugins left out or kept old, a failed update, the host's errors; dropped by default (stderr is the launcher's). */ + readonly onError?: (error: ClientError) => void; +}; + +/** A file the launcher read for the sandbox: its name (the path's last segment) and its bytes. */ +export type AttachedFile = { readonly name: string; readonly bytes: Uint8Array }; + +/** What the app's terminal code gets. */ +export type TerminalRuntime

    = { + readonly host: Client

    ["host"]; + readonly terminal: BridgeTerminal; + readonly theme: TerminalTheme; + /** Asks the launcher to copy `text` to the machine's clipboard; it asks the user. Whether it was done. */ + copy(text: string): Promise; + /** Asks the launcher to open `url` (http or https) in a browser; it asks the user. Whether it was done. */ + open(url: string): Promise; + /** + * Asks the launcher for the file at `path` on the machine (at most 16 MiB); it asks the user. Its name and bytes, or + * `undefined` when declined. + */ + attach(path: string): Promise; + /** Ends the terminal client with `code`. */ + exit(code: number): void; +}; + +/** Starts the terminal client inside the sandbox; resolves with its runtime once rendered. */ +export async function startTerminalRuntime

    ( + options: TerminalRuntimeOptions

    , +): Promise> { + const channel = new SandboxChannel(takeBridge()); + const { start } = channel; + const scripts = new SandboxScripts(channel); + const core: PluginFacet

    [] = [{ name: "chord.themes", facet: themesFacet

    () }, ...(options.core ?? [])]; + const client = await startClient

    ({ + scripts: TuiScripts, + base: start.base, + initial: scripts.run(start.combined), + modules: options.modules, + core, + open: (events) => channel.openSocket(events), + fetchScript: async (entry) => { + const text = (await channel.request({ kind: "fetch", path: entry.url })) as string; + const registered = scripts + .run(text) + .find(({ plugin, digest }) => plugin === entry.name && digest === entry.digest); + if (registered === undefined) throw new Error(`plugin "${entry.name}": ${entry.url} did not register`); + return registered; + }, + serverBase: () => + channel.request({ kind: "base" }).then( + (base) => base as string, + () => undefined, + ), + outdated: () => channel.outdated(), + ...(options.canRetry === undefined ? {} : { canRetry: options.canRetry }), + ...(options.onConnection === undefined ? {} : { onConnection: options.onConnection }), + environment: TERMINAL_ENVIRONMENT, + onError: options.onError ?? (() => {}), + }); + const theme = new TerminalTheme(FALLBACK_THEMES[start.mode], start.colors); + await followActiveTheme(client.host, start.mode, (active) => theme.set(active), BACKGROUND_CONTEXT); + const runtime: TerminalRuntime

    = { + host: client.host, + terminal: new BridgeTerminal(channel), + theme, + copy: (text) => channel.request({ kind: "copy", text }).then(Boolean), + open: (url) => channel.request({ kind: "open", url }).then(Boolean), + // The launcher's reply: its own data, `null` when declined. + attach: (path) => + channel.request({ kind: "attach", path }).then((file) => (file as AttachedFile | null) ?? undefined), + exit: (code) => channel.exit(code), + }; + options.render(runtime); + return runtime; +} + +/** The bridge the bootstrap left on `globalThis`, taken off it (plugin code never finds it there). */ +function takeBridge(): Bridge { + const scope = globalThis as Record; + const bridge = scope[BRIDGE] as Bridge | undefined; + if (bridge === undefined) throw new Error("startTerminalRuntime: not inside the terminal's sandbox (no bridge)"); + delete scope[BRIDGE]; + return bridge; +} + +/** A terminal has no page wakes: the loop retries on its backoff alone. */ +const TERMINAL_ENVIRONMENT = { + now: () => Date.now(), + after: (ms: number, callback: () => void) => { + const timer = setTimeout(callback, ms); + return () => clearTimeout(timer); + }, + online: () => true, + onWake: () => () => {}, +}; + +/** Runs registered scripts' text in the sandbox and collects what registered (`chordFacet`). */ +class SandboxScripts { + readonly #channel: SandboxChannel; + /** The scripts the text being run registers. */ + #collected: RegisteredScript[] = []; + + constructor(channel: SandboxChannel) { + this.#channel = channel; + (globalThis as Record)[REGISTER] = (plugin: string, digest: string, run: FacetFunction) => { + this.#collected.push({ plugin, digest, run }); + }; + } + + /** Runs `text` (one or more registered scripts): what it registered. */ + run(text: string): RegisteredScript[] { + this.#collected = []; + this.#channel.evaluate(text, "chord-plugins.js"); + return this.#collected; + } +} diff --git a/packages/chord/src/next-tui/scope.ts b/packages/chord/src/next-tui/scope.ts new file mode 100644 index 000000000..8302e4d02 --- /dev/null +++ b/packages/chord/src/next-tui/scope.ts @@ -0,0 +1,81 @@ +// `TuiScope` (slice 11 §4, decision "Slice 11, P9"; p2's `TuiScope`): a terminal component's Chord resources, the +// counterpart of the React bindings for terminal UI libraries. It holds references, follows watchables and registries, +// calls one `changed` callback (the app's redraw) on every change, and releases everything at `dispose`. References +// wait and rebind by themselves (no retry loop, unlike p2's); no terminal library dependency. + +import { BACKGROUND_CONTEXT, withAbortSignal } from "../context/index.ts"; +import type { Contract, ParamsOf } from "../next/contracts/index.ts"; +import type { Principal } from "../next/data/index.ts"; +import type { Held, Host } from "../next/host/index.ts"; +import type { Registry } from "../next/registries/index.ts"; +import type { NativeWatchable, Watchable } from "../watch.ts"; + +/** A component's scope: what it holds and follows, until `dispose`. */ +export class TuiScope

    { + readonly #host: Host

    ; + readonly #changed: () => void; + readonly #abort = new AbortController(); + readonly #releases: (() => void)[] = []; + + constructor(host: Host

    , changed: () => void) { + this.#host = host; + this.#changed = changed; + } + + /** A reference to `Type` with `params`, held until `dispose`; its state changes call `changed`. */ + hold(Type: C, params: ParamsOf): Held { + const held = this.#host.hold(Type, params, this.#context()); + this.#releases.push(() => held.release()); + this.watch(held.state); + return held; + } + + /** + * Follows `source` (a member, a derive, `ref.state`) until `dispose`: its changes call `changed`. Returns a reader of + * its current value (`undefined` while it has none: nothing throws in a redraw). + */ + watch(source: Watchable | NativeWatchable): () => T | undefined { + source.watch(this.#context()).then( + (watch) => { + (watch.start as (listener: () => Promise) => void)(async () => { + if (!this.#abort.signal.aborted) this.#changed(); + }); + // The value it acquired may be newer than what was drawn (a reference bound since): draw it. + if (!this.#abort.signal.aborted) this.#changed(); + }, + // Disposed (or the source ended) before the watch started: nothing to follow. + () => {}, + ); + return () => { + try { + return source.value; + } catch { + return undefined; + } + }; + } + + /** Registry `R`'s composed value, followed until `dispose`; a reader, `undefined` until it is usable. */ + registry(R: Registry): () => V | undefined { + let read: () => V | undefined = () => undefined; + this.#host.registry(R, this.#context()).then( + (value) => { + read = this.watch(value); + if (!this.#abort.signal.aborted) this.#changed(); + }, + () => {}, + ); + return () => read(); + } + + /** Releases every reference and ends every watch; `changed` is not called after it. */ + dispose(): void { + this.#abort.abort(); + for (const release of this.#releases.splice(0)) release(); + } + + /** The context of the scope's work: aborted at `dispose`. */ + #context() { + return withAbortSignal(this.#abort.signal, BACKGROUND_CONTEXT); + } +} diff --git a/packages/chord/src/next-tui/terminal.ts b/packages/chord/src/next-tui/terminal.ts new file mode 100644 index 000000000..a29723cf4 --- /dev/null +++ b/packages/chord/src/next-tui/terminal.ts @@ -0,0 +1,45 @@ +// The terminal as the app's terminal code sees it inside the sandbox (slice 11 §4): its size, writing frames, and +// input and resizes, all through the bridge; the launcher holds the real terminal. Library-agnostic: an app adapts it +// to its terminal UI library (pi-tui's `Terminal` in a few lines). + +import type { SandboxChannel } from "./channel.ts"; + +/** The bridged terminal. */ +export class BridgeTerminal { + readonly #channel: SandboxChannel; + #columns: number; + #rows: number; + + constructor(channel: SandboxChannel) { + this.#channel = channel; + this.#columns = channel.start.columns; + this.#rows = channel.start.rows; + channel.onResize((columns, rows) => { + this.#columns = columns; + this.#rows = rows; + }); + } + + get columns(): number { + return this.#columns; + } + + get rows(): number { + return this.#rows; + } + + /** Writes `data` (text and escape sequences) to the terminal. */ + write(data: string): void { + this.#channel.write(data); + } + + /** Calls `listener` with each input (keys, pastes, as the terminal sends them) until the returned removal. */ + onInput(listener: (data: string) => void): () => void { + return this.#channel.onInput(listener); + } + + /** Calls `listener` after each resize (the new size read from `columns` and `rows`) until the returned removal. */ + onResize(listener: () => void): () => void { + return this.#channel.onResize(() => listener()); + } +} diff --git a/packages/chord/src/next-tui/theme.ts b/packages/chord/src/next-tui/theme.ts new file mode 100644 index 000000000..724356e3d --- /dev/null +++ b/packages/chord/src/next-tui/theme.ts @@ -0,0 +1,44 @@ +// The terminal's theme lookup (slice 11 §4b, decision "Looks and themes"): colours by role from the active theme, as +// the terminal shows them (true colour, else the nearest of 256; a theme's `tui` overrides first). A plugin asks for a +// role (`theme.fg("accent", "Notes")`), never writes raw colours; a new theme applies at the next redraw. + +import { type ColorDepth, type Theme, terminalColor } from "../next/surfaces/index.ts"; + +/** Colours by role for the terminal. */ +export class TerminalTheme { + readonly #depth: ColorDepth; + #theme: Theme; + readonly #listeners = new Set<() => void>(); + + constructor(theme: Theme, depth: ColorDepth) { + this.#theme = theme; + this.#depth = depth; + } + + /** The active theme. */ + get theme(): Theme { + return this.#theme; + } + + /** `text` in `role`'s colour (the default colour after it). */ + fg(role: string, text: string): string { + return `\x1b[${terminalColor(this.#theme, role, this.#depth, "fg")}m${text}\x1b[39m`; + } + + /** `text` on `role`'s colour as its background. */ + bg(role: string, text: string): string { + return `\x1b[${terminalColor(this.#theme, role, this.#depth, "bg")}m${text}\x1b[49m`; + } + + /** Calls `listener` when the active theme changes (to redraw) until the returned removal. */ + onChange(listener: () => void): () => void { + this.#listeners.add(listener); + return () => this.#listeners.delete(listener); + } + + /** The active theme is now `theme`. */ + set(theme: Theme): void { + this.#theme = theme; + for (const listener of this.#listeners) listener(); + } +} diff --git a/packages/chord/src/next-web/README.md b/packages/chord/src/next-web/README.md new file mode 100644 index 000000000..0c946b782 --- /dev/null +++ b/packages/chord/src/next-web/README.md @@ -0,0 +1,25 @@ +# web (browser side) + +`startPage` (slice 11 §2) for the app's web entry, the base bundle the server builds: `@earendil-works/chord/next/web` +once v2 has its entry point. The server side is `next-node/surfaces` (`webSurface`). + +```ts +startPage

    ({ + connect: () => new WebSocket(`${location.origin.replace(/^http/, "ws")}/chord`), + modules: { "@acme/app/v3": api, "@acme/app/v3/web": apiWeb, react: React, "react/jsx-runtime": jsx }, + core: [{ name: "@acme/web", facet: webCore }], + canRetry: async () => true, // false: stop reconnecting (a guest's ended share) + onConnection: (state) => setBanner(state), // connecting | connected | waiting + render: (host) => createRoot(root).render(), +}); +``` + +- The page's inline bootstrap (`PageAssets.bootstrap`, the response's nonce) defines `chordFacet` before the combined + file runs; its scripts queue, and `startPage` takes them (`scripts.ts`). Later scripts load with ``; + +export default defineFacet((host, self) => { + host.serve(Channels, () => ({ variant: "main", members: { names: constant(CHANNELS) } })); + self.contribute(Http, { + routes: { + "GET /": { public: true, handler: async (request) => ({ kind: "html", body: signIn(request.nonce) }) }, + "GET /files/:channel/:id": async (request, ctx) => { + const room = await ctx.get(Room, { id: request.params.channel! }); + const file = room.messages.value.find((message) => message.file?.id === request.params.id)?.file; + if (file === undefined) return { kind: "error", status: 404 }; + return { + kind: "bytes", + mime: file.mime, + size: file.size, + inline: file.mime.startsWith("image/") ? true : undefined, + download: file.mime.startsWith("image/") ? undefined : file.name, + read: (offset, length) => + room.read({ id: file.id, offset, length: Math.min(length, 1024 * 1024) }, BACKGROUND_CONTEXT), + }; + }, + }, + }); +}); diff --git a/packages/chord/test/next/examples/59-chat/plugins/chat/worker.ts b/packages/chord/test/next/examples/59-chat/plugins/chat/worker.ts new file mode 100644 index 000000000..7f78cf8cb --- /dev/null +++ b/packages/chord/test/next/examples/59-chat/plugins/chat/worker.ts @@ -0,0 +1,78 @@ +// The chat plugin on a channel's worker: the room's messages in a document of the worker's store (kept by a local +// object: a persisted state needs a stable identity), and each principal's `Room` posting as them and sharing files +// through the kit's `Files` of this worker. + +import { + Attachment, + arr, + defineFacet, + defineState, + local, + Message, + method, + num, + obj, + opt, + Room, + str, +} from "@chat/app/v1"; +import { Files } from "@chat/app/v1/worker"; + +/** The newest messages a channel keeps. */ +const KEPT = 500; + +const Log = defineState("chat.log", { + schema: obj({ next: num(), messages: arr(Message) }), + version: 1, + initial: () => ({ next: 1, messages: [] }), + persist: true, + checkpointWhen: (_value, _ops, { deltasSinceBase }) => deltasSinceBase >= 50, +}); + +const Store = local("chat.store", { + version: 1, + params: {}, + members: { messages: arr(Message), add: method(obj({ author: str(), text: str(), file: opt(Attachment) })) }, +}); + +export default defineFacet((host) => { + host.serve(Store, (_params, self) => { + const log = self.state(Log); + return { + messages: self.derive([log], ([value]) => value.messages), + add: async ({ author, text, file }, ctx) => { + await log.change((draft) => { + draft.messages.push({ + id: draft.next++, + author, + text, + at: Date.now(), + ...(file === undefined ? {} : { file }), + }); + if (draft.messages.length > KEPT) draft.messages.splice(0, draft.messages.length - KEPT); + }, ctx); + return null; + }, + }; + }); + host.serve(Room, (_params, principal, self) => { + const store = self.get(Store, {}); + const files = self.get(Files, {}); + const author = (principal as { readonly name?: string }).name ?? principal.kind; + return { + variant: "main", + members: { + messages: self.derive([store.messages], ([messages]) => messages), + post: (text, ctx) => store.add({ author, text }, ctx), + begin: (file, ctx) => files.begin(file, ctx), + append: (chunk, ctx) => files.append(chunk, ctx), + share: async ({ upload, text }, ctx) => { + const id = await files.finish(upload, ctx); + const { name, mime, size } = files.list.value.find((file) => file.id === id)!; + return store.add({ author, text, file: { id, name, mime, size } }, ctx); + }, + read: (range, ctx) => files.read(range, ctx), + }, + }; + }); +}); diff --git a/packages/chord/test/next/examples/59-chat/plugins/login.server.ts b/packages/chord/test/next/examples/59-chat/plugins/login.server.ts new file mode 100644 index 000000000..c6a5a9779 --- /dev/null +++ b/packages/chord/test/next/examples/59-chat/plugins/login.server.ts @@ -0,0 +1,5 @@ +// The kit's pick-a-name login, added to the chat as a one-line plugin. + +import { login } from "@chat/app/v1/server"; + +export default login; diff --git a/packages/chord/test/next/examples/59-chat/start.ts b/packages/chord/test/next/examples/59-chat/start.ts new file mode 100644 index 000000000..304449280 --- /dev/null +++ b/packages/chord/test/next/examples/59-chat/start.ts @@ -0,0 +1,76 @@ +// The chat's start: one server keeping the channel list and the surfaces, one worker per channel, a web page at `/chat` +// (the sign-in page at `/` comes from the chat plugin) and a terminal client. No harness: the app is its API, two +// plugins and its two client entries. + +import { cp, mkdtemp } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import type { Context } from "../../../../src/context/index.ts"; +import { type App, startServer } from "../../../../src/next-node/app/index.ts"; +import { type PageAssets, pageScripts } from "../../../../src/next-node/surfaces/index.ts"; +import type { ChatPrincipal } from "./app/principals.ts"; +import * as v1server from "./app/v1/server.ts"; +import * as v1worker from "./app/v1/worker.ts"; +import * as v1 from "./app/v1.ts"; + +/** The chat's API, as the server and the channel workers have it. */ +export const API = { + name: "@chat/app", + versions: ["v1"], + modules: { "@chat/app/v1": v1, "@chat/app/v1/server": v1server, "@chat/app/v1/worker": v1worker }, +}; + +/** The chat's facet kinds. */ +export const KINDS = { + server: { platform: "node" }, + worker: { platform: "node" }, + web: { platform: "browser", packages: { react: {}, "react/jsx-runtime": {} } }, + tui: { platform: "node" }, +} as const; + +/** The chat page: the app's template around the surface's assets. */ +function page(request: { readonly nonce: string }, assets: PageAssets): string { + return `Chat${pageScripts(assets, request.nonce)}

    `; +} + +/** A data directory and a copy of the chat's plugins, in a new temporary directory. */ +export async function chatDirectories(): Promise<{ + readonly base: string; + readonly dataDir: string; + readonly root: string; +}> { + const base = await mkdtemp(join(tmpdir(), "chat-")); + const root = join(base, "plugins"); + await cp(fileURLToPath(new URL("./plugins/", import.meta.url)), root, { recursive: true }); + return { base, dataDir: join(base, "data"), root }; +} + +/** Starts the chat on a free loopback port. */ +export function startChat(dataDir: string, root: string, context: Context): Promise> { + return startServer( + { + name: "chat", + dataDir, + root, + listen: { host: "127.0.0.1", port: 0 }, + api: API, + kinds: KINDS, + principal: { server: { kind: "server" }, worker: (address) => ({ kind: "worker", channel: address.key }) }, + workers: [ + { + kind: v1.Channel, + entry: fileURLToPath(new URL("./worker.ts", import.meta.url)), + execArgv: ["--conditions=source", "--experimental-strip-types", "--no-warnings"], + start: () => ({}), + // Anyone signed in may join a channel. + authorize: (principal) => principal.kind === "user" || principal.kind === "owner", + }, + ], + web: { entry: fileURLToPath(new URL("./web/main.ts", import.meta.url)), page, paths: ["/chat"] }, + tui: { entry: fileURLToPath(new URL("./tui/main.ts", import.meta.url)) }, + onError: (error) => console.error("chat:", error.kind, error.error), + }, + context, + ); +} diff --git a/packages/chord/test/next/examples/59-chat/tui/main.ts b/packages/chord/test/next/examples/59-chat/tui/main.ts new file mode 100644 index 000000000..b3dd20f52 --- /dev/null +++ b/packages/chord/test/next/examples/59-chat/tui/main.ts @@ -0,0 +1,90 @@ +// The chat in a terminal (the terminal base bundle, run in the launcher's sandbox): the channels (Tab switches), the +// current room's messages (an image as a placeholder, other files as their link), and a line to type into. Enter +// posts; `/attach ` asks the launcher for a file of this machine (it asks the user) and shares it in chunks; +// Ctrl-C quits. + +import type { Held } from "../../../../../src/next/index.ts"; +import { type AttachedFile, startTerminalRuntime, TuiScope } from "../../../../../src/next-tui/index.ts"; +import * as v1 from "../app/v1.ts"; + +const { BACKGROUND_CONTEXT: context, Channels, Room } = v1; + +/** The size of one upload chunk. */ +const CHUNK = 256 * 1024; + +/** Shares `file` in `room`, in chunks. */ +async function share(room: Held, file: AttachedFile): Promise { + const mime = /\.(png|jpe?g|gif|webp)$/i.test(file.name) + ? `image/${file.name.split(".").pop()!.toLowerCase()}` + : "application/octet-stream"; + const id = await room.begin({ name: file.name, mime }, context); + for (let offset = 0; offset < file.bytes.length; offset += CHUNK) { + await room.append({ upload: id, offset, bytes: file.bytes.subarray(offset, offset + CHUNK) }, context); + } + await room.share({ upload: id, text: "" }, context); +} + +/** One message as a line. */ +function line(message: v1.Message, channel: string, theme: { fg(role: string, text: string): string }): string { + const { file } = message; + const attached = + file === undefined + ? "" + : file.mime.startsWith("image/") + ? ` ${theme.fg("muted", `[image: ${file.name}]`)}` + : ` ${theme.fg("accent", `[${file.name}] /files/${channel}/${file.id}`)}`; + return `${theme.fg("accent", message.author)} ${message.text}${attached}`; +} + +void startTerminalRuntime({ + modules: { "@chat/app/v1": v1 }, + render: (runtime) => { + const { terminal, theme } = runtime; + const scope = new TuiScope(runtime.host, () => draw()); + const channels = scope.watch(scope.hold(Channels, {}).names); + // Each channel's room held once, on first view. + const rooms = new Map; messages: () => readonly v1.Message[] | undefined }>(); + const roomOf = (id: string) => { + let held = rooms.get(id); + if (held === undefined) { + const room = scope.hold(Room, { id }); + held = { room, messages: scope.watch(room.messages) }; + rooms.set(id, held); + } + return held; + }; + let channel = "general"; + let draft = ""; + const draw = (): void => { + const tabs = (channels() ?? []).map((name) => + name === channel ? theme.fg("accent", `#${name}`) : `#${name}`, + ); + const shown = (roomOf(channel).messages() ?? []) + .slice(-(terminal.rows - 3)) + .map((message) => line(message, channel, theme)); + terminal.write(`\x1b[2J\x1b[H${tabs.join(" ")}\r\n${shown.join("\r\n")}\r\n> ${draft}`); + }; + const send = (text: string): void => { + if (text.startsWith("/attach ")) { + const { room } = roomOf(channel); + void runtime + .attach(text.slice(8).trim()) + .then((file) => (file === undefined ? undefined : share(room, file))); + } else if (text !== "") void roomOf(channel).room.post(text, context); + }; + terminal.onInput((data) => { + if (data === "\x03") runtime.exit(0); + else if (data === "\t") { + const names = channels() ?? []; + channel = names[(names.indexOf(channel) + 1) % names.length] ?? channel; + } else if (data === "\r") { + send(draft); + draft = ""; + } else if (data === "\x7f") draft = draft.slice(0, -1); + else if (!data.startsWith("\x1b")) draft += data; + draw(); + }); + theme.onChange(draw); + draw(); + }, +}); diff --git a/packages/chord/test/next/examples/59-chat/web/main.ts b/packages/chord/test/next/examples/59-chat/web/main.ts new file mode 100644 index 000000000..478f2b218 --- /dev/null +++ b/packages/chord/test/next/examples/59-chat/web/main.ts @@ -0,0 +1,95 @@ +// The chat's page (the web base bundle): the channels as tabs, the current room's messages (images inline, other files +// as links), a message box and a file picker. A file goes up in chunks through the room (into its worker's files). + +import * as React from "react"; +import { createElement as h, useState } from "react"; +import * as jsx from "react/jsx-runtime"; +import { createRoot } from "react-dom/client"; +import type { Held } from "../../../../../src/next/index.ts"; +import { HostProvider, useHold, useWatch } from "../../../../../src/next-react/index.ts"; +import { startPage } from "../../../../../src/next-web/index.ts"; +import * as v1 from "../app/v1.ts"; +import "./style.css"; + +const { BACKGROUND_CONTEXT: context, Channels, Room } = v1; + +/** The size of one upload chunk. */ +const CHUNK = 256 * 1024; + +/** Sends `file` to `room` in chunks, then shares it. */ +async function upload(room: Held, file: File): Promise { + const id = await room.begin({ name: file.name, mime: file.type || "application/octet-stream" }, context); + for (let offset = 0; offset < file.size; offset += CHUNK) { + const bytes = new Uint8Array(await file.slice(offset, offset + CHUNK).arrayBuffer()); + await room.append({ upload: id, offset, bytes }, context); + } + await room.share({ upload: id, text: "" }, context); +} + +function Message({ channel, message }: { channel: string; message: v1.Message }) { + const { file } = message; + const url = file === undefined ? "" : `/files/${channel}/${file.id}`; + const body = + file === undefined + ? null + : file.mime.startsWith("image/") + ? h("img", { src: url, alt: file.name }) + : h("a", { href: url }, `${file.name} (${file.size} bytes)`); + return h("li", null, h("b", null, message.author), " ", message.text, body); +} + +function RoomView({ channel }: { channel: string }) { + const room = useHold(Room, { id: channel }); + const messages = useWatch(room?.messages) ?? []; + const [text, setText] = useState(""); + const send = (event: { preventDefault(): void }) => { + event.preventDefault(); + if (room === undefined || text === "") return; + void room.post(text, context); + setText(""); + }; + return h( + "section", + null, + h("ul", null, ...messages.map((message) => h(Message, { key: message.id, channel, message }))), + h( + "form", + { onSubmit: send }, + h("input", { value: text, onChange: (event: { target: { value: string } }) => setText(event.target.value) }), + h("input", { + type: "file", + onChange: (event: { target: { files: FileList | null } }) => { + const file = event.target.files?.[0]; + if (room !== undefined && file !== undefined) void upload(room, file); + }, + }), + ), + ); +} + +function Chat() { + const channels = useWatch(useHold(Channels, {})?.names) ?? []; + const [channel, setChannel] = useState("general"); + return h( + "main", + null, + h( + "nav", + null, + ...channels.map((name) => + h( + "button", + { key: name, type: "button", onClick: () => setChannel(name), "aria-pressed": name === channel }, + `#${name}`, + ), + ), + ), + h(RoomView, { key: channel, channel }), + ); +} + +void startPage({ + connect: () => new WebSocket(`${location.origin.replace(/^http/, "ws")}/chord`), + modules: { "@chat/app/v1": v1, react: React, "react/jsx-runtime": jsx }, + render: (host) => createRoot(document.getElementById("root")!).render(h(HostProvider, { host }, h(Chat))), +}); diff --git a/packages/chord/test/next/examples/59-chat/web/style.css b/packages/chord/test/next/examples/59-chat/web/style.css new file mode 100644 index 000000000..ccfea4873 --- /dev/null +++ b/packages/chord/test/next/examples/59-chat/web/style.css @@ -0,0 +1,7 @@ +/* The chat's look: the active theme's colours (the kit writes them on :root). */ +body { margin: 0; font: 15px system-ui, sans-serif; background: var(--bg); color: var(--fg); } +main { max-width: 40rem; margin: 1rem auto; } +nav button[aria-pressed="true"] { color: var(--accent); } +ul { list-style: none; padding: 0; } +li img { display: block; max-width: 16rem; margin-top: 0.25rem; } +a { color: var(--accent); } diff --git a/packages/chord/test/next/examples/59-chat/worker.ts b/packages/chord/test/next/examples/59-chat/worker.ts new file mode 100644 index 000000000..982eefcd2 --- /dev/null +++ b/packages/chord/test/next/examples/59-chat/worker.ts @@ -0,0 +1,6 @@ +// A channel's worker: the kit's `startWorker` with the chat's API. + +import { startWorker } from "../../../../src/next-node/app/index.ts"; +import { API, KINDS } from "./start.ts"; + +startWorker({ api: API, kinds: KINDS, onError: (error) => console.error("channel:", error) }); diff --git a/packages/chord/test/next/link/values.test.ts b/packages/chord/test/next/link/values.test.ts index 84656b516..a92b5562d 100644 --- a/packages/chord/test/next/link/values.test.ts +++ b/packages/chord/test/next/link/values.test.ts @@ -29,6 +29,14 @@ describe("payloads", () => { expect(read.thumbs).toEqual(thumbs); }); + it("carry any Uint8Array subclass as bytes: a Node Buffer, whose toJSON the replacer never sees (slice 11)", () => { + const buffer = Buffer.from([1, 2, 3]); + const { json, segments } = payloadPart({ bytes: buffer, at: [buffer] }, true); + expect(json).toBe('{"bytes":{"$bytes":0},"at":[{"$bytes":1}]}'); + expect(segments![0]).toBe(buffer); + expect(payloadPart(buffer, true)).toEqual({ json: '{"$bytes":0}', segments: [buffer] }); + }); + it("marked as carrying bytes refuse the reserved key $bytes with invalid/args", () => { let thrown: unknown; try { diff --git a/packages/chord/test/next/surfaces/active-theme.test.ts b/packages/chord/test/next/surfaces/active-theme.test.ts new file mode 100644 index 000000000..540306f98 --- /dev/null +++ b/packages/chord/test/next/surfaces/active-theme.test.ts @@ -0,0 +1,123 @@ +// Tests of the active theme of a client (slice 11 §4b) and of theme choices on the server: a client follows the +// `Themes` registry, its principal's `ThemeChoice` and its mode; choices are kept per principal across restarts. + +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { + createHost, + defineFacet, + followActiveTheme, + hostKind, + memoryLinkPair, + openStore, + sqliteBackend, + type Theme, + ThemeChoice, + Themes, + themeChoiceFacet, + themesFacet, +} from "../../../src/next/index.ts"; +import { openNodeSqliteDatabase } from "../../../src/sqlite/node.ts"; + +type TestPrincipal = { readonly kind: "server" } | { readonly kind: "user"; readonly id: string }; + +const COLORS = { + accent: "#111111", + bg: "#000000", + fg: "#ffffff", + muted: "#888888", + error: "#f00", + warning: "#ff0", + success: "#0f0", +}; + +const theme = (id: string, mode: Theme["mode"]): Theme => ({ id, name: id, mode, colors: COLORS }); + +/** A server with theme choices, and a client of `user` with themes `themes` contributed by a plugin. */ +async function setup(user: string, themes: readonly Theme[]) { + const store = await openStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), context); + const server = createHost({ + address: hostKind("server")(), + principal: { kind: "server" }, + store, + onError: () => {}, + }); + await server.load([{ name: "chord.themes.choice", facet: themeChoiceFacet() }], context); + const client = createHost({ onError: () => {} }); + await client.load( + [ + { name: "chord.themes", facet: themesFacet() }, + ...themes.map((each, index) => ({ + name: `@test/theme-${index}`, + facet: defineFacet((_host, self) => self.contribute(Themes, each)), + })), + ], + context, + ); + const [mine, theirs] = memoryLinkPair(); + server.accept(theirs, { principal: { kind: "user", id: user } }, context); + client.connect(mine, { principal: { kind: "server" } }, context); + return { server, client }; +} + +/** Waits until `check` holds. */ +async function until(check: () => boolean): Promise { + for (let index = 0; index < 200 && !check(); index++) await new Promise((resolve) => setTimeout(resolve, 5)); + expect(check()).toBe(true); +} + +describe("following the active theme", () => { + it("follows the registry, the principal's pick and the mode", async () => { + const { server, client } = await setup("ada", [ + theme("night", "dark"), + theme("dusk", "dark"), + theme("paper", "light"), + ]); + const seen: string[] = []; + const follower = await followActiveTheme(client, "dark", (active) => seen.push(active.id), context); + await until(() => seen.at(-1) === "night"); + const choice = await client.get(ThemeChoice, {}, context); + await choice.choose({ dark: "dusk", light: null }, context); + await until(() => seen.at(-1) === "dusk"); + follower.setMode("light"); + expect(seen.at(-1)).toBe("paper"); + follower.stop(); + choice.release(); + await client.close(context); + await server.close(context); + }); +}); + +describe("theme choices", () => { + it("refuse an id over 128 characters (Opus M4, Fable L7)", async () => { + const { server, client } = await setup("ada", []); + const choice = await client.get(ThemeChoice, {}, context); + await expect(choice.choose({ dark: "x".repeat(129), light: null }, context)).rejects.toMatchObject({ + code: "invalid", + }); + await choice.choose({ dark: "x".repeat(128), light: null }, context); + choice.release(); + await client.close(context); + await server.close(context); + }); + + it("are kept per principal", async () => { + const { server, client } = await setup("ada", []); + const choice = await client.get(ThemeChoice, {}, context); + expect(choice.pick.value).toEqual({ dark: null, light: null }); + await choice.choose({ dark: "dusk", light: "paper" }, context); + await until(() => choice.pick.value.dark === "dusk"); + const other = createHost({ onError: () => {} }); + await other.load([], context); + const [mine, theirs] = memoryLinkPair(); + server.accept(theirs, { principal: { kind: "user", id: "grace" } }, context); + other.connect(mine, { principal: { kind: "server" } }, context); + const graces = await other.get(ThemeChoice, {}, context); + expect(graces.pick.value).toEqual({ dark: null, light: null }); + choice.release(); + graces.release(); + await other.close(context); + await client.close(context); + await server.close(context); + }); +}); diff --git a/packages/chord/test/next/surfaces/client.test.ts b/packages/chord/test/next/surfaces/client.test.ts new file mode 100644 index 000000000..82017c549 --- /dev/null +++ b/packages/chord/test/next/surfaces/client.test.ts @@ -0,0 +1,179 @@ +// Tests of a client host with its plugins (slice 11 §2, §4, decisions "Slice 11, P4" to "P6"), over an in-memory link +// to a server serving the scripts object: the initial scripts loaded with the core facets, a new set applied in place +// (only changed scripts fetched; a failed one keeps its old facet), a new base or an outdated server ending the client. + +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import * as chordNext from "../../../src/next/index.ts"; +import { + constant, + createHost, + defineFacet, + hostKind, + local, + memoryLinkPair, + type RegisteredScript, + type ScriptSet, + type SocketEvents, + startClient, + str, + WebScripts, +} from "../../../src/next/index.ts"; +import { publishTo, scriptsFacet } from "../../../src/next-node/surfaces/current.ts"; + +type TestPrincipal = { readonly kind: "server" } | { readonly kind: "user"; readonly id: string }; + +const Greeting = local("test.greeting", { version: 1, params: {}, members: { text: str() } }); + +/** A registered script of plugin `plugin` (digest `digest`) serving `Greeting` with `text`; `broken` throws instead. */ +function script(plugin: string, digest: string, text: string): RegisteredScript { + return { + plugin, + digest, + run: (require, module) => { + if (text === "broken") throw new Error("broken plugin"); + const chord = require("@test/chord") as typeof chordNext; + const facet = chord.defineFacet((host) => host.serve(Greeting, () => ({ text: chord.constant(text) }))); + Object.assign(module, { exports: { default: facet } }); + }, + }; +} + +/** The set of `scripts` under base `base`. */ +function setOf(base: string, scripts: readonly RegisteredScript[]): ScriptSet { + return { + base, + combined: "/combined.js", + plugins: scripts.map(({ plugin, digest }) => ({ name: plugin, digest, url: `/${plugin}.${digest}.js` })), + }; +} + +/** A server serving `WebScripts` with `initial`, and a client started against it with `scripts` available to fetch. */ +async function setup(initial: readonly RegisteredScript[], available: readonly RegisteredScript[] = []) { + const server = createHost({ + address: hostKind("server")(), + principal: { kind: "server" }, + onError: () => {}, + }); + await server.load( + [{ name: "chord.web", facet: scriptsFacet("web", { set: setOf("B1", initial) }) }], + context, + ); + const fetched: string[] = []; + const errors: chordNext.ClientError[] = []; + const outdated: string[] = []; + const sockets: { events: SocketEvents; link: chordNext.Link }[] = []; + let serverBase = "B1"; + const client = await startClient({ + scripts: WebScripts, + base: "B1", + initial, + modules: { "@test/chord": chordNext }, + core: [ + { name: "@test/core", facet: defineFacet((host) => host.serve(Echo, () => ({ text: constant("core") }))) }, + ], + open: (events) => { + const [mine, theirs] = memoryLinkPair(); + server.accept(theirs, { principal: { kind: "user", id: "u" } }, context); + sockets.push({ events, link: mine }); + queueMicrotask(() => events.opened()); + return mine; + }, + fetchScript: async (entry) => { + fetched.push(entry.url); + const found = available.find(({ plugin, digest }) => plugin === entry.name && digest === entry.digest); + if (found === undefined) throw new Error(`no ${entry.url}`); + return found; + }, + serverBase: async () => serverBase, + outdated: () => outdated.push("outdated"), + environment: { + now: () => Date.now(), + after: (ms, callback) => { + const timer = setTimeout(callback, ms); + return () => clearTimeout(timer); + }, + online: () => true, + onWake: () => () => {}, + }, + onError: (error) => errors.push(error), + }); + return { + server, + client, + fetched, + errors, + outdated, + sockets, + setServerBase: (base: string) => { + serverBase = base; + }, + }; +} + +const Echo = local("test.echo", { version: 1, params: {}, members: { text: str() } }); + +/** Waits until `check` holds. */ +async function until(check: () => boolean): Promise { + for (let index = 0; index < 200 && !check(); index++) await new Promise((resolve) => setTimeout(resolve, 5)); + expect(check()).toBe(true); +} + +describe("loading", () => { + it("loads the initial scripts with the core facets", async () => { + const { client, fetched, server } = await setup([script("@test/a", "d1", "hello")]); + expect((await client.host.get(Greeting, {}, context)).text.value).toBe("hello"); + expect((await client.host.get(Echo, {}, context)).text.value).toBe("core"); + expect(fetched).toEqual([]); + await client.stop(context); + await server.close(context); + }); +}); + +describe("applying a new set in place", () => { + it("fetches only changed scripts, reloads the host, keeps an unchanged plugin's facet", async () => { + const a1 = script("@test/a", "d1", "hello"); + const a2 = script("@test/a", "d2", "hello again"); + const { client, fetched, server } = await setup([a1], [a2]); + const greeting = await client.host.get(Greeting, {}, context); + await publishTo(server, "web", setOf("B1", [a2]), context); + await until(() => greeting.text.value === "hello again"); + expect(fetched).toEqual(["/@test/a.d2.js"]); + await client.stop(context); + await server.close(context); + }); + + it("keeps a plugin's old facet when its new script fails, and reports it", async () => { + const a1 = script("@test/a", "d1", "hello"); + const broken = script("@test/a", "d2", "broken"); + const { client, server, errors } = await setup([a1], [broken]); + const greeting = await client.host.get(Greeting, {}, context); + await publishTo(server, "web", setOf("B1", [broken]), context); + await until(() => errors.length > 0); + expect(errors[0]).toMatchObject({ kind: "plugins", failed: [{ plugin: "@test/a" }] }); + expect(greeting.text.value).toBe("hello"); + await client.stop(context); + await server.close(context); + }); +}); + +describe("outdated clients", () => { + it("calls outdated for a set with another base", async () => { + const { client, server, outdated } = await setup([]); + await publishTo(server, "web", setOf("B2", []), context); + await until(() => outdated.length === 1); + await client.stop(context); + await server.close(context); + }); + + it("asks the server's base before reconnecting: another one calls outdated and stops", async () => { + const { client, server, outdated, sockets, setServerBase } = await setup([]); + setServerBase("B2"); + sockets[0]!.link.close(); + await until(() => outdated.length === 1); + await new Promise((resolve) => setTimeout(resolve, 400)); + expect(sockets).toHaveLength(1); + await client.stop(context); + await server.close(context); + }); +}); diff --git a/packages/chord/test/next/surfaces/connection.test.ts b/packages/chord/test/next/surfaces/connection.test.ts new file mode 100644 index 000000000..68171f865 --- /dev/null +++ b/packages/chord/test/next/surfaces/connection.test.ts @@ -0,0 +1,214 @@ +// Tests of the connection loop (slice 11 §2, decision "Slice 11, P6"): reconnect pauses (250 ms doubling to 5 s, reset +// by a stable connection, the longest while offline), wakes (a pending retry at once; an open socket silent past the +// server's pings replaced), the connect timeout, `canRetry` stopping it, and the states it reports. + +import { describe, expect, it } from "vitest"; +import { + CONNECT_TIMEOUT_MS, + type ConnectionState, + keepConnected, + type LoopEnvironment, + type SocketEvents, + STABLE_MS, + STALE_MS, +} from "../../../src/next/surfaces/connection.ts"; + +/** A clock and timers the test advances, wakes it sends, and the sockets the loop opened. */ +class FakeWorld { + now = 0; + online = true; + readonly #timers: { at: number; callback: () => void; cancelled: boolean }[] = []; + #wake: (() => void) | undefined; + readonly sockets: FakeSocket[] = []; + readonly opens: number[] = []; + + readonly environment: LoopEnvironment = { + now: () => this.now, + after: (ms, callback) => { + const timer = { at: this.now + ms, callback, cancelled: false }; + this.#timers.push(timer); + return () => { + timer.cancelled = true; + }; + }, + online: () => this.online, + onWake: (listener) => { + this.#wake = listener; + return () => { + this.#wake = undefined; + }; + }, + }; + + open = (events: SocketEvents): FakeSocket => { + const socket = new FakeSocket(events); + this.sockets.push(socket); + this.opens.push(this.now); + return socket; + }; + + /** Advances the clock by `ms`, running due timers in order; settles promises after each. */ + async advance(ms: number): Promise { + const end = this.now + ms; + for (;;) { + await settled(); + const due = this.#timers.filter(({ at, cancelled }) => !cancelled && at <= end).sort((a, b) => a.at - b.at)[0]; + if (due === undefined) break; + this.now = due.at; + due.cancelled = true; + due.callback(); + } + this.now = end; + await settled(); + } + + wake(): void { + this.#wake?.(); + } + + get last(): FakeSocket { + return this.sockets.at(-1)!; + } +} + +/** One socket and its link: the test opens, feeds and closes it. */ +class FakeSocket { + readonly events: SocketEvents; + closed = false; + #onClose: (() => void) | undefined; + + constructor(events: SocketEvents) { + this.events = events; + } + + onClose(listener: () => void): void { + this.#onClose = listener; + } + + close(): void { + if (this.closed) return; + this.closed = true; + this.#onClose?.(); + } +} + +async function settled(): Promise { + for (let index = 0; index < 5; index++) await Promise.resolve(); +} + +/** A loop in `world` recording its states; `retry` answers `canRetry`. */ +function loop(world: FakeWorld, retry: () => Promise = async () => true) { + const states: ConnectionState[] = []; + const handle = keepConnected({ + open: world.open, + onState: (state) => states.push(state), + canRetry: retry, + environment: world.environment, + }); + return { states, handle }; +} + +describe("reconnecting", () => { + it("reconnects after a growing pause, 250 ms doubling to 5 s, and reports its states", async () => { + const world = new FakeWorld(); + const { states } = loop(world); + const pauses: number[] = []; + for (let attempt = 0; attempt < 7; attempt++) { + const closedAt = world.now; + world.last.close(); + await world.advance(5000); + pauses.push(world.opens.at(-1)! - closedAt); + } + expect(pauses).toEqual([250, 500, 1000, 2000, 4000, 5000, 5000]); + expect(states.slice(0, 4)).toEqual(["connecting", "waiting", "connecting", "waiting"]); + }); + + it("starts the pauses again after a connection that stayed open 10 s", async () => { + const world = new FakeWorld(); + const { states } = loop(world); + for (let attempt = 0; attempt < 3; attempt++) { + world.last.close(); + await world.advance(5000); + } + world.last.events.opened(); + expect(states.at(-1)).toBe("connected"); + await world.advance(STABLE_MS); + const before = world.now; + world.last.close(); + await world.advance(5000); + expect(world.opens.at(-1)! - before).toBe(250); + }); + + it("waits the longest pause while offline, and retries at once on a wake", async () => { + const world = new FakeWorld(); + loop(world); + world.online = false; + world.last.close(); + await world.advance(1000); + expect(world.sockets).toHaveLength(1); + world.wake(); + await settled(); + expect(world.sockets).toHaveLength(2); + }); + + it("gives up a socket not open after 20 s", async () => { + const world = new FakeWorld(); + loop(world); + await world.advance(CONNECT_TIMEOUT_MS - 1); + expect(world.sockets[0]!.closed).toBe(false); + await world.advance(1); + expect(world.sockets[0]!.closed).toBe(true); + }); + + it("stops when canRetry says no", async () => { + const world = new FakeWorld(); + const { states } = loop(world, async () => false); + world.last.close(); + await world.advance(60_000); + expect(world.sockets).toHaveLength(1); + expect(states.at(-1)).toBe("waiting"); + }); + + it("retries at once when woken while canRetry is deciding", async () => { + const world = new FakeWorld(); + let answer!: (retry: boolean) => void; + loop( + world, + () => + new Promise((resolve) => { + answer = resolve; + }), + ); + world.last.close(); + world.wake(); + answer(true); + await world.advance(0); + expect(world.sockets).toHaveLength(2); + }); +}); + +describe("a silent socket", () => { + it("is replaced on a wake once its next ping is overdue; one with recent frames stays", async () => { + const world = new FakeWorld(); + loop(world); + world.last.events.opened(); + await world.advance(10_000); + world.last.events.received(); + world.wake(); + await world.advance(STALE_MS); + expect(world.sockets[0]!.closed).toBe(false); + world.wake(); + await world.advance(STALE_MS + 1000); + expect(world.sockets[0]!.closed).toBe(true); + }); + + it("closes the link at stop and opens no other", async () => { + const world = new FakeWorld(); + const { handle } = loop(world); + world.last.events.opened(); + handle.stop(); + expect(world.last.closed).toBe(true); + await world.advance(60_000); + expect(world.sockets).toHaveLength(1); + }); +}); diff --git a/packages/chord/test/next/surfaces/registered.test.ts b/packages/chord/test/next/surfaces/registered.test.ts new file mode 100644 index 000000000..1ae367fcd --- /dev/null +++ b/packages/chord/test/next/surfaces/registered.test.ts @@ -0,0 +1,57 @@ +// Tests of scripts that register themselves (slice 11 §1, §2): the served text calls `chordFacet` with the plugin, the +// digest and the build's function; evaluating it answers only the client's modules (an ES module namespace as a +// CommonJS import sees it) and refuses a default export that is no facet. + +import { Script } from "node:vm"; +import { describe, expect, it } from "vitest"; +import * as chordNext from "../../../src/next/index.ts"; +import { + evaluateRegistered, + type FacetFunction, + REGISTER, + type RegisteredScript, + registeredSource, +} from "../../../src/next/surfaces/registered.ts"; + +/** Runs a registered script's text and returns what it registered. */ +function run(text: string): RegisteredScript[] { + const registered: RegisteredScript[] = []; + (globalThis as Record)[REGISTER] = (plugin: string, digest: string, f: FacetFunction) => + registered.push({ plugin, digest, run: f }); + new Script(text).runInThisContext(); + delete (globalThis as Record)[REGISTER]; + return registered; +} + +/** A built script's function expression, as esbuild wraps a facet: it requires `specifier`, exports a facet. */ +const BUILT = (specifier: string) => + `(function (require, module, exports) { +const chord = require(${JSON.stringify(specifier)}); +module.exports = { default: chord.defineFacet(() => {}) }; +//# sourceMappingURL=data:application/json;base64,e30= +})`; + +describe("registered scripts", () => { + it("register their plugin, digest and function, the source map comment kept inside the call", () => { + const text = registeredSource("@acme/notes", "abc123", BUILT("@acme/app/v1")); + expect(text.startsWith('chordFacet("@acme/notes","abc123",(function (require, module, exports) {')).toBe(true); + const [script] = run(text); + expect([script!.plugin, script!.digest]).toEqual(["@acme/notes", "abc123"]); + const facet = evaluateRegistered(script!, { "@acme/app/v1": chordNext }); + expect(typeof facet.setup).toBe("function"); + }); + + it("answer only the client's modules, naming the plugin", () => { + const [script] = run(registeredSource("@acme/notes", "d", BUILT("node:fs"))); + expect(() => evaluateRegistered(script!, { "@acme/app/v1": chordNext })).toThrow( + 'plugin "@acme/notes": "node:fs" is not one of this client\'s host packages', + ); + }); + + it("refuse a default export that is no facet", () => { + const [script] = run( + registeredSource("@acme/bad", "d", "(function (require, module) { module.exports = { default: 42 }; })"), + ); + expect(() => evaluateRegistered(script!, {})).toThrow('plugin "@acme/bad": its default export is not a facet'); + }); +}); diff --git a/packages/chord/test/next/surfaces/themes.test.ts b/packages/chord/test/next/surfaces/themes.test.ts new file mode 100644 index 000000000..2731b21c1 --- /dev/null +++ b/packages/chord/test/next/surfaces/themes.test.ts @@ -0,0 +1,70 @@ +// Tests of themes (slice 11 §4b): the active theme for a mode and a pick, CSS variables of a theme, and terminal +// colours by role (true colour, the nearest of 256, `tui` overrides first, the fallback for a missing role). + +import { describe, expect, it } from "vitest"; +import { + activeTheme, + FALLBACK_THEMES, + type Theme, + terminalColor, + themeVariables, +} from "../../../src/next/surfaces/themes.ts"; + +const COLORS = { + accent: "#e8590c", + bg: "#111317", + fg: "#e8e8e8", + muted: "#8a8f98", + error: "#ff0000", + warning: "#ffaa00", + success: "#00ff00", +}; + +const SOLARIZED: Theme = { id: "solarized", name: "Solarized", mode: "dark", colors: { ...COLORS, link: "#268bd2" } }; + +const PAPER: Theme = { id: "paper", name: "Paper", mode: "light", colors: COLORS, web: { radius: "10px" } }; + +const NIGHT: Theme = { id: "night", name: "Night", mode: "dark", colors: COLORS, tui: { accent: "208" } }; + +describe("the active theme", () => { + it("is the picked one of the mode, else the first of the mode, else the fallback", () => { + const themes = [SOLARIZED, PAPER, NIGHT]; + expect(activeTheme(themes, { dark: "night", light: null }, "dark").id).toBe("night"); + expect(activeTheme(themes, { dark: null, light: null }, "dark").id).toBe("solarized"); + expect(activeTheme(themes, { dark: "paper", light: null }, "dark").id).toBe("solarized"); + expect(activeTheme(themes, { dark: null, light: "gone" }, "light").id).toBe("paper"); + expect(activeTheme([SOLARIZED], { dark: null, light: null }, "light")).toBe(FALLBACK_THEMES.light); + }); +}); + +describe("CSS variables", () => { + it("gives every role and every web extra a custom property", () => { + expect(themeVariables(SOLARIZED)).toMatchObject({ + "--accent": "#e8590c", + "--bg": "#111317", + "--link": "#268bd2", + }); + expect(themeVariables(PAPER)["--radius"]).toBe("10px"); + }); +}); + +describe("terminal colours", () => { + it("gives true colour, or the nearest of 256, for a role", () => { + expect(terminalColor(SOLARIZED, "accent", "truecolor", "fg")).toBe("38;2;232;89;12"); + expect(terminalColor(SOLARIZED, "bg", "truecolor", "bg")).toBe("48;2;17;19;23"); + expect(terminalColor(SOLARIZED, "error", "256", "fg")).toBe("38;5;196"); + expect(terminalColor(SOLARIZED, "fg", "256", "fg")).toBe("38;5;254"); + expect(terminalColor(SOLARIZED, "bg", "256", "bg")).toBe("48;5;233"); + }); + + it("takes the theme's tui override first, the fallback for a missing role, the default for an unknown one", () => { + expect(terminalColor(NIGHT, "accent", "truecolor", "fg")).toBe("38;5;208"); + const partial = { ...SOLARIZED, colors: { ...COLORS, muted: "not a colour" } }; + expect(terminalColor(partial, "muted", "truecolor", "fg")).toBe("39"); + expect(terminalColor(SOLARIZED, "nobody", "256", "bg")).toBe("49"); + const fallback = FALLBACK_THEMES.dark.colors.accent.slice(1); + const rgb = [0, 2, 4].map((at) => Number.parseInt(fallback.slice(at, at + 2), 16)).join(";"); + const missing = { ...SOLARIZED, colors: { ...COLORS, accent: undefined as unknown as string } }; + expect(terminalColor(missing, "accent", "truecolor", "fg")).toBe(`38;2;${rgb}`); + }); +}); diff --git a/packages/chord/test/next/tui/frames.test.ts b/packages/chord/test/next/tui/frames.test.ts new file mode 100644 index 000000000..7d63caaf9 --- /dev/null +++ b/packages/chord/test/next/tui/frames.test.ts @@ -0,0 +1,26 @@ +// Tests of frames for the terminal (slice 11 §4; fix round 1, Fable L9): a write over the launcher's 64 KiB arrives +// as frames each under it, cut only between escape sequences and surrogate pairs, the filter keeping every one whole. + +import { describe, expect, it } from "vitest"; +import { filterOutput, MAX_FRAME } from "../../../src/next-node/tui/filter.ts"; +import { FRAME_UNITS, splitFrames } from "../../../src/next-tui/frames.ts"; + +describe("splitting frames", () => { + it("keeps a small write whole", () => { + expect(splitFrames("hello")).toEqual(["hello"]); + }); + + it("cuts a large true-colour paint between sequences, each frame under 64 KiB and passing the filter whole", () => { + let paint = ""; + for (let cell = 0; cell < 20_000; cell++) + paint += `\x1b[38;2;${cell % 256};12;200m${cell % 7 === 0 ? "😀" : "é"}`; + const frames = splitFrames(paint); + expect(frames.length).toBeGreaterThan(1); + expect(frames.join("")).toBe(paint); + for (const frame of frames) { + expect(frame.length).toBeLessThanOrEqual(FRAME_UNITS); + expect(Buffer.byteLength(frame)).toBeLessThanOrEqual(MAX_FRAME); + expect(filterOutput(frame)).toBe(frame); + } + }); +}); diff --git a/packages/chord/test/next/tui/scope.test.ts b/packages/chord/test/next/tui/scope.test.ts new file mode 100644 index 000000000..43375c862 --- /dev/null +++ b/packages/chord/test/next/tui/scope.test.ts @@ -0,0 +1,80 @@ +// Tests of `TuiScope` (slice 11 §4, decision "Slice 11, P9"): references held for a component, watchables and +// registries followed, one `changed` callback for every change, and everything released at `dispose`. + +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { createHost, defineFacet, defineRegistry, local, method, str } from "../../../src/next/index.ts"; +import { TuiScope } from "../../../src/next-tui/scope.ts"; + +const Notes = local("test.notes", { version: 1, params: {}, members: { text: str(), write: method(str()) } }); + +const Panels = defineRegistry("test.panels", { version: 1 }); + +/** A host serving `Notes` (a writable text) and `Panels` with one contribution. */ +async function host() { + const created = createHost({ onError: () => {} }); + await created.load( + [ + { + name: "@test/notes", + facet: defineFacet((h, self) => { + h.serve(Notes, (_params, object) => { + const text = object.writable("first"); + return { + text: object.derive([text], ([value]) => value), + write: async (next) => { + text.set(next); + return null; + }, + }; + }); + h.serve(Panels, { compose: (table) => table.map(({ value }) => value) }); + self.contribute(Panels, "notes"); + }), + }, + ], + context, + ); + return created; +} + +/** Waits until `check` holds. */ +async function until(check: () => boolean): Promise { + for (let index = 0; index < 200 && !check(); index++) await new Promise((resolve) => setTimeout(resolve, 5)); + expect(check()).toBe(true); +} + +describe("a component's scope", () => { + it("holds, follows and calls changed on every change; releases everything at dispose", async () => { + const h = await host(); + let changes = 0; + const scope = new TuiScope(h, () => changes++); + const notes = scope.hold(Notes, {}); + const text = scope.watch(notes.text); + const panels = scope.registry(Panels); + await until(() => text() === "first" && panels()?.[0] === "notes"); + const before = changes; + await notes.write("second", context); + await until(() => text() === "second" && changes > before); + scope.dispose(); + expect(notes.state.value.kind).toBe("ended"); + const after = changes; + await new Promise((resolve) => setTimeout(resolve, 20)); + expect(changes).toBe(after); + await h.close(context); + }); + + it("reads undefined from a watchable without a value, never throwing", async () => { + const h = await host(); + const scope = new TuiScope(h, () => {}); + const read = scope.watch({ + get value(): string { + throw new Error("not ready"); + }, + watch: () => Promise.reject(new Error("not ready")), + }); + expect(read()).toBeUndefined(); + scope.dispose(); + await h.close(context); + }); +}); diff --git a/tsconfig.json b/tsconfig.json index e8e536464..c032b98e2 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -13,9 +13,16 @@ "@earendil-works/chord/next/plugins/runtime": ["./packages/chord/src/next-node/plugins/runtime.ts"], "@acme/app/*": ["./packages/chord/test/next/examples/49-plugin-api/app/*.ts"], "@notes/app/*": ["./packages/chord/test/next/examples/53-app/app/*.ts"], + "@board/app/*": ["./packages/chord/test/next/examples/56-web/app/*.ts"], + "@chat/app/*": ["./packages/chord/test/next/examples/59-chat/app/*.ts"], "@earendil-works/chord/next/http": ["./packages/chord/src/next-node/http/index.ts"], "@earendil-works/chord/next/workers": ["./packages/chord/src/next-node/workers/index.ts"], "@earendil-works/chord/next/app": ["./packages/chord/src/next-node/app/index.ts"], + "@earendil-works/chord/next/web": ["./packages/chord/src/next-web/index.ts"], + "@earendil-works/chord/next/web/server": ["./packages/chord/src/next-node/surfaces/index.ts"], + "@earendil-works/chord/next/react": ["./packages/chord/src/next-react/index.ts"], + "@earendil-works/chord/next/tui": ["./packages/chord/src/next-tui/index.ts"], + "@earendil-works/chord/next/tui/launcher": ["./packages/chord/src/next-node/tui/index.ts"], "@earendil-works/chord/node": ["./packages/chord/src/node.ts"], "@earendil-works/chord/output": ["./packages/chord/src/output.ts"], "@earendil-works/chord/telemetry-sqlite": ["./packages/chord/src/telemetry-sqlite/index.ts"], From 1d81a69727d1b7987fd0277af9d1917c0cb16f3c Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 2 Nov 2026 21:40:16 +0000 Subject: [PATCH 174/175] docs(chord): Chord v2 becomes the package's root --- packages/chord/docs/next/decisions.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 20049a107..a9647718a 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -2396,6 +2396,16 @@ validation and copies the v2 kernel added around Delta. `t.render` in its own language (checked at runtime across a link: an unknown key shows the key). Never translated: the model's prompts and answers, log lines, error codes. No translator file format now; an export/import tool to JSON when someone needs it. Built after slice 11 (carry-over "After slice 11"). +- **Chord v2 becomes the package's root (owner, 2026-11-03: "src/old", "subpaths", "yes", "yes", "no, chord is + wip"; for merging pi-private into pi this week; trial `/home/badlogic/chord-root-move/report.md`):** `src/next` is + the package's `src` (its main export), the Node parts `src/next-node` are `src/node` with short subpaths (`/http`, + `/workers`, `/plugins`, `/plugins/runtime`, `/app`, and slice 11's `/web`, `/web/server`, `/react`, `/tui`, + `/tui/launcher`), the shared foundations Durable and env use get their own subpaths (`/json`, `/watch`, next to + `/context`, `/delta`, `/sqlite`), `test/next` and `docs/next` move up (`test/`, `docs/`). Chord v1, which only p2 + still runs on, moves to `src/old` (exported as `/old`, `/old/node`; its tests `test/old`, its docs `docs/old`), not + into p2 (its vitest tests and about 140 imports would need porting); it is deleted in the change that moves p2 to + v2. No version bump: Chord is work in progress. Done by a re-runnable script (move, import and link rewrites, + generated exports and aliases) on the base that includes slice 11; p2's imports updated in private. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a From 0a4fad5489f303fe023e3948d473731d7b2f8acb Mon Sep 17 00:00:00 2001 From: David Brailovsky Date: Mon, 2 Nov 2026 22:43:27 +0100 Subject: [PATCH 175/175] fix(ai): leave room for OpenRouter's input overcount when clamping max tokens (#11712) OpenRouter checks input + max_tokens against the context window before routing, using its own tokenizer. For prose it counted up to 12% more input tokens than the provider. pi left only 4096 tokens spare, so models whose output limit is close to the window (e.g. DeepSeek flash) were rejected with a 400, which pi then treated as a context overflow and compacted. The default output limit now leaves 15% of the estimated input unused (at least 4096). When that would leave less than 32768 tokens of output, only the fixed 4096 margin is kept and output is capped at 32768. fixes #10497 --- packages/ai/CHANGELOG.md | 1 + packages/ai/src/api/simple-options.ts | 17 +++++- packages/ai/test/clamp-max-tokens.test.ts | 71 +++++++++++++++++++++++ 3 files changed, 87 insertions(+), 2 deletions(-) create mode 100644 packages/ai/test/clamp-max-tokens.test.ts diff --git a/packages/ai/CHANGELOG.md b/packages/ai/CHANGELOG.md index 1823bff41..b6e781ae5 100644 --- a/packages/ai/CHANGELOG.md +++ b/packages/ai/CHANGELOG.md @@ -6,6 +6,7 @@ - Fixed OpenCode Go Claude Haiku 5.5 failing with a 400 (`Invalid signature in thinking block ... bound to a different conversation`) after the tool list, system prompt, or earlier history changed, which left the session failing on every retry. OpenCode Go and Zen Claude 5.5 models, Opus 5, and Fable 5.1 now use per-message effort and send `thinking.block_binding: drop_block`, like the Anthropic provider ([#10954](https://github.com/earendil-works/pi/issues/10954)) - Fixed Anthropic non-strict tool schemas dropping root `$defs` and `definitions`, which left property `$ref`s pointing at nothing. Claude then produced wrong arguments for such tools (common in MCP servers), for example a JSON string instead of an object, and some Anthropic-compatible gateways rejected the request outright ([#9557](https://github.com/earendil-works/pi/issues/9557)) +- Fixed OpenRouter rejecting requests with a 400 (`This endpoint's maximum context length is ...`) for models whose output limit is close to the context window, such as DeepSeek flash. OpenRouter counts the input with its own tokenizer, which can be several percent higher than the provider's count, and pi left only 4,096 tokens spare. pi then treated the 400 as a context overflow and compacted the session, even at around 20% of the window. The default output limit now leaves 15% of the estimated input unused, and near a full window at most 32,768 output tokens are requested ([#10497](https://github.com/earendil-works/pi/issues/10497)) ## [1.5.0] - 2026-11-02 diff --git a/packages/ai/src/api/simple-options.ts b/packages/ai/src/api/simple-options.ts index 85bfa78f9..375c9e4be 100644 --- a/packages/ai/src/api/simple-options.ts +++ b/packages/ai/src/api/simple-options.ts @@ -13,12 +13,25 @@ import type { import { estimateContextTokens } from "../utils/estimate.ts"; const CONTEXT_SAFETY_TOKENS = 4096; +// OpenRouter counts the input with its own tokenizer and rejects input + max_tokens above the +// window before routing. It counted prose up to 12% above the provider's count (#10497). +const CONTEXT_SAFETY_RATIO = 0.15; +// The 15% margin is kept only while it leaves at least this much output. Otherwise (a nearly full +// large window, or any small window such as a 32k local model) only the fixed margin is kept, and +// output is capped at this size so an OpenRouter overcount still fits for a while before compaction. +const MIN_OUTPUT_WITH_RATIO_MARGIN = 32768; const MIN_MAX_TOKENS = 1; +/** Fit maxTokens into the context window, leaving room for the input being counted higher than estimated. */ export function clampMaxTokensToContext(model: Model, context: TranscriptContext, maxTokens: number): number { if (model.contextWindow <= 0) return Math.max(MIN_MAX_TOKENS, maxTokens); - const available = model.contextWindow - estimateContextTokens(context).tokens - CONTEXT_SAFETY_TOKENS; - return Math.min(maxTokens, Math.max(MIN_MAX_TOKENS, available)); + const input = estimateContextTokens(context).tokens; + const margin = Math.max(CONTEXT_SAFETY_TOKENS, Math.ceil(input * CONTEXT_SAFETY_RATIO)); + let room = model.contextWindow - input - margin; + if (room < MIN_OUTPUT_WITH_RATIO_MARGIN) { + room = Math.min(MIN_OUTPUT_WITH_RATIO_MARGIN, model.contextWindow - input - CONTEXT_SAFETY_TOKENS); + } + return Math.min(maxTokens, Math.max(MIN_MAX_TOKENS, room)); } export function resolveSamplingParams( diff --git a/packages/ai/test/clamp-max-tokens.test.ts b/packages/ai/test/clamp-max-tokens.test.ts new file mode 100644 index 000000000..ada7a22a4 --- /dev/null +++ b/packages/ai/test/clamp-max-tokens.test.ts @@ -0,0 +1,71 @@ +import { describe, expect, it } from "vitest"; +import { clampMaxTokensToContext } from "../src/api/simple-options.ts"; +import { getModel } from "../src/compat.ts"; +import type { AssistantMessage, Model } from "../src/types.ts"; +import { normalizeContext } from "../src/utils/transcript.ts"; + +const WINDOW = 1_048_576; + +function modelWith(maxTokens: number, contextWindow = WINDOW): Model<"openai-completions"> { + const { compat: _compat, ...base } = getModel("openai", "gpt-4o-mini")!; + return { ...base, api: "openai-completions", contextWindow, maxTokens }; +} + +// The last assistant usage anchors the estimate, so the estimated input is exactly `inputTokens`. +function contextWithInput(inputTokens: number) { + const assistant: AssistantMessage = { + role: "assistant", + content: [{ type: "text", text: "ok" }], + api: "openai-completions", + provider: "openai", + model: "gpt-4o-mini", + usage: { + input: inputTokens, + output: 0, + cacheRead: 0, + cacheWrite: 0, + totalTokens: inputTokens, + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, + }, + stopReason: "stop", + timestamp: 2, + }; + return normalizeContext({ messages: [{ role: "user", content: "hi", timestamp: 1 }, assistant] }); +} + +describe("clampMaxTokensToContext", () => { + it("leaves 15% of the input unused so an OpenRouter overcount still fits (#10497)", () => { + // The failing request from #10497: OpenRouter counted 229,747 input tokens where pi estimated 220,815. + const maxTokens = clampMaxTokensToContext(modelWith(943_718), contextWithInput(220_815), 943_718); + expect(maxTokens).toBe(WINDOW - 220_815 - 33_123); + expect(229_747 + maxTokens).toBeLessThanOrEqual(WINDOW); + }); + + it("leaves at least 4096 tokens unused for small inputs", () => { + expect(clampMaxTokensToContext(modelWith(WINDOW), contextWithInput(10_000), WINDOW)).toBe(WINDOW - 10_000 - 4096); + }); + + it("keeps the model output limit when it already fits", () => { + expect(clampMaxTokensToContext(modelWith(64_000), contextWithInput(100_000), 64_000)).toBe(64_000); + }); + + it("sends up to 32768 tokens when the proportional margin leaves no room", () => { + expect(clampMaxTokensToContext(modelWith(943_718), contextWithInput(912_000), 943_718)).toBe(32_768); + }); + + it("uses the real room minus 4096 when less than 32768 tokens fit", () => { + expect(clampMaxTokensToContext(modelWith(943_718), contextWithInput(1_030_000), 943_718)).toBe( + WINDOW - 1_030_000 - 4096, + ); + }); + + it("keeps only the fixed 4096 margin for small windows", () => { + expect(clampMaxTokensToContext(modelWith(32_768, 32_768), contextWithInput(20_000), 32_768)).toBe( + 32_768 - 20_000 - 4096, + ); + }); + + it("sends 1 token when the input already exceeds the window", () => { + expect(clampMaxTokensToContext(modelWith(943_718), contextWithInput(1_050_000), 943_718)).toBe(1); + }); +});