sscoble/pi-mono
1
0
Fork 0
mirror of https://github.com/badlogic/pi-mono.git synced 2026-11-05 23:37:25 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4

docs(chord): slice 10 points D9-D12, RouteHandle.retry, design §23 folded

Browse source
This commit is contained in:
Mario Zechner 2026-10-30 21:12:13 +00:00
parent f8497996fe
commit 9e4868a605
4 changed files with 93 additions and 37 deletions
Download patch file Download diff file Expand all files Collapse all files

21
packages/chord/docs/next/decisions.md
View file

@ -1907,6 +1907,25 @@ validation and copies the v2 kernel added around Delta.
trailing `/*` (the rest, possibly empty, `params["*"]`); the most specific match wins, segment by segment from the
left (exact before `:name` before `*`); one pattern from two contributions is a reported conflict, the first by
plugin name wins; `HEAD` falls back to the `GET` route.
(D9, "a") Worker keep-alive: the child reports `host.activity` and the app's `busy` counts (a JSON watchable of
numbers, e.g. Durable's active tasks) whenever they change; the server stops a worker whose counts were all zero for
`idleMs` (default 5 min, checked every second), never while it starts; a stop sends `shutdown` (the host closes),
the process tree killed after 5 s.
(D10, "a") A failed worker start uses the kernel's route backoff (1 s doubling to 60 s while holds exist,
`resolve_failed` reported); no sticky failure; `workers.state` shows the error and when it is tried again.
(D10b, "b") `host.route(Kind, options, ctx)` returns a `RouteHandle` with `retry(address)`: it ends that address's
backoff, so `resolve` runs now if the address has holders (a Retry button for a worker that failed to start, wired
by the app to the handle; the workers component has no `retry` of its own).
(D11, "a") Workers never build: the server's plugins component discovers and builds every plugin; a worker's start
message carries its kind's current scripts (`Built`, plain data) and a reload ships new ones (D12); the child's
`plugins` only evaluates (no esbuild, no watcher). So plugins are defined where the server can read them, one set for
every host (owner: "no cwd specific plugins, which might be alright; that way workers can live anywhere"); project
plugins (design §26 #14) would be discovered and built by the server too. A worker on another machine needs the
plugin directories there only for `chord.external` packages and `pluginFile`.
(D12, "a") A rollout step returns one report per host, `{ hosts: [{ host, report }] }` (7c's `RolloutStep` changes;
the in-process server step returns a list of one); the workers step reloads every running worker in parallel; a
worker that closed during it is listed with its error and does not stop the rollout; a plugin's dirty mark clears
only when it applied on every host of every step.
- **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):**
a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into
each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a
@ -2512,7 +2531,7 @@ From `research/chord-on-durable-objects.md`, adopted for the Chord v2 kernel bec
beyond a use (a pinned Durable phase holding an old table is fine). Natives reach other plugins only this way: no
`opaque` members, no natives in method results. What a native's old value does after its owner restarts is the
owner's `close` (a model runtime keeps working until collected; a closed SSH connection fails its in-flight `exec`).
- **HTTP is an optional Chord component** (`@earendil-works/chord/next/http`): a route gets a plain request summary
- **HTTP is an optional Chord component** (**superseded in part** by "Slice 10's points": `html` instead of `page`, request bodies through `json`/`body`, authentication one app hook, plugin sockets instead of a channel component until a second user) (`@earendil-works/chord/next/http`): a route gets a plain request summary
(method, path params, query, principal, selected headers; parsed and capped JSON bodies; uploads handed over in
chunks) and returns a response description (`json`, `page`, `bytes` with `size`, `mime` and `read(offset, length)`,
`redirect`, `error`). The component does ranges (206/416), HEAD, download headers, streaming, abort on disconnect and

67
packages/chord/docs/next/design.md
View file

@ -236,7 +236,7 @@ parent link closes. `resolve` returns the link before the worker has loaded (the
| `reload([{ name, facet }], context)` | the complete new plugin set after `load` (a facet passed as the same object is an unchanged plugin): staging, then one commit, then every object of a changed or removed plugin takes its fate; resolves with a report (a plugin whose new code fails keeps its old code, listed in `failed`); one at a time (section 21, slice 7b) |
| `connect(link, { principal, trusted?, codec? }, context)` | the link to the parent, at most one open; once it closed (a drop), `connect` takes the next, and every remote reference over the parent is held again on it. `principal` = who the parent is |
| `accept(link, { principal, trusted?, codec? }, context)` | a link from a child; holds and calls on it act as `principal` |
| `route(Kind, { authorize, resolve }, context)` | how to reach hosts of a kind (below); called before `load`, once per kind; work arriving on its links runs under `context` |
| `route(Kind, { authorize, resolve }, context)` | how to reach hosts of a kind (below); called before `load`, once per kind; work arriving on its links runs under `context`; returns a `RouteHandle` whose `retry(address)` ends that address's backoff (slice 10 D10b) |
| `get(Type, params, context, { readyTimeout? })` | a reference once its object is live, waiting through `loading`, `failed { transient }` and `failed { waiting }` up to its `readyTimeout` (section 7) |
| `hold(Type, params, context, { readyTimeout? })` | a reference at once that follows its object until released (section 7) |
| `close(context)` | closes the links (children, routes, then the parent), then ends every object at once and tears them down, the last to go live first (reason `close`); rejects with an `AggregateError` of the native closes that threw |
@ -308,7 +308,8 @@ host.route(Session, {
address's link closing while holds over it exist, starts the **per-address backoff** (1 s doubling to 60 s; a map
keyed by address, separate from objects' identity backoff). Meanwhile the holds are `loading`. Only a link that
completed its handshake resets the backoff. Without holds for the address nothing is resolved again, so passive
viewers recover without a user action while they hold.
viewers recover without a user action while they hold. `RouteHandle.retry(address)` (slice 10 D10b) ends an
address's backoff, so `resolve` runs at once if it has holders (a Retry button for a worker that failed to start).
- Chord never closes a route link itself except at `host.close`; stopping idle workers is the app's (`host.activity`).
- Work arriving on a route's links runs under the context given to `host.route(Kind, options, context)` (slice 4b).
@ -2099,46 +2100,68 @@ validation, no clone of snapshots or ops.
## 23. Optional components
Deferred: the HTTP and channel components and middleware phases are built after the kernel fixes and a first real
proof (harness, one tool with an approval pause, a code mode reload); the playground's web shell serves its pages
with plain app code until then. The design below stays the target.
The app kit's Node components (slice 10, `slice-10.md`, decision "Slice 10's points"): `chord/next/http` and
`chord/next/workers` in `src/next-node/`; the web page host and the TUI are slice 11. Each is ordinary code on Chord's
public API; nothing in the kernel depends on them.
### HTTP (`@earendil-works/chord/next/http`)
A route is `(req, ctx) => Promise<HttpResponse>`. `req` is a plain request summary (method, path params, query,
principal, provenance such as local or relayed, selected headers; parsed and capped JSON bodies; uploads handed over in
chunks); `ctx.principal` is the request's principal (from the request resolvers), so `ctx.get(Notes, …)` gives that
principal's view. A route returns a description:
`httpServer(host, { listen, authenticate, headers?, links? }, ctx)` listens first, holds every request and upgrade
until `load` resolved, then serves. `authenticate(request, ctx) => P | undefined` is the app's one hook (D3: who a
request is, and which credential wins, is app policy; no Chord registry of resolvers); `headers` are added to every
response, also those no route makes; `links.path` (`/chord`) upgrades to `host.accept(webSocketLink(ws), { principal,
limits })`, a socket without a principal refused. `ws` is an optional peer dependency (D7).
The registry `Http` (`"chord.http"`) is defined by the component and served by its own facet (`http.facet`, loaded
with the plugins, D1); plugins contribute `routes`, `wraps` and `sockets`. A route is `(req, ctx) =>
Promise<HttpResponse>`, run in a `host.root` scope with the request's principal (`ctx.principal`) and a signal aborted
when the client disconnects; `req` has `method`, `path`, `params`, `query`, `headers`, `origin`, `json(limit?)` and
`body(limit?)` (D4: nothing read until asked; 1 MiB by default; past the limit 413). Patterns (D8): exact segments,
`:name`, a trailing `/*`; the most specific match wins, segment by segment from the left; one pattern from two
contributions is a reported conflict, the first by plugin name wins; `HEAD` falls back to `GET`. A request uses the
router as it was when it started.
```ts
| { kind: "json"; value } | { kind: "page"; title; body }
| { kind: "bytes"; name; mime; size; read(offset, length) }
| { kind: "redirect"; to } | { kind: "error"; status }
type HttpResponse = { readonly headers?: Readonly<Record<string, string>> } & ( // D5
| { kind: "json"; value: JsonValue; status?: number } | { kind: "html"; body: string; status?: number }
| { kind: "bytes"; mime: string; size: number; read(offset, length, ctx): Promise<Uint8Array>; download?: string; ranges?: boolean }
| { kind: "redirect"; to: string; status?: 302 | 303 | 307 } | { kind: "error"; status: number; message?: string });
```
The component does ranges (206/416), HEAD, download headers (safe filename, `nosniff`, CSP, caching), streaming,
abort on disconnect and the CSP nonce once, instead of every plugin. Contributions hold `routes` (composed with
`keyed`; a duplicate pattern is a reported conflict, the first by plugin name wins) and `wraps`:
The component does ranges (206/416), `HEAD`, safe download headers, 64 KiB reads with backpressure stopped on
disconnect, and a 500 without the message for a throwing route; `fileResponse(path, options)` serves a file.
```ts
type Wrap = {
name: string;
phase: "observe" | "headers" | "transform"; // owner-defined phases, outermost first
phase: "observe" | "headers" | "transform"; // the component's phases, outermost first
scope?: "own" | "all"; // default "own": only this contributor's routes
route?: string; // only this route; a missing route is a reported conflict
wrap: (req, next, ctx) => Promise<HttpResponse>;
};
```
Within a phase, wraps run in contributor order (no numeric priorities: plugins have no single registration order, as
Express or Hono apps do; Fastify's lifecycle phases and encapsulation are the model). Wraps may add a security boundary
on top of an owner's checks, never instead of them. A request uses the router as it was when it started. Request
resolvers (who sent a request: cookie, loopback, relay) are hooks over the summary.
Wraps (D2) run within a phase in contributor order (no numeric priorities: plugins have no single registration
order; Fastify's lifecycle phases and encapsulation are the model). Wraps may add a security boundary on top of a
route's checks, never instead of them. `sockets` (D6): `{ "/terminal": (socket, req, ctx) => … }`, a `ws` WebSocket
after the origin check and `authenticate` (no principal: 403 before plugin code), for a plugin's own protocol; wraps do
not apply.
### Workers (`@earendil-works/chord/next/workers`)
`createWorkers({ entry, start, plugins, idleMs?, onOutput? }, ctx)` gives a route's `resolve` (a forked worker per
address, IPC, a start message with the app's start data and the worker kind's built scripts, `ready` within 60 s), a
`state` watchable per address and `stop(address)`; `workerMain(fn)` is the child's side. One live worker per address
(a kernel lock taken by the child before it opens its store). Keep-alive (D9): the child reports `host.activity` and
the app's `busy` counts; the server stops a worker idle (all zero) for `idleMs` (5 min). A failed start uses the
route's backoff (D10); `RouteHandle.retry(address)` (D10b) ends it. Workers never build (D11): the server's plugins
component builds every plugin; workers evaluate shipped scripts, and `workers.step(name)` ships a rollout's scripts to
every running worker (D12: one report per host).
### Channels
WebSocket byte channels (`send`, `onData`, `close`) for terminals, separate from object links, with their own flow
control (acknowledged windows).
WebSocket byte channels with acknowledged windows, separate from object links, wait for a second user after the
terminals (slice 10 D6); they would be built on top of plugin sockets.
### Chunked transfers

35
packages/chord/docs/next/slice-10.md
View file

@ -1,6 +1,6 @@
# Slice 10: HTTP and workers (spec)
Status: **draft** (owner, 2026-10-30: "do it"; the points marked **Decision needed** go to the owner one at a time).
Status: **points decided** (owner, 2026-10-30: D1-D12 in the decision "Slice 10's points"); next: spec reviews.
Slice 10 builds two Node components of the app kit (decision "Slice 7b's open points" (6)): `chord/next/http` (the
server's HTTP and WebSocket entry, routes contributed by plugins through a registry, responses as descriptions) and
`chord/next/workers` (a worker process per host address, started on demand, stopped when idle, its plugins shipped from
@ -144,7 +144,8 @@ const workers = createWorkers({
idleMs: 300_000,
onOutput: (address, stream, chunk) => logs.append(address.key, stream, chunk),
}, ctx);
server.route(Session, { authorize: app.authorizeSession, resolve: workers.resolve }, ctx);
const sessions = server.route(Session, { authorize: app.authorizeSession, resolve: workers.resolve }, ctx);
sessions.retry(Session({ id })); // a Retry button: ends the address's backoff (D10b)
workers.state // JSON watchable: { [address key]: { status, pid?, startedAt?, error? } }
await workers.stop(Session({ id }), ctx); // archive; the next hold starts it again
```
@ -195,19 +196,27 @@ returns `{ hosts: [{ host, report }] }`, D12). A worker starting during or after
start message. A worker that fails to apply (closed, aborted) is listed and does not stop the rollout; a plugin kept old
on any worker stays dirty (7c's rule).
> **Decision needed D9: keep-alive inputs.** Chord's `host.activity` plus app counts (`busy`: a JSON watchable of
> numbers, e.g. Durable's active tasks) reported to the server; `idleMs` default 5 min; checked every second.
> Recommended (p2's rule, with Chord's counts instead of p2's own).
> **Decided D9 (owner, 2026-10-30: "a"):** Chord's `host.activity` plus the app's `busy` counts (a JSON watchable of
> numbers, e.g. Durable's active tasks), reported to the server when they change; `idleMs` default 5 min, checked every
> second; never while the worker starts.
> **Decision needed D10: failed starts.** p2 keeps a failed start sticky until the owner presses Retry; with Chord's
> route backoff a failed start is retried after 1 s, doubling to 60 s, while holds exist. Recommended: the kernel's
> backoff (no sticky error); `state` shows the error, and `workers.retry(address)` skips the backoff (a Retry button).
> **Decided D10 (owner, 2026-10-30: "a"):** the kernel's route backoff (1 s doubling to 60 s while holds exist,
> `resolve_failed`), no sticky failure; `state` shows the error and when the next try is.
>
> **Decided D10b (owner, 2026-10-30: "b"):** a kernel change: `host.route(Kind, options, ctx)` returns a `RouteHandle`
> with `retry(address)`, which ends that address's backoff so `resolve` runs now if the address has holders (nothing
> without holders); the app wires its Retry button to the handle (`sessions.retry(Session({ id }))`); the workers
> component has no `retry` of its own.
> **Decision needed D11: scripts shipped to workers.** As above, against each worker building its own plugins (240 ms
> of esbuild per cold start, and a second build per worker for one edit). Recommended.
> **Decided D11 (owner, 2026-10-30: "a"):** workers never build; the server discovers and builds every plugin, one set
> for every host; a worker's start message carries its kind's scripts and a reload ships new ones. Plugins are defined
> where the server can read them (owner: "no cwd specific plugins, which might be alright; that way workers can live
> anywhere"). A worker on another machine needs the plugin directories only for `chord.external` packages and
> `pluginFile` (both resolve in the plugin's directory).
> **Decision needed D12: a step over many hosts.** `RolloutStep.reload` returns `{ hosts: [{ host, report }] }` instead
> of one `ReloadReport` (7c's in-process step returns one host). Recommended.
> **Decided D12 (owner, 2026-10-30: "a"):** `RolloutStep.reload` returns `{ hosts: [{ host, report }] }` (7c's
> in-process step returns a list of one); workers reload in parallel; one that closed during the step is listed with its
> error and does not stop the rollout.
## 5. Not in slice 10
@ -222,7 +231,7 @@ transfers as a Chord helper (an owner pattern, design §23, unchanged); telemetr
made safe, a disconnect stopping `read`); bodies past their limit; a throwing route (500, reported); the links
endpoint (`host.accept`, a refused socket, the output cap); plugin sockets; a reload swapping a route while a request
runs on the old router; close. Workers: a start on the first hold, the lock, `ready` timeout, idle stop and a hold
keeping it, `busy` counts, a crash (reported, re-resolved while held), `stop`, `retry`, the rollout step (a worker
keeping it, `busy` counts, a crash (reported, re-resolved while held), `stop`, `RouteHandle.retry` (D10b), the rollout step (a worker
starting during it gets the new scripts), the child's shutdown and disconnect.
- GC: requests (their root scopes and bodies) and stopped workers' state collected; no listener left on a closed socket.
- Examples (each a folder, as 7c's): `40-http` (routes, a download with ranges, a socket), `41-workers` (a server

7
packages/chord/docs/next/slices.md
View file

@ -139,7 +139,7 @@ pattern, no code. Decided cuts are noted where they apply.
| 22 | message format, header schema, unknown ids, one message per commit per link, consumers apply ops, plugin errors | 3a |
| 22 | relays, size limits per part, compression at the origin, adapted bindings get diffs | 4a (relays), 4b (limits, compression), 7b |
| 22 | trace carriers (R5) | 9 |
| 23 | HTTP, channels, chunked transfers | after slice 9 (with the first real proof) |
| 23 | HTTP (routes, wraps, responses as descriptions, request bodies, plugin sockets), workers; channels and chunked transfers stay owner patterns until a second user | 10 |
| 24 | Durable integration: `chord(fn, context)` (Q6), a `RegistryReader` over composed registries; the harness object and its model and env registries are p3's (base code, owner, 2026-10-27) | 8 (Chord parts), p3 |
| 24 | `StoreStorage` | 6b |
| 27 | versions table: contracts (L1 done), registries 8, documents 6, jobs 5, link protocol 3a | as listed |
@ -180,6 +180,11 @@ deleted (the slice's spec and code are the record). Moved here from `carry-over.
- **`rebuild` of tables** (key and column-type changes): `evolveTable` has no edit for them; build one only when a
real case needs it. (Slice 6a's second spec review; out of slice 7b's scope, `slice-7b.md` §12.)
### After slice 11
- **Small examples** (owner, 2026-10-30: the 7c examples are complete but complex): once HTTP, workers, web and TUI
exist, a pass of small, easy-to-read examples, each showing one thing a plugin author or app does.
### Slice 9: telemetry
- Spans for routine calls and job runs (from slice 5).