diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index e6034d216..e3061f3c1 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -1907,6 +1907,25 @@ validation and copies the v2 kernel added around Delta. trailing `/*` (the rest, possibly empty, `params["*"]`); the most specific match wins, segment by segment from the left (exact before `:name` before `*`); one pattern from two contributions is a reported conflict, the first by plugin name wins; `HEAD` falls back to the `GET` route. + (D9, "a") Worker keep-alive: the child reports `host.activity` and the app's `busy` counts (a JSON watchable of + numbers, e.g. Durable's active tasks) whenever they change; the server stops a worker whose counts were all zero for + `idleMs` (default 5 min, checked every second), never while it starts; a stop sends `shutdown` (the host closes), + the process tree killed after 5 s. + (D10, "a") A failed worker start uses the kernel's route backoff (1 s doubling to 60 s while holds exist, + `resolve_failed` reported); no sticky failure; `workers.state` shows the error and when it is tried again. + (D10b, "b") `host.route(Kind, options, ctx)` returns a `RouteHandle` with `retry(address)`: it ends that address's + backoff, so `resolve` runs now if the address has holders (a Retry button for a worker that failed to start, wired + by the app to the handle; the workers component has no `retry` of its own). + (D11, "a") Workers never build: the server's plugins component discovers and builds every plugin; a worker's start + message carries its kind's current scripts (`Built`, plain data) and a reload ships new ones (D12); the child's + `plugins` only evaluates (no esbuild, no watcher). So plugins are defined where the server can read them, one set for + every host (owner: "no cwd specific plugins, which might be alright; that way workers can live anywhere"); project + plugins (design §26 #14) would be discovered and built by the server too. A worker on another machine needs the + plugin directories there only for `chord.external` packages and `pluginFile`. + (D12, "a") A rollout step returns one report per host, `{ hosts: [{ host, report }] }` (7c's `RolloutStep` changes; + the in-process server step returns a list of one); the workers step reloads every running worker in parallel; a + worker that closed during it is listed with its error and does not stop the rollout; a plugin's dirty mark clears + only when it applied on every host of every step. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a @@ -2512,7 +2531,7 @@ From `research/chord-on-durable-objects.md`, adopted for the Chord v2 kernel bec beyond a use (a pinned Durable phase holding an old table is fine). Natives reach other plugins only this way: no `opaque` members, no natives in method results. What a native's old value does after its owner restarts is the owner's `close` (a model runtime keeps working until collected; a closed SSH connection fails its in-flight `exec`). -- **HTTP is an optional Chord component** (`@earendil-works/chord/next/http`): a route gets a plain request summary +- **HTTP is an optional Chord component** (**superseded in part** by "Slice 10's points": `html` instead of `page`, request bodies through `json`/`body`, authentication one app hook, plugin sockets instead of a channel component until a second user) (`@earendil-works/chord/next/http`): a route gets a plain request summary (method, path params, query, principal, selected headers; parsed and capped JSON bodies; uploads handed over in chunks) and returns a response description (`json`, `page`, `bytes` with `size`, `mime` and `read(offset, length)`, `redirect`, `error`). The component does ranges (206/416), HEAD, download headers, streaming, abort on disconnect and diff --git a/packages/chord/docs/next/design.md b/packages/chord/docs/next/design.md index 303b3fb50..f10c3b56c 100644 --- a/packages/chord/docs/next/design.md +++ b/packages/chord/docs/next/design.md @@ -236,7 +236,7 @@ parent link closes. `resolve` returns the link before the worker has loaded (the | `reload([{ name, facet }], context)` | the complete new plugin set after `load` (a facet passed as the same object is an unchanged plugin): staging, then one commit, then every object of a changed or removed plugin takes its fate; resolves with a report (a plugin whose new code fails keeps its old code, listed in `failed`); one at a time (section 21, slice 7b) | | `connect(link, { principal, trusted?, codec? }, context)` | the link to the parent, at most one open; once it closed (a drop), `connect` takes the next, and every remote reference over the parent is held again on it. `principal` = who the parent is | | `accept(link, { principal, trusted?, codec? }, context)` | a link from a child; holds and calls on it act as `principal` | -| `route(Kind, { authorize, resolve }, context)` | how to reach hosts of a kind (below); called before `load`, once per kind; work arriving on its links runs under `context` | +| `route(Kind, { authorize, resolve }, context)` | how to reach hosts of a kind (below); called before `load`, once per kind; work arriving on its links runs under `context`; returns a `RouteHandle` whose `retry(address)` ends that address's backoff (slice 10 D10b) | | `get(Type, params, context, { readyTimeout? })` | a reference once its object is live, waiting through `loading`, `failed { transient }` and `failed { waiting }` up to its `readyTimeout` (section 7) | | `hold(Type, params, context, { readyTimeout? })` | a reference at once that follows its object until released (section 7) | | `close(context)` | closes the links (children, routes, then the parent), then ends every object at once and tears them down, the last to go live first (reason `close`); rejects with an `AggregateError` of the native closes that threw | @@ -308,7 +308,8 @@ host.route(Session, { address's link closing while holds over it exist, starts the **per-address backoff** (1 s doubling to 60 s; a map keyed by address, separate from objects' identity backoff). Meanwhile the holds are `loading`. Only a link that completed its handshake resets the backoff. Without holds for the address nothing is resolved again, so passive - viewers recover without a user action while they hold. + viewers recover without a user action while they hold. `RouteHandle.retry(address)` (slice 10 D10b) ends an + address's backoff, so `resolve` runs at once if it has holders (a Retry button for a worker that failed to start). - Chord never closes a route link itself except at `host.close`; stopping idle workers is the app's (`host.activity`). - Work arriving on a route's links runs under the context given to `host.route(Kind, options, context)` (slice 4b). @@ -2099,46 +2100,68 @@ validation, no clone of snapshots or ops. ## 23. Optional components -Deferred: the HTTP and channel components and middleware phases are built after the kernel fixes and a first real -proof (harness, one tool with an approval pause, a code mode reload); the playground's web shell serves its pages -with plain app code until then. The design below stays the target. +The app kit's Node components (slice 10, `slice-10.md`, decision "Slice 10's points"): `chord/next/http` and +`chord/next/workers` in `src/next-node/`; the web page host and the TUI are slice 11. Each is ordinary code on Chord's +public API; nothing in the kernel depends on them. ### HTTP (`@earendil-works/chord/next/http`) -A route is `(req, ctx) => Promise`. `req` is a plain request summary (method, path params, query, -principal, provenance such as local or relayed, selected headers; parsed and capped JSON bodies; uploads handed over in -chunks); `ctx.principal` is the request's principal (from the request resolvers), so `ctx.get(Notes, …)` gives that -principal's view. A route returns a description: +`httpServer(host, { listen, authenticate, headers?, links? }, ctx)` listens first, holds every request and upgrade +until `load` resolved, then serves. `authenticate(request, ctx) => P | undefined` is the app's one hook (D3: who a +request is, and which credential wins, is app policy; no Chord registry of resolvers); `headers` are added to every +response, also those no route makes; `links.path` (`/chord`) upgrades to `host.accept(webSocketLink(ws), { principal, +limits })`, a socket without a principal refused. `ws` is an optional peer dependency (D7). + +The registry `Http` (`"chord.http"`) is defined by the component and served by its own facet (`http.facet`, loaded +with the plugins, D1); plugins contribute `routes`, `wraps` and `sockets`. A route is `(req, ctx) => +Promise`, run in a `host.root` scope with the request's principal (`ctx.principal`) and a signal aborted +when the client disconnects; `req` has `method`, `path`, `params`, `query`, `headers`, `origin`, `json(limit?)` and +`body(limit?)` (D4: nothing read until asked; 1 MiB by default; past the limit 413). Patterns (D8): exact segments, +`:name`, a trailing `/*`; the most specific match wins, segment by segment from the left; one pattern from two +contributions is a reported conflict, the first by plugin name wins; `HEAD` falls back to `GET`. A request uses the +router as it was when it started. ```ts -| { kind: "json"; value } | { kind: "page"; title; body } -| { kind: "bytes"; name; mime; size; read(offset, length) } -| { kind: "redirect"; to } | { kind: "error"; status } +type HttpResponse = { readonly headers?: Readonly> } & ( // D5 + | { kind: "json"; value: JsonValue; status?: number } | { kind: "html"; body: string; status?: number } + | { kind: "bytes"; mime: string; size: number; read(offset, length, ctx): Promise; download?: string; ranges?: boolean } + | { kind: "redirect"; to: string; status?: 302 | 303 | 307 } | { kind: "error"; status: number; message?: string }); ``` -The component does ranges (206/416), HEAD, download headers (safe filename, `nosniff`, CSP, caching), streaming, -abort on disconnect and the CSP nonce once, instead of every plugin. Contributions hold `routes` (composed with -`keyed`; a duplicate pattern is a reported conflict, the first by plugin name wins) and `wraps`: +The component does ranges (206/416), `HEAD`, safe download headers, 64 KiB reads with backpressure stopped on +disconnect, and a 500 without the message for a throwing route; `fileResponse(path, options)` serves a file. ```ts type Wrap = { name: string; - phase: "observe" | "headers" | "transform"; // owner-defined phases, outermost first + phase: "observe" | "headers" | "transform"; // the component's phases, outermost first scope?: "own" | "all"; // default "own": only this contributor's routes route?: string; // only this route; a missing route is a reported conflict wrap: (req, next, ctx) => Promise; }; ``` -Within a phase, wraps run in contributor order (no numeric priorities: plugins have no single registration order, as -Express or Hono apps do; Fastify's lifecycle phases and encapsulation are the model). Wraps may add a security boundary -on top of an owner's checks, never instead of them. A request uses the router as it was when it started. Request -resolvers (who sent a request: cookie, loopback, relay) are hooks over the summary. +Wraps (D2) run within a phase in contributor order (no numeric priorities: plugins have no single registration +order; Fastify's lifecycle phases and encapsulation are the model). Wraps may add a security boundary on top of a +route's checks, never instead of them. `sockets` (D6): `{ "/terminal": (socket, req, ctx) => … }`, a `ws` WebSocket +after the origin check and `authenticate` (no principal: 403 before plugin code), for a plugin's own protocol; wraps do +not apply. + +### Workers (`@earendil-works/chord/next/workers`) + +`createWorkers({ entry, start, plugins, idleMs?, onOutput? }, ctx)` gives a route's `resolve` (a forked worker per +address, IPC, a start message with the app's start data and the worker kind's built scripts, `ready` within 60 s), a +`state` watchable per address and `stop(address)`; `workerMain(fn)` is the child's side. One live worker per address +(a kernel lock taken by the child before it opens its store). Keep-alive (D9): the child reports `host.activity` and +the app's `busy` counts; the server stops a worker idle (all zero) for `idleMs` (5 min). A failed start uses the +route's backoff (D10); `RouteHandle.retry(address)` (D10b) ends it. Workers never build (D11): the server's plugins +component builds every plugin; workers evaluate shipped scripts, and `workers.step(name)` ships a rollout's scripts to +every running worker (D12: one report per host). ### Channels -WebSocket byte channels (`send`, `onData`, `close`) for terminals, separate from object links, with their own flow -control (acknowledged windows). +WebSocket byte channels with acknowledged windows, separate from object links, wait for a second user after the +terminals (slice 10 D6); they would be built on top of plugin sockets. ### Chunked transfers diff --git a/packages/chord/docs/next/slice-10.md b/packages/chord/docs/next/slice-10.md index ce1f56e13..5d9eb88d3 100644 --- a/packages/chord/docs/next/slice-10.md +++ b/packages/chord/docs/next/slice-10.md @@ -1,6 +1,6 @@ # Slice 10: HTTP and workers (spec) -Status: **draft** (owner, 2026-10-30: "do it"; the points marked **Decision needed** go to the owner one at a time). +Status: **points decided** (owner, 2026-10-30: D1-D12 in the decision "Slice 10's points"); next: spec reviews. Slice 10 builds two Node components of the app kit (decision "Slice 7b's open points" (6)): `chord/next/http` (the server's HTTP and WebSocket entry, routes contributed by plugins through a registry, responses as descriptions) and `chord/next/workers` (a worker process per host address, started on demand, stopped when idle, its plugins shipped from @@ -144,7 +144,8 @@ const workers = createWorkers({ idleMs: 300_000, onOutput: (address, stream, chunk) => logs.append(address.key, stream, chunk), }, ctx); -server.route(Session, { authorize: app.authorizeSession, resolve: workers.resolve }, ctx); +const sessions = server.route(Session, { authorize: app.authorizeSession, resolve: workers.resolve }, ctx); +sessions.retry(Session({ id })); // a Retry button: ends the address's backoff (D10b) workers.state // JSON watchable: { [address key]: { status, pid?, startedAt?, error? } } await workers.stop(Session({ id }), ctx); // archive; the next hold starts it again ``` @@ -195,19 +196,27 @@ returns `{ hosts: [{ host, report }] }`, D12). A worker starting during or after start message. A worker that fails to apply (closed, aborted) is listed and does not stop the rollout; a plugin kept old on any worker stays dirty (7c's rule). -> **Decision needed D9: keep-alive inputs.** Chord's `host.activity` plus app counts (`busy`: a JSON watchable of -> numbers, e.g. Durable's active tasks) reported to the server; `idleMs` default 5 min; checked every second. -> Recommended (p2's rule, with Chord's counts instead of p2's own). +> **Decided D9 (owner, 2026-10-30: "a"):** Chord's `host.activity` plus the app's `busy` counts (a JSON watchable of +> numbers, e.g. Durable's active tasks), reported to the server when they change; `idleMs` default 5 min, checked every +> second; never while the worker starts. -> **Decision needed D10: failed starts.** p2 keeps a failed start sticky until the owner presses Retry; with Chord's -> route backoff a failed start is retried after 1 s, doubling to 60 s, while holds exist. Recommended: the kernel's -> backoff (no sticky error); `state` shows the error, and `workers.retry(address)` skips the backoff (a Retry button). +> **Decided D10 (owner, 2026-10-30: "a"):** the kernel's route backoff (1 s doubling to 60 s while holds exist, +> `resolve_failed`), no sticky failure; `state` shows the error and when the next try is. +> +> **Decided D10b (owner, 2026-10-30: "b"):** a kernel change: `host.route(Kind, options, ctx)` returns a `RouteHandle` +> with `retry(address)`, which ends that address's backoff so `resolve` runs now if the address has holders (nothing +> without holders); the app wires its Retry button to the handle (`sessions.retry(Session({ id }))`); the workers +> component has no `retry` of its own. -> **Decision needed D11: scripts shipped to workers.** As above, against each worker building its own plugins (240 ms -> of esbuild per cold start, and a second build per worker for one edit). Recommended. +> **Decided D11 (owner, 2026-10-30: "a"):** workers never build; the server discovers and builds every plugin, one set +> for every host; a worker's start message carries its kind's scripts and a reload ships new ones. Plugins are defined +> where the server can read them (owner: "no cwd specific plugins, which might be alright; that way workers can live +> anywhere"). A worker on another machine needs the plugin directories only for `chord.external` packages and +> `pluginFile` (both resolve in the plugin's directory). -> **Decision needed D12: a step over many hosts.** `RolloutStep.reload` returns `{ hosts: [{ host, report }] }` instead -> of one `ReloadReport` (7c's in-process step returns one host). Recommended. +> **Decided D12 (owner, 2026-10-30: "a"):** `RolloutStep.reload` returns `{ hosts: [{ host, report }] }` (7c's +> in-process step returns a list of one); workers reload in parallel; one that closed during the step is listed with its +> error and does not stop the rollout. ## 5. Not in slice 10 @@ -222,7 +231,7 @@ transfers as a Chord helper (an owner pattern, design §23, unchanged); telemetr made safe, a disconnect stopping `read`); bodies past their limit; a throwing route (500, reported); the links endpoint (`host.accept`, a refused socket, the output cap); plugin sockets; a reload swapping a route while a request runs on the old router; close. Workers: a start on the first hold, the lock, `ready` timeout, idle stop and a hold - keeping it, `busy` counts, a crash (reported, re-resolved while held), `stop`, `retry`, the rollout step (a worker + keeping it, `busy` counts, a crash (reported, re-resolved while held), `stop`, `RouteHandle.retry` (D10b), the rollout step (a worker starting during it gets the new scripts), the child's shutdown and disconnect. - GC: requests (their root scopes and bodies) and stopped workers' state collected; no listener left on a closed socket. - Examples (each a folder, as 7c's): `40-http` (routes, a download with ranges, a socket), `41-workers` (a server diff --git a/packages/chord/docs/next/slices.md b/packages/chord/docs/next/slices.md index 8960bc67d..941d7c436 100644 --- a/packages/chord/docs/next/slices.md +++ b/packages/chord/docs/next/slices.md @@ -139,7 +139,7 @@ pattern, no code. Decided cuts are noted where they apply. | 22 | message format, header schema, unknown ids, one message per commit per link, consumers apply ops, plugin errors | 3a | | 22 | relays, size limits per part, compression at the origin, adapted bindings get diffs | 4a (relays), 4b (limits, compression), 7b | | 22 | trace carriers (R5) | 9 | -| 23 | HTTP, channels, chunked transfers | after slice 9 (with the first real proof) | +| 23 | HTTP (routes, wraps, responses as descriptions, request bodies, plugin sockets), workers; channels and chunked transfers stay owner patterns until a second user | 10 | | 24 | Durable integration: `chord(fn, context)` (Q6), a `RegistryReader` over composed registries; the harness object and its model and env registries are p3's (base code, owner, 2026-10-27) | 8 (Chord parts), p3 | | 24 | `StoreStorage` | 6b | | 27 | versions table: contracts (L1 done), registries 8, documents 6, jobs 5, link protocol 3a | as listed | @@ -180,6 +180,11 @@ deleted (the slice's spec and code are the record). Moved here from `carry-over. - **`rebuild` of tables** (key and column-type changes): `evolveTable` has no edit for them; build one only when a real case needs it. (Slice 6a's second spec review; out of slice 7b's scope, `slice-7b.md` §12.) +### After slice 11 + +- **Small examples** (owner, 2026-10-30: the 7c examples are complete but complex): once HTTP, workers, web and TUI + exist, a pass of small, easy-to-read examples, each showing one thing a plugin author or app does. + ### Slice 9: telemetry - Spans for routine calls and job runs (from slice 5).