From 378d45a05b3a2c3249804c7dedc88a58ea424728 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Wed, 4 Nov 2026 10:02:47 +0000 Subject: [PATCH] feat(chord): telemetry, spans across hosts (slice 9) Chord's telemetry schema; spans for calls, methods, relays, holds, activations, natives, loads, reloads, resolves, revocations, routines, jobs and HTTP requests; trace carriers on hold and call (bounded, printable ASCII) continued by relays and workers; events and host-wide metrics; the app kit passes an adapter to its hosts. renderTrace escapes control characters. Example 62 prints a trace across three hosts. --- packages/chord/CHANGELOG.md | 2 + packages/chord/docs/benchmarks.md | 40 ++ packages/chord/docs/design.md | 88 ++- .../chord/docs/guide-plugins-hosts/book.md | 56 +- .../chord/docs/guide-plugins-hosts/index.html | 44 +- packages/chord/docs/slices.md | 12 +- packages/chord/src/data/principal.ts | 3 +- packages/chord/src/data/validator.ts | 7 +- packages/chord/src/documents/definition.ts | 3 +- packages/chord/src/documents/documents.ts | 22 +- packages/chord/src/documents/persistence.ts | 3 +- packages/chord/src/documents/transaction.ts | 10 + packages/chord/src/host/README.md | 4 +- packages/chord/src/host/attempt.ts | 140 ++++- packages/chord/src/host/ctx.ts | 99 +++- packages/chord/src/host/host.ts | 154 +++++- packages/chord/src/host/internal.ts | 2 +- packages/chord/src/host/native.ts | 51 +- packages/chord/src/host/object.ts | 60 +- packages/chord/src/host/route.ts | 55 +- packages/chord/src/host/routines.ts | 94 +++- packages/chord/src/index.ts | 1 + packages/chord/src/jobs/job.ts | 107 +++- packages/chord/src/jobs/records.ts | 4 +- packages/chord/src/jobs/run.ts | 6 +- packages/chord/src/link/binary.ts | 26 +- packages/chord/src/link/channel.ts | 16 +- packages/chord/src/link/message.ts | 45 +- packages/chord/src/link/part.ts | 7 +- packages/chord/src/link/text.ts | 43 +- packages/chord/src/node/app/app.ts | 9 + packages/chord/src/node/app/worker.ts | 8 + packages/chord/src/node/http/http.ts | 62 ++- packages/chord/src/remote/consumer.ts | 17 +- packages/chord/src/remote/hold.ts | 18 +- packages/chord/src/remote/host-link.ts | 4 +- packages/chord/src/remote/provider.ts | 87 ++- packages/chord/src/remote/relay.ts | 119 +++- packages/chord/src/remote/remote-object.ts | 94 +++- packages/chord/src/state/batch.ts | 7 +- packages/chord/src/store/archive.ts | 5 +- packages/chord/src/store/json-column.ts | 3 +- packages/chord/src/store/live-query.ts | 3 +- packages/chord/src/store/store.ts | 13 +- packages/chord/src/telemetry-sqlite/index.ts | 4 + packages/chord/src/telemetry-sqlite/store.ts | 17 +- packages/chord/src/telemetry.ts | 514 ++++++++++++++++++ packages/chord/test/bench/telemetry.ts | 195 +++++++ .../chord/test/documents/documents.test.ts | 43 +- .../examples/62-telemetry/app/principals.ts | 7 + .../test/examples/62-telemetry/app/v1.ts | 17 + .../chord/test/examples/62-telemetry/main.ts | 79 +++ .../62-telemetry/plugins/tally/package.json | 5 + .../62-telemetry/plugins/tally/worker.ts | 46 ++ .../chord/test/examples/62-telemetry/start.ts | 61 +++ .../test/examples/62-telemetry/worker.ts | 9 + packages/chord/test/host/attempt.test.ts | 131 ++++- packages/chord/test/host/ctx.test.ts | 103 +++- packages/chord/test/host/host.test.ts | 76 ++- packages/chord/test/host/native.test.ts | 86 +++ packages/chord/test/host/route.test.ts | 35 +- packages/chord/test/host/routines.test.ts | 60 +- packages/chord/test/jobs/job.test.ts | 101 +++- packages/chord/test/jobs/persisted.test.ts | 42 +- packages/chord/test/link/channel.test.ts | 2 + packages/chord/test/link/message.test.ts | 103 ++++ packages/chord/test/node/http/http.test.ts | 89 +++ packages/chord/test/remote/provider.test.ts | 76 +++ .../chord/test/remote/remote-object.test.ts | 66 ++- packages/chord/test/support.ts | 28 +- packages/chord/test/telemetry-sqlite.test.ts | 14 + packages/chord/test/telemetry.test.ts | 457 ++++++++++++++++ packages/telemetry/CHANGELOG.md | 4 + packages/telemetry/src/render.ts | 27 +- packages/telemetry/test/render.test.ts | 19 + tsconfig.json | 1 + 76 files changed, 3929 insertions(+), 241 deletions(-) create mode 100644 packages/chord/src/telemetry.ts create mode 100644 packages/chord/test/bench/telemetry.ts create mode 100644 packages/chord/test/examples/62-telemetry/app/principals.ts create mode 100644 packages/chord/test/examples/62-telemetry/app/v1.ts create mode 100644 packages/chord/test/examples/62-telemetry/main.ts create mode 100644 packages/chord/test/examples/62-telemetry/plugins/tally/package.json create mode 100644 packages/chord/test/examples/62-telemetry/plugins/tally/worker.ts create mode 100644 packages/chord/test/examples/62-telemetry/start.ts create mode 100644 packages/chord/test/examples/62-telemetry/worker.ts create mode 100644 packages/chord/test/telemetry.test.ts diff --git a/packages/chord/CHANGELOG.md b/packages/chord/CHANGELOG.md index 55812b7ef..e6b0922b9 100644 --- a/packages/chord/CHANGELOG.md +++ b/packages/chord/CHANGELOG.md @@ -21,6 +21,8 @@ - Chord v2 gains live queries: `self.query(table, q, limit)` in the factory of a table's owner is a watchable of the query's rows (at most `limit`, in its order), usable as a state member and a derive source. Chord runs it before the object goes live and again inside the commit step of every commit of the object that writes the table, publishing only the diff in that commit's batch; a commit writing no table with a live query runs nothing (`chord.query.runs` counts the runs). Only the object's own tables; the limit is required (`invalid/args` at declaration). A run that fails (a row the code cannot read) is a failed member, reported once as the new `HostError` `query_failed { object, table, error }`, and heals at the next good run. A live query is no native, so an object showing table rows live swaps on reload instead of restarting, as reading them in `self.open` made it. Each touching commit runs the query once (100 rows: about 60 µs, 80 µs watched): not for tables written hundreds of times a second. - Chord v2 gains messages and languages in the app kit (design §23): `defineMessages(namespace, { en: {...}, de: {...} })` returns a typed `t(key, ...args)` (the first catalog types the others: a missing key, an unknown one or a wrong argument is a type error; values are strings or functions), `t.message(key, ...args)` builds a message as data (`MessageData`, `{ namespace, key, params }`, for state members typed with `MessageSchema`), `t.render(message)` renders one (a foreign namespace or an unknown key from a link shows the key), `t.translations` lists its catalogs, and `MessagesOf` types a translation. The `Translations` registry takes a list of `{ namespace, locale, messages, override? }` per contributor (one catalog per namespace and locale: the first by plugin name wins, reported; `override: true` wins); a missing key falls back down the locale's chain (`pt-BR`, `pt`) to the first catalog, reported once. `LocaleChoice` keeps one locale per principal on the server (the theme choice's store, now shared as one preference); a client follows it with `followActiveLanguage`, defaulting to `navigator.languages` on a page and the system locale in the terminal (`startTerminal`'s `locales`), matched against the catalogs, switched live. `useMessages(t)` (`/react`) and `TuiScope.messages(t)` (`/tui`) render in the host's language and re-render or redraw on a change; `startPage` sets `lang` and `dir` on the page (`followLanguage`, `/web`); `formatTime` and `formatNumber` use `Intl` in the language being rendered. The kit serves `Translations` on the server, workers, pages and terminals and `LocaleChoice` on the server; client errors gain `{ kind: "messages", report }`. +- Chord v2 gains telemetry across hosts (slice 9): `CHORD_TELEMETRY_SCHEMA` (root export) types every span, event and metric Chord records. With a host adapter (`createHost({ telemetry })`), calls (`chord.call`), methods (`chord.method`, local and remote), relays (`chord.relay`), holds (`chord.hold`), activations (`chord.activate`, ending `live`, `failed`, `parked` or `ended`, with `chord.retry` and `chord.park` events), natives' opens and closes, `load`, `reload` (a `chord.reload.fate` event per object), resolves (a worker start included), revocations, routine calls, job phases (`intent`, `run`, `finish`: a persisted run resumed in a new attempt or process stays under its stored carrier) and HTTP requests (`chord.http`: method, route pattern, status, never the path) are spans; failures carry a fixed category and the message cut at 500 characters, a cancel is no error; commits are `chord.commit { documents, ops, duration }` events of the span they ran under. `hold` and `call` carry the caller's trace carrier (`trace`, at most 8 keys, keys and values of at most 256 characters of printable ASCII, else the message fails the message schema and the link closes; the binary codec's kind byte flag `0x40`), relays forward their own span's, and the provider runs the method under it (an absent or unknown carrier: the adapter's root); a reference's carrier rides on its first hold only, a re-hold after a reconnect being a fresh root linked to it. A hold's `type` and its address's `kind` are bounded at 256 characters by the message schema, and span attributes a peer chose at 256 characters. `ctx.trace(name, attributes, fn)` runs `fn` in a child span with a child ctx. Host-wide metrics: `chord.link.bytes`, `chord.link.state_messages`, `chord.commit.duration`, `chord.calls.in_flight`, `chord.holds.active`, `chord.links.open`, `chord.park.duration`, `chord.backpressure.wait`. Without an adapter nothing is started, built or sent: a call over a link costs what it did before. `createHttp`, `startServer` and `startWorker` take a `telemetry` adapter (`createHttp` continues a W3C `traceparent` only from a request a contributed relay verified: an app's proxy must strip it from browsers). A host with an adapter counts the cost counters of its links' and root scopes' work on it also when the app passed a context without a span. The SQLite telemetry store keeps spans of one millisecond in the order they started. + ### Changed - Chord v2's HTTP component denies by default (slice 13): a request or plugin socket without a principal gets 401 before plugin code runs unless its route or socket is marked `public: true` (`{ handler, public: true }`; a `crossSite` route is public), and the `/chord` links endpoint refuses requests without a principal. diff --git a/packages/chord/docs/benchmarks.md b/packages/chord/docs/benchmarks.md index 4f5003322..749e26188 100644 --- a/packages/chord/docs/benchmarks.md +++ b/packages/chord/docs/benchmarks.md @@ -994,3 +994,43 @@ The diff depends on the reuse: with every row decoded anew, `diffRevisions` comp previous 100 by content and the watched commit took 885 µs; with unchanged rows the same objects it compares them by identity. "There may be dragons": at 100 rows a table written more than a few thousand times a second spends most of its time in the query. + +## Slice 9: telemetry (2026-11-04) + +`test/bench/telemetry.ts`, Node 24.4.1, CPU 10, medians of nine bursts after a warm-up, three runs each; heap +allocated per operation includes what the collections during a burst freed. "Before" is the same script (`--noop`, +its imports pointed at a checkout of the commit before slice 9) run alternately with this one. + +A call over a memory link (a client host holding a remote counter on a server host; `bump` returns at once, `add` +commits one document change): + +| Call | Before slice 9 | NOOP adapter | In-memory adapter on both hosts | +| --- | --- | --- | --- | +| over a link | 10.46-10.64 µs, 13,083 B | 10.24-10.30 µs, 12,766 B | 31.3 µs, 28,886 B | +| that commits | 32.91-34.06 µs, 28,898 B | 31.44-31.61 µs, 28,598 B | 64.6 µs, 53,902 B | + +With the NOOP adapter the span code is within noise (the target): every span site is one identity check of its parent +against the NOOP context's span starter, and nothing else runs (no span, no attribute object, no carrier). A call +allocates 317 B less than before slice 9 (300 B less for one that commits), from the review's three code shapes (fix +round): the call message takes its optional `trace` as a plain key (a key holding `undefined`, which both codecs and the +schema treat as absent) instead of a conditional spread, `ChordObject.call`'s optional `parent` has no default +expression (an async function with a non-simple parameter list allocated 48 B more), and `ProviderSide.#run` is a +`.then` chain instead of an `async` function (its frame, about 85 B, held the extra argument). The first build's 206 B +more per call were these shapes, not the `trace` field. Recording costs what the adapter does: the in-memory one keeps +every span and measurement (each call records `chord.call`, `chord.method`, the commit's event, the host metrics, +`chord.link.bytes` per message and, since the fix round, the links' cost counters on the host's adapter), about 21 µs +and 16 KB per call here. + +Container width (R24, design §12): one item of a flat array changed per commit (a new text each time), alone and +under a watched `mapShared` derive of the array (one mapper run per change: only the changed item is mapped again; +the derive still visits every element once by identity and `diffRevisions` compares the arrays): + +| Width | One item changed | Under a watched `mapShared` derive | +| --- | --- | --- | +| 1,000 | 43.7-45.8 µs, 27.8 KB | 89.0-97.5 µs, 40.8 KB | +| 10,000 | 84.5-88.7 µs, 131.7 KB | 299.8-310.0 µs, 262.8 KB | +| 100,000 | 552.5-670.1 µs, 819.7 KB | 3,711-4,197 µs, 1,624.8 KB | + +A change costs O(width): the array's slots are copied (about 8 bytes per element) and a derive over it walks them. The +documented limit: up to about 10,000 elements a list in one document stays below 0.3 ms per change with a derive; past +that, keyed documents or a bounded window (design §12) keep commits O(change). diff --git a/packages/chord/docs/design.md b/packages/chord/docs/design.md index e272d9004..f3c165e54 100644 --- a/packages/chord/docs/design.md +++ b/packages/chord/docs/design.md @@ -912,8 +912,11 @@ Every handler gets a `ctx` (a `Context`): `telemetry` (the current span, section cancelled or the object ended; there is no per-call deadline), `principal` (the caller as its link stamped it; on a local object's handler the host's principal, none on a client), `incarnation` (the calling host's, section 4), and `get(Type, params, { readyTimeout? })` (references as the caller sees them, released when the handler settles; a -`get` after that throws `released`). Later: `ctx.trace(name, attributes, fn)` gives `fn` a child `ctx` under a child -span (slice 9, cut C11 to decide); in a job's `run` also `jobId` and `state` (slice 5). Pass `ctx` to everything you wait +`get` after that throws `released`), and `trace(name, attributes, fn)`: `fn` in a child span of `ctx.telemetry` with a +child `ctx` (the same principal, incarnation, signal and `get`, its references the handler's), so calls inside are +parented under it; a throw marks the span failed (`trace_failed`) and is rethrown; without an adapter that records, +`fn` gets `ctx` itself (slice 9; round-3 cut C11 kept, as this section already stated it); in a job's `run` also `jobId` +and `state` (slice 5). Pass `ctx` to everything you wait on. Code outside objects gets a `ctx` from `host.root` (slice 8). Durable's `Context` is the same module (`../context`); bridging is explicit. No code versions, no request ids. @@ -2207,8 +2210,17 @@ validation, no clone of snapshots or ops. The asynchronous inflate runs one part at a time; the queue behind it is bounded by the sending peer's window and N, so only a misbehaving parent or route peer could grow it (accepted: those are the app's own hosts). The adapter reports "too large" and "corrupt" as a tagged result. Limits and the threshold count bytes. No per-link compression. -- **Trace carriers (R5; slice 9)** are an optional bounded field of `call` and `hold` (key count, string lengths); - relays read them from the envelope. +- **Trace carriers (R5; slice 9)** are an optional field `trace` of `call` and `hold`: an adapter's string record of at + most 8 keys, keys and values of at most 256 characters of printable ASCII (keys without spaces; W3C's headers and + Chord's adapters fit); past the bounds, with a control character or not a string record, the message fails `MESSAGE` + as any malformed header and the link closes (a trusted link skips the check; the adapter's `extract` then checks it). + A carrier names stored trace and parent ids, so no control character reaches a renderer (and `renderTrace` escapes + them anyway). Names a relay records unparsed are bounded the same way: a hold's `type` and its address's `kind` have + at most 256 characters. The + binary codec flags it in the kind byte (`0x40`, beside the origin's `0x80`) and writes it as JSON after the origin; + the text codec writes it as the last element (a hold's after `limit`, absent ones before it `null`; a call's after its + payload, whose two fields are then positional, `null` when absent). Relays read it from the envelope and forward + their own span's carrier (a relay recording nothing forwards it as it came). `PROTOCOL_VERSION` stays 1. - **Adapted bindings get diffs** (slice 7b; decision "References are the only indirection" (5)): a consumer bound through `accepts` attaches to the object instance's shared mapping for its version: per member one derive of the instance's member through `(current) => old` (a changed member), of its whole state (a removed member), or the @@ -2598,7 +2610,6 @@ this document): | 16 | a Durable Object profile | 24 | when a DO target becomes concrete | | 24 | a per-type drain override: a host option or fixed (no install watchdog: `reload` takes the app's context, decision "Slice 7b's open points" (4); the routine handover uses `closeTimeout`; a failing job `finish` backs off 1-60 s and blocks after five failures, decided) | 3 | if a real case needs it | | 28 | Astra's circuit breaker (a hard transport-buffer cap plus a stall deadline); the heartbeat is built, the cap was never decided | 3 | if a real case needs it | -| 29 | round-3 cut C11 (`ctx.trace`); C3 and C4 are decided (§14, §15) | 8 | slice 9 | Apps (p3), from the functionality map, decided per domain when p3 is designed: @@ -2736,7 +2747,72 @@ Chord reports traces, events and metrics through `@earendil-works/pi-telemetry`; (`pi-telemetry/render`) prints links and never follows them. A web trace viewer in the playground reads the store. - **Durable** (done): submit and phase spans, stored carriers on submissions and tasks (`TaskOptions.trace`, creation links), events after commit, report events; pi-ai emits `pi.ai.request`/`pi.ai.poll` from one observer in - `lazyStream`. Persisted Chord jobs store their start carrier. + `lazyStream`. Persisted Chord jobs store their start carrier. A worker's Harness gets its host's adapter + (`HarnessOptions.telemetry`) and a method that submits input passes its `ctx` (its `telemetry` the `chord.method` + span), so `pi.durable.submit` links to it and starts its own trace, as decided. + +**Built (slice 9, `slice-9.md`; `src/telemetry.ts`):** + +- **The schema:** `CHORD_TELEMETRY_SCHEMA` (root export, `defineTelemetrySchema`) declares every span, event and metric + Chord records with its attributes; the cost counters (`chord.data.*`, `chord.link.*`, `chord.store.*`, + `chord.query.runs`, `chord.cycle.visits`, `chord.derive.*`, `chord.registry.*`, `chord.contract.adapts`) are recorded + through it (`chordMetrics(metrics)`: typed instruments made once per adapter's metrics object and name, a `WeakMap`; + the NOOP metrics share one set). Ids (object labels, run ids, address keys) are span attributes only. +- **Spans:** `chord.call` (a consumer's call of a remote member, until its reply), `chord.method` (a provider running a + method: under the call's carrier over a link, the caller's span on the same host; in `Attempt.call`, so local and + remote calls alike), `chord.relay` (a relayed hold until it is held upstream, a relayed call until its reply is + forwarded; the member is `hold` or `#`: a relay never knows names), `chord.hold` (a remote reference's hold + until its first `bound` or failure; its carrier rides on the reference's first hold only: a re-hold after a reconnect + is a fresh `chord.hold` root of the host's adapter linked to the first, and a relay's re-hold upstream after a drop a + fresh `chord.relay` root linked to its first, so a page kept open for days never adds children under a span long + ended), `chord.activate`, + `chord.open` (under the activation), `chord.close` (a root), `chord.load`, `chord.reload` (a `chord.reload.fate` + event per object), `chord.resolve`, `chord.revoke` (recorded after the synchronous revocation, its `count`), + `chord.routine` (a root per call), `chord.job` (`intent` under the starter, `run` and `finish` under the start's + carrier), `chord.http`. Outcomes are attributes; a failure is `status: "error"` with a fixed category (`call_failed`, + `method_failed`, `relay_failed`, `hold_failed`, `activation_failed`, `open_failed`, `close_failed`, `load_failed`, + `reload_failed`, `resolve_failed`, `routine_failed`, `job_failed`, `http_failed` for a status of 500 or more, + `trace_failed`) and the message cut at 500 characters; a cancel (the caller's signal, `aborted`, an open its attempt + abandoned) sets status ok explicitly (an adapter marks a span failed when its callback throws). String attributes a + peer chose (an object's label with its params key, an address key, a principal's kind, names from the wire) are cut + at 256 characters: stores keep them for days. +- **Activations:** one span per attempt on the host's adapter, linked to the object's first trigger (the span that + created it, or a hold's carrier), a retry's to the attempt before it; it ends `live`, `failed` (a transient failure's + backoff a `chord.retry` event) or `ended`, or `parked` with a `chord.park { cause }` event, and a parked attempt's own + steps resume in a new span linked to the parked one (spans stay short); one that fails or ends while still parked + records it in a short span linked to the parked one. A getter binding to an object while it + activates links its span to the activation (shared work); a route's first hold links to its resolve and back. +- **Parents across processes are carriers, not spans:** a settled span starts no recorded children, so whatever runs + later under work that has ended (a job's run and finish, a re-hold) goes through a carrier extracted with the host's + adapter. Every job run keeps its start's carrier (a persisted one stores it): a run resumed in a new attempt or + process stays in its trace. +- **`ctx.trace`** gives `fn` a child of the same kind (a job run's keeps `jobId` and `state`), and a synchronous throw of + `fn` rejects whether or not anything records: recording changes nothing `fn` sees but its span. +- **Routines:** each call's span links to the span of the commit that changed its source when the watch frame's + context names one (the newest folded frame's; the first call, with the current value, has none), and a call that + threw carries its backoff as a `chord.retry` event. +- **Cost (§30):** a span site checks `untraced(parent)` (the parent starts spans as the NOOP context does: no adapter can + record) and then + starts no span, builds no attribute object and injects no carrier; with the NOOP adapter a call over a link is within + noise of the code before slice 9 (`benchmarks.md`). A recording host's call costs its spans (`OpenSpan`, a span kept + open until its work's promise settles). +- **Events and metrics:** `chord.commit { documents, ops, duration }` on the span of the context a transaction ran + under (a method, a routine, a job phase, an HTTP request), only for a commit that changed something; the host-wide + metrics go to the host's adapter (`host.metrics`), not to a context: `chord.link.bytes { direction }` (every encoded + message of the host's links), `chord.link.state_messages`, `chord.commit.duration`, `chord.calls.in_flight`, + `chord.holds.active`, `chord.links.open`, `chord.park.duration`, `chord.backpressure.wait`. The cost counters still + count through the context of the work; a host with an adapter gives its links' and root scopes' work, when the app + passed a context without a span (`BACKGROUND_CONTEXT`, as the kit does), a span that records nothing but carries the + adapter's metrics (`countingOnly`), so they reach the adapter in an app too. +- **Components:** `createHttp({ telemetry })` makes each routed request a `chord.http` root (method, route pattern, + status; never the path), its scope's `ctx` under it; a W3C `traceparent` (with its `tracestate`) is extracted (as the + carrier `{ traceparent, tracestate }`) only from a request a contributed relay verified, each header at most 256 and + 512 characters of printable ASCII (else ignored). A verified relay vouches for the proxy, not the browser behind it: + an app's proxy must strip `traceparent` and `tracestate` from browsers' requests (or set its own), or a browser + chooses the trace its requests join. `startServer({ telemetry })` passes the adapter to + its host and HTTP; `startWorker({ telemetry })` to its host. Each process exports to the store it is given; a shared + SQLite file per data directory is the app's choice (example 62). The SQLite store orders spans of one millisecond as + their recorder started them (by start time, then id length and id). ## 30. Cost model diff --git a/packages/chord/docs/guide-plugins-hosts/book.md b/packages/chord/docs/guide-plugins-hosts/book.md index a66d72075..6561d2ceb 100644 --- a/packages/chord/docs/guide-plugins-hosts/book.md +++ b/packages/chord/docs/guide-plugins-hosts/book.md @@ -13,7 +13,7 @@ note: | `slice-10.md`, `slice-12.md`, `slice-13.md`, the decision log and the code in `packages/chord/src`. API names are the current ones. The early chapters wire hosts by hand to show the parts; the last chapter shows the app kit (`chord/app`), which does that wiring for an app. -colophon: Hosts and Plugins · third edition · 2026-11-03 · Chord v2 slices 1-10, 12 and 13, messages and languages +colophon: Hosts and Plugins · fourth edition · 2026-11-04 · Chord v2 slices 1-13, messages and languages, telemetry --- # Before we begin {.preface} @@ -809,6 +809,60 @@ writes them as one archive (a manifest, the store's export, the files) and stops own store at one consistent point. `app.importData(path, Project({ id: "copy" }), ctx)` makes a new project from it and refuses one that already has data (example 55). +# Seeing a Request Across Hosts {#telemetry lede="One trace from a click in a client through the server's relay into a worker's method: adapters, carriers and spans."} + +A tab calls `notes.add(…)`. The call leaves the tab, the server relays it, the project's worker runs the method, the +method commits. When it is slow, where did the time go? Each host is a different process, so no single log has the +answer. Chord's telemetry makes the four steps one **trace**: a tree of **spans** (named, timed pieces of work), each +host recording its own with its own **adapter**, joined by **carriers** (a few strings naming the caller's span) that +ride on the wire with every `hold` and `call`. + +Give each host an adapter. With the app kit that is one option per process; the SQLite store +(`@earendil-works/chord/telemetry-sqlite`) records settled spans and metrics, and here every process of the app opens +the same file in the data directory (example 62): + +```ts +// start.ts: the server +const telemetry = await createSqliteTelemetry(await openNodeSqliteDatabase(join(dataDir, "telemetry.sqlite"))); +await startServer({ ...options, telemetry }, ctx); +// worker.ts: each worker, its own recorder on the same file (the path in its environment) +startWorker({ api, kinds, telemetry: await openTelemetry(process.env.TALLY_TELEMETRY!), onError }); +``` + +Nothing else changes in plugin code. A method's `ctx.telemetry` is its `chord.method` span, so whatever it calls or +commits is parented under it, on its host or the next one. `ctx.trace(name, attributes, fn)` adds a span of your own: + +```ts +add: async (n, ctx) => { + await ctx.trace("tally.check", { n }, async () => { + if (!Number.isSafeInteger(n) || n <= 0) throw new Error("add a positive count"); + }); + return store.add(n, ctx); +}, +``` + +The store renders the click's trace across the three hosts (the client's spans, the server's relay, the worker's +methods and the commit, an event of the method that made it): + +``` +tally.click + chord.hold type="tally.counter" object="tally.counter(t1)" address.kind="tally" outcome="bound" + chord.relay type="tally.counter" member="hold" upstream.kind="tally" outcome="ok" + → link to chord.resolve + chord.call type="tally.counter" member="add" object="tally.counter(t1)" address.kind="tally" outcome="ok" + chord.relay type="tally.counter" member="#0" upstream.kind="tally" outcome="ok" + chord.method type="tally.counter" member="add" object="tally.counter(t1)" principal.kind="owner" outcome="ok" + tally.check n=3 + chord.method type="tally.store" member="add" object="tally.store" principal.kind="worker" outcome="ok" + · chord.commit documents=1 ops=1 duration=1.9 +``` + +Work shared by several requests is its own trace, linked rather than nested: the worker's start (`chord.resolve`), +an object's activation (`chord.activate`, linked to whoever asked first). Routines and job phases are roots too; a job's +`run` and `finish` stay under the carrier its start stored, even when a restarted worker resumes the run. Metrics +(`chord.link.bytes`, `chord.commit.duration`, `chord.calls.in_flight`, ...) are aggregated per 10 s; ids are never their +attributes. Without an adapter (the default), Chord starts no span and sends no carrier: a call costs what it did before. + # Reference {.appendix} | Call | What it does | diff --git a/packages/chord/docs/guide-plugins-hosts/index.html b/packages/chord/docs/guide-plugins-hosts/index.html index b96ccf5c3..7c4184fe6 100644 --- a/packages/chord/docs/guide-plugins-hosts/index.html +++ b/packages/chord/docs/guide-plugins-hosts/index.html @@ -131,6 +131,7 @@ are the current ones. The early chapters wire hosts by hand to show the parts; t
  • VIII. Changing a Plugin
  • IX. Base Code and Plugins
  • X. The App Kit
    • +
  • XI. Seeing a Request Across Hosts
  • Appendix A: Reference
    • @@ -727,6 +728,47 @@ own store at one consistent point. app.importData(path, Project({ id: &quo refuses one that already has data (example 55).

    +
    XI

    Seeing a Request Across Hosts

    One trace from a click in a client through the server's relay into a worker's method: adapters, carriers and spans.

    +

    A tab calls notes.add(…). The call leaves the tab, the server relays it, the project's worker runs the method, the +method commits. When it is slow, where did the time go? Each host is a different process, so no single log has the +answer. Chord's telemetry makes the four steps one trace: a tree of spans (named, timed pieces of work), each +host recording its own with its own adapter, joined by carriers (a few strings naming the caller's span) that +ride on the wire with every hold and call.

    +

    Give each host an adapter. With the app kit that is one option per process; the SQLite store +(@earendil-works/chord/telemetry-sqlite) records settled spans and metrics, and here every process of the app opens +the same file in the data directory (example 62):

    +
    // start.ts: the server
+const telemetry = await createSqliteTelemetry(await openNodeSqliteDatabase(join(dataDir, "telemetry.sqlite")));
+await startServer({ ...options, telemetry }, ctx);
+// worker.ts: each worker, its own recorder on the same file (the path in its environment)
+startWorker({ api, kinds, telemetry: await openTelemetry(process.env.TALLY_TELEMETRY!), onError });
    +

    Nothing else changes in plugin code. A method's ctx.telemetry is its chord.method span, so whatever it calls or +commits is parented under it, on its host or the next one. ctx.trace(name, attributes, fn) adds a span of your own:

    +
    add: async (n, ctx) => {
+	await ctx.trace("tally.check", { n }, async () => {
+		if (!Number.isSafeInteger(n) || n <= 0) throw new Error("add a positive count");
+	});
+	return store.add(n, ctx);
+},
    +

    The store renders the click's trace across the three hosts (the client's spans, the server's relay, the worker's +methods and the commit, an event of the method that made it):

    +
    tally.click
+  chord.hold type="tally.counter" object="tally.counter(t1)" address.kind="tally" outcome="bound"
+    chord.relay type="tally.counter" member="hold" upstream.kind="tally" outcome="ok"
+      → link to chord.resolve
+  chord.call type="tally.counter" member="add" object="tally.counter(t1)" address.kind="tally" outcome="ok"
+    chord.relay type="tally.counter" member="#0" upstream.kind="tally" outcome="ok"
+      chord.method type="tally.counter" member="add" object="tally.counter(t1)" principal.kind="owner" outcome="ok"
+        tally.check n=3
+        chord.method type="tally.store" member="add" object="tally.store" principal.kind="worker" outcome="ok"
+          · chord.commit documents=1 ops=1 duration=1.9
    +

    Work shared by several requests is its own trace, linked rather than nested: the worker's start (chord.resolve), +an object's activation (chord.activate, linked to whoever asked first). Routines and job phases are roots too; a job's +run and finish stay under the carrier its start stored, even when a restarted worker resumes the run. Metrics +(chord.link.bytes, chord.commit.duration, chord.calls.in_flight, ...) are aggregated per 10 s; ids are never their +attributes. Without an adapter (the default), Chord starts no span and sends no carrier: a call costs what it did before.

    +
    +
    A

    Reference

    @@ -798,7 +840,7 @@ refuses one that already has data (example 55).

    -

    Hosts and Plugins · third edition · 2026-11-03 · Chord v2 slices 1-10, 12 and 13, messages and languages

    +

    Hosts and Plugins · fourth edition · 2026-11-04 · Chord v2 slices 1-13, messages and languages, telemetry