docs(chord): slice 9 spec, telemetry with spans across hosts

packages/chord/docs/slice-9.md

@@ -0,0 +1,110 @@
+# Slice 9: telemetry (spec)
+
+Status: **draft for the build** (owner, 2026-11-03: "build i18n, then slice 9 (we said we want cross host spans)";
+2026-11-04: "go ahead"). Telemetry was decided on 2026-10-18 (decisions under "## Telemetry" in `decisions.md`, from
+"Context" to "Telemetry store") and is stated in design.md §29 and §30 (cost guards). Chord v2 so far records only
+metrics (cost counters such as `chord.data.validations`, `chord.link.encodes`) and stores a persisted job run's start
+carrier (slice 6b). This slice adds the spans, events and metrics of §29, carriers across links, and Chord's telemetry
+schema, so one trace follows a call from a page through the server's relay into a worker's method, Durable's phases
+and pi-ai's model request.
+
+Read in full before writing code: this spec; `decisions.md` (the telemetry decisions of 2026-10-18, R5 in "R4 and R5,
+protocol header schema", and every later entry that touches telemetry, links, relays, workers or jobs); `design.md`
+(all of it; §22 wire, §29 telemetry and §30 cost model in particular); `slices.md` (the feature map rows for §8
+`ctx.trace`, §12 the container-width benchmark, §22 trace carriers, §29; the carry-over "Slice 9"); pi-telemetry
+(`packages/telemetry/src`: `TelemetrySpan`, `TelemetryContext`, propagation, links, metrics, typed schemas, the in-memory
+adapter, `render.ts`, the conformance tests); Chord's `src/telemetry-sqlite/`; Durable's `harness/telemetry.ts` and how
+Durable starts its spans (the model for a typed schema); pi-ai's `pi.ai.request` wrapper; Chord's `src/context`,
+`src/host` (ctx, object, attempt, root, host, routines, jobs), `src/link`, `src/remote`, `src/node/http`,
+`src/node/workers`. Root `AGENTS.md` "Chord v2 Code Style" (binding).
+
+## 1. Chord's telemetry schema
+
+`CHORD_TELEMETRY_SCHEMA` (`src/telemetry.ts`, exported from the root), typed with pi-telemetry's
+`defineTelemetrySchema` like `DURABLE_TELEMETRY_SCHEMA`: every span, event and metric name Chord emits with its
+attributes. The existing cost counters (`chord.data.*`, `chord.link.*`, `chord.store.*`, `chord.query.runs`,
+`chord.cycle.visits`) are declared in it and recorded through it. Ids (object identity, hold, call, address key) are
+span attributes only, never metric attributes.
+
+## 2. Spans
+
+All spans are short and in one process (decision "Durable and AI spans"). Names and their parent:
+
+| Span | Where | Parent | Attributes (besides ids) |
+| --- | --- | --- | --- |
+| `chord.call` | a consumer's call of a remote member (`ref.method(args, ctx)`) | the caller's `ctx.telemetry` | type, member, address kind |
+| `chord.method` | a provider running a method | the call's carrier (remote) or the caller's span (local) | type, member, principal kind, outcome |
+| `chord.relay` | a relay forwarding a hold or a call | the arriving carrier | type, member or `hold`, upstream address kind |
+| `chord.hold` | a consumer's hold until bound (the first `bound` or failure) | the holder's span | type, outcome |
+| `chord.activate` | one activation attempt of an object (factory, natives opened, live or failed) | root, linked to its first trigger | type, attempt, outcome (live, failed, parked) |
+| `chord.open` / `chord.close` | one native's open / close | the attempt's span / root | type, native index |
+| `chord.load`, `chord.reload` | a host's load / reload | the caller's span | plugins, fates counts |
+| `chord.resolve` | a route resolving an address (a worker start included) | root, linked to its first trigger | kind, outcome |
+| `chord.revoke` | a revocation | the caller's span | count |
+| `chord.routine` | one routine run (`self.on`, `onLive` handler) | root, linked to the commit that triggered it when known | type, routine |
+| `chord.job` | one job run phase (intent, run, finish) | the stored carrier (persisted) or the starter's span | job, phase, outcome |
+
+- **Shared work** (one activation for several getters, a worker start for several holds) is its own root span linked
+  to its first trigger; each waiting span links to it (decision "Across links" / "Shared work").
+- **Failed spans** carry `status: "error"` with a fixed category (`activation_failed`, `call_failed`, ...) and the
+  error message truncated to 500 characters; cancelling is not an error (decision "Errors in telemetry").
+- **`ctx.trace(name, attributes, fn)`** (round-3 cut C11, decided here as kept: design §29 already states it): a child
+  span of the handler's `ctx.telemetry`, `fn` gets a child `ctx` (principal and `get` kept), so calls inside are parented.
+
+## 3. Across hosts
+
+- **Carriers on the wire:** `call` and `hold` get an optional `trace` field (R5: a `TelemetryCarrier`, bounded: at most
+  8 keys, keys and values at most 256 characters; over the bounds the message fails the `MESSAGE` schema as any malformed
+  header). The consumer injects `ctx.telemetry`'s carrier when its adapter has propagation; the provider extracts it
+  (malformed or absent: the adapter's root, never a throw) and starts `chord.method` under it. The binary codec gets a
+  flag bit for the field; the text codec an optional element. `PROTOCOL_VERSION` stays 1 (nothing is deployed outside
+  our own apps; the carry-over rule for bumps applies once p3 ships).
+- **Relays** read the carrier from the envelope, start `chord.relay` under it, and forward the relay span's carrier
+  upstream (the envelope rewrite already copies the header once).
+- **Workers** (`chord/workers`): a worker start is a `chord.resolve` root span; a held worker's methods are
+  `chord.method` under the forwarded carrier, so a trace reads tab → server relay → worker method.
+- **HTTP** (`chord/http`): a request is a `chord.http` span (method, route pattern, status; the path never, it may
+  carry ids or secrets), its handler's `ctx` under it; an incoming W3C `traceparent` header is extracted only from a
+  contributed relay that `verify`s (never from browsers).
+- **Durable and pi-ai** already accept a span (`HarnessOptions.telemetry`, pi-ai's `telemetry` option): a worker's
+  Harness gets its host's adapter, and a method that submits input passes its `ctx.telemetry`, so `pi.durable.submit`
+  is a child of `chord.method` (Durable starts its own trace per input submission and links to the trigger, as decided).
+
+## 4. Events and metrics
+
+- **Events** on the enclosing span: `chord.commit` (documents, ops, duration) for commits inside a method or routine,
+  `chord.retry` (backoff), `chord.park` (cause), `chord.reload.fate` (one per object fate: swapped, restarted, parked).
+- **Metrics** (low cardinality, never ids): `chord.link.state_messages`, `chord.link.bytes` (encoded WebSocket frames,
+  sizes the code already has), `chord.commit.duration`, `chord.calls.in_flight` (up/down), `chord.holds.active`,
+  `chord.links.open`, `chord.park.duration`, `chord.backpressure.wait`. Nothing is serialized or walked to measure.
+
+## 5. Storage and reading
+
+`chord/telemetry-sqlite` exists (decision "Telemetry store"); check it against §29 and fix gaps (links inline, metrics
+per 10 s series, retention, bounded queue). The app kit: `startServer` and `startWorker` take an optional `telemetry`
+adapter (default NOOP) and pass it to their hosts, HTTP and workers; a worker's adapter comes from its `startWorker`
+options (the kit does not ship spans across processes itself: each process exports to the store it is given; a shared
+SQLite file per data directory is the app's choice, documented with a short example).
+
+## 6. Cost
+
+With the NOOP adapter every span site costs one call and an empty object; no carrier is injected (no propagation),
+no attribute objects are built when the adapter cannot record (check pi-telemetry's API for the cheapest form; no
+`isRecording` exists: decided). Cost guards: the slice's tests assert through the in-memory adapter that a call over a
+relay produces exactly the spans of §2 and no extra encodes; a benchmark (`test/bench/telemetry.ts`, by hand) measures
+a call over a link with the NOOP adapter versus without the span code (target: within noise) and with the in-memory
+adapter; results in `docs/benchmarks.md`. The §12 container-width benchmark (R24) is carried over into this slice's
+benchmark file if not done yet (check `benchmarks.md`).
+
+## 7. Tests and examples
+
+- One test file per source file, one `describe` per mechanism; the in-memory adapter and `renderTrace` for assertions.
+- A cross-host test: a client calls a method over a relay into a worker (memory links or the workers component), the
+  method commits and calls a local object; the store-rendered trace shows client `chord.call` → server `chord.relay` →
+  worker `chord.method` → `chord.commit` event, one trace id; a malformed carrier from a client gives a new root.
+- Jobs: a persisted run resumed in a new attempt stays under its stored carrier (slice 6b's field, now used).
+- Example `62-telemetry`: a kit app (server and a worker kind) with the SQLite telemetry store, a page or a client
+  calling a worker method that submits to a fake Durable harness or does a model-free piece of work; it prints the
+  rendered trace across the three hosts and a metric query.
+- Docs: design.md §29 marked built with anything refined; the guide gets a short chapter "Seeing a request across
+  hosts"; CHANGELOG under Unreleased; delete the carry-over section "Slice 9".