mirror of
https://github.com/badlogic/pi-mono.git
synced 2026-11-05 23:37:25 +00:00
docs(chord): slice 6b spec revised after the first spec review
This commit is contained in:
Mario Zechner
parent
a2c5657c86
commit
7f3cfb060d
1 changed files with 257 additions and 186 deletions
|
|
@ -1,245 +1,316 @@
|
|||
# Slice 6b: Durable and jobs on the store (spec)
|
||||
|
||||
Status: **decided, ready for the spec reviews** (2026-10-27; D1-D9 decided by the owner), written by the redesign session after reading
|
||||
Durable's `Storage` interface (`packages/durable/src/types.ts`), its SQLite storage, schema and migrations, the storage
|
||||
conformance suite and benchmark, `session/session.ts`, `session/transaction.ts`, `session/forks.ts`, and Chord v2's
|
||||
store engine, applier, schema, commit step and jobs, in full. Durable's memory, JSONL and IndexedDB storages were not
|
||||
read (they stay as they are, S15).
|
||||
Status: **revised after the first spec review** (2026-10-27): D1-D9 and R1-R4 decided by the owner (decision log,
|
||||
entries "Slice 6b …" and "owner names and claims"); the four reviews (`/tmp/s6b-review-{astra,fable,opus,sol}.md`)
|
||||
kept every decision and found the gaps this revision closes. Written by the redesign session after reading, in full,
|
||||
Durable's `types.ts`, its SQLite storage, schema and migrations, the storage conformance suite and benchmark,
|
||||
`session/session.ts`, `session/transaction.ts`, `session/forks.ts`, the scheduler's restarts, `writeAtClose` and `abort`,
|
||||
and Chord v2's store, engine, applier, commit step and jobs. Durable's memory, JSONL and IndexedDB storages stay as they
|
||||
are (S15).
|
||||
|
||||
The single source for building 6b, together with: `design.md` read in full; the decision log
|
||||
(`packages/chord/docs/next/decisions.md`, later entries win), in particular "The store", S1, S2, S4, S10, S11, S13, S15,
|
||||
W1, W12, W13, "Slice 6 first decisions", "Durable observers", "Durable's synchronous observers", "Text is plain UTF-8",
|
||||
"Failures are watch calls", the "One vocabulary" entries, "The performance and memory bar"; `slice-6a.md` (what 6a
|
||||
built); the feature map and the carry-over of `slices.md` (slice 6b); the root `AGENTS.md` "Chord v2 Code Style".
|
||||
The single source for building 6b, together with `design.md` read in full; the decision log `decisions.md` (later
|
||||
entries win), in particular "The store", S1, S2, S4, S10, S11, S13, S15, W1, W12, "Slice 6 first decisions", "Durable's
|
||||
synchronous observers", "Text is plain UTF-8", "Failures are watch calls", the "One vocabulary" entries, "The performance
|
||||
and memory bar", and every "Slice 6b" entry; `slice-6a.md` (what 6a built); the feature map and the carry-over of
|
||||
`slices.md` (6b); the root `AGENTS.md` "Chord v2 Code Style".
|
||||
|
||||
6b puts Durable on the store (`StoreStorage`), follows Durable's views as Chord sources (`self.follow`), persists jobs,
|
||||
and extends the applier to every write kind. After 6b a worker has one store: Chord's objects and Durable's Session
|
||||
commit through one queue, and a commit of either is one seq in one outbox.
|
||||
and extends the applier to every write kind. A worker then has one store: Chord's objects and Durable's Session commit
|
||||
through one queue, and a commit of either is one seq in one outbox.
|
||||
|
||||
## What an app can do after this slice
|
||||
|
||||
```ts
|
||||
// the start code of a session's worker (p3 base code)
|
||||
// p3 worker base code
|
||||
const store = await openStore(sqliteBackend(db), ctx);
|
||||
const worker = createHost<P>({ address: Session({ id }), principal, store, onError });
|
||||
const host = createHost<P>({ address: Session({ id }), principal, store, onError });
|
||||
await host.load([{ name: "@p3/core", facet: coreWorkerFacet({ init, store }) }, ...plugins], ctx);
|
||||
|
||||
// the core facet: the harness object opens Durable on the same store
|
||||
host.serve(Harness, (_, self) => {
|
||||
// the core facet: the harness object opens Durable on the same store, its data under the harness's own type
|
||||
host.serve(Harness, (_, self) => { // local, empty params (P3)
|
||||
const harness = self.open(async (context) =>
|
||||
Harness.open(await StoreStorage.open(store, Harness.name, context), { models, registry, … }, context),
|
||||
(h, _reason, context) => h.close(context));
|
||||
const view = self.follow((observer, ctx) => harness.value.observeConversation(id, observer, ctx)); // section 2
|
||||
return { … };
|
||||
const view = self.follow((observer, ctx) => harness.value.observeConversation(rootId, observer, ctx, { min: 50 }));
|
||||
return { view };
|
||||
});
|
||||
|
||||
// a persisted job: survives a crash
|
||||
// a persisted job: survives a crash, a clean close and its object's retry
|
||||
const ImportJob = defineJob("notes.import", {
|
||||
version: 1, args: obj({ path: str() }), result: obj({ count: int() }),
|
||||
state: ImportState, persist: { resume: "rerun" },
|
||||
state: ImportState /* persist: true, checkpointWhen */, persist: { resume: "rerun" },
|
||||
});
|
||||
const imports = self.job(ImportJob, { run, finish });
|
||||
await imports.abort(id, ctx); // stored; Decision needed (D7): the result
|
||||
await imports.abort(id, ctx); // "marked" | "terminal", stored
|
||||
await ctx.state.change((d) => { d.output = text; }, ctx, { persistence: { kind: "deferred" } });
|
||||
```
|
||||
|
||||
## 1. `StoreStorage`: Durable's `Storage` on the store
|
||||
## 1. Changes to 6a's store
|
||||
|
||||
Durable's `Storage` (types.ts, "Atomic persistence boundary for Session records") implemented over a `ChordStore`, so
|
||||
Durable's Session runs unchanged on it and passes Durable's storage conformance suite
|
||||
(`createStorageConformance`), across reopen too.
|
||||
Every item below changes 6a's code; nothing is deployed, so the schema and the archive format change in place.
|
||||
|
||||
**Where it lives (D1, owner, 2026-10-27: "they are only used if durable is used"):** `StoreStorage` belongs to
|
||||
Durable (D9 makes `durableView` a generic Chord primitive, `self.follow`), in its own `src` (Chord v2 is in pi upstream since `93029a0f1`), importing
|
||||
`@earendil-works/chord/next` (Chord's `./next` package export comes with them). Durable already depends on Chord and
|
||||
Chord imports nothing of Durable (S13); Durable's conformance suite runs in Durable's tests and CI.
|
||||
### 1.1 Names and claims (decisions "R2", "owner names and claims")
|
||||
|
||||
### 1.1 Records
|
||||
- Tables are `"<owner>:<table>"` (`"notes.store:edits"`), document owners `"<owner>"`; no host kind. The applier and
|
||||
import check two name parts.
|
||||
- Every owner name is claimed once per store (in memory, for the store's lifetime): an object claims its type when it
|
||||
first declares persisted data; `StoreStorage.open` claims its name; a second claim is refused `invalid/args`; a
|
||||
claim is released by its owner's end (`StoreStorage.close`, the object's end), so a new Session can reopen over the
|
||||
same store.
|
||||
|
||||
Durable's four record tables (conversations, entries, tasks, submissions) and its ID namespace.
|
||||
### 1.2 One id allocator (D3)
|
||||
|
||||
**Layout (D2, owner, 2026-10-27: "the whole purpose of this slice"):** declared tables of the store, written as
|
||||
ordinary `put` writes: `defineTable`s that `StoreStorage` declares for itself under its owner name (`StoreStorage.open(store, owner, ctx)`,
|
||||
the name claimed once per store; SQL names `"<owner>:<table>"`, e.g. `"p3.harness:entries"`; decision "owner names
|
||||
and claims"), each row the record's indexed columns (`id`, `conversation_id`, `kind`, `status`, …) plus
|
||||
the whole record in a `json(...)` column. The outbox, export, import and the applier handle them as every table;
|
||||
Durable's layout changes later through `evolveTable`. The partial head index becomes an index on `(conversation_id,
|
||||
has_head, id)` (index columns are non-`opt`); indexed strings that must stay lossless (`kind`, `requestId`, document
|
||||
kinds and keys) keep Durable's JSON encoding in their columns.
|
||||
- The store owns one synchronous allocator: `mintId()` takes the next id; the commit step takes its documents' ids from
|
||||
the same allocator when a creation applies (no copy in the `Step`, no overwrite after `COMMIT`); explicit ids in
|
||||
staged writes raise it at staging (`max`); a refused step leaves its ids unused (gaps, S10). Each commit persists
|
||||
the allocator's current value.
|
||||
- It starts at 2 (id 1 is Durable's root conversation). It saturates at `Number.MAX_SAFE_INTEGER`: an explicit write of
|
||||
that id is accepted, the persisted value stays `MAX_SAFE_INTEGER`, every later `mintId` rejects "ID space is
|
||||
exhausted", and a Chord creation at the saturated allocator refuses its own commit (1.4).
|
||||
- Outbox commits carry the allocator's value after them (`{ seq, nextId, writes }`); the archive manifest's
|
||||
`nextDocumentId` is `nextId`; import and batch apply restore it, so a replica mints above every id it holds.
|
||||
|
||||
- **Writes:** every `StorageWrite` of a table record becomes a `put` of its row (Durable replaces complete records;
|
||||
conversations and entries are immutable and written once). `record_ids` is a declared table too (`id`,
|
||||
`record_type`), one `put` per new id (decision "Slice 6 first decisions" (3)).
|
||||
- **Reads:** `conversation`, `entry` (with its `commitSeq`: a column), `findLatestHeadMarker`, `scanEntries` (both
|
||||
orders, the fork chain walked as `SqliteStorage` does), `task`, `scanTasks`, `submission`, `scanSubmissions`,
|
||||
`submissionByRequest`, `scanConversations`: queries over the declared tables, Durable's cursors (`{ after, order }`,
|
||||
`scan.ts`) kept as they are (built from the last id, not from Chord's table cursors).
|
||||
- **Validation in the step** (D4): what `SqliteStorage` checks inside its transaction (`checkGlobalIds`,
|
||||
`prepareDocumentActions`, `checkDocumentActions`) runs inside the store's step, which rolls back a refused commit.
|
||||
### 1.3 The commit step
|
||||
|
||||
### 1.2 IDs
|
||||
- **A staged write returns the writes it recorded:** `StagedWrite.apply(step): Promise<readonly StoreWrite[]>`.
|
||||
`StoreStorage` stages one composite write per Durable commit (1.3 of section 2).
|
||||
- **Order:** within the step, writes apply in the order staged, except that a staged write that frees a live address
|
||||
(a retirement of another incarnation) is applied before the creation at that address (rule 9's reason, the partial
|
||||
unique index); Chord's own staging keeps 6a's order. The outbox records the writes in applied order; the applier
|
||||
replays them in that order (no re-sort).
|
||||
- **Stamps:** a staged write can take values only the step knows (`step.seq`): Durable's `entries.commit_seq`. The
|
||||
outbox records the resolved values.
|
||||
- **Adopt in the owner's batch:** the host gives the store its batch when it attaches (`attach({ failed, batch })`);
|
||||
an adapter's fence runs its `adopt` in `batch.within(…)` (joins a running batch, opens one otherwise); Chord's own
|
||||
transactions keep their `batch.commit`. Without a host, `adopt` runs plain.
|
||||
|
||||
**One counter per store (D3, owner, 2026-10-27):** the store's counter (`next_document_id` renamed `next_id`, the
|
||||
manifest's `nextDocumentId` renamed `nextId`) numbers every id: `StoreStorage.mintId()` takes from it in memory,
|
||||
explicit ids in Durable's writes raise it, Chord's commit step takes its documents' ids from it, and every commit
|
||||
persists it as a high-water mark. `record_ids` stays for Durable's "ID n already belongs to entry" check. Durable's ids
|
||||
rise in its creation order (its scans and windows rely on it); Chord's ids are gaps between them. Spec reviewers check
|
||||
that nothing relies on Chord's document ids rising with seq.
|
||||
### 1.4 Errors: Durable's Storage semantics (D4)
|
||||
|
||||
### 1.3 Documents
|
||||
- The step records the error its own code threw inside the transaction callback. A rejection of the backend
|
||||
transaction that is that very error is a **refusal**: the facade rolled back (a failed `ROLLBACK` reaches us as
|
||||
another error, an `AggregateError` on Node), the commit rejects, nothing is adopted, the allocator keeps its raised
|
||||
value, the store goes on. Any other rejection (`BEGIN`, `COMMIT`, `ROLLBACK`, a throwing `adopt`) is an unknown
|
||||
outcome: the store stops (`storage_failed`). The Cloudflare facade is checked to keep the same identity; a test
|
||||
injects failures at `BEGIN`, in the body, at `COMMIT` and at `ROLLBACK` on both drivers' facades.
|
||||
- **The refusal's error reaches the owner unchanged** (Durable's suite matches its messages); the store classifies it
|
||||
for Chord's owners: an `int` `CHECK` failure (SQLite's extended code `SQLITE_CONSTRAINT_CHECK`) is `invalid/args` and
|
||||
fails only that transaction; every other refusal of a Chord commit (the engine's counted `UPDATE`/`DELETE`, the
|
||||
live-address index, a primary key, the saturated allocator, a backend error in a statement) fails the object live
|
||||
(`object_failed`, `broken`: memory and disk disagree).
|
||||
- **6a's owners:** a refused `self.tx` rejects with the error above and the object fails as above; a refused
|
||||
migration commit (a chunk, (A) or (Z)) fails the activation `broken`, the table stays `migrating`, committed chunks
|
||||
stay; a refused job start rejects the start; a refused outcome or finish commit fails the object (records and
|
||||
memory disagree); a refused commit that carried held changes (D8) fails the object.
|
||||
- **Reads:** a read's callback returns data, never throws for a request: an adapter's request checks (an unknown
|
||||
conversation, a cursor, history a document does not keep) happen after the guarded read returned, from tagged
|
||||
results, so only a backend error fails the store. The store offers a read transaction for reads of several queries
|
||||
that must see one state (S4).
|
||||
|
||||
Durable's documents on the documents engine (S13): address `(namespace "durable", owner, kind, key?)`, owner the
|
||||
scope as a string (`session`, `conversation:7`, `task:31`), kind and key JSON-encoded as Durable's indexed strings.
|
||||
6a built `latest`; 6b adds what Durable needs:
|
||||
### 1.5 The documents engine
|
||||
|
||||
- **`rewindable` records** keep every revision and are not deleted at retirement: `retired_at` is set, revisions
|
||||
stay (decision "Writes carry their versions" already defined the archive fields).
|
||||
- **Reads at a point:** `findDocument(address, at)`, `document(id, at)` (the base at or below `at` plus the deltas up
|
||||
to it, in one read transaction, S4), `scanDocuments(scope, at, kind?)`, with Durable's half-open lifetime
|
||||
(`createdAt <= at < retiredAt`); a historical read of a `latest` (or session or task) record is a
|
||||
`StorageRequestError` ("does not retain historical content").
|
||||
- **`document.copy`** (S11): `StoreStorage` reads the source at its point before the step and enters the step with a
|
||||
plain create carrying the base; the conformance rules (source not changed in the copy batch, a matching record) are
|
||||
checked there.
|
||||
- **Stored version and `deltasSinceBase`** as `StoredDocument` reports them; Durable's records carry no fingerprint:
|
||||
the engine's fingerprint column takes `""` for Durable records.
|
||||
- **Export** carries every revision of a `rewindable` record, retired ones included, with its seq (format 1's
|
||||
fields); import and the applier take them.
|
||||
- **Addresses:** `(owner, namespace, kind, key?)`: `owner` the claimed name; `namespace` the adapter's partition
|
||||
(Chord: `chord` for documents, `chord.jobs` and `chord.job-states` for persisted runs; Durable: its scope,
|
||||
`session`, `conversation:<id>`, `task:<id>`); `kind` and `key` as the adapter writes them (Durable JSON-encodes both,
|
||||
as its indexed strings). One live record per address (the partial unique index).
|
||||
- **`history` per record (R1):** `none` (Chord: older revisions dropped at a new base; row and revisions deleted at
|
||||
retirement), `latest` (Durable's session, task and latest conversation documents: older revisions dropped; the row
|
||||
kept with `retired_at` at retirement), `rewindable` (nothing dropped; the row kept). `retireDocument` reads the record
|
||||
and applies its rule.
|
||||
- **Explicit ids:** a creation may carry its id (Durable mints ahead); otherwise the allocator gives one.
|
||||
- **Changes without a known revision:** an adapter whose writes carry no revision (Durable) stages a change that reads
|
||||
the record in the step and uses its revision; the counted conditional `UPDATE` stays the guard.
|
||||
- **`meta`:** Durable's `fork` policy and `history` details live in the record's `meta`.
|
||||
- **Reads at a point:** `find(address, at)`, `load(id, at)` (the base at or below `at` plus the deltas up to it, its
|
||||
version the base's), `list(owner, namespace, at, kind?, limit, after?)` by id, with the half-open lifetime
|
||||
(`createdAt <= at < retiredAt`); records carry `retiredAt`. New indexes (store schema version 2):
|
||||
`(owner, namespace, kind, key, created_at)`, `(owner, namespace, id)`, revisions `(document_id, kind, seq)`; their
|
||||
cost on Chord's commit gate is measured.
|
||||
|
||||
### 1.4 Commits
|
||||
### 1.6 The archive and batch apply
|
||||
|
||||
- `StoreStorage.commit(writes, adopt, ctx)`: validates (1.1), resolves copies (1.3), stages the writes (table `put`s,
|
||||
document creates, changes and retirements) and calls `store.commit(staged, { admit: () => true, adopt: (seq) =>
|
||||
adopt(seq) }, ctx)`. The store's step runs `adopt` inside `host.batch.commit(...)` when the store has a host, so
|
||||
Durable's observers mark Chord sources in the step (W1, "Slice 6 first decisions" (1)); `adopt` never throws
|
||||
(Durable's rule).
|
||||
- **Durable's seq is the store's seq** (gaps allowed: Chord's commits take seqs too).
|
||||
- **Errors:**
|
||||
- **Format 1 gains** (nothing deployed): `document.create` carries `history`, `retiredAt?` and `meta`; a
|
||||
`document.revision { id, seq, kind, version, content }` part per kept revision, in seq order after its create; for
|
||||
`none` and `latest` records only the current base and its deltas are exported (as 6a), for `rewindable` every
|
||||
revision, retired records included. Import checks: revisions in seq order, a base first, `seq <= lastSeq`, lifetimes
|
||||
at one address never overlapping (a second live record is still refused), versions per revision; the `end` count
|
||||
includes revisions.
|
||||
- **Batch apply** (`applyBatch(store, commits, ctx)`, internal): only on a store with no attached host (no owner's
|
||||
memory to update); `commits[0].seq` follows `lastSeq` (S5); one backend transaction for the batch, every write applied
|
||||
with its commit's seq, `lastSeq` and `nextId` set from the last commit; the applier handles every `StoreWrite` kind
|
||||
(`alter`, `dropIndex`, `update`/`updateWhere`/`delete`/`deleteWhere` with their recorded counts and versions,
|
||||
`document.change`, `document.retire`, `document.revision`), checked against the store's records (a trust boundary);
|
||||
a count that differs from the recorded one stops applying and marks the store diverged (S17): every later
|
||||
`applyBatch` rejects. A test replays a store's outbox into an empty store and compares the exports, manifest
|
||||
included.
|
||||
|
||||
**Errors (D4, owner, 2026-10-27):** the store takes Durable's `Storage` semantics (design §10 "Store errors"): a
|
||||
step that fails before `COMMIT` rolls back and rejects that commit only; the store stops only on an unknown outcome
|
||||
(`COMMIT` or `ROLLBACK` failed), a throwing `adopt`, or a read's backend error. Owners react as Durable's Session:
|
||||
Durable's Session fails (`GuardedStorage`, unchanged); a Chord object whose commit the step refused fails live
|
||||
(`object_failed`, `broken`); an `int` `CHECK` failure fails only its transaction (`invalid/args`). `StoreStorage` runs
|
||||
Durable's commit checks in the step, as `SqliteStorage` does, so the conformance suite's refused commits leave the
|
||||
store usable. 6b changes 6a's step and object failure accordingly.
|
||||
## 2. `StoreStorage`: Durable's `Storage` on the store (D1, D2)
|
||||
|
||||
- `mintId`, `close(context)` (closes nothing of the store: the app owns it; later calls reject "closed"), reads after
|
||||
a store failure reject as Durable's `GuardedStorage` expects.
|
||||
In Durable's `src` (an ordinary pi module), importing `@earendil-works/chord/next` (Chord's `./next` export comes
|
||||
with it: the store and the adapter surface below, marked internal for adapters). It passes Durable's storage
|
||||
conformance suite on SQLite `:memory:` and across reopen (a wrapper that closes and reopens the store file after
|
||||
every commit, Durable's reopen harness), and Durable's Session and harness test files run on it.
|
||||
|
||||
### 1.5 Statements per commit
|
||||
### 2.1 Opening and the adapter surface
|
||||
|
||||
A target to measure, as 6a's: an entry append is the `record_ids` put, the entry put, `lastSeq` (plus the outbox); a
|
||||
document delta is 6a's three statements. `SqliteStorage` does the same work today (`record_ids`, the row, metadata).
|
||||
`StoreStorage.open(store, owner, ctx)`: claims `owner` (1.1); declares its tables through the store's table
|
||||
preparation (the path `self.table` takes: created at the newest version or migrated); raises the allocator to
|
||||
`max(record_ids.id) + 1` (one index read; covers stores built by import or batch apply); attaches to the store's host
|
||||
batch if there is one. Chord exports for it: `StoreOwnerTables` (declare and prepare tables under an owner),
|
||||
staged table writes and engine writes, the step reader, the read transaction, `mintId`, the commit with a fence. Its
|
||||
`close(context)` releases the claim and refuses every later call ("StoreStorage is closed"); the store stays open (the
|
||||
app owns it).
|
||||
|
||||
## 2. `self.follow`: Durable's views as Chord sources (D9, owner, 2026-10-27)
|
||||
### 2.2 Tables
|
||||
|
||||
Durable's synchronous observers (`observeConversation`, `observeTaskGraph`, `observeDoc`; decision "Durable's
|
||||
synchronous observers") have Chord's watch shapes (`advance(call, ctx)` with a `WatchCall`, `end(end)` with a
|
||||
`WatchEnd`, pi `5f312dac4`), so Chord follows them with one generic primitive and Durable needs no adapter:
|
||||
Every row carries the record's indexed columns and the whole record as `json(unknown(), { version: 1 })`; indexed
|
||||
strings that must stay lossless (`kind`, `request_id`) hold Durable's JSON encoding; index columns are non-`opt`, so
|
||||
absent values take sentinels that no encoded value can be (`0` for an absent owner id, ids start at 2; `""` for an
|
||||
absent request id, JSON-encoded strings start with `"`).
|
||||
|
||||
| Table | Key | Columns | Indexes |
|
||||
| --- | --- | --- | --- |
|
||||
| `record_ids` | `id` | `record_type str` | |
|
||||
| `conversations` | `id` | `owner_conversation_id int`, `owner_task_id int`, `record` | `(owner_conversation_id, id)`, `(owner_task_id, id)` |
|
||||
| `entries` | `id` | `conversation_id int`, `has_head int` (0/1), `commit_seq int` (stamped), `record` | `(conversation_id, id)`, `(conversation_id, has_head, id)` |
|
||||
| `tasks` | `id` | `conversation_id int`, `kind str`, `status str`, `abort_requested int`, `background int`, `record` | `(status, id)`, `(conversation_id, id)`, `(kind, id)`, `(abort_requested, id)`, `(background, id)` |
|
||||
| `submissions` | `id` | `conversation_id int`, `request_id str`, `status str`, `record` | `(conversation_id, request_id)`, `(conversation_id, id)`, `(status, id)` |
|
||||
|
||||
### 2.3 Commits
|
||||
|
||||
`commit(writes, adopt, ctx)` stages one composite write and calls the store's commit with `{ admit: () => true,
|
||||
adopt: (seq) => adopt(seq) }` (adopted in the host's batch, 1.3). Inside the step the composite write does what
|
||||
`SqliteStorage` does inside its transaction, with the step's reader: `checkGlobalIds` (an id already claimed by
|
||||
another record type, a conversation, entry or document written twice: Durable's messages), the per-document actions
|
||||
of `prepareDocumentActions`/`checkDocumentActions` (one content command per document, a delta on its version, one live
|
||||
incarnation per address: "already has a current incarnation"), then the table `put`s (`record_ids` only for ids not yet
|
||||
claimed by the same type), then per document: its creation (explicit id) or content, then its retirement; a
|
||||
create-plus-retire is an empty lifetime (`createdAt = retiredAt = seq`). Copies (`document.copy`) are resolved before
|
||||
the step (S11): the source read at its point, checked against the copy's record ("changed in the copy batch",
|
||||
kind, key, history, fork), entered as a creation carrying the base. A refused check rejects with Durable's message (a
|
||||
refusal, 1.4): the store goes on, Durable's Session fails (`GuardedStorage`).
|
||||
|
||||
### 2.4 Reads
|
||||
|
||||
`conversation`, `entry` (both overloads, `commitSeq` from its column), `findLatestHeadMarker`, `scanEntries` (both
|
||||
orders, the fork chain walked as `SqliteStorage` does), `task`, `scanTasks`, `submission`, `scanSubmissions`,
|
||||
`submissionByRequest`, `scanConversations`: queries over the tables, Durable's cursors (`scan.ts`) as they are;
|
||||
`findDocument`, `document` (one read transaction), `scanDocuments`: the engine's reads at a point (1.5). Request errors
|
||||
(`StorageRequestError`) are thrown after the guarded read (1.4).
|
||||
|
||||
### 2.5 Cost
|
||||
|
||||
Statements per commit, as the gate measures them: an entry append, a `record_ids` check read, the `record_ids` put,
|
||||
the entry put, the meta update (4; `SqliteStorage` 5); a document delta, the record read, the revision insert, the
|
||||
counted update, the meta update (about 4-5; `SqliteStorage` about 7). Per `put` one compiled row check and the JSON
|
||||
encoding of indexed strings.
|
||||
|
||||
## 3. `self.follow`: Durable's views as Chord sources (D9, R4)
|
||||
|
||||
`self.follow(attach)` in a factory returns a `Watchable` (a state member, a derive source, an `on` source);
|
||||
`attach(observer, ctx)` returns `Promise<{ value, detach } | undefined>` (Durable's `Observation`; `undefined` for an
|
||||
absent `observeDoc` address). In 6b it is used inside the object that opened Durable (a native's value); reaching
|
||||
Durable from other objects is slice 8's (the harness in a registry entry, carry-over).
|
||||
|
||||
- **Activation:** after the natives opened, before the members are checked and derives compute: Chord makes the
|
||||
observer and calls `attach`; calls arriving before it resolves are queued and applied in order on `value`; a rejected
|
||||
or `undefined` attach fails the activation like a failed dependency (`transient`); a failed activation detaches
|
||||
what attached. The attempt's end detaches.
|
||||
- **Calls:** a call inside a running batch (Durable's `adopt` on the store) applies the queued calls first, then marks
|
||||
the source in that batch; a call outside any batch (Durable's deferred commits, `#hold`, and the task graph's
|
||||
`blocked` refresh, even on the store; any Durable storage that is not the store) is queued and one batch on the next
|
||||
microtask applies everything queued. A derive over two views of one commit sees both changes in one computation.
|
||||
- **Ends (R4):** `failed` fails the object `transient` (its retry reopens Durable and attaches again); `ended` is final:
|
||||
a pending retirement call reaches the source's watches first (Durable's `endAfterDelivery`), then they end `ended`;
|
||||
the member keeps its last value; a new `watch` on an ended follow source opens at that value and ends `ended` at once.
|
||||
|
||||
## 4. Persisted jobs (D5-D8, R3)
|
||||
|
||||
`persist: { resume: "rerun" | "abandon" }` only in `local` types with empty params (P3, at `self.job`); a persisted job
|
||||
needs a state definition with `persist: true` and `checkpointWhen` if it has state, and a JSON-only result (no `bytes()`,
|
||||
refused at declaration).
|
||||
|
||||
### 4.1 Records
|
||||
|
||||
Each unsettled run is a keyed persisted document in namespace `chord.jobs` (kind the job name, key the run id,
|
||||
`history: none`); its state is a keyed persisted document in namespace `chord.job-states` (same kind and key), so two
|
||||
jobs sharing a state definition never meet. The record:
|
||||
|
||||
```ts
|
||||
host.serve(ChatView, ({ conversation }, self) => {
|
||||
const harness = self.get(Harness, {});
|
||||
const view = self.follow((observer, ctx) =>
|
||||
harness.value.observeConversation(conversation, observer, ctx, { min: 50 }));
|
||||
return { view, title: self.derive([view], ([v]) => v.title) };
|
||||
});
|
||||
{ version, fingerprint, args, key?, trace?,
|
||||
phase: "pending" | "running" | "stopped" | "finishing" | "blocked",
|
||||
restarts, finishFailures, abort?: true, blocked?: "finish_failed" | "version",
|
||||
outcome?: { kind: "done", result } | { kind: "failed", error: StoredJobError } }
|
||||
// StoredJobError: { kind: "threw", error: WireError } | { kind: "aborted", reason } | { kind: "restart_limit", restarts }
|
||||
```
|
||||
|
||||
- `self.follow(attach)` returns a `Watchable`: a state member, a derive source, an `on` source. `attach(observer,
|
||||
ctx)` returns `Promise<{ value, detach }>` (Durable's `Observation`).
|
||||
- **Attaching** is that one call during activation: Chord makes the observer, calls `attach`, takes `value` as the
|
||||
source's value (calls arriving before the promise resolves are held and applied after it), and the object goes live.
|
||||
The attempt's end calls `detach()`. An attach that rejects, or resolves `undefined` (`observeDoc` of an absent
|
||||
address), fails the activation like a failed dependency.
|
||||
- **Calls:** `advance` records the call. Inside a batch (on the store, Durable's `adopt` runs in the host's batch) the
|
||||
source is marked in the step and its frame goes out in that flush; outside a batch (a Durable storage that is not
|
||||
the store) the source collects the calls and opens one batch on the next microtask. A derive over two views of one
|
||||
Durable commit sees both changes in one computation.
|
||||
- **End:** Durable's failures are final: `end({ kind: "failed" })` ends the watches `failed`, `ended` ends them
|
||||
`ended`.
|
||||
Errors are stored in the wire error format (a `PluginError`'s code, message and details survive); a stored outcome is
|
||||
rebuilt as the in-memory one.
|
||||
|
||||
## 3. Persisted jobs
|
||||
### 4.2 Transitions
|
||||
|
||||
Design §15 "Persistence": `persist: { resume: "rerun" | "abandon" }`, only in `local` objects with empty params (P3,
|
||||
at declaration); a run's record and state document in the owner's store; a recorded outcome resumes only `finish`; an
|
||||
abort request is persisted; a restart limit; the job fingerprint stored with every record; the start trace carrier.
|
||||
| Event | Writes (one commit unless noted) | After |
|
||||
| --- | --- | --- |
|
||||
| start | record `pending`, state `initial()` | invoked on the live attempt |
|
||||
| reservation (before `run`) | record `running` | `run` invoked (a crash after it counts) |
|
||||
| `ctx.state.change` | state; the record's `restarts: 0` only when it was not 0 | |
|
||||
| abort (D7) | record `abort: true` (on the line); the signal fires after the commit, outside the batch; the line is released, then `abort` joins `run` | `"marked"` |
|
||||
| `run` settles | held state flushed (D8), record `finishing` with the outcome | `finish` on the live attempt |
|
||||
| `finish` fails | record `finishFailures + 1`; at 5 `blocked: "finish_failed"` | retried after the backoff, a blocked run on reload |
|
||||
| `finish` succeeds | record deleted, state retired (Q9), with what `finish` wrote | the run settles |
|
||||
| clean stop (host close, a native restart, R3's seal) | the stopped runs written `stopped`, held state flushed, by the close writer | |
|
||||
| activation (once per object, not per retry) | each `running` record: `restarts + 1` (or, at 5, `failed { restart_limit }` and `finishing`, `job_interrupted` reported); `stopped` and `pending`: back to `pending` | maps, keys, active count, `running` rebuilt before anything runs; `rerun` invokes, `abandon` records `failed { aborted, restart }`; an abort-marked run finishes `aborted` without `run`; `finishing` resumes `finish`; `blocked` stays |
|
||||
|
||||
- **Where records live:**
|
||||
- **The close writer:** at the object's end, after the drain and before the documents drop, one commit past the seal
|
||||
(Durable's `writeAtClose`): the runs stopped by the end written `stopped`, held state flushed; a run whose `run`
|
||||
outlived the drain is fenced and written `stopped` too; blocked persisted runs keep their records (dropped only by
|
||||
`abort`). In-memory runs keep 6a's rules.
|
||||
- **Pinning:** records and state documents of unsettled runs stay in memory (never evicted): `ctx.state.value` and
|
||||
`running` read them synchronously.
|
||||
- **Fingerprint:** the job's fingerprint (args, result, state) is stored; a record of the job's version with another
|
||||
fingerprint is `blocked: "version"` (resolved by slice 7).
|
||||
|
||||
**Where records live (D5, owner, 2026-10-27):** each unsettled run's record is a keyed persisted document of its
|
||||
object in the engine's namespace `chord.jobs` (owner the object's owner string, kind the job name, key the run id:
|
||||
`{ version, fingerprint, args, key?, phase, outcome?, abort?, restarts, trace? }`); Chord sets that namespace, a plugin
|
||||
never does (its documents are always `chord`), so nothing can collide. Its state document stays an ordinary document
|
||||
of the run. Both change in the transactions that change them today (start, state change, outcome, finish, block,
|
||||
drop); `finish` deletes the record and retires the state in its commit (Q9), so only unsettled runs are stored. At the
|
||||
object's first activation the records are loaded through the engine's key listing and a load per run (no new read).
|
||||
### 4.3 Deferred state (D8)
|
||||
|
||||
- **Resume at activation:** a run recorded `finishing` resumes `finish` only; a run recorded `running` was cut off by
|
||||
a crash: `rerun` invokes `run` again with the stored args and state, `abandon` records `failed { aborted, reason:
|
||||
"restart" }` and finishes.
|
||||
- **Restart limit** (aligned with Durable's `restarts`):
|
||||
|
||||
**The same rule as Durable (D6, owner, 2026-10-27: "yes, same rule for chord"):** at activation a run recorded
|
||||
`running` was cut off by a crash: its `restarts` rises by one (stored), then it resumes (`rerun` invokes `run` again,
|
||||
`abandon` finishes with `failed { aborted, reason: "restart" }`); a clean close (`host.close`, a native restart on
|
||||
reload) writes its stopped runs back so the next activation counts nothing (Durable's `writeAtClose`); a
|
||||
`ctx.state.change` resets the count (progress); at five in a row the run ends with a new `JobError` `{ kind:
|
||||
"restart_limit", restarts }`, its `finish` runs, and `job_interrupted` is reported (Durable's `restart_limit`,
|
||||
`task_interrupted`). No `blocked { restart_limit }`: supersedes design §15's parking, as Durable dropped parking in
|
||||
#11139.
|
||||
|
||||
- **Abort** (owner, 2026-10-27: async and stored in 6b):
|
||||
|
||||
**`job.abort(id, ctx): Promise<"marked" | "terminal">` (D7, owner, 2026-10-27), Durable's words and behaviour:**
|
||||
a running run gets its abort stored (a rerun after a crash then finishes `aborted` instead of running again) and its
|
||||
signal fired, and `abort` resolves `"marked"` once `run` settled (as `abortTask` joins the run); a finishing run is only
|
||||
marked (`"marked"`, as Durable's `completing`); a blocked run is dropped at once (`"marked"`, as Durable's orphan); an id
|
||||
with no unsettled run resolves `"terminal"` (Chord keeps no settled runs, Q9, so a settled and an unknown id are the
|
||||
same; Durable throws for an unknown one).
|
||||
|
||||
- **Deferred state persistence:**
|
||||
|
||||
**Per change, Durable's shape (D8, owner, 2026-10-27):** `ctx.state.change(fn, ctx, { persistence: { kind:
|
||||
"deferred", withinMs? } })` (default 1,000 ms, as Durable's `CommitPersistence`): adopted and published at once,
|
||||
stored with the next immediate change of the document, before the outcome is recorded (so `outcome.state` has it), at
|
||||
close, or after `withinMs`; a crash loses what is not stored. Only run state documents; other persisted documents get
|
||||
it when a real case needs it.
|
||||
|
||||
- **Fingerprint:** the job's fingerprint (args, result and state schemas) stored with every record; a record of the
|
||||
same version with another fingerprint is `blocked { version }` (slice 7's family; the carry-over's "a job
|
||||
fingerprint over args, result and state").
|
||||
- **Trace carrier:** the start's carrier (design §29) stored with the record; used in slice 9.
|
||||
|
||||
## 4. The applier for batch apply
|
||||
|
||||
6a's applier applies an archive's writes; 6b extends it to every `StoreWrite` kind (`alter`, `dropIndex`, `update`,
|
||||
`updateWhere`, `delete`, `deleteWhere` with their recorded counts and versions, `document.change`, `document.retire`),
|
||||
checked against the store's records (a trust boundary), one code path with import. A test replays a store's outbox
|
||||
into an empty store and compares the two stores' exports. Applying a batch with a different count than recorded stops
|
||||
applying (S17).
|
||||
As Durable's holds: the slot adopts and publishes the change at once (`valueVersion` ahead of what is stored), keeps
|
||||
the held ops in a `PendingOps` (a base flag when `checkpointWhen` selected one), and the run's documents stay pinned;
|
||||
the next immediate change of the state document stores the held ops and its own as one delta on the stored revision
|
||||
(or a base); one foreground timer per object for the earliest deadline (`withinMs`, default 1,000 ms, `Infinity` waits
|
||||
for the others; a negative or non-number `withinMs` is `invalid/args`); the outcome's commit and the close writer
|
||||
store what is held; a held change of a fenced invocation is still stored; a crash loses it. A deferred change that
|
||||
would reset `restarts` is stored at once (the reset must be durable). On an in-memory job the option is accepted and
|
||||
does nothing.
|
||||
|
||||
## 5. Benchmarks and tests
|
||||
|
||||
- **The gate** (decisions "The commit-step benchmark", "The performance and memory bar"): Durable's own storage
|
||||
benchmark (`test/storage.bench.ts`: the write and read cases of `testing/storage-benchmark.ts`) on `StoreStorage`
|
||||
against `SqliteStorage`, and Durable's `test/conversation-view-bench.ts` on both: each hot path within about 5%,
|
||||
memory (allocation per op, retained after GC) no worse beyond noise; persisted job starts and state changes against
|
||||
in-memory ones.
|
||||
- **Tests:** Durable's storage conformance suite on `StoreStorage` (memory `:memory:` and across reopen); Durable's
|
||||
Session and harness test files that take a storage, run on `StoreStorage`; `self.follow` (one frame per commit, a
|
||||
derive over two views, outside the store, early calls held, failure and end, an absent `observeDoc`); persisted jobs (rerun, abandon, a recorded outcome
|
||||
resumes `finish`, restart counting and reset, the limit, abort stored, deferred state, fingerprints); the applier
|
||||
(every kind, the outbox replay, refused counts); export and import with Durable records and `rewindable` history;
|
||||
GC (observers detached with their attempt).
|
||||
- **Examples:** `26-durable-on-the-store.ts`, `27-durable-views.ts`, `28-persisted-jobs.ts`.
|
||||
- **The gate** ("The performance and memory bar"): Durable's storage benchmark (`test/storage.bench.ts`, every read and
|
||||
write case, 1k and 10k) and `test/conversation-view-bench.ts` on `StoreStorage` against `SqliteStorage`, outbox off
|
||||
(its cost reported separately): each hot path within about 5%, allocation per op and retained memory no worse beyond
|
||||
noise; Chord's 6a commit gate again with the new indexes and the allocator; persisted job start and state change
|
||||
reported against in-memory ones (a disk commit is not held to the in-memory number).
|
||||
- **Tests:** the conformance suite (both fixtures); the D4 matrix (`BEGIN`, body, `COMMIT`, `ROLLBACK` failures; a
|
||||
refused Durable commit keeps the store; a refused Chord commit fails its object; a `CHECK` failure only its
|
||||
transaction; a refused migration chunk; a request error inside a multi-query read keeps the store); the allocator (a
|
||||
mint during a running Chord step, saturation across reopen, a refused step's gap, replay restoring `nextId`);
|
||||
create-plus-retire and change-plus-retire through the outbox and the applier; reads at a point on retired `latest`
|
||||
and `rewindable` records; export and import of rewindable history and mixed Chord and Durable data; `applyBatch`
|
||||
(every kind, the replay equality, a diverged count); `self.follow` (one frame per commit, a derive over two views,
|
||||
calls outside a batch on the store, early calls, an absent `observeDoc`, `failed` and `ended`, a retirement before
|
||||
the end); persisted jobs (every transition of 4.2, a crash between reservation and `run`, a crash during the close
|
||||
writer, a seal and retry, restart counting and its reset, the limit, the finish failure count across a crash, abort
|
||||
joined and stored, abort racing an outcome, deferred state merged, flushed at the outcome and at close, pinned past
|
||||
the retention); GC (closed `StoreStorage`, ended follow sources, cancelled abort waiters, deferred timers, settled and
|
||||
dropped runs).
|
||||
- **Examples:** `26-durable-on-the-store.ts` (a worker with Durable's Session and a Chord store object on one store,
|
||||
exporting the archive), `27-durable-views.ts`, `28-persisted-jobs.ts`.
|
||||
|
||||
## Carry-over taken
|
||||
|
||||
From `slices.md` "Slice 6b": persisted jobs (all of it), `StoreStorage` (ids, fork copies, `adopt` in the batch),
|
||||
the Chord source over Durable's views (`self.follow`, the batch on and off the store), the applier for batch apply; `job.abort` async and
|
||||
stored (decision of 2026-10-27).
|
||||
From `slices.md` "Slice 6b": persisted jobs (all of it, the finish failure count included), `StoreStorage` (ids, fork
|
||||
copies, `adopt` in the batch), the Chord source over Durable's views (`self.follow`), the applier for batch apply;
|
||||
`job.abort` async and stored.
|
||||
|
||||
## Carry-over left
|
||||
|
||||
Slice 7: a job fingerprint mismatch's `blocked { version }` resolution on reload; runs of jobs the code no longer
|
||||
declares (with D5's records). Slice 9: the trace carrier's use.
|
||||
Slice 7: `blocked: "version"` resolved on reload; runs of jobs the code no longer declares; blocked runs retried on
|
||||
reload. Slice 8: Durable's views from other objects (the harness in a registry entry). Slice 9: the trace carrier's
|
||||
use.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue