pi-mono/packages/chord/docs/slice-6b.md

Slice 6b: Durable and jobs on the store (spec)

Status: built (2026-10-28, pi 14b4c56e5: one build, a build review and two fix rounds by the four spec reviewers; reopen, the one-entry commit's mean and the resume watch's CPU accepted by the owner). Was: ready to build (2026-10-27).

The single source for building 6b, together with design.md read in full; the decision log decisions.md (later entries win), in particular "The store", S1, S2, S4, S10, S11, S13, S15, W1, W12, "Slice 6 first decisions", "Durable's synchronous observers", "Text is plain UTF-8", "Failures are watch calls", the "One vocabulary" entries, "The performance and memory bar", and every "Slice 6b" entry; slice-6a.md (what 6a built); the feature map and the carry-over of slices.md (6b); the root AGENTS.md "Chord v2 Code Style".

6b puts Durable on the store (StoreStorage), follows Durable's views as Chord sources (self.follow), persists jobs, and extends the applier to every write kind. A worker then has one store: Chord's objects and Durable's Session commit through one queue, and a commit of either is one seq in one outbox.

What an app can do after this slice

// p3 worker base code
const store = await openStore(sqliteBackend(db), ctx);
const host = createHost<P>({ address: Session({ id }), principal, store, onError });
await host.load([{ name: "@p3/core", facet: coreWorkerFacet({ init, store }) }, ...plugins], ctx);

// the core facet: the harness object opens Durable on the same store, its data under the harness's own type
host.serve(Harness, (_, self) => {                     // local, empty params (P3)
  const harness = self.open(async (context) =>
    Harness.open(await StoreStorage.open(store, Harness.name, context), { models, registry, … }, context),
    (h, _reason, context) => h.close(context));
  const view = self.follow((observer, ctx) => harness.value.observeConversation(rootId, observer, ctx, { min: 50 }));
  return { view };
});

// a persisted job: survives a crash, a clean close and its object's retry
const ImportJob = defineJob("notes.import", {
  version: 1, args: obj({ path: str() }), result: obj({ count: int() }),
  state: ImportState /* persist: true, checkpointWhen */, persist: { resume: "rerun" },
});
const imports = self.job(ImportJob, { run, finish });
await imports.abort(id, ctx);                          // "marked" | "terminal", stored
await ctx.state.change((d) => { d.output = text; }, ctx, { persistence: { kind: "deferred" } });

1. Changes to 6a's store

Every item below changes 6a's code; nothing is deployed, so the schema and the archive format change in place.

1.1 Names and claims (decisions "R2", "owner names and claims")

• Tables are "<owner>:<table>" ("notes.store:edits"), document owners "<owner>"; no host kind. The applier and import check two name parts.
• Claims (owner, 2026-10-27, decision "owner names and claims", refined): what can collide is claimed once per store, in memory: a table by its SQL name, documents by (owner, namespace). An object claims its tables and (type, "chord") (plus chord.jobs and chord.job-states with persisted jobs) when it declares them; StoreStorage.open claims its five tables and its document namespaces (Durable's scopes contain a :, Chord's namespaces never do). A second claim is refused invalid/args; a claim is released by its owner's end (StoreStorage.close after its calls underway drained, the object's end), so a new Session can reopen over the same store. So the object that opens Durable can declare its own persisted data under the same owner name, and two harnesses can share a store under two owner names.

1.2 One id allocator (D3)

• The store owns one synchronous allocator: mintId() takes the next id; the commit step takes its documents' ids from the same allocator when a creation applies (no copy in the Step, no overwrite after COMMIT); explicit ids in staged writes raise it at staging (max); a refused step leaves its ids unused (gaps, S10). Each commit persists the allocator's current value.
• It starts at 2 (id 1 is Durable's root conversation). It saturates at Number.MAX_SAFE_INTEGER: an explicit write of that id is accepted, the persisted value stays MAX_SAFE_INTEGER, every later mintId rejects "ID space is exhausted", and a Chord creation at the saturated allocator refuses its own commit (1.4).
• Each commit takes the allocator's value once, after its writes applied, for the outbox commit ({ seq, nextId, writes }) and the metadata row (mints meanwhile only raise it further); an export reads it once with lastSeq; the archive manifest's nextDocumentId is nextId; import and batch apply restore it, so a replica mints above every id it holds.

1.3 The commit step

• A staged write returns the writes it recorded: StagedWrite.apply(step): Promise<readonly StoreWrite[]>. StoreStorage stages one composite write per Durable commit (1.3 of section 2).
• Order: within the step, writes apply in the order staged, except that a staged write that frees a live address (a retirement of another incarnation) is applied before the creation at that address (rule 9's reason, the partial unique index); Chord's own staging keeps 6a's order. The outbox records the writes in applied order; the applier replays them in that order (no re-sort).
• Stamps: a staged write can take values only the step knows (step.seq): Durable's entries.commit_seq. The outbox records the resolved values.
• Adopt in the owner's batch: the host gives the store its batch when it attaches (attach({ failed, batch })); an adapter's fence runs its adopt in batch.within(…) (joins a running batch, opens one otherwise); Chord's own transactions keep their batch.commit. Without a host, adopt runs plain.

1.4 Errors: Durable's Storage semantics (D4)

• The step records the error its own code threw inside the transaction callback. A rejection of the backend transaction that is that very error is a refusal: the facade rolled back (a failed ROLLBACK reaches us as another error, an AggregateError on Node), the commit rejects, nothing is adopted, the allocator keeps its raised value, the store goes on. Any other rejection (BEGIN, COMMIT, ROLLBACK, a throwing adopt) is an unknown outcome: the store stops (storage_failed). The Cloudflare facade is checked to keep the same identity; a test injects failures at BEGIN, in the body, at COMMIT and at ROLLBACK on both drivers' facades.
• The refusal's error reaches the owner unchanged (Durable's suite matches its messages); the store classifies it for Chord's owners: an int CHECK failure (SQLite's extended code SQLITE_CONSTRAINT_CHECK on Node; the error message on the Durable Object facade, which has no extended codes) is invalid/args and fails only that transaction; every other refusal of a Chord commit (the engine's counted UPDATE/DELETE, the live-address index, a primary key, the saturated allocator, a backend error in a statement) fails the object live (object_failed, broken: memory and disk disagree).
• 6a's owners: a refused self.tx rejects with the error above and the object fails as above; a refused migration commit (a chunk, (A) or (Z)) is rolled back whole, fails the activation broken, the table stays migrating, earlier committed chunks stay; a refused job start rejects the start; a refused outcome or finish commit fails the object (records and memory disagree); a refused commit that carried held changes (D8) fails the object.
• Reads: a read's callback returns data, never throws for a request: an adapter's request checks (an unknown conversation, a cursor, history a document does not keep) happen after the guarded read returned, from tagged results, so only a backend error fails the store. The store offers a read transaction for reads of several queries that must see one state (S4).

1.5 The documents engine

• Addresses: (owner, namespace, kind, key?): owner the owner name; namespace the adapter's partition, claimed with the owner (Chord: chord for documents, chord.jobs and chord.job-states for persisted runs; Durable: the document's scope, session, conversation:<id>, task:<id>, which Durable's reads by scope need); kind and key as the adapter writes them (Durable JSON-encodes both, as its indexed strings). One live record per address (the partial unique index).
• history per record (R1): none (Chord: older revisions dropped at a new base; row and revisions deleted at retirement), latest (Durable's session, task and latest conversation documents: older revisions dropped; the row kept with retired_at at retirement), rewindable (nothing dropped; the row kept). retireDocument reads the record and applies its rule.
• Explicit ids: a creation may carry its id (Durable mints ahead); otherwise the allocator gives one.
• Changes without a known revision: an adapter whose writes carry no revision (Durable) stages a change that reads the record in the step and uses its revision; the counted conditional UPDATE stays the guard.
• meta: Durable's fork policy and history details live in the record's meta.
• Reads at a point: find(address, at), load(id, at) (the base at or below at plus the deltas up to it, its version the base's), list(owner, namespace, at, kind?, limit, after?) by id, with the half-open lifetime (createdAt <= at < retiredAt); records carry retiredAt. New indexes (store schema version 2): (owner, namespace, kind, key, created_at), (owner, namespace, id), revisions (document_id, kind, seq); their cost on Chord's commit gate is measured.

1.6 The archive and batch apply

• Format 1 gains (nothing deployed): document.create carries history, retiredAt? and meta; a document.revision { id, seq, content } (content a DocumentContent: { kind: "base", version, value } or { kind: "delta", version, ops }) part per kept revision, in seq order after its create; for live none and latest records only the current base and its deltas are exported (as 6a); a retired latest record is exported as its record alone (no base: its content was dropped at retirement); for rewindable every revision, retired records included. Import checks: revisions in seq order, a base first, seq <= lastSeq, lifetimes at one address never overlapping (a second live record is still refused), versions per revision; the end count includes revisions.
• Batch apply (applyBatch(store, commits, ctx), internal): refused while any owner holds a claim on the store (an object or a StoreStorage: its memory would go stale) and run through the store's commit queue; commits[0].seq follows lastSeq (S5); one backend transaction for the batch, every write applied with its commit's seq, lastSeq and nextId set from the last commit; the applier handles every StoreWrite kind (alter, dropIndex, update/updateWhere/delete/deleteWhere with their recorded counts and versions, document.change, document.retire, document.revision), checked against the store's records (a trust boundary); a count that differs from the recorded one stops applying and marks the store diverged (S17): every later applyBatch rejects. A test replays a store's outbox into an empty store and compares the exports, manifest included.

2. StoreStorage: Durable's Storage on the store (D1, D2)

In Durable's src (an ordinary pi module), importing Chord v2 (@earendil-works/chord, then /next) for the store and the adapter surface below, marked internal for adapters. It passes Durable's storage conformance suite on SQLite :memory: and across reopen (a wrapper that closes and reopens the store file after every commit, Durable's reopen harness), and Durable's Session and harness test files run on it.

2.1 Opening and the adapter surface

StoreStorage.open(store, owner, ctx): claims owner (1.1); declares its tables through the store's table preparation (the path self.table takes: created at the newest version or migrated); raises the allocator to max(record_ids.id) + 1, saturating at Number.MAX_SAFE_INTEGER (one index read; covers stores built by import or batch apply); attaches to the store's host batch if there is one. Chord exports for it: StoreOwnerTables (declare and prepare tables under an owner), staged table writes and engine writes, the step reader, the read transaction, mintId, the commit with a fence. Its close(context) refuses new calls ("StoreStorage is closed"), waits for its calls underway, then releases its claims; the store stays open (the app owns it).

2.2 Tables

Every row carries the record's indexed columns and the whole record as json(unknown(), { version: 1 }); indexed strings that must stay lossless (kind, request_id) hold Durable's JSON encoding; index columns are non-opt, so absent values take sentinels that no encoded value can be (0 for an absent owner id, ids start at 2; "" for an absent request id, JSON-encoded strings start with "). SQLite's partial head index is the table entry_heads (builder, 2026-10-27: an index on (conversation_id, has_head, id) cost about 5.5 µs per commit; the table is written only for entries with a head and is faster on both paths).

Table	Key	Columns	Indexes
record_ids	id	record_type str
conversations	id	owner_conversation_id int, owner_task_id int, record	(owner_conversation_id, id), (owner_task_id, id)
entries	id	conversation_id int, commit_seq int (stamped), record	(conversation_id, id)
entry_heads	id	conversation_id int (one row per entry with a head, written with it)	(conversation_id, id)
tasks	id	conversation_id int, kind str, status str, abort_requested int, background int, record	(status, id), (conversation_id, id), (kind, id), (abort_requested, id), (background, id)
submissions	id	conversation_id int, request_id str, status str, record	(conversation_id, request_id), (conversation_id, id), (status, id)

2.3 Commits

commit(writes, adopt, ctx) stages one composite write and calls the store's commit with { admit: () => true, adopt: (seq) => adopt(seq) } (adopted in the host's batch, 1.3). Inside the step the composite write does what SqliteStorage does inside its transaction, with the step's reader: checkGlobalIds (an id already claimed by another record type, a conversation, entry or document written twice: Durable's messages), the per-document actions of prepareDocumentActions/checkDocumentActions (one content command per document, a delta on its version, one live incarnation per address: "already has a current incarnation"), then the table puts (record_ids only for ids not yet claimed by the same type), then the documents in two passes: first every document that already exists (its content, then its retirement), which frees the addresses a creation of this commit takes, then the creations and copies (each with its retirement, a create-plus-retire being an empty lifetime, createdAt = retiredAt = seq). Durable's "reconstructs rewindable documents" case (create secondId at address A, retire firstId at A, change firstId) needs this order. Copies (document.copy) are resolved before the step (S11): the source read at its point, checked against the copy's record ("changed in the copy batch", kind, key, history, fork), entered as a creation carrying the base. A refused check rejects with Durable's message (a refusal, 1.4): the store goes on, Durable's Session fails (GuardedStorage).

2.4 Reads

conversation, entry (both overloads, commitSeq from its column), findLatestHeadMarker, scanEntries (both orders, the fork chain walked as SqliteStorage does), task, scanTasks, submission, scanSubmissions, submissionByRequest, scanConversations: queries over the tables, Durable's cursors (scan.ts) as they are; findDocument, document (one read transaction), scanDocuments: the engine's reads at a point (1.5). Request errors (StorageRequestError) are thrown after the guarded read (1.4).

2.5 Cost

Statements per commit, as the gate measures them: an entry append, a record_ids check read, the record_ids put, the entry put, the meta update (4; SqliteStorage 5); a document delta, the record read, the revision insert, the counted update, the meta update (about 4-5; SqliteStorage about 7). Per put one compiled row check and the JSON encoding of indexed strings.

3. self.follow: Durable's views as Chord sources (D9, R4)

self.follow(attach) in a factory returns a Watchable (a state member, a derive source, an on source); attach(observer, ctx) returns Promise<{ value, detach } | undefined> (Durable's Observation; undefined for an absent observeDoc address). In 6b it is used inside the object that opened Durable (a native's value); reaching Durable from other objects is slice 8's (the harness in a registry entry, carry-over).

• Activation: after the natives opened, before the members are checked and derives compute: Chord makes the observer and calls attach; calls arriving before it resolves are queued and applied in order on value; a rejected or undefined attach fails the activation like a failed dependency (transient); a failed activation detaches what attached. The attempt's end detaches.
• Calls: a call inside a running batch (Durable's adopt on the store) applies the queued calls first, then marks the source in that batch; a call outside any batch (Durable's deferred commits, #hold, and the task graph's blocked refresh, even on the store; any Durable storage that is not the store) is queued and one batch on the next microtask applies everything queued. A derive over two views of one commit sees both changes in one computation.
• Ends (R4): failed fails the live object transient (reported once as object_failed with that failure, its identity's backoff, several sources ending at once fail it once; its retry reopens Durable and attaches again); ended is final: a pending retirement call reaches the source's watches first (Durable's endAfterDelivery), then they end ended; the member keeps its last value; a new watch on an ended follow source opens at that value and ends ended at once.

4. Persisted jobs (D5-D8, R3)

persist: { resume: "rerun" | "abandon" } only in local types with empty params (P3, at self.job); a persisted job needs a state definition with persist: true and checkpointWhen if it has state, and a JSON-only result (no bytes(), refused at declaration).

4.1 Records

Each unsettled run is a keyed persisted document in namespace chord.jobs (kind the job name, key the run id, history: none); its state is a keyed persisted document in namespace chord.job-states (same kind and key), so two jobs sharing a state definition never meet. The record:

{ version, fingerprint, args, key?, trace?,
  phase: "pending" | "running" | "stopped" | "finishing" | "blocked",
  restarts, finishFailures, abort?: true, blocked?: "finish_failed" | "version",
  outcome?: { kind: "done", result } | { kind: "failed", error: StoredJobError } }
// StoredJobError: { kind: "threw", error: WireError } | { kind: "aborted", reason } | { kind: "restart_limit", restarts }

Errors are stored in the wire error format (a PluginError's code, message and details survive); a stored outcome is rebuilt as the in-memory one.

4.2 Transitions

Event	Writes (one commit unless noted)	After
start	record pending, state initial()	invoked on the live attempt
reservation (before run)	record running	run invoked (a crash after it counts)
ctx.state.change	state; the record's restarts: 0 only when it was not 0
abort (D7)	record abort: true (on the line); the signal fires after the commit, outside the batch; the line is released, then abort joins run	"marked"
run settles	held state flushed (D8), record finishing with the outcome	finish on the live attempt
finish fails	record finishFailures + 1; at 5 blocked: "finish_failed"	retried after the backoff, a blocked run on reload
finish succeeds	record deleted, state retired (Q9), with what finish wrote	the run settles
clean stop (host close, a native restart, R3's seal)	the stopped runs written stopped, held state flushed, by the close writer
activation (records loaded once per object; each live attempt resumes what waits)	each running record: restarts + 1 (or, at 5, failed { restart_limit } and finishing, job_interrupted reported); stopped and pending: back to pending	maps, keys, active count, running rebuilt before anything runs; rerun invokes, abandon records failed { aborted, restart }; an abort-marked run finishes aborted without run; finishing resumes finish; blocked stays; a record of another job version, or of the same version with another fingerprint, is blocked: "version"

• A retry after a seal (R3): the runs the seal stopped are resumed by the next live attempt only once their old invocation settled (bounded by closeTimeout, as the drain; an invocation outliving it is fenced, its run resumes then), so a run that ignores its signal never runs twice at once.
• The close writer: at the object's end, after the drain and before the documents drop, one commit past the seal (Durable's writeAtClose): the runs stopped by the end written stopped, held state flushed; a run whose run outlived the drain is fenced and written stopped too; blocked persisted runs keep their records (dropped only by abort). In-memory runs keep 6a's rules.
• Pinning: records and state documents of unsettled runs stay in memory (never evicted): ctx.state.value and running read them synchronously.
• Fingerprint: the job's fingerprint (args, result, state) is stored; a mismatch is blocked: "version" (resolved by slice 7). A run ending at the limit reports job_interrupted { object, job, id, restarts } (a new HostError).

4.3 Deferred state (D8)

As Durable's holds: the slot adopts and publishes the change at once (valueVersion ahead of what is stored), keeps the held ops in a PendingOps (a base flag when checkpointWhen selected one), and the run's documents stay pinned; the next immediate change of the state document stores the held ops and its own as one delta on the stored revision (or a base); one foreground timer per object for the earliest deadline (withinMs, default 1,000 ms, Infinity waits for the others; a negative or non-number withinMs is invalid/args); the outcome's commit and the close writer store what is held; a held change of a fenced invocation is still stored; a crash loses it. A deferred change that would reset restarts is stored at once (the reset must be durable). On an in-memory job the option is accepted and does nothing.

5. Benchmarks and tests

• The gate ("The performance and memory bar"): Durable's storage benchmark (test/storage.bench.ts, every read and write case, 1k and 10k) and test/conversation-view-bench.ts on StoreStorage against SqliteStorage, outbox off (its cost reported separately): each hot path within about 5%, allocation per op and retained memory no worse beyond noise; Chord's 6a commit gate again with the new indexes and the allocator; persisted job start and state change reported against in-memory ones (a disk commit is not held to the in-memory number).
• Tests: the conformance suite (both fixtures); the D4 matrix (BEGIN, body, COMMIT, ROLLBACK failures; a refused Durable commit keeps the store; a refused Chord commit fails its object; a CHECK failure only its transaction; a refused migration chunk; a request error inside a multi-query read keeps the store); the allocator (a mint during a running Chord step, saturation across reopen, a refused step's gap, replay restoring nextId); create-plus-retire and change-plus-retire through the outbox and the applier; reads at a point on retired latest and rewindable records; export and import of rewindable history and mixed Chord and Durable data; applyBatch (every kind, the replay equality, a diverged count); self.follow (one frame per commit, a derive over two views, calls outside a batch on the store, early calls, an absent observeDoc, failed and ended, a retirement before the end); persisted jobs (every transition of 4.2, a crash between reservation and run, a crash during the close writer, a seal and retry, restart counting and its reset, the limit, the finish failure count across a crash, abort joined and stored, abort racing an outcome, deferred state merged, flushed at the outcome and at close, pinned past the retention); GC (closed StoreStorage, ended follow sources, cancelled abort waiters, deferred timers, settled and dropped runs).
• Examples: Durable's test/examples/35-durable-on-the-store.ts (a worker with Durable's Session and a Chord store object on one store, exporting the archive) and 36-durable-views.ts (Chord's tests import nothing of Durable); Chord's 28-persisted-jobs.ts.

Carry-over taken

From slices.md "Slice 6b": persisted jobs (all of it, the finish failure count included), StoreStorage (ids, fork copies, adopt in the batch), the Chord source over Durable's views (self.follow), the applier for batch apply; job.abort async and stored.

Carry-over left

Slice 7: blocked: "version" resolved on reload; runs of jobs the code no longer declares; blocked runs retried on reload. Slice 8: Durable's views from other objects (the harness in a registry entry). Slice 9: the trace carrier's use.