pi-mono/packages/chord/src/delta/README.md

Chord Delta

Chord Delta produces immutable JSON revisions and exact operation batches for ordered replicas. Import it from @earendil-works/chord/delta.

Immutability is an ownership contract. Nothing is frozen or defensively copied, so an illegal mutation is not detected. It silently corrupts state.

import { applyImmutable, applyImmutableBatches, track } from "@earendil-works/chord/delta";

const initial = { output: "", entries: [] as { id: number }[] };
const tracker = track(initial); // `initial` is transferred: never mutate it again

const change = tracker.beginChange();
change.state.output += "done\n"; // the draft is mutable only while the change is open
change.state.entries.push({ id: 1 }); // placed values are cloned; the caller keeps its object
const prepared = change.prepare(); // draft handles are unusable from here on

// tracker.value, prepared.base, prepared.value, and prepared.ops (including paths
// and payloads) are immutable by contract.
tracker.adopt(prepared); // tracker.value === prepared.value

// Shares containers with prepared.base and the op payloads: never mutate it either.
const replica = applyImmutable(prepared.base, prepared.ops);

Mutation rights

Value	May it be mutated?
Root passed to track(), prepareReplace(), replicatedState(), or replace()	No. Ownership moved to the tracker.
change.state and handles read from it	Yes, only while that change is open. After prepare(), abort(), or another adoption, every use throws. Writes through a handle whose element was removed from the draft are ignored.
External value after assigning or inserting it into a draft	Yes. The draft stored a validated clone.
tracker.value, prepared.base, retained older revisions	No.
prepared.value, prepared.ops, op tuples, paths, permutations, payloads	No. Payloads may be the same objects as parts of prepared.value.
applyImmutable() / applyImmutableBatches() inputs and result	No. The result shares containers with both inputs.
Values from replicated state and cells (value, listener values, in-process handles)	No. In-process consumers may share the provider's containers.
Mutable replica passed to apply()	Only through apply(), with batches it owns exclusively. Code that edits it between batches breaks convergence.
Batch passed to apply()	Consumed. apply() adopts payload containers into the replica. Use a detached copy for exactly one replica and never touch it again.

Replaying batches

Immutable replay is the preferred in-process path. One batch can fan out to any number of immutable replicas:

let replica = tracker.value; // shared with authority; never mutated
// For each adopted batch, in order:
replica = applyImmutable(replica, prepared.ops); // replica must equal prepared.base

When only the final result of an ordered backlog is needed, replay its batches without concatenating their operations. One copy-on-write scope is shared across the complete call, so no intermediate revision is exposed or safe to retain:

replica = applyImmutableBatches(
  replica,
  queuedFrames.map((frame) => frame.ops),
);

Use separate applyImmutable() calls when every intermediate revision is published or retained.

Mutable replay needs a detached starting root and a detached copy of every batch for every replica. Never apply one in-memory batch to two mutable replicas:

import { apply } from "@earendil-works/chord/delta";

const detach = <T>(value: T): T => JSON.parse(JSON.stringify(value)) as T;

let replica = detach(tracker.value); // exclusively owned starting root
// For each adopted batch, in order:
replica = apply(replica, detach(prepared.ops)); // fresh copy for this replica only
// Only apply() may change `replica`; do not edit it in application code.

Parsing a separately serialized message per replica is also a valid detachment. decode() does not copy payloads, so decoding one in-memory WireOp[] for two mutable replicas makes them alias each other.

Slow consumers

PendingOps holds the batches one consumer has not taken yet (a watch listener that is still running, a link whose window is full) and folds them while they wait, so what it hands over replays to the same value with fewer ops:

import { PendingOps } from "@earendil-works/chord/delta";

const pending = new PendingOps();
pending.push(prepared.ops); // for each adopted batch, while the consumer is busy
// When the consumer is ready again:
const batch = pending.take();
const ops = batch.kind === "ops" ? batch.ops : [["r", tracker.value]];

• A leaf op (an append, or a set of a primitive) folds into the last pending leaf op on its path: a + a join, s replaces s or a, an a after a string s appends into it. ["a", ["text"], "he"], ["s", ["n"], 1], ["a", ["text"], "y"] becomes ["a", ["text"], "hey"], ["s", ["n"], 1].
• Any other op (p, m, d, t, s of an object or array) ends folding into the ops before it, as it may change what a path names; r drops them.
• Past 10,000 pending ops after folding, only replace stays: the owner sends the current value as [["r", value]].
• One batch is handed over as received; received batches and op tuples are never mutated, so the same batch can feed other consumers.
• It folds ops of the tracker, diffRevisions() and appendText(); it does not validate or normalize hand-written ops.

Do and don't

• Do build a fresh root and then transfer it; validate untrusted roots at your ingestion boundary.
• Do keep structural sharing between revisions, for example replace(ctx, { ...state.value, changed }).
• Do clone or serialize before handing any published value to code that may mutate it.
• Do settle every change with prepare() or abort().
• Don't reuse one container at two places in a transferred root.
• Don't mutate anything the tracker owns, published, or received from a batch.
• Don't keep draft handles past their change or place a settled handle.
• Don't infer meaning from op shape. Only the resulting value is contractual.

Roots and placements

Full roots are trusted. track() and prepareReplace() take ownership in O(1) and never walk the tree. Roots must be alias-free, acyclic strict JSON: dense plain arrays, plain or null-prototype objects with enumerable own data properties, strings, booleans, finite numbers, and null. Violations have unspecified behavior. For example, a container shared by two keys can make prepared.value diverge from what replicas compute from prepared.ops.

Draft placements are copied, and that copy walk also validates. Placements come from property writes, index writes, push, unshift, splice, fill, and copyWithin. A non-strict value throws TypeError before the draft changes; for example, push(valid, invalid) inserts nothing. Rejected values include: cycles, accessors (never invoked), symbol keys, class instances, sparse or non-plain arrays, functions, bigint, NaN, infinities, and undefined. Null-prototype objects are accepted and preserved. Placing a draft handle clones its current content. Placing one value at several paths yields independent containers.

undefined rules:

• draft.obj.key = undefined deletes key.
• arr[i] = undefined, push(undefined), unshift, splice, and fill with undefined throw.
• undefined nested anywhere inside a placed object or array throws. It is not dropped like JSON.stringify does.

Arrays stay dense. Writing past the next index and deleting an element throw. Growing length inserts null; shrinking it removes elements.

Lifecycle

tracker.value is the latest adopted revision. beginChange() opens an overlay draft over it and never modifies it. A draft may stay open across await. A draft dropped without prepare() or abort() stays referenced by its tracker until the tracker's next beginChange() or adopt(). prepare() materializes the candidate and ops and does not change authority. adopt() checks the preparation and swaps the root pointer.

Several changes may be open or prepared from one revision. Adopting one makes all others stale: open drafts become unusable and their prepare() throws, and prepared competitors are rejected by adopt(). adopt() also rejects foreign, aborted, and already consumed preparations. Change.abort() or Prepared.abort() after prepare() prevents adoption. The candidate stays readable.

No-op batches:

• Empty ops means prepared.value === prepared.base. Writes restored to their original value and deeply equal container assignments usually normalize to empty.
• Equality ignores key order and prototype. A deeply equal assignment keeps the previous revision's order and prototype, even if the draft showed the new ones.
• Structural array edits may emit a nonempty exact batch even when the result is deeply equal.
• Adopting a no-op still advances tracker.revision and stales competitors. Replicated state does not publish no-ops.

prepareReplace(value) is a whole-root operation, not a diff. If value is deeply equal to the current root, ops is empty and the current root is kept (the comparison may traverse both trees). Otherwise ops is [["r", value]] and prepared.value === value, so replicas receive the complete root.

Array mutators: push, pop, shift, unshift, splice, reverse, sort, fill, and copyWithin. Held handles follow elements through reindexing.

Operations

Paths contain object keys and non-negative integer array indices.

Tuple	Meaning
["r", value]	Replace the complete value.
["s", path, value]	Set an object property or array element.
["d", path]	Delete an object property or remove an array element.
["a", path, text]	Append to a string.
["t", path, count]	Remove count UTF-16 code units from a string's front.
["p", path, index, remove, items]	Splice an array.
["m", path, permutation]	Reorder an array: new[i] = old[permutation[i]].

Batches are exact but not canonical. The same change may use different tuples, and large edit sets may fold into a region splice, an ancestor s, or r.

Reserved keys: the tracker never emits __proto__, constructor, or prototype as a path segment. A mutation at or below such a key is folded into a set of the nearest safe ancestor, or r at the root. apply(), applyImmutable(), applyImmutableBatches(), and decoder() reject those segments with UnsafePathError. The appliers write values as own data properties, never through a prototype setter. Appliers check op shape and path safety but not payload strictness. Chord's remote state applies batches decoded from JSON frames, so their payloads are strict JSON; validate payloads of batches from any other source.

encoder() interns repeated paths into WireOp tuples; decoder() validates and restores Op tuples. Use one encoder/decoder pair per ordered state stream. Path IDs span batches and an r resets both dictionaries. After a decode or apply error, discard the decoder and replica and recover from a later r.

Growing strings: use appendText

The footgun: appending to a string through a draft costs the string's whole length on every change, not the length of what you append.

const change = tracker.beginChange();
change.state.message.text += chunk;   // looks like O(chunk)
const prepared = change.prepare();     // emits ["a", ["message", "text"], chunk], but costs O(text.length)

The draft's proxy only sees the finished new string, never the chunk, so prepare() recovers the append by checking that the new string starts with the old one. That check reads the whole string, and V8 first copies a string built by += (a rope) into one flat piece. There is no fast path: measured on Node 24 and 26, 20,000 changes appending 40 characters each, up to 800 KB, take about 2.4 s with the check and about 1 ms without it. A stream of N chunks into one string therefore costs O(N x length) in total: quadratic.

The fix: hand Delta the chunk.

import { appendText } from "@earendil-works/chord/delta";

appendText(change.state.message, "text", chunk);   // records ["a", path, chunk]: O(chunk), no compare, no copy

• appendText(parent, key, text) takes the parent draft and a key whose type is exactly string (not optional, not a string literal type), so the compiler refuses appendText(block, "type", …).
• Several appends to one key in one change become one op. A plain assignment or deletion of the same key in that change falls back to the compare, as does an append after a plain write.
• The appended text is copied into a string of its own, so neither the op nor the new value keeps a larger string alive.
• Use it wherever a string grows by appends across many changes (streamed answers, tool output, logs). A one-off assignment is fine with =.
• Derives cannot use it: diffRevisions() compares two revisions it did not write, so a derive producing a growing string pays O(length) per delivery.

Limits and footguns

• No freezing. Mutating any immutable value corrupts state silently.
• Appending to a string through a draft (draft.text += chunk) costs O(string length) per change; use appendText (section "Growing strings").
• In-process sharing: local handles, cell subscribers, and replicatedState subscribers receive the provider's containers. A consumer mutation corrupts authority and makes remote replicas diverge.
• A change inside a large flat array copies that array's pointer storage for the new revision.
• Placement validation costs time and memory on bulk inserts. In a benchmark unshifting 100k three-field objects, it added about 19 ms and 20 MiB of transient heap.
• Draft reads go through proxies. In one measurement, 20,000 hot nested reads took about 9 ms through the tracker versus 0.08 ms on raw objects; compare with node --expose-gc packages/chord/test/delta-traversal.bench.ts --quick.
• External proxies as placements, argument coercion callbacks that mutate the draft, non-primitive array indices, and non-numeric comparator results are out of contract. So are sort comparators that mutate, prepare, abort, or adopt. Behavior is unspecified.
• Object identity is not replicated. Each path is an independent value placement.
• One tracker is one revision sequence. Delivery order and persistence belong to the surrounding protocol.