mirror of
https://github.com/badlogic/pi-mono.git
synced 2026-11-05 23:37:25 +00:00
docs: describe only the current Chord, Durable, and coding-agent code
Chord: README rewritten for the new runtime; notes.md and requests.md removed, their open limits and decisions folded into spec.md section 12; guide and spec status lines updated. Durable: drops the pico-v5 handoff and delta findings, renames the Pico spec, and uses current Chord terms. Delta README, issue categories, and PI_RADIUS_GATEWAY updated.
This commit is contained in:
parent
066ebe4ca1
commit
6db4070f17
15 changed files with 147 additions and 1988 deletions
|
|
@ -1,358 +0,0 @@
|
|||
# Chord delta investigation findings
|
||||
|
||||
## Decision and scope
|
||||
|
||||
The standalone ID-addressed graph tracker experiment was removed. It solved
|
||||
reference reassignment and replicated identity, but its memory, import, and
|
||||
hydration costs were unacceptable for the drawing workload. This rejects the
|
||||
measured implementation, not every possible graph representation.
|
||||
|
||||
Keep the existing tree/path delta implementation, its weak proxy-cache changes,
|
||||
and the spread-first cloning improvement. Keeping the weak-cache patch is not a
|
||||
claim that its performance is satisfactory: permanent retention improved greatly,
|
||||
but exhaustive reads still allocate gigabytes and became slower.
|
||||
|
||||
The graph prototype never had a package export or Chord service integration.
|
||||
Removing it does not change State/ReplicatedState or adopt graph semantics in the
|
||||
Pico specifications. Existing delta alias behavior remains a separate correctness
|
||||
problem; this investigation did not establish general mutable-graph correctness
|
||||
for the tree protocol.
|
||||
|
||||
## Workload and measurement rules
|
||||
|
||||
Measured on an Apple M5 Max with 128 GiB RAM, Node v26.0.0. The document contains
|
||||
20,000 strokes with 100 separately allocated `{ x, y, pressure }` point objects
|
||||
each: two million points, approximately 2.04 million total objects/arrays. It is
|
||||
not a packed numeric buffer. Full traversal uses a flat strokes array; mutation
|
||||
benchmarks partition strokes into layers and include a 200 KB string.
|
||||
|
||||
The plain document occupies about **139.5 MiB of JavaScript heap**. Its roughly
|
||||
66 MB serialized representation is a different measurement.
|
||||
|
||||
- Use independent processes, three trials per configuration, and rotated variant
|
||||
order. Report medians. Large graph/traversal workers used an 8 GiB V8 heap limit.
|
||||
- Separate initial import, snapshot publication, mutation, flush, JSON encoding,
|
||||
decoding, and replica application. Fixture construction is outside mutation
|
||||
timing; importing a newly assigned object is inside it.
|
||||
- Measure producer-only and producer-plus-replica memory separately. Release raw
|
||||
graph inputs and emitted batches in separate call frames before retained-heap
|
||||
measurements; otherwise temporary references can pin entire extra documents.
|
||||
- Ready/retained heap is measured after GC, relative to the warmed empty process.
|
||||
Sampled heap includes temporary allocations and uncollected garbage. It is not
|
||||
an exact allocation peak. Maximum RSS includes heap capacity and native/runtime
|
||||
allocations, not just live JavaScript objects.
|
||||
- Weak-reference reclamation requires real job/event-loop boundaries, not merely
|
||||
a synchronous `global.gc()` after a loop. Yield, collect, and let finalizers run.
|
||||
- Validate serialized replay outside timing. Graph correctness must compare
|
||||
identity/topology as well as values: deep equality alone misses lost aliases.
|
||||
- Raw-object and minimal-proxy controls perform no replication. They are local
|
||||
cost floors, not equivalent implementations.
|
||||
|
||||
The historical measurements below predate the final spread-first clone change
|
||||
unless explicitly stated. They are not new measurements of the current source.
|
||||
|
||||
## Operation log versus cloned baseline
|
||||
|
||||
The operation log at `2c995acf4` compares whole-container assignments immediately
|
||||
and coalesces recorded operations. The baseline design at `86bac52f9` marks dirty
|
||||
paths, compares against a retained copy at flush, and refreshes that copy.
|
||||
|
||||
A layer swap illustrates the difference. Two independently allocated layers can
|
||||
have equal geometry and different names. Both designs eventually emit just two
|
||||
name sets, **73 bytes**, but still inspect the geometry to discover that result.
|
||||
The baseline additionally refreshes its copy. Tiny output does not imply cheap
|
||||
mutation or flush.
|
||||
|
||||
For a fair clone-cost comparison, only the baseline's `cloneJson` helper was
|
||||
replaced with the exact helper from `2c995acf4`. No diff, coalescing, baseline
|
||||
refresh, or proxy behavior changed. This was the earlier dynamic-assignment
|
||||
helper, not the later spread-first helper.
|
||||
|
||||
Mutation plus flush, milliseconds:
|
||||
|
||||
| Workload | Original baseline | Matched-clone baseline | Operation log |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| One stroke swap | 0.114 | 0.086 | 0.085 |
|
||||
| Swap 1,000-stroke layers | 89.940 | 47.924 | 21.048 |
|
||||
| Swap 10,000-stroke layers | 906.390 | 483.264 | 214.425 |
|
||||
| Ten 1,000-stroke swaps before flush | 90.598 | 49.455 | 209.346 |
|
||||
| Sixty selections before flush | 0.121 | 0.101 | 2.346 |
|
||||
| Swap dissimilar 100-stroke layers | 12.097 | 7.515 | 168.273 |
|
||||
|
||||
Batching favors the baseline because it compares the final dirty region once;
|
||||
the operation log pays assignment-time comparison repeatedly. Ten swaps restore
|
||||
the original value and emit no operations in all three variants. The dissimilar
|
||||
case triggers the operation log's 4,096-metadata threshold and publishes a roughly
|
||||
66 MB root replacement; the baselines emit about 2 MB of patches. This difference
|
||||
is not caused by the cloning helper.
|
||||
|
||||
All 63 independent-process configurations completed. Small-fixture checks and
|
||||
full-size serialized replay checks passed; original and matched-clone baselines
|
||||
emitted identical batches in the checked cases. RHS references came from
|
||||
`tracker.target` to model unwrapping. These benchmarks do **not** establish general
|
||||
alias correctness.
|
||||
|
||||
### Semantics cannot be inferred from passing performance cases
|
||||
|
||||
Historical probes found replica divergences around aliases, held references,
|
||||
reindexing, and detach/reinsert, despite the existing tests passing. The baseline
|
||||
also diverged for shared initial objects and held elements after `unshift`.
|
||||
|
||||
For example, selecting an item by reference and subsequently editing it means
|
||||
two producer paths observe one object. A tree replica holds two separate copies;
|
||||
it needs updates at both paths. Keeping a reference across an array insertion
|
||||
also means subsequent writes must follow the object's new position, not its old
|
||||
index. Neither cloning a baseline nor caching a proxy at a path automatically
|
||||
solves these cases.
|
||||
|
||||
Other approaches considered:
|
||||
|
||||
- Deep copy-on-assignment changes alias semantics and adds traversal/allocation.
|
||||
Earlier clone measurements were about 7.4 ms for 1,000 strokes, 76.6 ms for
|
||||
10,000, and 153.8 ms for 20,000. Adding copies increased a large swap from about
|
||||
213 ms to 385 ms. Those figures predate spread-first cloning.
|
||||
- Persistent trees share unchanged structure and avoid a permanent complete
|
||||
baseline copy. They do not independently solve live aliases or relocation.
|
||||
- A path-based wire copy operation needs ordering barriers: existing operation
|
||||
coalescing can otherwise change the source before the copy executes.
|
||||
- Declared reference fields/application IDs, stricter tree ownership, and explicit
|
||||
path setters change the contract. No replacement API was selected; ordinary
|
||||
JavaScript mutation syntax remained the desired interface.
|
||||
|
||||
## What the graph experiment did
|
||||
|
||||
Each object/array became a canonical node with a numeric ID. Scalars were stored
|
||||
inline; object edges referenced canonical nodes. A map indexed nodes by ID, a
|
||||
weak map recognized imported raw objects, and lazy producer proxies exposed
|
||||
ordinary mutation syntax. This was normalized graph storage, not a reverse-edge
|
||||
index layered beside a second complete document.
|
||||
|
||||
Wire references used `["@", id]`, with separate object/array definitions and
|
||||
ID-addressed set, delete, splice, length, and release operations. Revisioned
|
||||
snapshots allowed recovery. Replicas allocated shells before linking definitions,
|
||||
materializing actual JavaScript objects with aliases and cycles. Incremental
|
||||
updates preserved held replica identities; rebasing reset identities.
|
||||
|
||||
Known-object assignments became reference writes. Swapping two huge layers no
|
||||
longer walked their descendants. Fresh independently reconstructed equal objects
|
||||
were new identities, however, and needed complete definitions rather than a
|
||||
small value diff. Detached nodes remained resident until explicit collection.
|
||||
|
||||
Fifty tests covered topology, cycles, held references, array operations, import
|
||||
rollback, revision recovery, collection, and deterministic mixed mutations. The
|
||||
full benchmark completed 120 independent processes with value validation and
|
||||
held-identity checks for graph swaps. The initial full-traversal comparison added
|
||||
12 processes. Correctness on these cases did not make the implementation cheap.
|
||||
|
||||
### Graph mutation and publication costs
|
||||
|
||||
Historical sibling delta before the weak-cache change versus graph, mutation
|
||||
plus flush:
|
||||
|
||||
| Workload | Tree delta ms | Graph ms | Tree bytes | Graph bytes |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| 100 scattered edits | 0.265 | 0.158 | 5,243 | 2,235 |
|
||||
| Swap 1,000-stroke layers | 21.523 | 0.002334 | 73 | 111 |
|
||||
| Swap 10,000-stroke layers | 221.373 | 0.002584 | 73 | 109 |
|
||||
| Ten 1,000-stroke swaps before flush | 211.370 | 0.015083 | 2 | 471 |
|
||||
| Sixty selections before flush | 0.768 | 0.046125 | 66 | 102 |
|
||||
| Append one new 100-point stroke | 0.011 | 0.095 | 3,357 | 7,567 |
|
||||
| Fresh equal-geometry 1,000-stroke layer | 12.311 | 78.716 | 37 | 7,440,956 |
|
||||
| Append ten characters to a 200 KB string | 0.010 | 0.000958 | 29 | 200,299 |
|
||||
|
||||
Graph bytes include its envelope; tree bytes are decoded operation arrays, not
|
||||
path-dictionary encoding. The formats provide different identity guarantees.
|
||||
The graph retained ordered reference writes for cancelled swaps. Its missing
|
||||
string append/truncate compression was a prototype limitation, not an inherent
|
||||
cost of graph addressing.
|
||||
|
||||
With JSON encode/decode and replica application included, the fresh-layer case
|
||||
cost **165.773 ms** for graph versus **12.017 ms** for tree delta. The 10,000-stroke
|
||||
swap pipeline cost **0.009958 ms** versus **214.119 ms**. Initial import and
|
||||
hydration are not included in those incremental numbers.
|
||||
|
||||
Twenty thousand hot nested reads cost about **20.84 ms** for graph, **9.12 ms** for
|
||||
tree delta, **6.79 ms** for a minimal proxy, and **0.082 ms** for raw objects. This
|
||||
small working set touched only 1,000 points and missed exhaustive-read allocation.
|
||||
|
||||
### Initial graph costs and lifetime
|
||||
|
||||
Representative scalar-workload measurements:
|
||||
|
||||
| Metric | Tree delta | Graph |
|
||||
| --- | ---: | ---: |
|
||||
| Ready producer heap | 139.5 MiB | 430.4 MiB |
|
||||
| Ready producer plus replica | 279.3 MiB | 719.3 MiB |
|
||||
| Initial import | 0.020 ms | 1,118.907 ms |
|
||||
| Snapshot flush | 169.218 ms | 594.904 ms |
|
||||
| Snapshot wire size | 66,364,338 bytes | 147,487,101 bytes |
|
||||
| Snapshot JSON encode | 176.961 ms | 366.942 ms |
|
||||
| Snapshot JSON decode | 159.388 ms | 1,179.166 ms |
|
||||
| Snapshot apply | 0.027 ms | 1,417.161 ms |
|
||||
| Pipeline-process maximum RSS | 852.8 MiB | 4,978.0 MiB |
|
||||
|
||||
Snapshot phase timings came from a separately requested rebase. Tree application
|
||||
adopts the already decoded snapshot; graph application validates and materializes
|
||||
objects. RSS includes earlier import/hydration allocations and is not a minimum
|
||||
live-set requirement.
|
||||
|
||||
Five fresh 1,000-stroke layer replacements grew graph producer heap from **430.2
|
||||
to 563.9 MiB**. Explicit collection removed 510,010 nodes in **705.5 ms**, emitted
|
||||
a **7.14 MB** release batch, and left **486.3 MiB** retained. Producer plus replica
|
||||
went from **719.2 to 967.0 MiB**, then **831.2 MiB** after collection. Lookup-table
|
||||
capacity and externally held values can prevent a return to the ready footprint.
|
||||
|
||||
Automatic collection, immutable historical snapshots, bounded logs, hardened
|
||||
transport limits, and service integration were not implemented.
|
||||
|
||||
## Full traversal exposed the memory problem
|
||||
|
||||
Reading every point through producer proxies allocated wrappers for nearly every
|
||||
container. Historical retained-heap results, after traversal and GC:
|
||||
|
||||
| Variant | Ready MiB | After full read MiB | Cold read ms | Warm read ms |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| Raw document | 139.47 | 139.49 | 8.00 | 2.95 |
|
||||
| Minimal cached read-only proxy | 139.47 | 452.50 | 705.10 | 550.35 |
|
||||
| Tree delta before weak caches | 139.26 | 3,525.93 | 2,720.98 | 470.89 |
|
||||
| Graph prototype | 430.39 | 2,159.75 | 1,437.65 | 388.74 |
|
||||
| Historical matched-clone baseline | 294.22 | 2,228.71 | 827.13 | 360.95 |
|
||||
|
||||
The baseline row was measured in a later three-process run using asynchronous GC
|
||||
boundaries, not extrapolated from its ready heap. Its immediately sampled heap
|
||||
was **2,245.37 MiB** and maximum RSS **2,456.30 MiB**. No diff was running: this was
|
||||
read-only traversal. All three historical trackers strongly cached wrappers.
|
||||
|
||||
### Existing delta: permanent retention fixed, allocation volume not fixed
|
||||
|
||||
A separate matched before/after run measured the weak-cache patch:
|
||||
|
||||
| Metric | Before | After |
|
||||
| --- | ---: | ---: |
|
||||
| Ready producer heap | 139.50 MiB | 139.50 MiB |
|
||||
| Retained after traversal, job boundaries, and GC | 3,526.17 MiB | 203.54 MiB |
|
||||
| Immediately sampled heap after cold read | 3,531.93 MiB | 2,491.78 MiB |
|
||||
| Maximum RSS | 3,858.11 MiB | 3,391.06 MiB |
|
||||
| Cold traversal | 2,397.40 ms | 6,298.76 ms |
|
||||
| Warm same-job traversal | 466.43 ms | 844.13 ms |
|
||||
|
||||
These are separate runs from the preceding table. Warm time is the median of
|
||||
three traversals per process after collection: the first reconstructs collected
|
||||
wrappers and the following two reuse same-job caches.
|
||||
|
||||
The patch separates public proxies from entry/placement metadata, uses weak
|
||||
ordinary caches and incarnation-safe finalizer cleanup, and keeps explicit alias
|
||||
metadata separately. Held descendants retain ancestor metadata so array
|
||||
reindexing still works if ancestor public proxies were collected. Assigned known
|
||||
proxies are unwrapped, avoiding wrappers embedded in raw targets. Placement reuse
|
||||
prevents duplicate splice emission after reindexing or reattachment. Shared
|
||||
handlers, lazy child maps, and job-local caches reduce some overhead.
|
||||
|
||||
But one point read can still allocate a proxy, handler, tracking entry, placement
|
||||
record, placement set, weak references, cache entries, and finalization records.
|
||||
Across roughly two million containers, this is gigabytes. `WeakRef` targets newly
|
||||
created or dereferenced during an uninterrupted synchronous traversal must remain
|
||||
alive through that job. The temporary strong caches also survive until scheduled
|
||||
microtask cleanup. Collection afterward does not prevent the allocation peak.
|
||||
|
||||
The remaining roughly **64 MiB** above the raw document after collection is
|
||||
bookkeeping/index overhead; its exact composition needs a heap profile. The
|
||||
roughly **1.2 KB per visited container** of additional immediately sampled heap is
|
||||
an aggregate average, not an allocation-by-allocation attribution.
|
||||
|
||||
Eleven isolated GC regressions cover unused proxy reclamation, tracker disposal,
|
||||
held identities and descendants, array reacquisition/reindexing, alias rewrapping,
|
||||
detach/reinsert, root rebasing, blocked views, and repeated wide-root read windows.
|
||||
Twelve windows over a 10,000-object root array retained about 0.35 MiB bookkeeping.
|
||||
Normal mutation correctness must not depend on finalizers running promptly.
|
||||
|
||||
This patch does not resolve the general tree-protocol alias problem. Bulk
|
||||
read-only inspection through `tracker.target` avoids proxy construction, but
|
||||
writing through it bypasses tracking. That escape hatch does not make normal
|
||||
producer-proxy traversal cheap.
|
||||
|
||||
## Clone layout: a separate, smaller improvement
|
||||
|
||||
Why did a 139.5 MiB document plus a baseline occupy 294.2 MiB rather than 279 MiB?
|
||||
V8 layout inspection on this Node version showed:
|
||||
|
||||
- A literal `{ x, y, pressure }` occupies 48 bytes, with three in-object slots.
|
||||
- Building `{}` by dynamic assignment occupies 56 bytes, with one unused slot.
|
||||
- Two million points times eight bytes accounts for **15.26 MiB** extra.
|
||||
|
||||
The old clone therefore occupied about **154.7 MiB**, not 139.5 MiB. The discrepancy
|
||||
was not hidden proxy allocation.
|
||||
|
||||
Shallow object spread followed by recursive replacement of object-valued children
|
||||
kept the compact layout. Arrays retain their recursive array branch; null-prototype
|
||||
objects retain a null prototype. Full-fixture clone-only measurements, three fresh
|
||||
processes per method:
|
||||
|
||||
| Clone construction | Additional retained heap | Median clone time |
|
||||
| --- | ---: | ---: |
|
||||
| Dynamic property assignment | 154.725 MiB | 162.338 ms |
|
||||
| Spread first, then recurse | 139.466 MiB | 93.924 ms |
|
||||
|
||||
This helper was adopted in existing delta. Regression tests cover detached nested
|
||||
values, null prototypes, inherited-name collisions, independent expansion of
|
||||
shared input values, and published payload isolation. V8 layouts/timings are
|
||||
engine-specific. `structuredClone()` still produced the 56-byte point layout in
|
||||
the layout probe; JSON round-trip produced 48-byte points but introduces text
|
||||
encoding/decoding and temporary text allocation.
|
||||
|
||||
A baseline using the new helper would be expected to need about **279 MiB** for
|
||||
document plus copy. That is an inference from measured clone sizes, not a rerun
|
||||
of the complete historical baseline. The improvement does not address gigabytes
|
||||
of proxy bookkeeping during traversal.
|
||||
|
||||
## Retained reproduction and provenance
|
||||
|
||||
From the repository root:
|
||||
|
||||
```sh
|
||||
node --expose-gc packages/chord/test/delta-traversal.bench.ts --quick --out /tmp/delta-traversal-quick.json
|
||||
node --expose-gc packages/chord/test/delta-traversal.bench.ts --out /tmp/delta-traversal.json
|
||||
node --expose-gc packages/chord/test/delta-traversal.bench.ts --modes delta --out /tmp/delta-only.json
|
||||
```
|
||||
|
||||
The retained benchmark supports raw, minimal-proxy, and current tree-delta modes.
|
||||
It checks traversal results, uses separate processes, records source hashes, and
|
||||
allows multiple event-loop/GC turns. It no longer contains the removed graph
|
||||
implementation or reproduces the historical graph/mutation comparison.
|
||||
|
||||
Targeted tests from `packages/chord`:
|
||||
|
||||
```sh
|
||||
node ../../node_modules/vitest/dist/cli.js --run test/delta.test.ts test/delta-clone.test.ts test/delta-tracker/retention.test.ts test/replicated-state.test.ts
|
||||
```
|
||||
|
||||
Historical source identifiers:
|
||||
|
||||
- Baseline: `86bac52f9`; matched clone helper / operation log: `2c995acf4`.
|
||||
- Pre-retention sibling delta SHA-256:
|
||||
`af008c77367a0ba362637b3b0422576c752a205339a33eb26479a41ac087957f`.
|
||||
- Weak-cache delta before spread-first cloning:
|
||||
`7a51c9295543b6fb47e06d8460f122242ee9a96a0ef5653d5595b8219792a6d8`.
|
||||
- Removed graph implementation:
|
||||
`90893bfa02a7aeb0fc1da9fdd62f53027ab3d22728337d00f7040c19344fec6c`.
|
||||
|
||||
The graph code was an uncommitted experiment; its hash identifies the measured
|
||||
source but does not provide a Git revision from which to restore it. Historical
|
||||
raw outputs were left in `/tmp/chord-graph-prototype/` (`full.json`,
|
||||
`traversal-final.json`, `retention-before.json`, `retention-final.json`, and
|
||||
`baseline-traversal.json`). Matched-baseline data was in
|
||||
`/tmp/pi-copy-cost.QAlME4/`; clone-only samples in `/tmp/chord-clone-compare.jsonl`.
|
||||
These are optional local artifacts, not durable dependencies of this document.
|
||||
|
||||
## Requirements for a future attempt
|
||||
|
||||
Start with exhaustive producer reads, not only small hot working sets and
|
||||
mutation/flush benchmarks. Measure cold allocation, warm access, cross-job
|
||||
reclamation, held identities, alias/reindex behavior, import, hydration, and
|
||||
fresh-object replacement separately. Never present post-GC retained heap as peak
|
||||
memory, tiny patches as proof of cheap comparison, or successful value replay as
|
||||
proof of identity correctness.
|
||||
|
||||
No measured design in this investigation simultaneously achieved low retained
|
||||
and transient memory, cheap ordinary reads, cheap large reference reassignment,
|
||||
and the desired broad mutable-JavaScript semantics.
|
||||
|
|
@ -1,913 +0,0 @@
|
|||
# Pico5 implementation handoff
|
||||
|
||||
`packages/durable/docs/spec.md` is normative. Implement this list in order.
|
||||
After every package: run its tests, run `npm run check`, and stop for user review.
|
||||
Do not redesign later packages while implementing the current one.
|
||||
|
||||
Pico3 is reference material only. Preserve useful behavior, not its capability
|
||||
facades, membranes, document routing, view projection, events, or clone chains.
|
||||
|
||||
## Status
|
||||
|
||||
- Obsolete `pico` and `pico4` prototypes were removed.
|
||||
- `pico3` remains.
|
||||
- Packages 1–23 are implemented in `packages/durable`; Package 10 was already satisfied by Chord's canonical structural diff implementation.
|
||||
|
||||
## 1. Records, cursors, and memory tables
|
||||
|
||||
Implement IDs, sequences, reserved root conversation ID `1`,
|
||||
`ConversationRecord`, `EntryRecord`, strict input/write `SubmissionRecord`
|
||||
values, live/terminal `TaskRecord` values, document records, storage writes, backend-opaque JSON cursors, and
|
||||
detached `MemoryStorage` tables.
|
||||
Reserve `Conversation` for the public conversation object, `Entry` for the typed
|
||||
entry definition, and `Task` for the typed executable definition returned by
|
||||
`defineTask()`.
|
||||
|
||||
Test reserved root identity and immutable creation, mixed atomic commits,
|
||||
rollback, detached reads/writes, cursor boundaries,
|
||||
fork-aware entry scans through deep ancestor caps, head lookup,
|
||||
entry-to-commit lookup, full task replacement, and submission replacement/
|
||||
request-ID lookup.
|
||||
|
||||
## 2. Memory document records
|
||||
|
||||
Add selected document base/delta writes, retirement, reincarnation,
|
||||
current/as-of membership, exact logical-address lookup, scoped scans, and
|
||||
materialized point-in-time reads. Storage keeps base/delta revisions private and
|
||||
returns a detached value plus its stored definition version. It applies Chord
|
||||
`Op[]` directly and receives no definition callbacks or unused candidate values.
|
||||
|
||||
Test Session-, conversation-, and task-scoped documents, half-open lifetimes,
|
||||
create-plus-retire, retired historical membership, family queries, current-only
|
||||
reclamation, version boundaries, detached ownership, and no scans of unrelated
|
||||
document records.
|
||||
|
||||
## 3. SQLite backend
|
||||
|
||||
Implement the complete storage contract with ordinary rows and indexed document
|
||||
records. Do not translate Chord operations into SQL JSON patches.
|
||||
|
||||
Run the memory conformance suite after reopen. Test SQL transaction rollback,
|
||||
recent/ancient as-of reads, query plans, latest reclamation, WAL checkpointing,
|
||||
deleted-page reuse, and representative storage sizes.
|
||||
|
||||
## 4. JSONL publication
|
||||
|
||||
Implement table writes in `main.jsonl`, one document sidecar per incarnation,
|
||||
one sidecar per live task, and one main marker per commit. Do not add a
|
||||
standalone-sidecar protocol.
|
||||
|
||||
Copy, rather than import, the current `ExecutionEnv`, `FileSystem`, `Shell`, Node
|
||||
implementation, and their required utility files from `packages/agent/src/harness`
|
||||
into `packages/durable/src/env`. Copy only the environment-related slice, not
|
||||
agent skills, prompts, telemetry, or tool definitions. Extend the copied
|
||||
filesystem contract with exact-byte file truncation and file flushing. JSONL
|
||||
depends only on `FileSystem`, not the broader `ExecutionEnv`. Keep the portable
|
||||
environment and JSONL entry points free of Node built-ins; expose Node
|
||||
implementations only from `/env/node` and `/storage/jsonl/node`. Do not use the
|
||||
Pico3 implementation as source material.
|
||||
|
||||
Refactor the current `MemoryStorage` state machinery into a two-phase prepared
|
||||
mutation: validation and detachment produce a candidate that can later be
|
||||
applied without failure. Build `MemoryStorage.commit()` on that pair, and reuse
|
||||
the same machinery for JSONL. JSONL must append every prepared sidecar record,
|
||||
append the main marker, and only then apply the prepared in-memory mutation.
|
||||
Serialization or preparation failure occurs before file I/O and does not poison
|
||||
the backend. Retained indexes/materializations remain detached from write
|
||||
arguments, and reads never expose backend-owned cached objects.
|
||||
|
||||
JSONL creation has `fsync?: boolean`, defaulting to `false`. With `false`, append
|
||||
sidecars and then the marker without an explicit flush. With `true`, append all
|
||||
affected sidecars, flush each affected sidecar, and then append the main marker.
|
||||
Do not explicitly flush `main.jsonl` for ordinary publication. A main-only commit
|
||||
has no sidecars to flush. Any uncertain publication append or flush failure
|
||||
poisons the open backend and publishes no prepared in-memory mutation. Package 5
|
||||
adds the separate post-publication flush required to authorize reclamation.
|
||||
|
||||
Fault-test torn/short sidecar writes, failures between sidecars, every marker
|
||||
boundary, unconfirmed tails, missing confirmed data, poisoned writes, exact-byte
|
||||
tail truncation, both fsync settings and their call ordering, detached retained
|
||||
state and reads, and browser-safe portable entry points. Run the complete storage
|
||||
conformance suite directly and after reopen.
|
||||
|
||||
## 5. JSONL reclamation
|
||||
|
||||
Implement task-document retirement and current-only base reclamation using
|
||||
committed markers and descriptor invalidation. Remove a sidecar directly when no
|
||||
records remain; otherwise use temporary replacement and rename. With `fsync:
|
||||
true`, flush `main.jsonl` once before destructive reclamation so the authorizing
|
||||
marker cannot disappear while cleanup survives; if that flush fails, skip
|
||||
reclamation without failing the already-published commit. Flush a non-empty
|
||||
temporary replacement before rename. This is not publication flushing or main
|
||||
compaction.
|
||||
|
||||
Crash-test every rewrite/rename boundary. Verify that rewindable history is
|
||||
never reclaimed and default no-fsync behavior matches the specification.
|
||||
|
||||
## 6–7. Tracker transaction core, definitions, and typed access
|
||||
|
||||
**Prerequisite:** `@earendil-works/chord/delta` exports the canonical
|
||||
Astra-immutable-optimized `track`, `Tracker`, `Change`, and `Prepared`, and its
|
||||
draft placements reject values that are not strict JSON.
|
||||
Experimental variants under other Delta directories are not Pico APIs.
|
||||
|
||||
Implement these packages as one milestone. Keep the implementation layers
|
||||
separate, but do not build a temporary untyped document-acquisition seam.
|
||||
Implement the generic `Tx` table surface these tests require: exact table reads,
|
||||
paginated conversation/entry/task scans, `ReadAfterWrite`, ID-creating writes,
|
||||
and full task replacement. Scans expose the Storage cursor and caller-selected
|
||||
limit; they never hide an unbounded full scan. Semantic
|
||||
conversation, entry, task, and scheduler behavior remains in Packages 13–17.
|
||||
|
||||
Keep one Astra-immutable tracker per loaded document. Its trusted immutable
|
||||
`value` is the current shareable revision. `prepare()` emits detached
|
||||
self-contained operations and computes the next revision with the optimized
|
||||
immutable applier; operation placement payloads and that revision may share
|
||||
containers, and neither may be mutated. `adopt()` validates ownership and
|
||||
revision, then only pointer-swaps to the already-computed value.
|
||||
|
||||
Implement scope-preserving singleton/family tokens and overloads for Session,
|
||||
conversation, and task owners. Only `tx.doc()` is get-or-create: singleton tokens
|
||||
supply `initial()`, while family calls always supply key and seed and use only the
|
||||
first seed when absent. Definitions are explicit typed arguments, not registered
|
||||
declarations; conflicting definitions claiming one persisted kind are
|
||||
unsupported caller misuse.
|
||||
|
||||
On first `tx.doc()` access, memoize the acquisition promise by logical address
|
||||
before awaiting it, then call `tracker.beginChange()`. Repeated access returns the
|
||||
same overlay draft for the whole possibly async Session callback. The Session
|
||||
line permits only one open change per tracker. A callback that settles with an
|
||||
unresolved acquisition rejects: seal `Tx`, abort open changes, drain and abort
|
||||
the pending acquisition, and observe its failure. Callback failure aborts every
|
||||
change. Callback success prepares every change before Storage admission.
|
||||
|
||||
Do not walk prepared operation payloads or selected bases for strict JSON; they
|
||||
are strict JSON by construction. Tracker branding and revision checks enforce
|
||||
ownership and staleness. Evaluate each staged document write exactly once and
|
||||
pass Storage only the selected base value or operation batch. Keep every previous immutable revision unchanged through Storage
|
||||
settlement. On success, adopt every prepared value by pointer swap and enqueue
|
||||
its immutable revision/operations publication before releasing the line. On
|
||||
Storage failure, abort prepared changes, poison the Session, and publish nothing.
|
||||
Preparation failures roll back normally; Package 8 adds checkpoint selection and
|
||||
its failure path.
|
||||
|
||||
Initializer, migration, and replacement roots are copied into exclusive kernel
|
||||
ownership with a strict-JSON check before becoming trusted immutable revisions.
|
||||
Loaded and fork-copy roots come detached from Storage and are tracked without
|
||||
another copy. Chord copies and strict-JSON-checks every draft placement and
|
||||
throws at the offending assignment. Astra empty batches suppress
|
||||
ordinary writes, while replayable nonempty structural no-ops remain valid writes
|
||||
and publications. No runtime freezing or second operation-payload copy is
|
||||
required.
|
||||
|
||||
Snapshot, source, and watch lookup never create and return `undefined` when
|
||||
absent. `snapshot()` returns the shareable immutable current revision; callers
|
||||
must copy before mutation. A read-only migration may cache its migrated immutable
|
||||
tracker together with the older stored-version marker, without writing; the next
|
||||
successful `tx.doc()` still writes the required current-version base.
|
||||
Transaction-staged creation or migration enters the shared cache only after its
|
||||
enclosing Storage commit succeeds. All cold loads run on the Session line; a
|
||||
loaded immutable revision may be read without copying.
|
||||
|
||||
Test callback failure; escaped-draft revocation at callback settlement; concurrent duplicate
|
||||
acquisition; callback failure and success with a pending acquisition; late
|
||||
acquisition after sealing; concurrent initialization once; initial bases; family
|
||||
first-seed wins; scope/token mismatch; non-creating reads; shared immutable
|
||||
snapshots and stable prior revisions; empty-batch suppression and replayable
|
||||
redundant structural no-ops; multi-document preparation failure; uncertain
|
||||
Storage failure poisoning; old-revision stability through Storage settlement;
|
||||
pointer-swap and replacement adoption; operation/revision payload sharing under
|
||||
the trusted no-mutation contract; non-JSON initializer and draft-placement
|
||||
rejection; assignment copying and repeated-placement
|
||||
independence; authority and prepared-draft non-escape; terminal-task rejection;
|
||||
task-derived conversation identity; retirement; reincarnation-bound sources;
|
||||
and unload/reload. Include create-task-then-document,
|
||||
document-after-terminal rejection, and create-document-then-terminal settlement
|
||||
in one transaction; internal candidate validation must not trigger
|
||||
`ReadAfterWrite`.
|
||||
|
||||
## 8. Checkpoints and migration
|
||||
|
||||
After tracker preparation, Session evaluates `checkpointWhen(value, ops, { deltasSinceBase })` exactly once
|
||||
for ordinary mutations and sends Storage only the selected base or delta.
|
||||
Implement required creation/version bases and lazy all-older-version migration
|
||||
on typed access; Harness open does not scan ordinary documents.
|
||||
|
||||
Test read-only in-memory migration, `tx.doc()` migration rollback and coalescing
|
||||
with later edits, rewindable migration on current/historical read, the first
|
||||
successful `tx.doc()` version base even without a JSON change, newer-version
|
||||
rejection, migrated state/watch hydration without a write, subsequent operations
|
||||
against that migrated baseline, stored-version fork copying, unaccessed and unavailable-definition
|
||||
preservation, predicate failure rollback before Storage admission, and checkpoint
|
||||
starvation without backend heuristics.
|
||||
|
||||
## 9. Conversation document forks
|
||||
|
||||
Using fixture conversations and entry-to-commit mappings, implement the `asOf`,
|
||||
`current`, and `initial` settings for singleton and family documents.
|
||||
|
||||
Test opaque stored-version copying without definitions, retired membership, new
|
||||
child incarnations, later lazy migration, lazy `initial` creation, and exclusion
|
||||
of task- and Session-scoped documents.
|
||||
|
||||
## 10. Chord structural array operations
|
||||
|
||||
**Chord-owned prerequisite/integration:** the canonical Astra-immutable operation
|
||||
generator must encode compact replayable array changes; Pico only verifies and
|
||||
consumes it.
|
||||
|
||||
Improve the canonical generator so ordinary positional mutations encode
|
||||
scattered removals without carrying retained payloads. Callers must not write
|
||||
operations manually.
|
||||
|
||||
Test front/tail/middle/scattered/all/no removal, retained 256 KiB and 1 MiB
|
||||
payloads, append plus removal, later nested/index writes, exact replay, unchanged
|
||||
previous immutable revisions, and equality between Astra's prepared candidate
|
||||
and immutable operation replay. One prepared document change remains one Session
|
||||
commit; no intermediate candidate is adopted or published.
|
||||
|
||||
## 11. Chord document state
|
||||
|
||||
Use Chord's existing `ReplicatedStateSource` attachment contract internally, but
|
||||
expose one already-attached read-only `DocumentState` per acquisition. It must
|
||||
atomically hydrate in O(1) from the current immutable revision and publish later
|
||||
exact committed immutable value/operation frames without another tracker, value
|
||||
copy, or re-diff. Disposal unregisters that one state. Pico remains the sole
|
||||
document mutator.
|
||||
|
||||
Test contiguous Chord delivery sequences, atomic hydrate/subscribe, a baseline
|
||||
that covers queued publication without duplicate application, exact value and
|
||||
operation reference sharing, retirement to `null`, incarnation-bound recreation,
|
||||
independent state disposal, migrated hydration, definition-free fork copies,
|
||||
tracker-cache unload, and trusted mutation footguns.
|
||||
|
||||
## 12. Document watches
|
||||
|
||||
Implement non-creating `watchDoc` as an incarnation-bound `WatchHandle` that
|
||||
returns `undefined` when absent and atomically captures the current immutable
|
||||
revision in O(1) while registering for later exact committed frames. Before
|
||||
`start()`, its value remains the acquisition revision. After start, invoke one
|
||||
serialized asynchronous listener with each exact value and operation batch,
|
||||
advancing `watch.value` immediately before the callback. Preserve commit Context
|
||||
values without inheriting producer cancellation.
|
||||
|
||||
Retain at most 100 pending frames, excluding the in-flight callback. Adding frame
|
||||
101 replaces the complete undelivered suffix with one root replacement carrying
|
||||
the newest exact immutable revision and Context. Do not estimate serialized
|
||||
bytes, call `JSON.stringify()` for accounting, copy values, replay operations, or
|
||||
re-diff revisions.
|
||||
|
||||
Test updates between acquisition/return/start; exact values and operation
|
||||
references; no callback overlap; listener-initiated commits; commits during an
|
||||
in-flight callback; overflow before start and behind an active callback;
|
||||
retirement folded into an overflow reset; replayable redundant structural
|
||||
commits; delivery Context value preservation; retained earlier revision
|
||||
stability; trusted mutation footguns; retirement before start and while active;
|
||||
recreation; idempotent stop; second-start rejection; cancellation during
|
||||
acquisition; cancellation/close during a callback without aborting or joining it;
|
||||
listener-error settlement; and invocation-owned cleanup in package 14.
|
||||
|
||||
Packages 13–20 are vertical milestones. Each must leave one real public path
|
||||
working end to end; do not defer all integration to the last package.
|
||||
|
||||
Cross-cutting constraints for these milestones:
|
||||
|
||||
- Introduce each persisted built-in document kind once, with its final schema,
|
||||
history/fork policy, migration, checkpoint predicate, and public mount path.
|
||||
Record that concrete protocol in the normative specification when the kind
|
||||
lands. Do not add temporary built-in kinds or later replace a fake schema.
|
||||
- Keep all visible progress durable. Tests may use faux models and fake effects,
|
||||
but production code must use pi-ai's exported `Models` interface and the real
|
||||
task chain; do not add a Pico model adapter or production fake successor.
|
||||
- Prepare every loaded `ConversationView` revision from the complete candidate
|
||||
Session commit before Storage admission. Never rebuild a view by subscribing
|
||||
to already-committed table/document publications.
|
||||
- A milestone may leave later operations unimplemented, but it must not expose a
|
||||
temporary public facade. Extend the final §2.2 objects as later behavior lands.
|
||||
|
||||
## 13. Openable Harness
|
||||
|
||||
Implement the first usable slice of the final §2.2 surface: `Harness.open/close`
|
||||
with a registry, `root(context, { init })`, `conversation(id)`,
|
||||
`createConversation({ ownership, init })`, and generic
|
||||
inherited Session document APIs (`Harness extends Session`). Implement the
|
||||
public `Entry` definition token and `Conversation` handles rather than adding an
|
||||
intermediate capability facade. `Conversation` provides `id`, conversation-bound
|
||||
`commit()`, fork-aware cursor `entries()`, `context()`, `fork(at, { ownership,
|
||||
init })`, and every configuration getter/setter.
|
||||
|
||||
Implement the §7.1 registry core: `createRegistry()`, `tools.add`, `tools.wrap`
|
||||
composition, `batch`, keyed ordering with remembered positions, `Registration`
|
||||
`dispose`, immutable `snapshot()`, and list helpers. Hook,
|
||||
task, and system prompt registrations are stored and listed now; their dispatch
|
||||
lands in Packages 14–16.
|
||||
|
||||
Bring conversation and entry semantics up to the normative contract: explicit
|
||||
ownership, fork-aware cursor pagination, head lookup, entry edits, and model
|
||||
context derivation. Context reduction includes newest-edit wins, positional PR
|
||||
#9548 `SystemMessage` replay, tool-result ordering, missing post-fork tool
|
||||
results, excluded stop reasons, and separate raw-history/model-context results.
|
||||
|
||||
Define the final `ConversationConfig` document from §2.2. Raw document writes
|
||||
are trusted and unchecked. Root creation uses the internal reserved-ID
|
||||
bootstrap in one commit together with the default configuration and `init`; do
|
||||
not expose a temporary root-creation API.
|
||||
|
||||
Open ignores live tasks in this milestone; Package 14 adds reconciliation.
|
||||
|
||||
Acceptance: open persistent storage, obtain the root, mutate configuration and
|
||||
history through public handles, create and fork conversations, close, reopen,
|
||||
and verify stable root/conversation identity and state. Test lazy root creation
|
||||
with `init` run once; atomic create/fork with `init`; actual forks; deep ancestor
|
||||
caps; same-commit entry prefixes; cursor boundaries; self-head resolution;
|
||||
newest-edit wins; raw head-to-tail transcript versus model context; model-less
|
||||
and excluded assistant entries; replacements/omissions; multiple heads;
|
||||
positional system messages; tool-result ordering; missing post-fork results;
|
||||
every configuration getter/setter; `setActiveTools` duplicate and newly-added
|
||||
unregistered rejection with stale names allowed; default active tools from the
|
||||
registry; as-of configuration inheritance including unavailable names; registry
|
||||
add/dispose/batch rollback/nested-batch rejection/keyed order/wrap composition;
|
||||
and reopen.
|
||||
|
||||
## 14. Durable task runtime
|
||||
|
||||
Implement `defineTask`, exhaustive phase maps, full checkpoint replacement,
|
||||
kind migration at reservation, runtime commits and memos, invocation lifetime
|
||||
gates, scheduler reservation, dependencies, terminal outcomes, typed waits,
|
||||
and joins. Complete the task-facing §2.2 methods: `resume`, `getTask`,
|
||||
`waitForTask`, `abortTask`, and the task-aware portion of idle waits. Task definitions come from the registry snapshot;
|
||||
add `RegistryReader.subscribe()` so registry changes wake the scheduler.
|
||||
|
||||
Include the execution-critical abort core: durable direct-task marks,
|
||||
signal-and-join of an active run, fresh abort invocation, run-commit rejection
|
||||
after a mark, and close precedence. Deep owned-subtree cascading and background
|
||||
boundaries remain Package 18. Open now reconciles every surviving `running` task
|
||||
to `pending`. Missing, too-old, and unmigratable definitions leave the task pending and
|
||||
blocked (§5.4); registry changes wake the scheduler to reconsider them. Aborting
|
||||
a blocked task settles it as `orphaned` with the full cleanup; nothing else
|
||||
orphans a task. Implement per-phase registry snapshots, refresh at
|
||||
every normal phase boundary, and hand over when the task definition object changed
|
||||
and the new definition can reserve the task.
|
||||
|
||||
Deferred to later packages; do not stub them in Package 14:
|
||||
- Orphaning writes the terminal `orphaned` record and retires task documents.
|
||||
Unanswered input submissions and clearing run control are added in Package
|
||||
15, when submissions and run control exist.
|
||||
- `TaskRuntime.hooks` (the hook runner) is added in Package 16 with hook
|
||||
dispatch. Package 14 adds `runtime.registry` (the phase's registry snapshot)
|
||||
and `runtime.models`.
|
||||
- `TaskRuntime.conversation()` returns a `ConversationHandle`, whose `submit`
|
||||
needs Package 15 and whose `abort`/`waitForIdle` need ownership traversal; it
|
||||
is added with the invocation-bound owned APIs in Package 18.
|
||||
- Idle waits count live non-background tasks directly (Harness-wide or in the
|
||||
addressed conversation). Traversal through owned conversations and background
|
||||
boundaries replaces this in Package 18.
|
||||
|
||||
Use a fake two-phase external effect to test the real runtime. Acceptance is an
|
||||
intent/effect/outcome task interrupted after intent, closed, reopened on the same
|
||||
storage, and safely resumed to a durable terminal receipt. Also test unchanged-
|
||||
checkpoint faulting, same-phase progress, cancellation precedence, thrown
|
||||
handlers, dependencies, result values and entry IDs, first-writer-wins memos,
|
||||
terminal checkpoint/memo removal, task-document retirement, close/reopen without
|
||||
abort marks or fabricated outcomes, no fresh phase/abort dispatch while closing,
|
||||
watch cleanup, blocked tasks unblocked
|
||||
by later registration, migration failure leaving the record unchanged, handover
|
||||
after a same-name task replacement, no handover to a missing or incompatible
|
||||
definition, no overlap between predecessor and successor
|
||||
invocations, abort of a blocked task settling as `orphaned` with full cleanup,
|
||||
run commits rejected after the mark, and crashes at every direct-task abort stage.
|
||||
|
||||
Implemented design notes: every task transition is decided by one callback
|
||||
serialized on the Session line (reservation, abort marks, runtime commits, and the
|
||||
step before each phase). `Tx` has no task replacement; `runtime.commit()`
|
||||
callbacks return the typed next state. `suspend()`, `hold()`, `quiescent()`, and
|
||||
a public `markTask()` were dropped from §2.2.
|
||||
|
||||
## 15. First runnable no-tool chat turn
|
||||
|
||||
Implement the smallest real input-to-answer vertical path. Extend Package 14's
|
||||
`orphaned` settlement to mark affected input submissions unanswered and clear
|
||||
matching run control in the same commit, through the Harness hook for
|
||||
scheduler-written outcomes (§5.4); the scheduler learns nothing about turns or
|
||||
task kinds. Implement the final `pi.live` document (§8.2) with `run` and
|
||||
`generation`, the built-in entry kinds (§8.1), and the `pi.generation` task
|
||||
(§8.3). The inbox document is defined in Package 17 with its first writer. Add
|
||||
input `Submission` admission, request-ID deduplication, reacquisition and
|
||||
waiting, idle placement, active-turn ownership, successful answer settlement,
|
||||
and terminal failure cleanup. Expose the final `SubmissionDraft` union. Idle
|
||||
write submissions append their entry and settle `done`. Until Package 17, every
|
||||
submission to a busy conversation rejects with `ConversationBusy` regardless of
|
||||
`whenBusy`, writing nothing. Busy steer/follow-up behavior, queued writes, and
|
||||
reset remain Package 17.
|
||||
|
||||
Also implement the decided surface changes: typed entry tokens
|
||||
(`defineEntry<D extends JsonValue>`, token-first `tx.appendEntry()` and
|
||||
`tx.entry()` overloads); removal of `input` from `createConversation()`;
|
||||
built-in tasks pre-registered by `createRegistry()` (undisposable, not
|
||||
overridable, required by `Harness.open`); the `TaskRuntime` additions
|
||||
`snapshot`/`snapshotAsOf`, `context()`, `now()`, and `report()`; and the
|
||||
`streamOptions`/`retry` configuration fields with their getters and setters.
|
||||
|
||||
Implement the §7.4 system prompt: rendering registered sections with `tag` and
|
||||
wrappers, and no-tool request preparation, including
|
||||
exact persisted rendered strings, positional system baselines/deltas with empty
|
||||
`content`, head-cut rebaselining with `ContextEdit` omissions, order-only
|
||||
two-entry rewrites, and section failure handling. No tools are offered in this
|
||||
package: preparation emits no tool declarations and passes `PromptInput.tools`
|
||||
as `[]`. Until Package 16, a `toolUse` response settles as the answer.
|
||||
Implement the ordinary generation phases needed for one response: preparation,
|
||||
request intent, durable throttled partials, attempts/retry classification,
|
||||
deferred handle polling/cancellation, assistant entry settlement, and input-
|
||||
submission completion.
|
||||
|
||||
Call pi-ai only through its exported `Models` methods: `getModel()`,
|
||||
`streamSimple()`, `fetchDeferred()`, and `cancelDeferred()`. The test double must
|
||||
implement that same interface; production code gets no adapter. A missing configured model produces
|
||||
the durable `no_model` failure. All progress exposed to observers is committed
|
||||
state, never raw provider frames.
|
||||
|
||||
Acceptance: `Harness.open → root → resume → submit(input) → Submission.wait →
|
||||
durable assistant answer → close/reopen`, using faux Models in tests. Exercise
|
||||
interruption and reopen before/after every implemented generation phase,
|
||||
aborted-partial conversion, deferred polling/cancellation, retryable and
|
||||
terminal model errors, no-visible-undurable updates, exact section order/value
|
||||
patches, complete post-head baselines, retained system deltas on both sides of a
|
||||
head marker, sections reading conversation documents through `input.read`,
|
||||
throwing sections, preparation
|
||||
rerun after a concurrent tail/config commit, fault and orphan turn cleanup,
|
||||
typed entry tokens, and durable submission settlement. This is also the first print-mode smoke path: print
|
||||
awaits its own input submission rather than global idle.
|
||||
|
||||
## 16. First coding-agent tool turn
|
||||
|
||||
Implement hook dispatch and `TaskRuntime.hooks`, deferred from Package 14
|
||||
(Session-wide and scoped to a conversation or its owned subtree, §7.2), and wire
|
||||
the real generation → tool tasks → post-tools → generation chain (§8.3–8.5).
|
||||
Neither side uses a production fake successor.
|
||||
|
||||
- Generation: tool declarations in preparation (§7.4: additions/removals,
|
||||
same-name replacement, order rewrite, complete baseline after a head cut,
|
||||
duplicate active names offered once, wrapped and failing wrappers,
|
||||
`PromptInput.tools`); `toolExecution` pinned in `request`/`poll`; the tool
|
||||
round commit (offered check against the committed context through `cutoff`,
|
||||
unaffected by `beforeRequest`; `tool_unavailable`
|
||||
results for calls not offered, parallel or `after`-chained sequential tool
|
||||
tasks, post-tools, `pi.live.tools`, run handed to post-tools);
|
||||
`beforeRequest`, `afterResponse`, and `onYield` continuations.
|
||||
- `pi.tool` (§8.4): input `{ assistant, callId }`, resolve/validate/
|
||||
`beforeTool`/validate/intent/execute/`afterTool`/result in one `call` handler,
|
||||
recovery-only `execute` with the stored/current replay rule, bounded output and
|
||||
details in the `pi.live.tools` slot, the terminal commit as final flush,
|
||||
diagnostics (`api.diagnostic()`, result diagnostics, Harness truncation and
|
||||
error codes, the `<harness>` block, `pi.tool-result` `data: { diagnostics }`),
|
||||
and the abort handler.
|
||||
- `pi.post-tools` (§8.5): `afterTools`, `addTools`, all-results `terminate`, the
|
||||
`postTools`/`final` boundaries (no inbox placement until Package 17), and the
|
||||
next generation. The `handoff` control moves to Package 17 with `reset()`,
|
||||
which defines the headed entry both write.
|
||||
- `pi.live.tools` and the checkpoint rule (§8.2); the scheduler cleanup hook for
|
||||
`pi.post-tools` (ends the run) and `pi.tool` (marks the slot `done`).
|
||||
- Surface: `ToolExecutionApi` without `conversation()`, which the `src` type
|
||||
drops until Package 18 adds it with `ConversationHandle`; `api.env` and `HarnessOptions.env`,
|
||||
`ToolRegistration.executionMode`, the `toolExecution` configuration field with
|
||||
its getter/setter, `TaskRuntime.getTask()` and `entry()`, `HookApi`, and the
|
||||
exported `GenerationTask`, `ToolTask`, and `PostToolsTask` tokens.
|
||||
- `@earendil-works/pi-durable/tools`: copy `read`, `bash`, `edit`, and `write`
|
||||
with their helpers from `packages/agent/src/harness/tools` and adapt them to
|
||||
`ToolRegistration` (`api.env`, `api.output`, `api.details`). Image reading in
|
||||
`read` is deferred; note it where the tool rejects or skips images. The env
|
||||
shell streams raw output (`onOutput`) and spills past thresholds; `output()` is
|
||||
the only place output is bounded, sanitized, and throttled. `prepareArguments`
|
||||
restores the edit tool's argument repair.
|
||||
|
||||
Acceptance: input → model tool call → registered local read/bash/edit operation →
|
||||
tool result → model answer → durable submission settlement. Run that path once
|
||||
normally and once interrupted/reopened. Test recovery from every tool and post-
|
||||
tools phase; offered-set enforcement, including a tool deactivated after
|
||||
preparation still executing; before/after hook rules, hook scopes, and hook
|
||||
memos; `onYield` continuations; both stored/current replay-policy directions;
|
||||
default and overridden `outputLimits`; output content fallback; details
|
||||
replacement, last-details fallback, and coalesced commit settlement; the terminal
|
||||
commit as final flush; abort/close with buffered output; invocation-bound
|
||||
watches; parallel and sequential rounds, including a per-tool sequential mode;
|
||||
unregistered active tools that are removed from the offered set, re-added after
|
||||
re-registration, and produce `tool_unavailable` results when called, without
|
||||
failing the request; a tool replaced mid-call finishing under the implementation
|
||||
it resolved; hooks surviving a task reload; order-only tool changes; atomic
|
||||
assistant/tool/post-tools commits; `terminate` only when every result requests
|
||||
it; `addTools`; tool fault/orphan leaving a synthesized context result;
|
||||
diagnostic ordering, the `<harness>` block, and entry `data`; and all
|
||||
positional tool-history cases. Also test the ported tools against
|
||||
`NodeExecutionEnv`, and a wrapper supplying a different `api.env`.
|
||||
|
||||
Document in the durable README that the package root loads TypeBox through the
|
||||
tool task's argument validation (pi-ai `validateToolArguments()`): about 23 MB of
|
||||
peak RSS unbundled, about 4 MB in a tree-shaken bundle.
|
||||
|
||||
Tool output benchmark (`test/*.bench.ts`, memory, SQLite, and JSONL): drive the
|
||||
real tool task, adaptive throttle, and `pi.live` commits with low (a line every
|
||||
few seconds), normal (a compiler or test run), and high (continuous `cat` of a
|
||||
large file) output rates, head and tail retention, one tool and several parallel
|
||||
tools, over one long round. Report commits, operation bytes written, stored
|
||||
size before and after the round's base, reclamation, heap and RSS, commit
|
||||
latency, and reopen/replay time mid-round. Assert that head output commits as
|
||||
Chord appends and that a sliding tail of non-repetitive output within 64 KiB
|
||||
commits as trim plus append; report how often repetitive output falls back to a
|
||||
full window set. The checkpoint rule stays "base whenever nothing runs" (§8.2) with
|
||||
no delta-count bound; the benchmark decides whether that holds. Assign
|
||||
`slot.output` as one string field so Chord can diff it; replacing the slot
|
||||
object records a full set.
|
||||
|
||||
Throughput target: a plain `cat` of a 1 GiB file of unique lines through the
|
||||
ported bash tool and the real Harness path (env capture and spill, `api.output`,
|
||||
throttled `pi.live` commits, result entry) completes in about 0.4 s, like the
|
||||
mini coding agent. Also drive `api.output()` directly with the same 1 GiB in
|
||||
64 KiB chunks: accepting a chunk must not cost work proportional to the
|
||||
retained window.
|
||||
|
||||
## 17. Live UI and product state
|
||||
|
||||
Complete submissions and the inbox (§6): the `pi.inbox` document created by the
|
||||
built-in setup; busy `steer`/`followUp`/`reject`; queued writes; admission to an
|
||||
idle conversation with a non-empty inbox (queue, then a final boundary in the
|
||||
same commit); withdrawal removing the item; `tx.placeSubmission()`; ordered
|
||||
`postTools`/`final` selection by the new `steeringMode`/`followUpMode`
|
||||
configuration fields (with getters/setters), writes before user items; stale
|
||||
head writes; a reset at `postTools` ending the run with `reset`; successor runs;
|
||||
and `onYield` applying only when the final boundary selected no user item and no
|
||||
reset. Only an answer, `terminate`, or `handoff` applies the final boundary;
|
||||
failure, a run task's abort, fault, and orphan leave the inbox alone.
|
||||
Successful inputs still require an answer; writes settle on placement and never
|
||||
start generation by themselves.
|
||||
|
||||
Define the `pi.reset` entry (§8.1), written by `Conversation.reset(handoff)`
|
||||
through a write submission and by the post-tools `handoff` control.
|
||||
|
||||
Usage (§8.6): the `pi.usage` document created by the built-in setup, updated in
|
||||
the same commit as every assistant entry (`appendAssistant()`) and every tool
|
||||
result with `usage` (`ToolExecutionResult.usage`, `appendToolResult()`), and
|
||||
`Harness.usage()` summing every conversation's document.
|
||||
|
||||
`ConversationView`, `viewState()`, and `watch()` (§9.3): one mount per
|
||||
conversation, built lazily on the Session line, advanced from
|
||||
`subscribeCommits()` publications, and dropped with its last observer.
|
||||
|
||||
Last: the experimental `watchEvents()` adapter (§9.4) with the snapshot event,
|
||||
translated message and tool deltas, and overflow-to-snapshot.
|
||||
|
||||
Examples next to the existing ones in `test/examples/`: a print demo awaiting
|
||||
its own `Submission`, a JSON demo with two modes, `--events` (adapter
|
||||
events as JSON lines) and `--ops` (raw `ConversationView` frames), an inbox
|
||||
demo, and a late-join demo attaching a view and an event stream mid-run. An
|
||||
interactive TUI demo and coding-agent integration are later work.
|
||||
|
||||
Table-test every submission transition, cross-type request-ID conflicts,
|
||||
interleaved queue selection under both queue modes, writes placed before user
|
||||
items, stale head writes, a reset at each boundary, handoff, `onYield` with and
|
||||
without queued items, compact positional removal of large payloads, withdrawal,
|
||||
reopen waits, queued items surviving a failed run and drained by the next
|
||||
submission, and orphan/fault cleanup. Test minimal Chord deltas for `pi.inbox`
|
||||
and `pi.usage` changes. Test one view revision per touching commit; atomic
|
||||
entry/document publication; fork-aware active entries and head cuts; mounted
|
||||
create/retire; empty-batch suppression; stable paths; structural sharing; watch
|
||||
overflow; and dropping the mount. Test the event translation of every
|
||||
`MessageChange` and output change, the snapshot at attachment and after
|
||||
overflow, and absence of raw provider frames.
|
||||
|
||||
## 18. Ownership and subagents
|
||||
|
||||
Complete the remaining owned-conversation and abort semantics: durable foreground
|
||||
subtree cascades, signal/join/fresh-abort across ownership edges, background
|
||||
boundaries, ordinary and full traversal, conversation abort, and exact idle
|
||||
waits. Finish invocation-bound owned APIs used by tools and the foreground and
|
||||
background subagent provisioning patterns, including atomic task/conversation/
|
||||
registry creation and request-ID-safe submission recovery.
|
||||
|
||||
An abort cascade withdraws queued inputs in every owned conversation it reaches
|
||||
(spec §5.4). There is no full-teardown primitive: `close()` stops everything
|
||||
without outcomes, and a host cancels everything by aborting what `inspect()`
|
||||
lists. No built-in subagent tool or supervisor task: two concise, product-style
|
||||
examples, `test/examples/22-subagent-foreground.ts` (child owned by the tool task,
|
||||
reported through `api.details({ conversationId })`, the UI attaching to the
|
||||
child's events) and `test/examples/23-subagent-background.ts` (reworked in
|
||||
Package 19: persistent subagents behind one `subagent` tool with spawn, send,
|
||||
wait, stop, and status actions; a background anchor task owns each child, and a
|
||||
background reporter task per message delivers it and posts the answer back as a
|
||||
follow-up input, request-ID-safe across restarts), show the patterns.
|
||||
|
||||
Test deep ownership trees, owner edges after terminal settlement, nested
|
||||
background boundaries, conversation abort/join with surviving passive writes and
|
||||
background tasks, cancellation of waiters without cancellation of work, and
|
||||
atomic cancellation intent. Test default non-inheritance, inheritance from the
|
||||
current committed tail, empty source conversations, document fork policies,
|
||||
configuration overrides through `init`, foreground subagent cascade, and
|
||||
background supervisor recovery before and after submission admission.
|
||||
|
||||
## 19. Structured concurrency
|
||||
|
||||
Implement spec §5.5 and its consequences, replacing `after` and `pi.post-tools`.
|
||||
No backward compatibility.
|
||||
|
||||
- Ownership: `TaskOptions.ownership` is required (`{ kind: "conversation" }` or
|
||||
`{ kind: "task", taskId }`); `TaskRecord.owner?: TaskId`; child tasks live in
|
||||
their owner's conversation; `background` only for conversation-owned tasks;
|
||||
new task- or conversation-ownership requires a live owner (not `completing`
|
||||
or `terminal`). Update every `createTask` call site, including tools'
|
||||
`api.createTask`, examples, and tests. Storage: persist `owner` and the new
|
||||
statuses (memory, JSONL, SQLite schema edited in place; no migrations while WIP), conformance cases. No owner
|
||||
index: the scheduler derives edges from the live tasks it loads.
|
||||
- States: real `waiting { checkpoint, on, policy }` and `completing { outcome }`
|
||||
statuses in `TaskState` and every backend and status scan; `NextTaskState`
|
||||
gains `waiting`; `runtime.outcomes()`; `inspect()` reports `completing`.
|
||||
Remove `after` (records, options, scheduler dependency handling).
|
||||
- Scheduler: a waiting task is never reserved until every task in `on` is
|
||||
terminal (then reserved directly from `waiting` to `running` at its
|
||||
checkpoint), unless an abort mark lets it reach its abort handler; `on` over
|
||||
tasks it does not own requires `allSettled`; `on` rejects missing tasks, the
|
||||
task itself, and its owner chain; failFast marks live owned siblings in `on` on a held or terminal
|
||||
non-`completed` outcome, in reconcile. Terminal commits (task- and
|
||||
scheduler-written) with live ordinary owned work become `completing`; a later
|
||||
scheduler commit finalizes them when that work is gone, re-evaluated on every
|
||||
commit and at open; scheduler-written holds defer their Harness cleanup
|
||||
(`settleSchedulerOutcome`) to the final commit, task-written holds land their
|
||||
other writes at hold; waiters and task-document retirement at the final
|
||||
commit. Abort invocations start only when ordinary owned work is gone
|
||||
(bottom-up, judged on committed records). Orphaning happens only where the
|
||||
abort invocation would start, or directly in `abortTask()` when nothing owned
|
||||
is live, so orphans never hold. Cascade (spec §5.4): only live owners cascade (abort mark or held
|
||||
non-`completed` outcome); terminal owners never do; ordinary traversal and
|
||||
idle follow task→task edges too. `Conversation.abort(context, { background:
|
||||
true })` snapshot semantics.
|
||||
- Built-ins: generation owns its tool tasks and waits `allSettled` in a new
|
||||
`tools` phase that runs the old post-tools body (§8.5); sequential rounds
|
||||
create one tool at a time from `pending`; the next generation is
|
||||
conversation-owned; the generation abort handler appends `aborted` results for
|
||||
unstarted calls. Remove `pi.post-tools`, `PostToolsTask`, `PostToolsHooks`
|
||||
(`afterTools` moves to `GenerationHooks`). Tool tasks that own conversations
|
||||
hold `completing` after their result entry. Events: `turn_end` when a
|
||||
generation's outcome is committed (hold or terminal, whichever first).
|
||||
- Docs: spec §5.5 and §12 footguns are written; update README (task ownership,
|
||||
waiting, `completing`, abort order), CHANGELOG, and examples 12/22/23 plus a new
|
||||
`test/examples/24-child-tasks.ts` (checkout with four payments: failFast,
|
||||
`abortTask(checkout)`, crash and reopen, each printing outcomes).
|
||||
|
||||
Exhaustive tests (new `test/harness-structured.test.ts` plus updates): ownership
|
||||
validation (missing ownership, cross-conversation child, background child,
|
||||
owner completing/terminal); waiting with allSettled and failFast (failed,
|
||||
aborted, faulted, orphaned children; held failures trigger failFast before the
|
||||
failing child drains); `on` with already-terminal and non-owned tasks
|
||||
(`allSettled` only); waiting on subsets in sequence; spawning without waiting;
|
||||
`completing` for task- and scheduler-written outcomes, including work created
|
||||
during the hold, finalization at open, `abortTask()` on a completing task, held
|
||||
failure aborting below, waiters and document retirement only at the final
|
||||
commit; bottom-up abort order across three levels (child terminal records before
|
||||
the parent's abort handler starts), including an aborted waiting parent; `on`
|
||||
validation (missing, self, owner chain, empty `on`); a waiting task whose
|
||||
definition is missing when it may resume (blocked, then orphaned when aborted);
|
||||
creating owned work in a finishing commit (task ownership rejects, a task in an
|
||||
owned conversation holds); cascades through
|
||||
task→task and task→conversation→task edges; background boundaries and the
|
||||
`{ background: true }` abort; a terminal owner never cascading (interrogating a
|
||||
finished or Esc'd subagent runs normally); crash/reopen at every point (before
|
||||
and after create+wait, after a child fails before siblings are marked, after all
|
||||
children terminal before the parent resumes, during holds and abort handlers);
|
||||
generation tool rounds parallel and sequential with Esc before, during, and
|
||||
after tools, unstarted sequential calls, faulted tool slots under generation
|
||||
abort, faulted generation keeping run control until its tools drain, `turn_end`
|
||||
before the successor's `turn_start` when a generation holds, and a tool holding
|
||||
`completing` while its subagent's extension work runs. Rerun every existing suite; migrate the post-tools,
|
||||
sequential-round, and dependency tests to the new shapes without weakening them.
|
||||
|
||||
## 20. Compaction and overflow
|
||||
|
||||
Implement spec §8.7 and its consequences. Compaction never appends to a busy
|
||||
conversation concurrently (§7.4 preparation does not recheck the transcript): a
|
||||
blocking compaction appends while its generation holds the run and waits for it;
|
||||
every other compaction places its summary through a write submission.
|
||||
|
||||
- Task: `pi.compaction` (`CompactionTask`, `CompactionHooks`, `CompactionInput`,
|
||||
`CompactionCheckpoint`, `CompactionResult`, `CompactionReason`) with phases
|
||||
`select`, `summarize`, `retry`, the abort handler, and the `beforeCompact`
|
||||
hook. Register it with the built-ins. `CompactionEntry` token for
|
||||
`pi.compaction` with `data: { reason }`.
|
||||
- `ContextView.contributions`: each active entry's contribution after all
|
||||
in-range edits (including edits on older in-range markers). Range selection is one pure function over those
|
||||
contributions and `keepRecentTokens`, shared by the generation's checks and
|
||||
`select`; the summarized messages are the prefix contributions ordered by
|
||||
§2.1 rules 7–8. The generation's size estimate (§8.3) uses the newest
|
||||
assistant appended after the head marker, not pi-ai's timestamp heuristic. Port the coding agent's serializer, summarization
|
||||
system prompt, and structured prompt (one prompt, with a line about carrying an
|
||||
earlier summary forward) into `src/harness/compaction.ts`; do not import from
|
||||
coding-agent.
|
||||
- Placement: direct append for a task-owned (blocking) compaction; for a
|
||||
conversation-owned one, factor the admission body of `Submissions.submit()` into
|
||||
a function over a `Tx` so the task's classifying commit admits the write
|
||||
(request ID `compaction:<taskId>`) with the same idle/queued/stale/final-boundary
|
||||
behavior. No new stale logic.
|
||||
- Config: `CompactionPolicy`, `DEFAULT_COMPACTION_POLICY`,
|
||||
`ConversationConfigState.compaction`, `getCompaction`/`setCompaction`.
|
||||
- Live: `pi.live.compactions` (`CompactionStatus[]`, task ID order, key removed
|
||||
when empty), added in the creating commit and removed in the outcome commit
|
||||
(also at a `completing` hold);
|
||||
`settleSchedulerOutcome` removes it for faulted/orphaned compactions.
|
||||
- Generation: `compacted`/`overflow` checkpoint fields; threshold checks in
|
||||
`prepare` (blocking child + wait; background conversation-owned task in the
|
||||
commit moving to `request`); overflow classification before the retry branch;
|
||||
failing an overflow generation whose compaction produced no entry.
|
||||
- `Conversation.compact()` admits the manual task (conversation-owned, not
|
||||
background) with its status in one commit and returns its ID.
|
||||
- Usage: every classified summarization response adds its usage to
|
||||
`pi.usage.models`.
|
||||
- Events: `compaction_start`/`compaction_end`, snapshot `compactions`, batch
|
||||
order per §9.4.
|
||||
- Docs: README (compaction, policy, events), CHANGELOG, and a new product-style
|
||||
example `test/examples/25-compaction.ts` (faux model: a long chat that crosses
|
||||
the background threshold, the summary lands at a boundary, a manual `compact()`
|
||||
while busy, and an overflow recovered by a blocking compaction, printing the
|
||||
model context before and after).
|
||||
|
||||
Not in v1 (note only): a cache-friendly summary request that resends the
|
||||
agent's exact last request plus one summarization user message with pi-ai
|
||||
`toolChoice: "none"`, so the prefix including tool declarations is a cache hit.
|
||||
It fits background compaction well below the window; overflow and a context
|
||||
without room for the prompt and summary keep the serialized request.
|
||||
|
||||
Exhaustive tests (new `test/harness-compaction.test.ts`, faux provider, plus
|
||||
updates):
|
||||
|
||||
- Range selection (pure): cut at a user entry, at an assistant entry mid-run
|
||||
(one prompt followed by many tool rounds), never at a tool result or system
|
||||
entry; a huge last tool result keeps its assistant; budget never reached and
|
||||
only-the-marker prefix are nothing to compact; an existing marker is
|
||||
summarized first and older in-range markers drop out; omitted and replaced
|
||||
entries use their edited contribution; excluded error/aborted assistants are
|
||||
not candidates; the §8.7 worked example.
|
||||
- Summarization request: exact messages (system prompt, serialized
|
||||
conversation, instructions line), tool-result truncation, no tools,
|
||||
`cacheRetention: "none"`, no `deferred`, `maxTokens` from `reserveTokens` and
|
||||
the model's max, pinned model/thinking/stream options surviving a config change
|
||||
mid-summary.
|
||||
- Summary entry: kind, wrapped text, `head` = first kept, `data.reason`; model
|
||||
context after placement is `[summary, kept...]` followed by a full system
|
||||
baseline on the next preparation; raw history unchanged and still scannable via
|
||||
`entries()`; `pi.usage.models` includes every attempt, including failed,
|
||||
retried, and stale ones; declines and hook-supplied summaries add none.
|
||||
- Outcomes: nothing to compact (`{}`), `beforeCompact` decline (`{}`) and
|
||||
supplied summary (no model call), first decision wins, hook throw reported and
|
||||
ignored; `no_model`; retryable error then success; retries exhausted;
|
||||
non-retryable error; `length` stop; tool call in the response; empty text.
|
||||
- Manual: idle with empty inbox (placed at once, `{ submissionId }` whose submission is `done`);
|
||||
idle with queued follow-ups after a failed run (final boundary places the
|
||||
summary first, then starts the follow-up in the compacted context); busy
|
||||
(queued, placed at the next `postTools` and the run continues in the compacted
|
||||
context; placed at `final`); stale because a reset landed while summarizing;
|
||||
stale because a later compaction cut further; an older summary with a later cut
|
||||
placed after a newer one; two queued summaries in one boundary with the
|
||||
second cutting before, at, and after the first (only "before" is stale); `Conversation.abort()` aborts a manual compaction and keeps an
|
||||
already queued summary; `Submission.abort()` withdraws a queued summary;
|
||||
`waitForIdle` waits for a manual compaction; `compact()` returns the ID
|
||||
`waitForTask` settles with.
|
||||
- Background threshold: starts only above the background threshold, only with
|
||||
`backgroundTokens > 0`, only when no status is listed, not when disabled, not
|
||||
without a cut, and not after a blocking compaction in the same generation; the
|
||||
generation does not wait; summary placed at the next boundary; survives
|
||||
`Conversation.abort()`; `abort({ background: true })` and `abortTask()` stop
|
||||
it; idle waits ignore it.
|
||||
- Blocking threshold: generation waits, nothing appended before the wait, then
|
||||
re-prepares with a baseline and sends; request proceeds after nothing to
|
||||
compact, decline, failure, fault, and direct `abortTask()` of the child; no
|
||||
second compaction when still above the threshold; a background compaction in
|
||||
flight ends stale; Esc during the blocking compaction aborts child then
|
||||
generation (child terminal before the generation's abort handler) and settles
|
||||
inputs `aborted`.
|
||||
- Overflow: error entry appended and excluded from the retry request, attempt
|
||||
unchanged, compaction then retry succeeds; second overflow fails
|
||||
`model_error` with its error entry; overflow with compaction disabled, without
|
||||
a cut, or after a threshold compaction fails; overflow whose compaction
|
||||
declines/fails/aborts fails `model_error` with the overflow text; an overflow
|
||||
error matching retryable patterns is still not retried; silent `stop`/`length`
|
||||
overflow is an ordinary answer.
|
||||
- Live status and events: status added in the creating commit with `blocking`,
|
||||
`attempt`, and `retry` during backoff, removed with every outcome
|
||||
(completed, failed, aborted, faulted, orphaned); several concurrent statuses;
|
||||
`compaction_start`/`compaction_end` in batch order; snapshot `compactions` for a late joiner during summarize and retry;
|
||||
view mount after a compaction head (kept entries stay mounted).
|
||||
- Recovery: crash and reopen in `select` (hook reruns), `summarize` (request
|
||||
resent once, no duplicate usage), `retry`, after placement, with the
|
||||
generation waiting on a blocking compaction, with a summary queued, and with a
|
||||
blocked compaction definition (orphaned on abort, status removed).
|
||||
- Estimate and cut edge cases: a background summary queued mid-run and placed
|
||||
at `postTools` makes the successor's estimate ignore the pre-compaction usage,
|
||||
so no blocking compaction starts (also with a fixed Harness clock); an
|
||||
assistant finishing while a summary is queued; raw order `assistant(call),
|
||||
user, toolResult, assistant` never cuts at that user; edits carried by an
|
||||
older in-range marker and a kept entry replacing a summarized one are reflected
|
||||
in the summarized messages and the hook's `messages`; kept `pi.system` deltas
|
||||
get omit edits in the next single baseline; compaction in a fork whose cut
|
||||
falls on a parent entry (head, stale checks, view mount).
|
||||
- Interactions: a summary queued before a reset in the same boundary (both
|
||||
placed, reset wins) and after it (stale); a manual compaction's status blocks
|
||||
a background start; a queued summary surviving a failed run is placed by the
|
||||
next submission's final boundary before the new input; the summarization
|
||||
request itself overflowing fails `model_error` and a waiting overflow
|
||||
generation fails with the original overflow text; a retryable error after an
|
||||
overflow compaction has the full retry budget; a background summary landing
|
||||
during a retry backoff stays queued while a blocking compaction wins; an
|
||||
application edit placed while summarizing is lost as §12 documents; a hook
|
||||
that creates owned work and supplies a summary (placement and status removal
|
||||
at hold, receipt after the child); `compact()` right after open enables
|
||||
scheduling; a rejected classifying commit leaves no usage, submission,
|
||||
summary, or outcome; idle admission that drains an older queued summary
|
||||
together with the current one.
|
||||
- Rerun every existing suite; generation, inbox, events, and view tests keep
|
||||
passing unchanged except for the new optional fields.
|
||||
|
||||
## 21. Extensions and per-conversation agents
|
||||
|
||||
Implement the design agreed with Mario, now specified in `spec.md` (§2.2,
|
||||
§5.1/§5.4 runtime surface, §6 queue modes, §7.1–§7.5, §8 where tasks read the
|
||||
agent and settings, §9.3/§9.4 view mount and events, §12 footguns): extensions
|
||||
installed by name replace keys, positions, `batch()`, wrappers-by-key, hook
|
||||
scopes, and public conversation setups; the rewindable `pi.agent` document
|
||||
(model, thinking level, extension and tool selection, `instructions`, `cwd`)
|
||||
replaces `pi.conversation.config` and its getters and setters; run policies
|
||||
become Harness-wide live `HarnessSettings`; the environment is built per call
|
||||
by one host `env` function. The `Tool` generic stays. `CodingTools` bundles
|
||||
read, write, edit, and bash; grep, find, and ls are not ported.
|
||||
|
||||
Implement in parts, each with tests: registry and extensions (install/replace,
|
||||
uninstall, catalogue views, task-name collisions, wrapper failures); `pi.agent`,
|
||||
`configure()`, and resolution (selection arrays and `add`/`remove`, same-name
|
||||
replacement, wrappers, uninstalled names, `addTools`); settings resolution and
|
||||
every reader (generation request options and retry, compaction thresholds, tool
|
||||
execution, queue modes on the line); `env` for tools, `prepare`, and
|
||||
`runtime.env()`; hooks resolved from the conversation's selected extensions per
|
||||
phase; reload of an extension while work runs (running phases keep their
|
||||
snapshot, pinned tools finish under old code, task definitions hand over at the
|
||||
next phase boundary as in §5.4); view and events. Rewrite the examples,
|
||||
including 22 and 23, to the new surface.
|
||||
|
||||
## 22. Lifecycle and final conformance
|
||||
|
||||
Package 21 of the previous plan was audited (three reviewers); these findings
|
||||
remain and apply to the new surface.
|
||||
|
||||
Bugs:
|
||||
- `runtime.now()` and `runtime.report()` stay usable after the invocation ends
|
||||
(`scheduler.ts` runtime); §5.4 says every runtime operation rejects.
|
||||
- A commit that only migrates a document through a newer token (version base, no
|
||||
ops) is not published (`transaction.ts` `publishes()`), so observers of the
|
||||
older shape keep a stale value until the next edit.
|
||||
- A failed `Harness.open` closes with the caller's possibly cancelled context,
|
||||
masking the original error; close without the caller's signal and rethrow.
|
||||
- `ConversationWatch` alias missing; stale TODO at `types.ts` about
|
||||
`subscribeCommits`; `resume()` doc comment omits progress calls.
|
||||
|
||||
Spec wording to settle with Mario first: close stops state and watch delivery at
|
||||
seal (§2.2 says states close after the join); cancelling `close()` cancels only
|
||||
that wait and a second `close()` awaits the same shutdown; add
|
||||
`Conversation.abort()` to the progress calls; "old and new Harness generations"
|
||||
means that after `await close()` no invocation code of the old Harness runs;
|
||||
"normative usage sequences" are the spec's example code blocks, copied into one
|
||||
compile-checked test.
|
||||
|
||||
Tests to add or strengthen: a committer cancelled while its commit is in
|
||||
Storage still settles durably; a cancelled `close()` still completes shutdown;
|
||||
a task handler, tool `execute`, and hook that ignore the signal keep `close()`
|
||||
pending and Storage open until they return (the current tests pass with
|
||||
`join()` deleted); a state receives no frame from a commit settling during
|
||||
close; watch acquisition cancelled mid-line leaves no subscription; conversation
|
||||
handle methods and `inspect()` (`scheduling: "closing"`) around close; each
|
||||
progress call enables scheduling on a paused Harness and each read-only viewer
|
||||
does not; registry change between open and resume; handover keeps memos;
|
||||
compile tests for every §3.3/§9.1 overload (including `snapshotAsOf` rejecting a
|
||||
non-rewindable token, token overloads of `tx.entry`/`appendEntry`), the
|
||||
`SubmissionDraft` never-fields, `waitForTask<R>`, `compact()` result type, and
|
||||
an erased extension/task with narrowed input, several phases, and custom hooks.
|
||||
|
||||
Already covered, rerun only: root-replacement delta versus checkpoint, blocked
|
||||
tasks surviving open, root identity across reopen, handover cases, a reopened
|
||||
interrupted turn. Run all package tests and the repository check, finish with a
|
||||
local coding-agent turn and a reopened interrupted turn through the public
|
||||
Harness, then stop for final review.
|
||||
|
||||
Done. The spec wording is settled as proposed (§2.2 close and progress calls,
|
||||
§5.1, §7.5, §9.1, §12). Close ends states and watches at the seal; a new Harness
|
||||
may open the Storage once the old `close()` resolved. The lifecycle tests are in
|
||||
`harness-lifecycle.test.ts`, and the spec's usage examples compile in `spec-usage.test.ts`.
|
||||
|
||||
## 23. Task graph view
|
||||
|
||||
A live, observable view of the Session's task graph for UIs and debugging, like
|
||||
`ConversationView` for a conversation: every live task with its owner edge
|
||||
(conversation or task), status (`running`, `waiting on …`, `completing`),
|
||||
background flag, and owned conversations, published after each commit as a
|
||||
replicated state or watch. `inspect()` already provides a one-off snapshot of
|
||||
the live task records. Specify and build it after the final conformance package.
|
||||
|
||||
Specified in `spec.md` §9.5: `Harness.taskGraph()` and `watchTaskGraph()` mount
|
||||
every live task keyed by ID with its committed status (phase, wait, held outcome
|
||||
status), owner edge, flags, and owned conversations. Derived states (`blocked`,
|
||||
`ready`, the live part of `on`) stay in `inspect()`. Implemented in
|
||||
`src/harness/task-graph.ts`, tested in `harness-task-graph.test.ts` and the
|
||||
lifecycle tests, shown in example 24.
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
# Pico5 specification
|
||||
# Durable specification
|
||||
|
||||
Pico5 is a durable, extensible agent harness. This document is normative.
|
||||
Pico5 uses existing package types as follows:
|
||||
`@earendil-works/pi-durable` is a durable, extensible agent harness. This document is normative.
|
||||
It uses existing package types as follows:
|
||||
|
||||
```ts
|
||||
import type {
|
||||
|
|
@ -34,7 +34,7 @@ type JsonObject = { [key: string]: JsonValue };
|
|||
type TaskOutcomeError = { message: string; detail?: JsonValue };
|
||||
```
|
||||
|
||||
Pico5 targets the transcript `SystemMessage` contract from pi-ai PR
|
||||
Durable targets the transcript `SystemMessage` contract from pi-ai PR
|
||||
[#9548](https://github.com/earendil-works/pi/pull/9548). `Message` includes that
|
||||
type once the PR lands.
|
||||
|
||||
|
|
@ -137,7 +137,7 @@ string adds or replaces a section, while `null` removes it. `toolsRemoved` is
|
|||
applied before `toolsAdded` within one message. Replaying every system message
|
||||
in transcript order yields the effective prompt and tool set.
|
||||
|
||||
Pico always writes `content: ""`. Every prompt part, including the preamble, is
|
||||
Durable always writes `content: ""`. Every prompt part, including the preamble, is
|
||||
a named section resolved from the conversation's agent (section 7.4).
|
||||
|
||||
```ts
|
||||
|
|
@ -295,9 +295,7 @@ separate consumers. Older stored history is available through the owning
|
|||
|
||||
### 2.2 Public Harness surface
|
||||
|
||||
This is the v1 host-facing API. Pico5 is not implemented yet, but implementations
|
||||
must expose this shape rather than inventing a different facade during package
|
||||
24.
|
||||
This is the v1 host-facing API.
|
||||
|
||||
```ts
|
||||
type ModelRef = {
|
||||
|
|
@ -628,10 +626,9 @@ declare const Harness: {
|
|||
};
|
||||
```
|
||||
|
||||
This intentionally retains the useful Pico3 host shape. It removes Pico3's
|
||||
fixed `rewindable()`/`sticky()` accessors, namespace router, semantic view events,
|
||||
and manual Chord view bridge. Typed Pico5 documents and the structural
|
||||
conversation watch replace those surfaces. `submit()` durably admits either a
|
||||
Typed documents and the structural conversation watch are the state surfaces;
|
||||
there are no fixed rewindable or sticky accessors, namespace router, semantic view
|
||||
events, or manual Chord view bridge. `submit()` durably admits either a
|
||||
user input or passive entry write and returns one `Submission` that tracks its
|
||||
settlement.
|
||||
|
||||
|
|
@ -1326,7 +1323,7 @@ alive at the target commit.
|
|||
|
||||
### 3.4 Mutation ownership
|
||||
|
||||
Pico uses Chord Delta's Astra-immutable transaction shape directly:
|
||||
Durable uses Chord Delta's immutable transaction shape directly:
|
||||
|
||||
```ts
|
||||
interface Prepared<T extends object> {
|
||||
|
|
@ -1358,7 +1355,7 @@ operations, and computes `value` with the optimized immutable applier before
|
|||
Storage admission. Unchanged subtrees are structurally shared with `base`.
|
||||
Operation placement payloads may also be shared with `value`; mutating either a
|
||||
prepared operation or an immutable revision is unsupported. Astra preserves an
|
||||
empty operation batch for changes it proves are no-ops, but Pico performs no
|
||||
empty operation batch for changes it proves are no-ops, but Durable performs no
|
||||
additional whole-value equality pass for nonempty structural batches.
|
||||
|
||||
`abort()` is idempotent. `adopt()` accepts only a prepared result from that
|
||||
|
|
@ -1917,9 +1914,8 @@ commits and joins invocations before closing storage. Already-running watch call
|
|||
caller-owned. Later runtime commits reject, and close writes no task outcome. Closing
|
||||
starts no fresh phase or abort invocation. It does not set abort marks,
|
||||
terminalize tasks, retire task documents, or publish document retirement. The
|
||||
hosting layer withdraws services and
|
||||
detaches clients before it closes the Harness, for example by closing its Chord
|
||||
host; reconnecting to a reopened Session
|
||||
hosting layer stops exposing Harness state to clients before it closes the
|
||||
Harness, for example by closing the Chord host whose objects serve it; reconnecting to a reopened Session
|
||||
hydrates the last committed state and resumes recovery from its durable
|
||||
checkpoints.
|
||||
|
||||
|
|
@ -3416,9 +3412,9 @@ is not a lock against `bash` or other processes.
|
|||
|
||||
### 7.4 System prompt and dynamic tools
|
||||
|
||||
Pico has no durable prompt sections. The conversation's resolved agent produces
|
||||
Durable has no durable prompt sections. The conversation's resolved agent produces
|
||||
the desired sections for each request; the transcript's `pi.system` entries are
|
||||
the only durable record of what the model saw. Pico stores prompt and tool
|
||||
the only durable record of what the model saw. Durable stores prompt and tool
|
||||
changes directly as PR #9548 `SystemMessage` values at their transcript
|
||||
positions, always with empty `content`:
|
||||
|
||||
|
|
@ -3541,7 +3537,7 @@ rebaseline adds every desired tool.
|
|||
The rendered strings stored in historical `SystemMessage.sections` remain
|
||||
authoritative even if the current renderer changes. Pi-ai decides whether to
|
||||
send the messages positionally to a capable provider or fold them into one
|
||||
leading system message; Pico does not rewrite its stored transcript for provider
|
||||
leading system message; Durable does not rewrite its stored transcript for provider
|
||||
compatibility.
|
||||
|
||||
### 7.5 Extension reload
|
||||
|
|
@ -3711,7 +3707,7 @@ The initial implementation provides:
|
|||
| `pi.tool` | validate, hook, execute, persist output and details, append result |
|
||||
| `pi.compaction` | select a transcript range, summarize, place a headed summary |
|
||||
|
||||
Generation uses `HarnessOptions.models` without a Pico-specific model adapter. It
|
||||
Generation uses `HarnessOptions.models` without a Durable-specific model adapter. It
|
||||
resolves `models.getModel(ref.provider, ref.modelId)`, builds a pi-ai `Context`
|
||||
from the prepared prompt/messages/tools, and calls `models.streamSimple()` with
|
||||
the task invocation's abort signal, the agent's thinking level, and the
|
||||
|
|
@ -4439,11 +4435,11 @@ interface DocumentObserver {
|
|||
}
|
||||
```
|
||||
|
||||
`documentState()` attaches Pico's committed document stream to Chord before
|
||||
`documentState()` attaches Durable's committed document stream to Chord before
|
||||
returning. The returned state is already hydrated with the exact shareable
|
||||
immutable revision, has no mutation methods, and publishes every later exact
|
||||
committed revision and operation batch without another tracker, value copy, or
|
||||
re-diff. Disposing it unregisters only that observation; Pico remains the sole
|
||||
re-diff. Disposing it unregisters only that observation; Durable remains the sole
|
||||
document mutator.
|
||||
|
||||
State and watch acquisition never create; absent lookup returns `undefined`.
|
||||
|
|
@ -4456,7 +4452,7 @@ Session close ends every state at its seal: the state keeps its last value and
|
|||
receives no frame from a commit that settles during close.
|
||||
|
||||
Each document state assigns its own in-memory contiguous Chord delivery sequence.
|
||||
Pico does not persist or expose that sequence through `WatchHandle`. States and watches retain immutable revisions independently of the loaded tracker
|
||||
Durable does not persist or expose that sequence through `WatchHandle`. States and watches retain immutable revisions independently of the loaded tracker
|
||||
cache. Reopen creates a new state lifetime and hydration.
|
||||
|
||||
Internally, state attachment uses Chord's synchronous atomic snapshot-and-register
|
||||
|
|
@ -4541,7 +4537,7 @@ type ConversationView = {
|
|||
```
|
||||
|
||||
These kinds and their fields are public protocol. Third-party documents are
|
||||
exposed through their own Chord services, not mounted.
|
||||
exposed through their own Chord objects, not mounted.
|
||||
|
||||
The Harness keeps at most one mount per conversation. The first `viewState()`,
|
||||
`watch()`, or event attachment builds its revision on the Session line from the
|
||||
|
|
@ -4666,7 +4662,7 @@ interface AgentEventStream {
|
|||
```
|
||||
|
||||
Every event is relative to the `snapshot` and the events before it, never to
|
||||
Pico's document layout. `snapshot` replaces everything a consumer holds; every
|
||||
Durable's document layout. `snapshot` replaces everything a consumer holds; every
|
||||
other event applies on top of it. Attachment captures the snapshot and
|
||||
registers for later publications atomically on the Session line.
|
||||
|
||||
|
|
@ -5311,8 +5307,8 @@ These are contracts, not invitations to add defensive machinery:
|
|||
- **Non-cooperative code at close:** close joins every invocation. A task
|
||||
handler, tool, or hook that ignores its signal keeps `close()` pending and
|
||||
Storage open until it returns; cancelling the close wait does not end it.
|
||||
- **Services outliving the Harness:** withdraw Chord services and detach clients
|
||||
before closing the Harness. A state ended by close keeps its last value and
|
||||
- **Objects outliving the Harness:** close the Chord objects and links that expose
|
||||
Harness state before closing the Harness. A state ended by close keeps its last value and
|
||||
never updates again.
|
||||
- **Fatal storage errors:** after a storage error the Session has failed and
|
||||
closed itself; reopen it (section 10.1). Retrying the failed call cannot
|
||||
|
|
@ -5323,7 +5319,7 @@ These are contracts, not invitations to add defensive machinery:
|
|||
|
||||
## 13. Non-goals
|
||||
|
||||
Pico5 initially has no:
|
||||
Durable has no:
|
||||
|
||||
- whole-Session DOM;
|
||||
- visible-undurable publication;
|
||||
|
|
@ -5334,5 +5330,5 @@ Pico5 initially has no:
|
|||
- CRDT/offline multi-writer merge;
|
||||
- SQL translation of arbitrary Chord operations;
|
||||
- JSONL global compaction or automatic corruption repair;
|
||||
- compatibility layer for removed Pico prototypes;
|
||||
- compatibility layer for earlier harness prototypes;
|
||||
- forced termination of non-cooperative extension code inside one process.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue