docs(agent): add pico design drafts

packages/agent/docs/pico/pico-work.md

@@ -0,0 +1,240 @@
+# pico implementation plan
+
+Clean room. Nothing from pico2 is copied; its tests are read for the bugs they caught (a failed
+generation that never ended its run, refs inside task patches, a collapse deadlocking appends,
+quiescence firing during a retry wait, watch dropping late deltas) so that packages 6, 9, 12 and
+16 each carry a case for them.
+
+Bottom up. Each package is a module with its own tests and no forward references; the suite is
+green after every one. The reference for every detail is `pico-v3.md`; `pico-usage-guide.md`
+shows the surface each package has to end up with.
+
+## 1. Types and ids
+
+`Id`, `Entry`, `Task`, `TaskRole`, `Conversation`, `Address`/`Value`/`List`/`Scope`,
+`Write`/`CommitBatch`, `Page`/`Cursor` and the query shapes (§2, §4.1, §5.1, §7.2–7.3). No code.
+
+Test: it compiles; a fixture file with one instance of every record shape.
+
+## 2. Memory storage
+
+`Storage` and `MemoryStorage`: `commit` numbering from `lastSeq`, point reads, `scanEntries` /
+`scanTasks` / `scanConversations` with cursors, `newestHead`, value versions and list elements read
+at a position, `ownedFrom`, the set of entry kinds written.
+
+Tests: every row of the §7.2 query table against hand-built batches; ids are `lastSeq + 1 + i`; a
+live-task scan decodes no terminal rows; `remove` and `clear` hide by position.
+
+## 3. The line and `Tx`
+
+`commit(plan)` on a serialized line: buffered writes, ids final at call time, state-before-entries
+throws, a throwing plan discards everything, publish after persist, `kick` when a batch touched a
+task. `value` / `list` / `entry` / `task` / `patch` / `settle` build the batch of §7.3.
+
+Tests: concurrent commits serialize; a rejected commit consumes no ids; each builder verb produces
+the expected write; reads inside a plan see committed state only.
+
+## 4. Entry kinds and context
+
+`EntryKind`, the registry, the built-in kinds (`user`, `assistant`, `tool_result`, `system`,
+`notice`, `summary`, `handoff`, `reset`) with `project` and `head`; context = newest head + range;
+`compose`; projection to pi-ai messages with tool results ordered by call index; the `system`
+kind's epoch fold (newest baseline plus the deltas after it, older deltas dropped) and its
+projection into the baseline slot / `SystemMessage`s.
+
+Tests: summary keeps the tail; handoff and reset; repeated compaction subsumes; a head number below
+the previous head's id is rejected; the fold on a context that kept an old delta; tool-result order.
+
+## 5. Forks and historical reads
+
+`createConversation` with `parent`, the shared prefix in `scanEntries`, capped-source lookup for
+values and lists, refusal of a fork inside an unresolved exchange.
+
+Tests: a fork sees the head and the values in force at its entry; heads the source adds later are
+invisible; deep fork chains; state committed after the entry (a model change) is not in the fork.
+
+## 6. Task kinds and the driver
+
+`TaskKind`, the registry, `TaskContext` (`commit`, `scratch`, `sleep`, `signal`, `config`,
+`hooks`), the driver of §6.1 (`owned`, `Wake`, attached scopes, `drive` as a waiter, poison guards,
+abort join and re-read), scopes of §6.2. Test kinds only: a counter, a blocker, a kind that
+returns without changing status, a kind that ignores its signal.
+
+Tests: roles drive execute / recover; a blocked execute holds nothing up; the spin guard; `after`
+gates a start; `drive` resolves on foreground idle while a background task keeps running and the
+loop keeps serving it; abort marks in both race orders; reopen recovers inflight; a mark written
+by one process is applied by the next.
+
+## 7. Scratch
+
+Scratch batches, one task per batch, `ScratchTx`, retire on settle, sidecar-independent (memory).
+
+Tests: a crash before settle keeps scratch; settle deletes it; a write after settle is rejected; a
+new attempt clears its list.
+
+## 8. Harness shell and handles
+
+`Harness.open` (built-in registries, `kinds` and `replace` options, kinds-set check, `inspect`,
+`drive`, `close`, `shutdown`), `ConversationHandle` (`commit`, `value` / `list`, `config` /
+`settings`, `fork` with `abort`, `abort`, `hooks.on` scoped with `subtree`), `abortTask`,
+`conversations` with `parent` / independent filtering. No agent behaviour yet.
+
+Tests: open on empty vs existing storage; an unknown kind rejects without scanning entries;
+replace by name keeps `h.kinds.<name>` consistent; `settings` round-trips; scoped hooks run after
+harness-wide ones, innermost last.
+
+## 9. Generation kind
+
+Statuses pending → streaming → done / failed / retry_wait / deferred / aborted on a faux provider;
+config capture (model, thinking, selected tools, profile, budget); `system_instructions` with
+sections merged across handlers and the diff writing `system` entries; `before_request`,
+`after_response`, `on_yield`; retry sleeps in execute; recover from frames; usage recorded per
+attempt.
+
+Tests: the six system-entry traces from the guide (tool added, removed, host section changed, MCP
+schema changed, plugin section on and off, `addTools`); fresh baseline after a head; a fork diffs
+against its own config; a restart emits only the changed section; in-band `aborted` stop reason;
+crash while streaming publishes the partial; retry budget exhausted → failed with no successor.
+
+## 10. Tools, post_tools, exchanges
+
+The tool kind with the sink (`ToolOutput`, `ToolOutputState`, limits enforced by the sink, `diag`,
+`delegate`, `handoff`, `addTools`, `terminate`), `before_tool` (fail-closed) and `after_tool`,
+replay policy on recover; post_tools with `after`, terminate / handoff / steer / next generation;
+`accept` idle vs busy; `prompt` and `answerTo`.
+
+Tests: parallel tools completing in either order; sequential via `after`; error results for every
+call on abort; `new_context` resets after the exchange, never inside it; `addTools` → the next
+turn's `toolsAdded`; a throwing tool → error result, `terminate` still honoured; truncation diag
+from the sink; `accept` with the same `requestId` twice → same entry, `answerTo` finds it after
+further turns.
+
+## 11. Inbox
+
+`pi.inbox` as a conversation list, the modes, the three dequeue points, `cancelQueued`, abort
+draining steer and followUp.
+
+Tests: the modes table; cancel vs land in both orders; input queued during a collapse lands.
+
+## 12. Collapse
+
+Manual, threshold and overflow; `before_collapse`; publish only if no newer head; the overflow
+chain (generation settles → collapse → new generation carrying the attempt).
+
+Tests: a summary lands under a running generation and later entries stay in context; a stale
+summary fails; overflow retries once and no live task ever waits on the collapse; threshold before
+a turn; abort of a running collapse leaves appends flowing.
+
+## 13. Subagents
+
+The `subagent` tool (`run`, `spawn`, `send`, `status`, `wait`, `stop`), ownership links, foreground
+reach through live owners, `run`'s recover driving the child again, `spawn` initialization of
+config, `answerTo` on the child.
+
+Tests: restart in the middle of `run`; parent abort reaches a `run` child and spares a `spawn`
+child; `stop`; nested children; the child's first turn writes its own baseline.
+
+## 14. Jobs and the budget
+
+`jobKind` on `ExecutionEnv.exec` with output into its scratch, `waitForTask` / `jobOutput`, `bash`
+delegating first and waiting with the budget, `notify` and the `notice` entry, the `job` tool,
+schedules.
+
+Tests: budget expiry settles the call with `delegated` and the job continues; the notice appears
+on completion; a schedule loops and `abortTask` ends it; recover → `lost` or rerun by policy; a
+user abort during the wait kills a non-backgrounded job.
+
+## 15. Previews
+
+`ctx.preview` as a Chord tracker per task; the generation applies stream events to a partial
+message, the tool's preview is its sink state, the job's the same; `preview.init` on attach and
+reopen; flush after each scratch commit.
+
+Tests: one token → one `a` op and nothing else; no `r` mid-stream; init after reopen yields a base
+equal to the live preview; a sliding tool tail → `t` + `a`.
+
+## 16. Watch
+
+`ConversationView`, `ConversationEvent`, the exported kind-free `applyEvent`, `WatchHandle`
+(capture on the line, bounded buffering, `resnapshot`, `unsubscribe`), the session watch with
+`report` and `usage`, the usage ledger (`pi.usage` + totals).
+
+Tests: the fold is correct (view after N events equals a fresh capture, randomized); all events of
+one commit delivered together; a thin-client reducer over a recorded stream with no kinds loaded;
+lag → fault → resnapshot; `resnapshot` from inside the listener; usage totals equal the ledger
+fold, failed and aborted attempts included.
+
+## 17. JSONL storage
+
+Append-only batches, replay on open, torn tail discarded, malformed line fails open, the kinds set
+from replay, values as Chord deltas (full when first or small, ops otherwise), scratch sidecars
+retired after the main-file settle.
+
+Tests: conformance against memory (one mutation stream, identical query results); linear file
+growth under repeated sets of a large value; crash between the settle write and the sidecar unlink.
+
+## 18. SQLite storage
+
+Tables and indexes of §7.5, scratch rows deleted in the settle transaction, storage version and
+`migrate`.
+
+Tests: conformance three ways; a cold reopen decodes only live rows (count them); a version
+mismatch rejects.
+
+## 19. Race matrix and telemetry
+
+The §10.2 matrix in both orders with faux clocks, fake processes and storage barriers; spans per
+task call and per commit.
+
+Tests: the matrix; span tree for one turn with a tool and a retry.
+
+## 20. Clients
+
+mini (`worker/run.ts`, `worker/lane-service.ts`, TUI `apply(view)`), the experimental agent's four
+seam files (`session-worker.ts`, `agent-controller-provider.ts`, `models-provider.ts`,
+`transcript-provider.ts`), real providers. This is the gate: system deltas on a live model, a
+retry, a spawned subagent surviving a restart, speculative compaction under a running turn.
+
+Packages 1–16 are what the gate needs; 17–19 can land after it.
+
+## What comes from the lane harness, and how
+
+Clean room means no imports from `src/harness/runtime`, `session`, `agent-harness.ts` or the
+`dom`/`pico`/`pico2` spikes. Where the old code has something worth keeping, it is **copied** into
+`src/harness/pico/` and owned there; pico must build with the rest of `src/harness` deleted.
+
+Copy (under `packages/agent/src/harness/` unless noted):
+
+| what | from | into | package |
+|---|---|---|---|
+| `ExecutionEnv` / `FileSystem` / `Shell` types, the Node env, capture and spill | `types.ts`, `env/`, `tools/tool-context.ts` (03-execenv) | `pico/env/` | 10, 14 |
+| shell output limits, `applyShellOutputUpdate`, truncation totals | `utils/` | `pico/env/output.ts` | 10 |
+| the built-in tools (`read`, `write`, `edit`, `bash`, `image`) | `tools/*.ts` | `pico/tools/`, rewritten to the sink signature | 10, 14 |
+| system prompt builder, skills, context files, templates | `system-prompt.ts`, `skills.ts`, `prompt-templates.ts`; the keyed-sections builder in `coding-agent` | the host's `system_instructions` handler, not the harness | 9, 20 |
+| telemetry span helpers | `telemetry.ts` | `pico/telemetry.ts` | 19 |
+
+Depend on, as packages (they are not the old harness):
+
+- `@earendil-works/chord/delta` (15, 16, 17)
+- `@earendil-works/pi-ai`: `faux` provider for tests, `utils/estimate` for thresholds, `SystemMessage` (4, 9, 12)
+
+Read before writing the equivalent, then close the file:
+
+- `runtime/drive/retry.ts`, `deferred.ts`, `response.ts`: retryable-error classification, overflow detection, deferred polling (9)
+- `compaction/`, `runtime/drive/boundary.ts`: the exchange-cut rule and the summarizer prompt (12)
+- `runtime/drive/recovery.ts`, `restore.ts`: the per-phase recovery case list (9, 10, 14)
+- `runtime/drive/tool-placement.ts`: the result order projection must reproduce (4)
+- `hooks.ts`, `docs/harness.md` §before_tool: the exact decision shape (10)
+- `session/`, `test/harness/jsonl-*.test.ts`: torn-tail and malformed-line rules (17)
+- `session/`, `test/harness/mutation-line.test.ts`: line discipline edge cases (3)
+- `agent-harness.ts`: `LaneSnapshot` / `HarnessEventPayload`, for the `toLaneSnapshot(view)` shim only (20)
+
+Tests to port (`packages/agent/test/harness/`): the three conformance suites as the model for 17–18;
+`compaction`, `branch`, `context`, `execution-*`, `values`, `mutation-line` as the parity source for
+19; `system-prompt`, `prompt-templates` for the host handler in 9; `tools`, `truncate`,
+`output-capture` for 10; pico v1's 27 ported scenarios as the shortest list of behaviours to
+re-prove.
+
+Do not read: `runtime/lane.ts`, `reducer.ts`, `drive/reconcile.ts`, `structural.ts`,
+`terminal.ts`, `checkpoint.ts`, and the `dom` spike. They are the operation state machines and the
+reconciler this design replaces.