pi-mono/apps/p2/docs/p3-walkthrough.md

# p3 / Chord v2: the whole target design as a walkthrough

A reading aid for checking completeness, not a new source. Sources: `redesign-decisions.md` (wins on conflict),
`target-design.md`, `questions.md`, `plugin-loading.md`, `p3-functions/`, `changes.md`. Marks: **[D]** decided,
**[B]** built in today's Chord, **[S]** superseded by a later decision, **[O]** open, **[P]** proposed, not decided.
Section 17 lists conflicts between the documents found while writing this.

---

## 1. Plan

- **[D] p3 next to p2**, as `apps/p3`, own data directory. p2 keeps running unchanged. Plugins are ported one at a time.
- **[D] No migration** of p2 data: p3 starts empty (no `seed`, no catalog/shares/subagents import, no legacy parsing).
- **[D] Chord v2 lives inside p3** until stable, then moves to `packages/chord` (pi first, CI) and p2 is deleted.
- **[D] Durable boundary:** Durable keeps importing today's Chord (`delta`, `context`, `output`); p3 hands Durable only
  plain data and Durable's own types. Chord v2 takes `delta`/`output` over unchanged. `OutputBuffer`/`Progress` already
  moved to `@earendil-works/chord/output` [B].
- **[D] No backward compatibility** anywhere unless asked.
- **[D] Approach:** functionality first (`p3-functions/`: 22 domains), then objects per domain (owner, host,
  pure/native, relationships), then target design v2 written from the decision log, then reviewers judge it.

## 2. Concepts

| concept | what it is |
|---|---|
| facet | a plugin's code for one host kind; its setup is a **pure** host-lifetime object [D] |
| object type | contract: name, input (address), variants, version, host [B] |
| object | one instance per (type, input, principal key) |
| principal | who calls; stamped by links, typed by the app [B] |
| reference | what `get` returns; keeps the object alive; live and hydrated [B] |
| document | named JSON value an object owns, written in transactions |
| derive | value computed from documents/readables |
| native | a value opened with `self.open` in an object (socket, process, DB handle) [D] |
| routine | standing behaviour, `self.routines.*` |
| job | owned work started by a method, outlives the call |
| registry | an owner's extension point, versioned immutable table |
| host value | typed native the host provides before facets load |
| `ctx` | per call: signal, principal, call-scoped `get`; in jobs `jobId`, `progress` |

## 3. Object types and principals [B]

- `local` (same host, unvalidated) or `remote` (over links, variants per principal, TypeBox-validated).
- `host` declares placement (`hostNamed`, `hostFamily`); changing it is breaking. Callers don't pass `at`.
- Same version binds if the provider covers the caller; breaking change bumps `version`; `accepts: { [old]: adapter }`
  keeps old consumers, else `contract_mismatch`.
- References are a union discriminated by `variant`; strict narrowing with `inVariant` / `expectVariant` / `RefIn`
  (entry 8). Weakening type checking was rejected.
- Principals: `createHost<P>()`, every principal has a `kind`. p2: `owner` (`access: local | relay | device`),
  `guest`, `worker`, `server`, `host`. State `init` (2b). Origin ids per client (entry 8). Protocol 4.
- `host.serve(Type, factory, { lifetime: "referenced" | "host", linger? })`; `"host"` checked by types [B].

## 4. Factories, natives, the facet [D]

- **Factory principle:** a factory is synchronous and only declares (references, documents, natives, derives,
  routines, jobs, registrations). Chord activates. Handles it gets are valid for the object's whole life.
- **Natives only in objects:** `self.open(async ({ signal }) => value, close)`. Chord opens during activation (object
  goes live after all opens), closes in reverse order when the object ends. The value never changes during the
  object's life.
  - Gone [S]: `self.resource`, resource `version`, `replace`, `deps`, reopen, stale handles, "read `.value` at use"
    (target-design §12).
- **No natives across plugins:** others reach a native only through the owning object's methods and state.
- **The facet is a pure object:** documents, references, routines, registrations; no `self.open`. Served objects may
  read/write the facet's documents; one transaction may span own docs + facet docs (lock order: own line, then
  facet). The facet may not reach into served objects' documents.
- Between plugins only: object types, registries, host values.
- Rejected: ceremony (`native/` dirs, `defineNative`, lints).

## 5. Ownership [D]

- Every thing has exactly one owning object, the smallest it belongs to. The owner holds the data, enforces rules in
  its methods, offers operations. Nobody reaches past it.
- Ownership nests: session owns its conversations and location; a conversation owns its content and rules (read-only,
  follows session cwd), set in the creating commit (Durable `createConversation` with `init`).
- Methods return data; objects are handed out as addresses (`ctx.get(Type, address)`).
- Registries add behaviour (tools, routes, renderers, commands, send routers, input gates); never guard owned data.
- Core owners always exist; plugin owners may be missing (→ optional references [O]).
- Avoid a core god object.
- **Porting guideline:** what core enforces on data lives in a format core owns, written by the plugin at creation.
  Example: one-shot / cwd-following children marked in a core per-conversation policy document; `ReadOnlyConversations`
  and `SessionCwdFollowers` go away.

## 6. References

- [B] `self.get` (factory, released with the object), `await ctx.get` (call), `host.get`/`host.hold` (outside).
- [B] Awaited reference is live and hydrated; `state: false` for method-only use.
- [B] Link drop: values stay (`loading`), re-hold automatically, calls wait.
- [D] Failed object ends its references with the cause; consumers re-hold with backoff (web gets the TUI's backoff
  instead of fixed 500 ms).
- [D] **Backoff per address** (1 s doubling to 60 s): a `get` during backoff returns the stored error; concurrent gets
  share one attempt. Cut short by: plugin reload, a dependency of the failed attempt changing, `ref.retry()`,
  `self.retryOn(...readables)`.
- [D] Consumers handle errors as on a disconnect. No call/request-id framework (Chat's `requestId` stays).

## 7. Documents and transactions [D]

- `defineState(name, { schema, version, initial, migrate?, keyed?, persist?, checkpointWhen? })`.
- `self.state(Def)` → one handle. Singleton: loaded at activation, `handle.value` sync, exposable readable. Keyed:
  `await handle.get(key)`, `for await (key of handle.keys())` (paged from storage). No live key list; live collections
  are singleton docs; large ones are paged methods.
- Writes only via `await tx.draft(handle, key?, { tags? })` in `self.tx(ctx, async tx => …)`; `tx.retire`.
- **Durable's rule:** inside a tx only `await tx.*`; no other objects, models, processes, network, nested tx. Fetch
  before; decide inside.
- Commit: per-object line; storage batch serialized per host; adopt; notify once; one wire batch.
- Shorthands: `handle.change(ctx, key?, fn)`, `handle.retire(ctx, key?)`.
- Versions: pure `migrate`, on access; old code on newer doc throws; no `migrate` → access refused, data kept.
- Tags: typed (`defineTag<T>`), set explicitly on create; `store.retireByTag`. p2 `SessionTag`.
- Lifetime: in-memory docs live with their object identity; persisted until retired / session delete.
- `persist` only on stable identities (`lifetime: "host"` or `storeKey(input)`).
- Rejected: optimistic transactions; Durable's Session as object store; volatile documents.

## 8. Store and storage [D]

- Chord owns the store: lines, serialized commits, migrate on access, `checkpointWhen`, retire, tags, invariants.
- Eviction as in Durable for keyed persisted docs (10 min unused and unwatched; LRU, one timer).
- Storage error ends the store; host reports and restarts.
- `DocumentStorage` interface (`commit`, `find`, `load`, `scan`, `byTag`, `mintId`); memory implementation in Chord;
  p2/p3 adapts Durable's SQLite `Storage` (`kind = <definition>.<objectHash>`); tag index in the adapter.
- Conformance suite in `@earendil-works/chord/testing`.
- Persistence split: app state in Chord documents; only transcript-bound data in Durable.
- Kinds of state: document, derive, source readable, paged method, plugin database (as a native).

## 9. Derives [D]

- `self.derive(name, sources, (values, previous) => …)`; `previous` kept across factory re-runs by name.

## 10. Jobs [D]

- `defineJob(name, { version, args, progress, result, concurrency?, persist?, accepts? })`.
- Sandwich: **intent** `tx.start(job, args, { key? })` (busy check and dedupe atomic, run starts after commit) →
  **run** `run(args, ctx)` (anything; external writes keyed by `ctx.jobId`; keeps old code on reload) → **finish**
  `finish(outcome, tx)` in current code, ack in the same commit; retried with backoff if it throws.
- Outcome `{ ok, result } | { ok: false, error: threw | aborted | lost }`.
- `job.start(args, ctx, opts)` returns a run with `outcome(ctx)`; caller's abort ends only its wait.
- Abort is a request (`job.abort(id)` fires the signal).
- `running` readable `{ id, version, args, progress, phase }`; `concurrency` counts active runs; `phase: finishing |
  blocked`.
- Versions via `accepts`; stored outcomes never rewritten; missing adapter → `blocked`, not a refused reload.
- `persist: true` (stable identities): a live run at crash ends `lost`.
- Running jobs count for keep-alive and the relocation fence.
- Not jobs: request/response (method), input into running work (Questions), crash-resumable work (Durable task).
- [O] jobs as objects with an "until done" lifetime.

## 11. Routines [D]

- `self.routines.on(readable|event, fn)`, `every(name, interval, fn)` (due time survives reloads), `loop(name, fn)`,
  `custom(name, setup)`.
- Start when the object goes live, stop when it ends, restart on reload of its plugin; serialized per routine (new
  instance waits for the old handler in flight, with a deadline).
- Reconcile from current value; no "in progress" flags; long work → job.
- No bare timers/callbacks outside routines, jobs or calls.
- [O] `overlap: "latest"`.

## 12. Registries and host values [D]

- `defineRegistry<V>(name, { version, accepts, conflict: "error" | "last" })`; facet registers; entry carries its
  build version; tables are immutable values swapped atomically; no late binding; no lifecycle in entries.
- `hostValue<T>(name, { version })`, provided before any facet on every host (worker: harness, environments, init,
  location, liveness, cwd followers; server: session hosts, server info, revocation; web/TUI: navigation, theme,
  clipboard). Replace `opaque` members.
- **Start barrier:** open Durable without scheduling → host values → load facets → check cwd/model → `resume()`.
- Rejected: late-bound entries, dynamic registries.

## 13. Reload [D]

- Explicit (`npm run reload -- <plugin>`), never on save. Behaves like a restart of the changed plugins.
- **Per host:** load (bundle; error rejects) → **staging** (facet setup and every live object's new factory with a
  recording `self`; nothing activated; a throw rejects) → **commit** (atomic) → after.
- **Rollback all-or-nothing per host:** any load/staging error leaves the host entirely on previous code. Status per
  host suffices.
- **Pure objects** swapped in place: new methods and derives; documents, references, jobs stay; routines restart
  serialized; new references go live; gate opens. Running calls finish on old code.
- **Native objects of the reloaded plugins restart:** running work aborted (`aborted`), natives close, factory runs,
  natives reopen. No per-module digest; split expensive natives into their own plugin. Open fails → that object
  `failed` with backoff; rest of host on new code.
- **Remote consumers** see a normal brief object restart: `loading`, values readable, calls park, re-hold; calls on the
  old instance end with an error.
- Errors: code errors (load, setup, factories) → rollback; activation errors (opens, new references, migrations) →
  per object; a method bug → that call.
- No factory trial, no migrate dry run, no consumer contract check during prepare.
- Removing a type: its objects end; in-memory docs go; persisted docs stay.
- Rejected: drain-based handover, version fences, `restart_required`, reload votes / cross-host transactions,
  revoking proxies on `.value`.

## 14. Rollout [D]

- Revision = (server incarnation, sequence) over the complete desired code set (every facet + digest).
- Server installs, then each worker, then clients; each reports `installed` / `installed_failed` / `rejected`. A
  starting or reconnecting host installs the current revision before anything is routed to it. Installed = code
  published (objects activate on demand).
- Mismatch responses carry the provider's revision and status: provider ahead or installing → `updating` (values
  readable, calls park); provider rejected → fail fast; same revision → `contract_mismatch` final. Deadline on waits.
- Server rejects → rollout stops.
- [O] check own host's status before waiting; serialize installs per host; digest of the code set (no reinstall after
  a server restart).

## 15. Wire, presence, Questions, ctx

- **Wire [D]:** documents are state members; `snap` on hold, `ops` per commit, diff ops for derives, throttled job
  progress, `reset` for migration, `error`. Commits atomic for consumers: one `state` frame with a `batch` per hold.
  [O] atomic per link, diffed ops for adapted bindings, size limit.
- **Presence [D]:** `host.references(Type)` readable (input, principal, clients); origin ids per client; **linger bit**
  (lingering keeps the worker alive, not a viewer). Replaces presence/viewer/login-abandon leases.
  [O] link drop counts as lingering; counts per origin.
- **Questions [D]:** p3 service for input into running work: shared store, per-principal asker/viewer views, typed
  questions (text, select, confirm), policies first/quorum/all over a live audience, `key` to reattach, deadline,
  authorization per question, secrets redacted. Used by login, tool approval, question tool.
- **ctx [D]:** `signal`, `principal`, `get`; in jobs `jobId`, `progress`. No code versions, no root ids.

## 16. Bytes and session files

- **[D] Bytes are a schema type** (`Uint8Array`): structured clone on IPC (`serialization: "advanced"`), binary frames
  on WebSockets. No base64 in JSON. Lives in Chord v2 inside p3 (not added to today's Chord).
- **[D] Large transfers chunked by the owner:** `begin` / `append({ upload, offset, bytes })` / `finish`,
  `read({ id, offset, length })`, ~1 MiB chunks; awaited `append` is backpressure; resume from stored offset after a
  link drop; unfinished uploads expire. Helper wraps the loop. No stream type in Chord.
- **[P] `SessionFiles {session}`** on the server: one owner of uploads (25 MiB, ≤20/message), sent-file snapshots
  (25 MiB), artifact versions (2 MiB, name + version); index readable without a worker; bytes under
  `<data>/sessions/<id>/files/`; copy on clone and fork, delete with the session. Not in it: workspace downloads
  (streamed), shell/codemode spill (in the env). `DownloadStore.save` (unused) goes.
  Pending sub-points: one preview limit (256 vs 512 KiB); fork's artifact list empty or copied; copying attachments
  into the remote env.

## 17. Session lifecycle [D]

- Archive retires nothing (stops the worker, ends guest access, closes forwards).
- Session delete (new) removes the whole session: tagged documents, the worker's database and document storage,
  files, catalog entry. [O] how: `SessionDeleted` registry vs. per-session objects + `SessionTag` + session directory;
  search index per session vs. global + hook.

## 18. Versions and adapters

| surface | on change |
|---|---|
| object type | `accepts` adapter, coverage check [B] |
| registry | `accepts` adapter |
| host value | `accepts`; harness refused on mismatch for now |
| document | pure `migrate` |
| job | `accepts` on outcomes and `running` |
| native | object restarts on reload (no version) |
| Durable tasks/docs | Durable `migrate` [B] |
| link protocol | refused; hosts update together [B] |
| UI components, API | exposed as they are; versioned with the SDK later |
| [O] file formats (`envs.json`, …) | unversioned today |

## 19. Plugin loading (`plugin-loading.md`, revision 4)

- Sources: built-in, global (`<data>/plugins`), global packages (npm/git, `--ignore-scripts`), project
  (`<cwd>/.pi/p2/plugins`, worker/web/tui only, never server).
- Discovery = directory listing, never parses code; server selects, workers scan nothing; dependencies resolved by
  contract name at runtime; missing provider fails at activation with the consumer's name.
- Names: `<plugin>.<name>` for contracts, registries, Durable extensions, entry kinds; tools `<plugin>_<name>`.
- Loader D: one base bundle per surface + one script per facet, host packages passed in, no module map (GC reclaims old
  generations); stable filename per (facet, digest), V8 compilation cache off. Vite removed after web D.
- Project trust: `trust.json` per directory, `ask`/`never`, pin-to-content mode; content-addressed snapshots; approved
  bytes = executed bytes; change → dirty → owner Reload. Project `.pi/skills`/`.pi/prompts` gated too.
- Scopes: a page/TUI worker holds one scope; `ClientBundle` pushes bundle hashes; TUI snapshots partitioned per scope.
- Leaving a scope: provenance map, fallback renderers, blocked tasks with Abort.
- Security table: trusting a project grants the owner's OS account and owner rights in the browser.
- Contracts are a versioned package, never host packages.
- [O] base skills on remote sessions; publishing; plugin identity across projects; guest opt-in; project root; browser
  compile cache. Project plugins as a whole: separate discussion.

## 20. Functionality map (`p3-functions/`)

22 domains; cross-cutting needs (one file store, per-session lifecycle, readers without workers, generation-fenced
reports, catch-up after commits, work outliving its caller, request-id idempotency, cancellation chains, keep-alive vs
presence, execution env following the session, single config writer, attribution, one permission matrix, revocation,
chunked bytes, one notice channel, UI state survival, TUI consent); ~25 duplicates; ~18 drop candidates.

## 21. Open (all)

- Optional references (a plugin owner may be missing; radius dies without ports today).
- Keep-alive computed by the worker itself.
- UI renderer reload vs. leaving (login aborted, TUI overlays close).
- File formats as versioned interfaces.
- Terminal host process as a revision artifact.
- Durable documents shared across plugins.
- Session delete mechanics (section 17).
- `close` with a reason (`reload` / `end` / `stop`).
- Call-scoped natives.
- Async activation guard for authority-carrying addresses.
- Invariants across several owners.
- Published projections with a generation fence.
- Connection pools; config files with a single writer.
- Rollout fixes (section 14); wire; presence; `overlap: "latest"`; project plugins; jobs as objects.
- Step 2 objects for every domain, starting with `SessionFiles`.

## 22. Tracked, not scheduled (`changes.md`)

- Leaks C3 (Chord kept listener), C5 (p2 guard listener per reload), C7 (harness keeps envs `close`); rerun GC tests.
- p2 hazards on interrupted calls (shell timer, dedupe in a closure, withdrawn text only in replies, web send while
  disconnected, missing request ids, dropped terminal keys, multi-step revocation/relocate/clone).
- Durable nested commits hang by convention only.
- #10959 (remote references accept a foreign ctx), owner fixes via subagent.

## 23. Rejected (all)

Drain-based handover; version fences without transactions; object restart for state changes; Durable Session as object
store; volatile documents; per-group lines; generic call/request ids; optimistic transactions; outbox for transient
delivery errors; revoking proxies on `.value`; late-bound registry entries; dynamic registries; `restart_required`;
detached references counting toward presence; contracts taken from the running p2; ceremony (`native/` dirs,
`defineNative`, lints); weakening type checking; "magic" input-field matching for delete; factory trial; migrate dry
run; consumer contract check during prepare; reload votes; `defaultProjectTrust: "always"`; loader variants A and B.

## 24. Conflicts found between the documents

1. `target-design.md` §15 and `plugin-loading.md` 7.5: "a facet whose setup throws keeps its previous version while the
   rest of the host moves on". The decision log replaced this with all-or-nothing rollback per host.
2. `questions.md` keeps its store in a facet `self.resource`. Facets may not open natives now, and `self.resource` is
   gone: the store must move into an object (e.g. a host-lifetime `QuestionStore` with `self.open`, or plain documents
   if in-memory is enough).
3. `target-design.md` §5 example (`self.resource` in the facet, `deps`) and §12 (resources replaced in place) are
   superseded by `self.open` in objects.
4. `target-design.md` §7 "handles belong to their code generation ... `stale`" assumed in-place resource replacement;
   with pure swap and native restart, whether old-generation handles still need `stale` is unclear.
5. `plugin-loading.md` 9.2a gap list mentions `opaque` members and unversioned registries; both are addressed by host
   values and versioned registries.