Commit graph

20 commits

Author SHA1 Message Date
Christian Klotz
9628c84809
feat(coding-agent): add prompt template documentation eval (#10261)
## Summary

- add live documentation comparisons for project-scoped and user-scoped
`/current-time` prompt templates
- require the generated command to expand to the exact requested prompt
text
- select the requested case from multi-case Vitest reports so skipped
siblings do not invalidate observations
- strengthen documentation-audit evidence and move the expensive audit
behind an explicit `eval:audit` command
- remove redundant eval oracles and test-only exports found during the
eval-suite audit

## Testing

- `npm run check`
- `./test.sh`
- `npm --prefix packages/evals test` (54 passed, also covered by
`./test.sh`)
- failed-status mutation: focused report test passed before the fixture
repair, failed after it, then passed with production restored
- prompt-template documentation eval, three repetitions per variant (6/6
pairs eligible)
- project scope: with docs 66.7%, without docs 100%; negative delta and
treatment flakiness
  - user scope: with docs 100%, without docs 100%; no lift
- model documentation eval (1/1 pair eligible; treatment 100%, control
0%)
- representative live documentation audit (`cli.md` passed)
2026-10-28 12:12:19 +02:00
Armin Ronacher
16292398af fix(coding-agent): send forced system prompts without recording them
Remove SystemMessage.replace. A prompt forced from before_agent_start is a rendering
of the current prompt for the run, not conversation state: persisting it as a
replacing system message wrote the full prompt and tool list on every change,
lost the structured sections while forced, and disabled mid-conversation system
messages for the rest of the session once a replacement existed.

The transcript now always records section patches. AgentSession wraps
transformContext and, when a forced prompt is active, collapses the system
messages into one head carrying the forced text and the current tools. Providers
receive the same request as before; nothing is persisted and the projection ends
with the run.

The evals harness validated the without_docs variant from the transcript, which
relied on the persisted replacement. Its transform extension now records the
prompt it forced and validates that instead.
2026-09-17 20:02:23 +02:00
Christian Klotz
5a3a03a7f5
fix(coding-agent): validate eval prompts from transcripts (#9706)
Preserve partial-run usage and session artifacts when post-run validation fails.
2026-09-17 14:12:10 +01:00
Armin Ronacher
1247476e6d fix(coding-agent): update eval prompt section markers 2026-09-16 17:47:13 +02:00
Christian Klotz
d7296c063b
feat(coding-agent): isolate documentation lift evals (#9635) 2026-09-15 17:54:18 +01:00
Christian Klotz
b215884021
feat(coding-agent): add customization documentation evals (#9491)
* fix(coding-agent): isolate extension eval documentation treatment

* feat(coding-agent): add custom provider documentation eval

* feat(coding-agent): add customization documentation evals

* feat(coding-agent): add repeatable eval reports
2026-09-11 17:29:34 +02:00
Christian Klotz
f3564a1d41
docs: validate documentation navigation and reachability (#9380)
* feat: use docs manifest for documentation evals

* feat: support nested documentation navigation

* docs(coding-agent): validate documentation reachability

* docs(coding-agent): remove catalog schema version
2026-09-09 17:34:42 +02:00
Christian Klotz
9211da1723
feat: add implementation-backed documentation evals (#9280)
* feat: add implementation-backed documentation evals

Generate one model-backed audit per cataloged documentation page and validate the catalog deterministically. Support terminating custom audit tools through the Pi eval harness.

* feat: inline documentation audit schema

* docs: document focused documentation eval command

* feat: simplify documentation audit setup
2026-09-07 20:21:39 +01:00
Christian Klotz
743e1595c4
fix(coding-agent): remove custom eval shuffling 2026-07-30 11:31:41 +03:00
Christian Klotz
ca11823f72
fix(coding-agent): streamline comparative eval ergonomics 2026-07-30 11:12:04 +03:00
Christian Klotz
fce7445b1c
fix(coding-agent): make eval comparison direction explicit 2026-07-30 10:58:07 +03:00
Christian Klotz
ac033d1865
fix(coding-agent): align eval comparison report output 2026-07-30 10:19:58 +03:00
Christian Klotz
29f97c20f3
fix(coding-agent): simplify harness table tests 2026-07-30 10:11:08 +03:00
Christian Klotz
d9ae6e2b07
fix(coding-agent): simplify comparative eval internals 2026-07-30 02:27:47 +03:00
Christian Klotz
6a3eae8e48
fix(coding-agent): normalize eval inputs before hashing 2026-07-30 02:13:02 +03:00
Christian Klotz
2a2e677bed
fix(coding-agent): make source artifacts language neutral 2026-07-30 01:59:30 +03:00
Christian Klotz
3631e82d0e
fix(coding-agent): generalize eval source artifacts 2026-07-30 01:56:06 +03:00
Christian Klotz
eb08aa8fa2
fix(coding-agent): specialize eval attachment types 2026-07-30 01:47:53 +03:00
Christian Klotz
af00848d6b
fix(coding-agent): record eval artifacts with explicit tasks 2026-07-30 01:30:18 +03:00
Christian Klotz
32f3a97282
feat(coding-agent): add comparative Pi eval harness 2026-07-30 01:17:54 +03:00