Armin Ronacher
13cbf77df2
Release v0.86.1
2026-09-20 13:04:11 +02:00
Armin Ronacher
ecac0a9c4e
Release v0.86.0
2026-09-20 01:00:41 +02:00
Armin Ronacher
16292398af
fix(coding-agent): send forced system prompts without recording them
...
Remove SystemMessage.replace. A prompt forced from before_agent_start is a rendering
of the current prompt for the run, not conversation state: persisting it as a
replacing system message wrote the full prompt and tool list on every change,
lost the structured sections while forced, and disabled mid-conversation system
messages for the rest of the session once a replacement existed.
The transcript now always records section patches. AgentSession wraps
transformContext and, when a forced prompt is active, collapses the system
messages into one head carrying the forced text and the current tools. Providers
receive the same request as before; nothing is persisted and the projection ends
with the run.
The evals harness validated the without_docs variant from the transcript, which
relied on the persisted replacement. Its transform extension now records the
prompt it forced and validates that instead.
2026-09-17 20:02:23 +02:00
Christian Klotz
5a3a03a7f5
fix(coding-agent): validate eval prompts from transcripts ( #9706 )
...
Preserve partial-run usage and session artifacts when post-run validation fails.
2026-09-17 14:12:10 +01:00
Christian Klotz
42cd371ba4
feat(coding-agent): add TUI context footer eval ( #9705 )
2026-09-17 13:21:31 +01:00
Armin Ronacher
1247476e6d
fix(coding-agent): update eval prompt section markers
2026-09-16 17:47:13 +02:00
Christian Klotz
d7296c063b
feat(coding-agent): isolate documentation lift evals ( #9635 )
2026-09-15 17:54:18 +01:00
Christian Klotz
b215884021
feat(coding-agent): add customization documentation evals ( #9491 )
...
* fix(coding-agent): isolate extension eval documentation treatment
* feat(coding-agent): add custom provider documentation eval
* feat(coding-agent): add customization documentation evals
* feat(coding-agent): add repeatable eval reports
2026-09-11 17:29:34 +02:00
Christian Klotz
f3564a1d41
docs: validate documentation navigation and reachability ( #9380 )
...
* feat: use docs manifest for documentation evals
* feat: support nested documentation navigation
* docs(coding-agent): validate documentation reachability
* docs(coding-agent): remove catalog schema version
2026-09-09 17:34:42 +02:00
Christian Klotz
9211da1723
feat: add implementation-backed documentation evals ( #9280 )
...
* feat: add implementation-backed documentation evals
Generate one model-backed audit per cataloged documentation page and validate the catalog deterministically. Support terminating custom audit tools through the Pi eval harness.
* feat: inline documentation audit schema
* docs: document focused documentation eval command
* feat: simplify documentation audit setup
2026-09-07 20:21:39 +01:00
Armin Ronacher
d981de1229
Release v0.85.1
2026-09-05 13:54:46 +02:00
Armin Ronacher
107d79f110
Release v0.85.0
2026-09-04 11:57:38 +02:00
Armin Ronacher
b79e4cc834
Release v0.84.4
2026-08-28 23:56:03 +02:00
Armin Ronacher
4e58f324fa
Release v0.84.3
2026-08-24 12:50:48 +02:00
Armin Ronacher
f8f03460a0
fix: reduce workspace dependency tree
2026-08-23 09:28:19 +02:00
David Brailovsky
914cf1472e
Release v0.84.2
2026-08-14 12:00:00 +02:00
Armin Ronacher
53fa77ccd8
Release v0.84.1
2026-08-07 07:48:13 +02:00
Armin Ronacher
a5f43bf8af
Release v0.84.0
2026-08-06 12:32:57 +02:00
Christian Klotz
7659e1c98d
Merge pull request #7289 from earendil-works/worktree/evals-baseline
...
feat(coding-agent): add comparative Pi eval harness
2026-07-30 11:46:45 +03:00
Christian Klotz
743e1595c4
fix(coding-agent): remove custom eval shuffling
2026-07-30 11:31:41 +03:00
Christian Klotz
ca11823f72
fix(coding-agent): streamline comparative eval ergonomics
2026-07-30 11:12:04 +03:00
Christian Klotz
fce7445b1c
fix(coding-agent): make eval comparison direction explicit
2026-07-30 10:58:07 +03:00
Christian Klotz
ac033d1865
fix(coding-agent): align eval comparison report output
2026-07-30 10:19:58 +03:00
Christian Klotz
29f97c20f3
fix(coding-agent): simplify harness table tests
2026-07-30 10:11:08 +03:00
Christian Klotz
d9ae6e2b07
fix(coding-agent): simplify comparative eval internals
2026-07-30 02:27:47 +03:00
Christian Klotz
6a3eae8e48
fix(coding-agent): normalize eval inputs before hashing
2026-07-30 02:13:02 +03:00
Christian Klotz
56e6a8a3fa
fix(coding-agent): record eval sessions after each test
2026-07-30 02:05:43 +03:00
Christian Klotz
2a2e677bed
fix(coding-agent): make source artifacts language neutral
2026-07-30 01:59:30 +03:00
Christian Klotz
3631e82d0e
fix(coding-agent): generalize eval source artifacts
2026-07-30 01:56:06 +03:00
Christian Klotz
eb08aa8fa2
fix(coding-agent): specialize eval attachment types
2026-07-30 01:47:53 +03:00
Christian Klotz
2eed6d219a
fix(coding-agent): remove redundant artifact names module
2026-07-30 01:37:23 +03:00
Christian Klotz
af00848d6b
fix(coding-agent): record eval artifacts with explicit tasks
2026-07-30 01:30:18 +03:00
Armin Ronacher
845d6ff1f6
Release v0.83.0
2026-07-30 00:24:19 +02:00
Christian Klotz
32f3a97282
feat(coding-agent): add comparative Pi eval harness
2026-07-30 01:17:54 +03:00
Christian Klotz
b0864a66cd
fix(coding-agent): preserve eval run diagnostics
2026-07-27 18:00:53 +03:00
Christian Klotz
b56a35f90f
fix(coding-agent): project typed eval outputs
2026-07-27 17:40:14 +03:00
Christian Klotz
af19a9eea8
fix(coding-agent): simplify extension eval prompt
2026-07-27 15:30:37 +03:00
Christian Klotz
95d1c60fe2
fix(coding-agent): simplify multi-step eval input
2026-07-27 15:16:01 +03:00
Christian Klotz
9423204c6b
fix(coding-agent): clarify eval harness errors
2026-07-27 15:02:23 +03:00
Christian Klotz
8a97632459
fix(coding-agent): keep eval scenarios in test cases
2026-07-27 14:50:51 +03:00
Christian Klotz
a0ac81c0a1
fix(coding-agent): distinguish eval invariants
2026-07-26 00:02:34 +03:00
Christian Klotz
4b8accab8b
fix(coding-agent): simplify eval normalization
2026-07-25 23:47:07 +03:00
Christian Klotz
73c1696d95
fix(coding-agent): use focused eval harness
2026-07-25 23:34:20 +03:00
Christian Klotz
66fbdb1c59
feat(coding-agent): add extension creation eval
2026-07-25 23:09:22 +03:00
Armin Ronacher
b4f293684b
Release v0.82.1
2026-07-25 14:37:11 +02:00
Armin Ronacher
6173017a7b
fix: drop harness-pi-ai and its legacy registry aliases from evals
...
The harness declared peer deps on the pre-rename @mariozechner/pi-ai and
@mariozechner/pi-agent-core names, so npm auto-installed our own code from the
registry under an alias. sync-versions bumped those aliases in lockstep, which
made every release fail because the new version is not published yet.
Build the eval harness on createHarness from vitest-evals core instead. The
pi-specific code was already ours; the harness only contributed agent-lifecycle
plumbing and unused tool replay.
2026-07-25 14:36:32 +02:00
Christian Klotz
eafe11fb94
feat(coding-agent): add vitest eval harness ( #7085 )
2026-07-25 08:22:32 +02:00
