pi-mono/packages/evals

Pi evals

Pi evals are behavioral, model-backed checks for Pi workflows. They adapt a real AgentSession to vitest-evals, run it in isolated temporary project and agent directories, and attach native Pi session artifacts. Use them to measure end-to-end behavior and compare prompts, tools, skills, models, or other harness configurations.

Running evals

Run from the repository root with a default provider and model:

npm run eval -- --provider openai --model gpt-5.6-sol

The equivalent environment variables are:

PI_PROVIDER=openai PI_MODEL=gpt-5.6-sol npm run eval

CLI values take precedence and become defaults for harnesses that do not select a model explicitly. Provider and model must be supplied together. The runner also allows no default when every executed harness configures its own model. Authentication comes from Pi's normal ModelRuntime, including Pi subscription credentials and provider API-key environment variables.

Additional arguments are forwarded to Vitest:

npm run eval -- src/extensions.eval.ts
npm run eval -- -t "creates and uses the extension"
npm run eval -- src/docs.eval.ts -t "session-format\.md"

Run all comparative customization evals five times in one invocation:

npm run eval -- \
  src/extensions.eval.ts src/models.eval.ts src/providers.eval.ts \
  --provider openai --model gpt-5.6-sol \
  --repetitions 5

--repetitions applies to suites declared with evalHarnessTable(...). An explicit repetitions value in a suite overrides the command-line default. PI_EVAL_REPETITIONS=5 is equivalent to the command-line option. Use one repetition while developing an eval and five when reporting lift.

Reports and artifacts

Each invocation prints a compound Eval Comparisons report after the Vitest results. When several comparative files run in the same invocation, this report contains one section for every eval set. For example, with illustrative values:

Eval Comparisons
  Add model to existing provider
     Baseline  system-prompt-without-docs
    Candidate  default-system-prompt (5/5 pairs)
    Pass rate  +60.0 pp (candidate 80.0%, baseline 20.0%)
       Tokens  +1200.0 (candidate 24000.0, baseline 22800.0)
      Latency  -850.0ms (candidate 14000.0ms, baseline 14850.0ms)
    Est. cost  +$0.0100 (candidate $0.1200, baseline $0.1100)

  Add OpenAI-compatible provider
    ...

  Add custom streaming provider
    ...

The runner prints the ignored .eval/ artifact directory at startup. It contains:

• report.txt: the terminal comparison report without color codes.
• report.json: the same aggregate comparison data as structured JSON.
• runs.jsonl: one record for every completed harness run.
• sessions/: native Pi session JSONL attachments.
• sources/: source attachments recorded by individual evals.

The report covers comparative suites using evalHarnessTable(...). Ordinary evals still appear in the Vitest summary and in runs.jsonl, but not in the baseline-versus-candidate comparison report. Artifacts may contain prompts, responses, source code, and tool output.

Writing evals

Follow vitest-evals for general suite, judge, assertion, and normalized trace guidance. Pi-specific evals use createPiCodingAgentHarness(...) from src/pi-harness.ts, with one harness bound to each describeEval(...) suite:

import { expect } from "vitest";
import { describeEval } from "vitest-evals";
import { createPiCodingAgentHarness } from "./pi-harness.ts";

const harness = createPiCodingAgentHarness({ noTools: "all" });

describeEval("Pi smoke", { harness }, (it) => {
	it("answers a factual question", async ({ run }) => {
		const result = await run("What is the capital of France? Reply with only the city name.");
		expect(result.output).toBe("Paris");
	});
});

Configuring the Pi harness

createPiCodingAgentHarness(...) accepts:

• name: stable harness identity used by reports and comparisons.
• model: optional { provider, id } selection. It overrides the runner's default model.
• noTools: Pi's tool-disable configuration.
• tools: optional allowlist of tool names available to the evaluated agent.
• customTools: custom tool definitions to register for the evaluated agent.
• transformSystemPrompt: transforms the complete default prompt before the eval starts.
• output: transforms the final response and AgentSession into a JSON-safe domain result.

An explicitly selected model makes model-comparison harnesses independent of the runner default:

const harness = createPiCodingAgentHarness({
	name: "claude-opus-4-6",
	model: { provider: "anthropic", id: "claude-opus-4-6" },
});

A run accepts either one prompt or a sequence of prompt and reload steps. Reload steps are useful when the preceding prompt creates or changes Pi resources:

const result = await run([
	{ type: "prompt", content: "Create a Pi extension." },
	{ type: "reload" },
	{ type: "prompt", content: "Use the extension." },
]);

Transforming harness output

Use output to expose scenario-specific, JSON-safe behavior without adding that behavior to the generic Pi adapter:

const harness = createPiCodingAgentHarness({
	output: ({ response, session }) => ({
		response,
		activeTools: session.getActiveToolNames(),
		extensionErrors: session.resourceLoader.getExtensions().errors,
	}),
});

Assert application behavior on result.output. Assert model and tool traces on result.session, using vitest-evals helpers such as toolCalls(...).

Writing comparative eval sets

Use evalHarnessTable(...) with Vitest's native describe.for(...) to run the same inputs against multiple harnesses. Harnesses may differ by prompt, tools, skills, model, or any other Pi configuration:

import { describe } from "vitest";
import { createJudge, describeEval } from "vitest-evals";
import { evalHarnessTable } from "./vitest-evals/harness-table.ts";

const TargetTaskJudge = createJudge<string, string>("TargetTaskJudge", ({ output }) => ({
	score: output === "expected result" ? 1 : 0,
}));

const harnessTable = evalHarnessTable(
	"target skill effectiveness",
	{
		baseline: withoutTargetSkillHarness,
		candidate: withTargetSkillHarness,
		repetitions: 6,
	},
);

describe.for(harnessTable)("$name repetition $repetition", ({ harness }) => {
	describeEval("target skill effectiveness", { harness, judges: [TargetTaskJudge], judgeThreshold: null }, (it) => {
		it("completes the target task", async ({ run }) => {
			await run("Complete the target task.");
		});
	});
});

Comparative suites should record correctness with deterministic or model-backed judges and set judgeThreshold: null. This keeps a low score as an observation instead of making the Vitest invocation fail. Use hard assertions only for suite invariants and infrastructure contracts. expect.soft(...) still fails the test and is not a scoring mechanism.

The Pi harness snapshots native session JSONL before deleting its temporary workspace. An eval-only afterEach hook registers that snapshot against the explicit Vitest test task before reporters run.

Harness names must be stable and unique within an eval set. The grouping key combines repetition with a non-empty string input.id when available, otherwise with a SHA-256 hash of strict canonical JSON input. Use candidate for one treatment or candidates for multiple treatments. Each candidate is compared only with the declared baseline. For each matched input and repetition, the reporter computes pass-rate lift from each run's recorded average judge score, treating a score of at least 1 as passing. Lift is the candidate pass rate minus the baseline pass rate, in percentage points. Missing judge scores are reported as incomplete observations. Tokens, latency, and estimated cost remain separate candidate-minus-baseline paired deltas; missing telemetry remains unavailable. If execution-order randomization becomes necessary, use Vitest's built-in sequence shuffling.

See the skill-eval-harness guidance for comparative-eval methodology, repetition strategy, trustworthy judges, and telemetry interpretation.