mirror of
https://github.com/badlogic/pi-mono.git
synced 2026-11-05 23:37:25 +00:00
fix(coding-agent): remove fixed output limits from summary requests (#11036)
Compaction summaries were limited to 0.8 * compaction.reserveTokens (13107 by default), turn-prefix summaries to 0.5 * reserveTokens, and branch summaries to a hardcoded 4096. On providers where reasoning shares the output limit (OpenAI Responses, adaptive Anthropic), high thinking levels used most of that limit and compaction failed with "generation hit the token cap", even though the model allows far more output. Tying the limit to reserveTokens assumed the summary request is as large as the full context, so its output had to fit in the reserved space. It is usually much smaller: recent messages kept by keepRecentTokens are not sent, a split turn goes to a separate request, tool results are truncated, and tool definitions and the normal system prompt are not included. Manual compaction can also run long before the context is full. The provider layer already sizes output from the actual request, using the model's output limit reduced to fit the remaining context, as for normal turns. Summary requests now set no maxTokens and rely on that. compaction.reserveTokens only controls when compaction starts. generateSummary() and generateSummaryWithUsage() now ignore the reserveTokens parameter. closes #9512, closes #9075, closes #10908
This commit is contained in:
David Brailovsky
GitHub
parent
90d863f7c1
commit
6fcb48670f
9 changed files with 44 additions and 111 deletions
|
|
@ -2,6 +2,14 @@
|
|||
|
||||
## [Unreleased]
|
||||
|
||||
### Changed
|
||||
|
||||
- `generateSummary()` and `generateSummaryWithUsage()` now ignore their `reserveTokens` parameter. Summary requests use the model's output limit instead.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed compaction and branch summaries failing with "generation hit the token cap" when reasoning used most of the fixed summary output limit. Summaries now use the model's output limit, and `compaction.reserveTokens` only controls when compaction starts ([#9512](https://github.com/earendil-works/pi/issues/9512), [#9075](https://github.com/earendil-works/pi/issues/9075)).
|
||||
|
||||
## [1.3.0] - 2026-10-15
|
||||
|
||||
### New Features
|
||||
|
|
|
|||
|
|
@ -32,7 +32,7 @@ Auto-compaction triggers when:
|
|||
contextTokens > contextWindow - reserveTokens
|
||||
```
|
||||
|
||||
By default, `reserveTokens` is 16384 tokens (configurable in `~/.pi/agent/settings.json` or `<project-dir>/.pi/settings.json`). This leaves room for the LLM's response.
|
||||
By default, `reserveTokens` is 16384 tokens (configurable in `~/.pi/agent/settings.json` or `<project-dir>/.pi/settings.json`). It only decides when compaction starts. Summary requests use the model's output limit, reduced when needed so input and output fit the context window.
|
||||
|
||||
During a multi-turn agent run, Pi checks the canonical projected context after tools finish and their results are appended, before starting the next assistant response. If the threshold is crossed, Pi compacts during `prepareNextTurn`, then performs the existing catch-up steering poll before `turn_start`. It skips this between-turn check when the completed tool batch terminates the run and no queued message requires another response. Pi also checks before a new user prompt and performs final-attempt overflow recovery after the low-level run ends.
|
||||
|
||||
|
|
@ -431,7 +431,7 @@ Configure compaction in `~/.pi/agent/settings.json` or `<project-dir>/.pi/settin
|
|||
| Setting | Default | Description |
|
||||
|---------|---------|-------------|
|
||||
| `enabled` | `true` | Enable auto-compaction |
|
||||
| `reserveTokens` | `16384` | Tokens to reserve for LLM response |
|
||||
| `reserveTokens` | `16384` | Free context tokens below which auto-compaction starts |
|
||||
| `keepRecentTokens` | `20000` | Recent tokens to keep (not summarized) |
|
||||
|
||||
Disable auto-compaction with `"enabled": false`. You can still compact manually with `/compact`.
|
||||
|
|
@ -454,7 +454,7 @@ Use `compaction.modelOverrides` to tune token budgets for different models:
|
|||
}
|
||||
```
|
||||
|
||||
For a model with a 1M context window, this override triggers compaction above 600K tokens and keeps the ordinary 20000 recent tokens. Other models retain the ordinary 16384-token reserve. `reserveTokens` also influences summarization output limits, capped by the model's maximum output tokens; it is not solely a trigger threshold.
|
||||
For a model with a 1M context window, this override triggers compaction above 600K tokens and keeps the ordinary 20000 recent tokens. Other models retain the ordinary 16384-token reserve.
|
||||
|
||||
Keys are exact, case-sensitive `provider/modelId` values, including any slashes within the model ID. Each `reserveTokens` and `keepRecentTokens` value falls back independently from the model override to the ordinary setting to the built-in default. Values must be non-negative safe integers. Invalid values in the matching model override produce an error when read; only omitted fields fall back to the ordinary setting. Model override entries must be objects. Invalid ordinary token settings produce an error when read, even if the active model has a valid override. Only omitted ordinary values use built-in defaults. `enabled` remains global, not model-specific.
|
||||
|
||||
|
|
|
|||
|
|
@ -71,7 +71,7 @@ CLI tool options override this setting for one invocation. `--tools` with only `
|
|||
| Setting | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `compaction.enabled` | boolean | `true` | Enable automatic compaction. |
|
||||
| `compaction.reserveTokens` | number | `16384` | Tokens reserved for the model response. |
|
||||
| `compaction.reserveTokens` | number | `16384` | Free context tokens below which automatic compaction starts. |
|
||||
| `compaction.keepRecentTokens` | number | `20000` | Recent tokens retained without summarization. |
|
||||
| `compaction.modelOverrides` | object | None | Per-model token settings keyed by exact `provider/modelId`. |
|
||||
|
||||
|
|
|
|||
|
|
@ -342,14 +342,12 @@ export async function generateBranchSummary(
|
|||
},
|
||||
];
|
||||
|
||||
const maxTokens = Math.min(4096, model.maxTokens > 0 ? model.maxTokens : Number.POSITIVE_INFINITY);
|
||||
|
||||
// Call LLM for summarization. Prefer the session stream function so SDK
|
||||
// request behavior (timeouts, retries, attribution headers) stays consistent
|
||||
// without running through agent state/events. Retried via completeSummarization
|
||||
// so transient stream drops reuse the configured retry policy.
|
||||
const context = normalizeContext({ systemPrompt: SUMMARIZATION_SYSTEM_PROMPT, messages: summarizationMessages });
|
||||
const requestOptions: SimpleStreamOptions = { apiKey, headers, env, signal, maxTokens };
|
||||
const requestOptions: SimpleStreamOptions = { apiKey, headers, env, signal };
|
||||
const response = await completeSummarization(model, context, requestOptions, streamFn, retry, callbacks);
|
||||
|
||||
// Check if aborted or errored
|
||||
|
|
|
|||
|
|
@ -594,7 +594,6 @@ export function getSummarizationFailure(response: AssistantMessage, label: strin
|
|||
|
||||
function createSummarizationOptions(
|
||||
model: Model<any>,
|
||||
maxTokens: number,
|
||||
apiKey: string | undefined,
|
||||
headers: Record<string, string> | undefined,
|
||||
env: Record<string, string> | undefined,
|
||||
|
|
@ -602,7 +601,7 @@ function createSummarizationOptions(
|
|||
thinkingLevel: ThinkingLevel | undefined,
|
||||
sessionId: string | undefined,
|
||||
): SimpleStreamOptions {
|
||||
const options: SimpleStreamOptions = { maxTokens, signal, apiKey, headers, env, sessionId };
|
||||
const options: SimpleStreamOptions = { signal, apiKey, headers, env, sessionId };
|
||||
if (model.reasoning && thinkingLevel && thinkingLevel !== "off") {
|
||||
options.reasoning = thinkingLevel;
|
||||
}
|
||||
|
|
@ -645,7 +644,8 @@ export async function completeSummarization(
|
|||
export async function generateSummary(
|
||||
currentMessages: AgentMessage[],
|
||||
model: Model<any>,
|
||||
reserveTokens: number,
|
||||
// Unused. Kept for existing callers; summaries use the model's output limit (#9512).
|
||||
_reserveTokens: number | undefined,
|
||||
apiKey: string | undefined,
|
||||
headers?: Record<string, string>,
|
||||
signal?: AbortSignal,
|
||||
|
|
@ -662,7 +662,7 @@ export async function generateSummary(
|
|||
await generateSummaryWithUsage(
|
||||
currentMessages,
|
||||
model,
|
||||
reserveTokens,
|
||||
_reserveTokens,
|
||||
apiKey,
|
||||
headers,
|
||||
signal,
|
||||
|
|
@ -696,7 +696,8 @@ function buildSummarizationContext(promptText: string): TranscriptContext {
|
|||
export async function generateSummaryWithUsage(
|
||||
currentMessages: AgentMessage[],
|
||||
model: Model<any>,
|
||||
reserveTokens: number,
|
||||
// Unused. Kept for existing callers; summaries use the model's output limit (#9512).
|
||||
_reserveTokens: number | undefined,
|
||||
apiKey: string | undefined,
|
||||
headers?: Record<string, string>,
|
||||
signal?: AbortSignal,
|
||||
|
|
@ -709,11 +710,6 @@ export async function generateSummaryWithUsage(
|
|||
callbacks?: RetryCallbacks,
|
||||
sessionId?: string,
|
||||
): Promise<{ text: string; usage: Usage }> {
|
||||
const maxTokens = Math.min(
|
||||
Math.floor(0.8 * reserveTokens),
|
||||
model.maxTokens > 0 ? model.maxTokens : Number.POSITIVE_INFINITY,
|
||||
);
|
||||
|
||||
// Use update prompt if we have a previous summary, otherwise initial prompt
|
||||
let basePrompt = previousSummary ? UPDATE_SUMMARIZATION_PROMPT : SUMMARIZATION_PROMPT;
|
||||
if (customInstructions) {
|
||||
|
|
@ -732,16 +728,7 @@ export async function generateSummaryWithUsage(
|
|||
}
|
||||
promptText += basePrompt;
|
||||
|
||||
const completionOptions = createSummarizationOptions(
|
||||
model,
|
||||
maxTokens,
|
||||
apiKey,
|
||||
headers,
|
||||
env,
|
||||
signal,
|
||||
thinkingLevel,
|
||||
sessionId,
|
||||
);
|
||||
const completionOptions = createSummarizationOptions(model, apiKey, headers, env, signal, thinkingLevel, sessionId);
|
||||
|
||||
const response = await completeSummarization(
|
||||
model,
|
||||
|
|
@ -984,7 +971,6 @@ export async function compact(
|
|||
tokensBefore,
|
||||
previousSummary,
|
||||
fileOps,
|
||||
settings,
|
||||
} = preparation;
|
||||
|
||||
// Generate summaries and merge into one
|
||||
|
|
@ -998,7 +984,7 @@ export async function compact(
|
|||
const historyResult = await generateSummaryWithUsage(
|
||||
messagesToSummarize,
|
||||
model,
|
||||
settings.reserveTokens,
|
||||
undefined,
|
||||
apiKey,
|
||||
headers,
|
||||
signal,
|
||||
|
|
@ -1017,7 +1003,6 @@ export async function compact(
|
|||
const turnPrefixResult = await generateTurnPrefixSummary(
|
||||
turnPrefixMessages,
|
||||
model,
|
||||
settings.reserveTokens,
|
||||
apiKey,
|
||||
headers,
|
||||
env,
|
||||
|
|
@ -1036,7 +1021,7 @@ export async function compact(
|
|||
const result = await generateSummaryWithUsage(
|
||||
messagesToSummarize,
|
||||
model,
|
||||
settings.reserveTokens,
|
||||
undefined,
|
||||
apiKey,
|
||||
headers,
|
||||
signal,
|
||||
|
|
@ -1076,7 +1061,6 @@ export async function compact(
|
|||
async function generateTurnPrefixSummary(
|
||||
messages: AgentMessage[],
|
||||
model: Model<any>,
|
||||
reserveTokens: number,
|
||||
apiKey: string | undefined,
|
||||
headers?: Record<string, string>,
|
||||
env?: Record<string, string>,
|
||||
|
|
@ -1087,10 +1071,6 @@ async function generateTurnPrefixSummary(
|
|||
callbacks?: RetryCallbacks,
|
||||
sessionId?: string,
|
||||
): Promise<{ text: string; usage: Usage }> {
|
||||
const maxTokens = Math.min(
|
||||
Math.floor(0.5 * reserveTokens),
|
||||
model.maxTokens > 0 ? model.maxTokens : Number.POSITIVE_INFINITY,
|
||||
); // Smaller budget for turn prefix
|
||||
const llmMessages = convertToLlm(messages);
|
||||
const conversationText = serializeConversation(llmMessages);
|
||||
const promptText = `# Conversation\n${conversationText}\n\n# Instructions\n${TURN_PREFIX_SUMMARIZATION_PROMPT}`;
|
||||
|
|
@ -1098,7 +1078,7 @@ async function generateTurnPrefixSummary(
|
|||
const response = await completeSummarization(
|
||||
model,
|
||||
buildSummarizationContext(promptText),
|
||||
createSummarizationOptions(model, maxTokens, apiKey, headers, env, signal, thinkingLevel, sessionId),
|
||||
createSummarizationOptions(model, apiKey, headers, env, signal, thinkingLevel, sessionId),
|
||||
streamFn,
|
||||
retry,
|
||||
callbacks,
|
||||
|
|
|
|||
|
|
@ -61,30 +61,9 @@ describe("branch summarization", () => {
|
|||
streamFn,
|
||||
});
|
||||
|
||||
expect(requestOptions?.maxTokens).toBe(4096);
|
||||
expect(requestOptions?.toolChoice).toBeUndefined();
|
||||
});
|
||||
|
||||
it("clamps the branch summary output cap to the model limit", async () => {
|
||||
let requestOptions: SimpleStreamOptions | undefined;
|
||||
const streamFn: StreamFn = (_model, _context, options) => {
|
||||
requestOptions = options;
|
||||
const stream = createAssistantMessageEventStream();
|
||||
queueMicrotask(() =>
|
||||
stream.push({ type: "done", reason: "stop", message: response([{ type: "text", text: "summary" }]) }),
|
||||
);
|
||||
return stream;
|
||||
};
|
||||
|
||||
await generateBranchSummary(entries, {
|
||||
model: { ...model, maxTokens: 1024 },
|
||||
signal: new AbortController().signal,
|
||||
streamFn,
|
||||
});
|
||||
|
||||
expect(requestOptions?.maxTokens).toBe(1024);
|
||||
});
|
||||
|
||||
it("rejects tool calls from branch summaries", async () => {
|
||||
const streamFn: StreamFn = () => {
|
||||
const stream = createAssistantMessageEventStream();
|
||||
|
|
|
|||
|
|
@ -77,7 +77,7 @@ describe("generateSummary reasoning options", () => {
|
|||
const result = await generateSummaryWithUsage(
|
||||
messages,
|
||||
createModel(true),
|
||||
2000,
|
||||
undefined,
|
||||
"test-key",
|
||||
undefined,
|
||||
undefined,
|
||||
|
|
@ -97,14 +97,14 @@ describe("generateSummary reasoning options", () => {
|
|||
});
|
||||
|
||||
it("preserves the string result from generateSummary", async () => {
|
||||
await expect(generateSummary(messages, createModel(false), 2000, "test-key")).resolves.toBe(
|
||||
await expect(generateSummary(messages, createModel(false), undefined, "test-key")).resolves.toBe(
|
||||
"## Goal\nTest summary",
|
||||
);
|
||||
});
|
||||
|
||||
it("uses fresh routing sessions without prompt caching", async () => {
|
||||
await generateSummary(messages, createModel(false), 2000, "test-key");
|
||||
await generateSummary(messages, createModel(false), 2000, "test-key");
|
||||
await generateSummary(messages, createModel(false), undefined, "test-key");
|
||||
await generateSummary(messages, createModel(false), undefined, "test-key");
|
||||
|
||||
const requestOptions = completeSimpleMock.mock.calls.map((call) => call[2]);
|
||||
expect(requestOptions).toHaveLength(2);
|
||||
|
|
@ -154,7 +154,7 @@ describe("generateSummary reasoning options", () => {
|
|||
it("rejects tool calls from conversation summaries", async () => {
|
||||
completeSimpleMock.mockResolvedValueOnce(mockToolCallResponse);
|
||||
|
||||
await expect(generateSummaryWithUsage(messages, createModel(false), 2000, "test-key")).rejects.toThrow(
|
||||
await expect(generateSummaryWithUsage(messages, createModel(false), undefined, "test-key")).rejects.toThrow(
|
||||
"Summarization attempted to call a tool",
|
||||
);
|
||||
});
|
||||
|
|
@ -183,7 +183,7 @@ describe("generateSummary reasoning options", () => {
|
|||
content: [{ type: "text", text: "partial" }],
|
||||
});
|
||||
|
||||
await expect(generateSummaryWithUsage(messages, createModel(false), 2000, "test-key")).rejects.toThrow(
|
||||
await expect(generateSummaryWithUsage(messages, createModel(false), undefined, "test-key")).rejects.toThrow(
|
||||
"generation hit the token cap",
|
||||
);
|
||||
});
|
||||
|
|
@ -213,7 +213,7 @@ describe("generateSummary reasoning options", () => {
|
|||
await generateSummary(
|
||||
messages,
|
||||
createModel(true),
|
||||
2000,
|
||||
undefined,
|
||||
"test-key",
|
||||
undefined,
|
||||
undefined,
|
||||
|
|
@ -233,7 +233,7 @@ describe("generateSummary reasoning options", () => {
|
|||
await generateSummary(
|
||||
messages,
|
||||
createModel(false),
|
||||
2000,
|
||||
undefined,
|
||||
"test-key",
|
||||
undefined,
|
||||
undefined,
|
||||
|
|
@ -261,7 +261,7 @@ describe("generateSummary reasoning options", () => {
|
|||
},
|
||||
],
|
||||
}),
|
||||
2000,
|
||||
undefined,
|
||||
"test-key",
|
||||
);
|
||||
|
||||
|
|
@ -270,13 +270,13 @@ describe("generateSummary reasoning options", () => {
|
|||
});
|
||||
|
||||
it("does not set Anthropic refusal fallback for models without allowed fallback targets", async () => {
|
||||
await generateSummary(messages, createModel(true), 2000, "test-key");
|
||||
await generateSummary(messages, createModel(true), undefined, "test-key");
|
||||
|
||||
expect(completeSimpleMock).toHaveBeenCalledTimes(1);
|
||||
expect(completeSimpleMock.mock.calls[0][2]).not.toHaveProperty("refusalFallbacks");
|
||||
});
|
||||
|
||||
it("clamps compaction summary maxTokens to the model output cap", async () => {
|
||||
it("combines usage from split-turn summaries", async () => {
|
||||
const preparation: CompactionPreparation = {
|
||||
firstKeptEntryId: "entry-keep",
|
||||
messagesToSummarize: messages,
|
||||
|
|
@ -284,10 +284,10 @@ describe("generateSummary reasoning options", () => {
|
|||
isSplitTurn: true,
|
||||
tokensBefore: 600000,
|
||||
fileOps: { read: new Set(), written: new Set(), edited: new Set() },
|
||||
settings: { enabled: true, reserveTokens: 500000, keepRecentTokens: 20000 },
|
||||
settings: { enabled: true, reserveTokens: 16384, keepRecentTokens: 20000 },
|
||||
};
|
||||
|
||||
const result = await compact(preparation, createModel(false, 128000), "test-key");
|
||||
const result = await compact(preparation, createModel(false), "test-key");
|
||||
|
||||
expect(result.usage).toEqual({
|
||||
...mockSummaryResponse.usage,
|
||||
|
|
@ -296,6 +296,5 @@ describe("generateSummary reasoning options", () => {
|
|||
totalTokens: 40,
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 },
|
||||
});
|
||||
expect(completeSimpleMock.mock.calls.map((call) => call[2]?.maxTokens)).toEqual([128000, 128000]);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -102,37 +102,6 @@ describe("AgentSession compaction model overrides", () => {
|
|||
},
|
||||
);
|
||||
|
||||
it.each(["manual", "automatic"] as const)("passes resolved budgets to built-in %s summarization", async (path) => {
|
||||
const harness = await createHarness({
|
||||
models: [{ id: "faux-1", contextWindow: 4000, maxTokens: 3000 }],
|
||||
tools: [],
|
||||
settings: {
|
||||
compaction: {
|
||||
reserveTokens: 10,
|
||||
modelOverrides: { "faux/faux-1": { reserveTokens: 2000, keepRecentTokens: 150 } },
|
||||
},
|
||||
},
|
||||
});
|
||||
harnesses.push(harness);
|
||||
const recentUserId = seedHistory(harness, 2500);
|
||||
const budgets: Array<number | undefined> = [];
|
||||
harness.setResponses([
|
||||
(_context, options) => {
|
||||
budgets.push(options?.maxTokens);
|
||||
return fauxAssistantMessage("built-in summary");
|
||||
},
|
||||
...(path === "automatic" ? [fauxAssistantMessage("done")] : []),
|
||||
]);
|
||||
if (path === "manual") await harness.session.compact();
|
||||
else await harness.session.prompt("continue");
|
||||
expect(budgets).toEqual([1600]);
|
||||
expect(harness.sessionManager.getEntries().find((entry) => entry.type === "compaction")).toMatchObject({
|
||||
firstKeptEntryId: recentUserId,
|
||||
summary: "built-in summary",
|
||||
});
|
||||
expect(harness.getPendingResponseCount()).toBe(0);
|
||||
});
|
||||
|
||||
it("uses the newly selected model without changing ordinary settings", async () => {
|
||||
const harness = await createHarness({
|
||||
models: [
|
||||
|
|
@ -193,14 +162,14 @@ describe("AgentSession compaction model overrides", () => {
|
|||
harness.session.agent.state.model = harness.getModel("second")!;
|
||||
return getAuth(model, ...args);
|
||||
});
|
||||
const requests: Array<{ id: string; maxTokens: number | undefined }> = [];
|
||||
const requests: string[] = [];
|
||||
harness.setResponses([
|
||||
(_context, options, _state, model) => {
|
||||
requests.push({ id: model.id, maxTokens: options?.maxTokens });
|
||||
(_context, _options, _state, model) => {
|
||||
requests.push(model.id);
|
||||
return fauxAssistantMessage("summary");
|
||||
},
|
||||
]);
|
||||
await harness.session.compact();
|
||||
expect(requests).toEqual([{ id: "first", maxTokens: 1600 }]);
|
||||
expect(requests).toEqual(["first"]);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -368,13 +368,13 @@ describe("AgentSession virtual models", () => {
|
|||
expect(reasons()).toEqual(["user", "user"]);
|
||||
});
|
||||
|
||||
it("routes compaction summaries before sizing them", async () => {
|
||||
it("routes compaction summaries to the resolved model", async () => {
|
||||
const summaries: string[] = [];
|
||||
const { harness, reasons } = await createRoutedHarness(defaultRoute, {
|
||||
settings: { compaction: { keepRecentTokens: 1 } },
|
||||
});
|
||||
const summary: FauxResponseStep = (_context, options, _state, model) => {
|
||||
summaries.push(`${model.id}:${options?.reasoning ?? "off"}:${options?.maxTokens}`);
|
||||
summaries.push(`${model.id}:${options?.reasoning ?? "off"}`);
|
||||
return fauxAssistantMessage("summary");
|
||||
};
|
||||
// Compaction summarizes the history and the split turn prefix with one routed model.
|
||||
|
|
@ -391,7 +391,7 @@ describe("AgentSession virtual models", () => {
|
|||
|
||||
expect(result.summary).toContain("summary");
|
||||
expect(reasons()).toEqual(["user", "user", "direct"]);
|
||||
// The router's thinking level applies, and the output budget respects the large model's 4000 tokens.
|
||||
expect(summaries).toEqual(["large:low:4000", "large:low:4000"]);
|
||||
// The router's model and thinking level apply. The provider layer sizes output from that model.
|
||||
expect(summaries).toEqual(["large:low", "large:low"]);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue