mirror of
https://github.com/badlogic/pi-mono.git
synced 2026-11-05 23:37:25 +00:00
feat(ai,coding-agent): image input for System One classifiers
Jev (TypeSafe, Workers AI), Clef (Workers AI), and OpenRouter decision models now accept ClassifierContext.images. Each service takes its own wire form: TypeSafe assets with the opt-in header, Clef images data URLs, or OpenRouter image_url parts in the state. A new SystemOneCompat (imageInput, sendImageOptInHeader) selects the form; catalog entries set it explicitly and models.json provider compat now applies to classifiers. Adds Clef Omni and Jev 1.14 on Workers AI and fixes Clef Flash and Workers AI Jev catalog pricing.
This commit is contained in:
Armin Ronacher
parent
02726086d1
commit
7f225f25d8
19 changed files with 687 additions and 36 deletions
|
|
@ -17,6 +17,9 @@
|
|||
- Failed model request spans report `provider_error` status with up to 500 error-message characters; cancelled requests keep `ok` status with `stop: "aborted"`.
|
||||
- Added `MiniMax-M3.1-Flash` to the `minimax` and `minimax-cn` providers, with thinking levels low through max sent as `output_config.effort`. Thinking cannot be turned off for this model. New MiniMax models from models.dev now appear without a manual allowlist entry; only the older M2, M2.1, M2.5, and M2.5-highspeed models stay hidden ([#11121](https://github.com/earendil-works/pi/issues/11121))
|
||||
- Sign in with ChatGPT: ChatGPT Business workspace usage-limit errors are not retried and tell the user to ask their workspace admin to increase the limit.
|
||||
- Added image input to System One classifiers: `ClassifierContext.images` now works with Jev on `typesafe` (experimental there, opted in with the `TypeSafe-Danger-Zone-Images` header) and `cloudflare-workers-ai`, with Clef on `cloudflare-workers-ai`, and with OpenRouter decision models whose catalog lists image input. Each service gets its own wire form: TypeSafe `assets`, Clef `images` data URLs, or OpenRouter `image_url` parts in the state.
|
||||
- Added `compat` to System One classifier models (`SystemOneCompat`, with `imageInput` and `sendImageOptInHeader`). Built-in catalog entries set it explicitly; models without it detect the form from the provider, `baseUrl`, and model ID.
|
||||
- Added the `@cf/cloudflare/clef-omni` and `typesafe/jev-1.14.0` classifier models to `cloudflare-workers-ai`.
|
||||
|
||||
### Changed
|
||||
|
||||
|
|
@ -24,6 +27,7 @@
|
|||
|
||||
### Fixed
|
||||
|
||||
- Fixed Cloudflare Workers AI classifier catalog data: Clef Flash now costs $0.038 per million input tokens with a 24,576-token context window, and Jev costs $0.042 per million input tokens instead of being listed as free.
|
||||
- Fixed whitespace-only tool output (for example a bare `\r\n` from bash) being sent as-is to OpenAI Completions, OpenAI Responses, and Google providers; it now uses the same placeholder as empty output, since some providers reject whitespace-only tool content with a 400 ([#8720](https://github.com/earendil-works/pi/issues/8720))
|
||||
- Fixed automatic retry not triggering for gateway upstream failures such as `upstream_error: Upstream request failed`, `upstream_stream_read_error`, and `Upstream HTTP/2 stream failed` ([#10920](https://github.com/earendil-works/pi/issues/10920))
|
||||
- Fixed automatic retry not triggering for `stream disconnected before completion` errors relayed by proxies that wrap the Codex CLI ([#11057](https://github.com/earendil-works/pi/issues/11057))
|
||||
|
|
|
|||
|
|
@ -895,8 +895,8 @@ Classifier models consume structured JSON state, and optionally images, and answ
|
|||
| Provider | Model IDs | Auth |
|
||||
| --- | --- | --- |
|
||||
| `typesafe` | `jev-latest` | `TYPESAFE_API_KEY` |
|
||||
| `openrouter` | `typesafe/jev-1.13`, `~typesafe/jev-latest` | `OPENROUTER_API_KEY` or OpenRouter OAuth |
|
||||
| `cloudflare-workers-ai` | `typesafe/jev`, `@cf/cloudflare/clef`, `@cf/cloudflare/clef-flash` | `CLOUDFLARE_API_KEY` and `CLOUDFLARE_ACCOUNT_ID` |
|
||||
| `openrouter` | `typesafe/jev-1.13`, `typesafe/jev-1.14`, `~typesafe/jev-latest`, `cloudflare/clef`, `cloudflare/clef-flash`, `cloudflare/clef-omni` | `OPENROUTER_API_KEY` or OpenRouter OAuth |
|
||||
| `cloudflare-workers-ai` | `typesafe/jev`, `typesafe/jev-1.14.0`, `@cf/cloudflare/clef`, `@cf/cloudflare/clef-flash`, `@cf/cloudflare/clef-omni` | `CLOUDFLARE_API_KEY` and `CLOUDFLARE_ACCOUNT_ID` |
|
||||
| `vercel-ai-gateway` | `typesafe-ai/jev` | `AI_GATEWAY_API_KEY` |
|
||||
| `opencode` | `jev-1.13`, `jev-1.13-free` | `OPENCODE_API_KEY` |
|
||||
| `openai` | `gpt-6-luna` | `OPENAI_API_KEY` |
|
||||
|
|
@ -941,6 +941,18 @@ When the service reports token counts, `result.usage` carries them with their co
|
|||
|
||||
`ClassifierContext.images` holds image blocks (`{ type: "image", data, mimeType }`) judged together with the state. Only models whose `input` includes `"image"` accept them; `classify()` returns an error result for other models.
|
||||
|
||||
### System One images
|
||||
|
||||
System One services take images in three forms. The model's `compat.imageInput` chooses the form (`"assets"`, `"images"`, or `"state-image-urls"`), and `compat.sendImageOptInHeader` controls TypeSafe's opt-in header:
|
||||
|
||||
- **TypeSafe** (`typesafe`, and Jev on Workers AI) takes PNG and JPEG images as `assets`, `{ "<image-1>": { type: "image", format: "png", data_base64 } }`. Every asset must be referenced from the request, so the state is sent as `["<image-1>", ..., state]`. Images are experimental on TypeSafe's own API and need the `TypeSafe-Danger-Zone-Images: true` header, which the API sends with requests that carry images.
|
||||
- **Clef on Workers AI** takes up to four PNG, JPEG, or WebP images as an `images` list of data URLs, next to the unchanged state.
|
||||
- **OpenRouter** takes images as `{ type: "image_url", image_url: { url } }` parts before the state: `[part, ..., state]`. It answers requests with `assets` but drops the images, without an error.
|
||||
|
||||
Built-in catalog entries set `compat` explicitly, so they keep their form when `baseUrl` points at a proxy. For models without it, the APIs detect the form: OpenRouter by provider `openrouter` or a `baseUrl` containing `openrouter.ai`, Clef by a model ID starting with `@cf/cloudflare/clef`, and TypeSafe's form with the opt-in header otherwise. A custom provider that reaches OpenRouter through a proxy matches neither check and must set `compat.imageInput: "state-image-urls"` and `compat.sendImageOptInHeader: false`; otherwise OpenRouter drops its images.
|
||||
|
||||
Vercel AI Gateway and OpenCode list their Jev models with text input only.
|
||||
|
||||
### OpenAI Decisions
|
||||
|
||||
The `openai-decisions` API calls OpenAI's [Decisions API](https://developers.openai.com/api/docs/guides/decisions) (`POST /v1/decisions`). GPT-6 Luna is its only model. The state is sent as JSON text; with images, the input is one user message with the state followed by the images as data URLs (at most 128). `choice` and `score` questions map to the Decisions types of the same name. `bool` questions become `predicate` questions; predicates have no criteria field, so the meanings of true and false are appended to the instructions. If OpenAI refuses to answer a question, the whole result is an error. Usage reports input tokens only, priced at the catalog input rate with OpenAI's long-context multiplier.
|
||||
|
|
|
|||
|
|
@ -2631,11 +2631,15 @@ async function loadModelsDevClassifierModels(): Promise<ClassifierModel<"typesaf
|
|||
api: "typesafe-system-one",
|
||||
provider: "typesafe",
|
||||
baseUrl: "https://api.typesafe.ai/v1/",
|
||||
input: metadata.modalities?.input?.includes("image") ? ["text", "image"] : ["text"],
|
||||
// models.dev lists text input only, but the API accepts image assets as an experimental
|
||||
// feature behind an opt-in header, which the typesafe-system-one API sends.
|
||||
// https://github.com/typesafe-ai/typesafe-sdk-js (v0.7.0 changelog)
|
||||
input: ["text", "image"],
|
||||
// The canonical models.dev entry has no direct-provider pricing. System One reports token usage,
|
||||
// so classify() results carry token counts but price them at zero.
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: metadata.limit?.context || 64000,
|
||||
compat: { imageInput: "assets", sendImageOptInHeader: true },
|
||||
},
|
||||
];
|
||||
} catch (error) {
|
||||
|
|
@ -2645,9 +2649,6 @@ async function loadModelsDevClassifierModels(): Promise<ClassifierModel<"typesaf
|
|||
}
|
||||
}
|
||||
|
||||
// Workers AI has no unauthenticated catalog and models.dev does not list its
|
||||
// System One models yet. Cloudflare publishes pricing only in the dashboard.
|
||||
// https://developers.cloudflare.com/ai/models/typesafe/jev/
|
||||
// OpenCode Zen serves Jev through its TypeSafe-compatible System One endpoint.
|
||||
// Neither its /inference/v1/models listing nor models.dev carries metadata for it.
|
||||
// https://opencode.ai/docs/zen
|
||||
|
|
@ -2676,11 +2677,14 @@ const OPENCODE_CLASSIFIER_MODELS: ClassifierModel<"typesafe-system-one">[] = [
|
|||
},
|
||||
];
|
||||
|
||||
// Workers AI has no unauthenticated catalog, so its System One models are listed here from
|
||||
// Cloudflare's model pages.
|
||||
const CLOUDFLARE_WORKERS_AI_CLASSIFIER_MODELS: ClassifierModel<"cloudflare-workers-ai-system-one">[] = [
|
||||
// Cloudflare-hosted Clef decision models. They accept images, but classifier
|
||||
// contexts carry text or JSON state only, so the catalog advertises text.
|
||||
// Pricing: https://developers.cloudflare.com/workers-ai/models/clef/
|
||||
// and https://developers.cloudflare.com/workers-ai/models/clef-flash/
|
||||
// Cloudflare-hosted Clef decision models. They take up to four PNG, JPEG, or WebP images as data URLs.
|
||||
// Clef Omni also takes audio and video, which classifier contexts do not carry.
|
||||
// Pricing and context: https://developers.cloudflare.com/workers-ai/models/clef/,
|
||||
// https://developers.cloudflare.com/workers-ai/models/clef-flash/,
|
||||
// and https://developers.cloudflare.com/workers-ai/models/clef-omni/
|
||||
{
|
||||
type: "classifier",
|
||||
id: "@cf/cloudflare/clef",
|
||||
|
|
@ -2688,9 +2692,10 @@ const CLOUDFLARE_WORKERS_AI_CLASSIFIER_MODELS: ClassifierModel<"cloudflare-worke
|
|||
api: "cloudflare-workers-ai-system-one",
|
||||
provider: "cloudflare-workers-ai",
|
||||
baseUrl: CLOUDFLARE_WORKERS_AI_REST_BASE_URL,
|
||||
input: ["text"],
|
||||
input: ["text", "image"],
|
||||
cost: { input: 0.24, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 65536,
|
||||
compat: { imageInput: "images" },
|
||||
},
|
||||
{
|
||||
type: "classifier",
|
||||
|
|
@ -2699,10 +2704,26 @@ const CLOUDFLARE_WORKERS_AI_CLASSIFIER_MODELS: ClassifierModel<"cloudflare-worke
|
|||
api: "cloudflare-workers-ai-system-one",
|
||||
provider: "cloudflare-workers-ai",
|
||||
baseUrl: CLOUDFLARE_WORKERS_AI_REST_BASE_URL,
|
||||
input: ["text"],
|
||||
cost: { input: 0.09, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 65536,
|
||||
input: ["text", "image"],
|
||||
cost: { input: 0.038, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 24576,
|
||||
compat: { imageInput: "images" },
|
||||
},
|
||||
{
|
||||
type: "classifier",
|
||||
id: "@cf/cloudflare/clef-omni",
|
||||
name: "Clef Omni",
|
||||
api: "cloudflare-workers-ai-system-one",
|
||||
provider: "cloudflare-workers-ai",
|
||||
baseUrl: CLOUDFLARE_WORKERS_AI_REST_BASE_URL,
|
||||
input: ["text", "image"],
|
||||
cost: { input: 0.15, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 64000,
|
||||
compat: { imageInput: "images" },
|
||||
},
|
||||
// TypeSafe's Jev models take PNG and JPEG images as System One assets.
|
||||
// https://developers.cloudflare.com/ai/models/typesafe/jev/ (jev-latest)
|
||||
// and https://developers.cloudflare.com/ai/models/typesafe/jev-1.14.0/
|
||||
{
|
||||
type: "classifier",
|
||||
id: "typesafe/jev",
|
||||
|
|
@ -2710,9 +2731,22 @@ const CLOUDFLARE_WORKERS_AI_CLASSIFIER_MODELS: ClassifierModel<"cloudflare-worke
|
|||
api: "cloudflare-workers-ai-system-one",
|
||||
provider: "cloudflare-workers-ai",
|
||||
baseUrl: CLOUDFLARE_WORKERS_AI_REST_BASE_URL,
|
||||
input: ["text"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
input: ["text", "image"],
|
||||
cost: { input: 0.042, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 32000,
|
||||
compat: { imageInput: "assets" },
|
||||
},
|
||||
{
|
||||
type: "classifier",
|
||||
id: "typesafe/jev-1.14.0",
|
||||
name: "Jev 1.14",
|
||||
api: "cloudflare-workers-ai-system-one",
|
||||
provider: "cloudflare-workers-ai",
|
||||
baseUrl: CLOUDFLARE_WORKERS_AI_REST_BASE_URL,
|
||||
input: ["text", "image"],
|
||||
cost: { input: 0.042, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 32000,
|
||||
compat: { imageInput: "assets" },
|
||||
},
|
||||
];
|
||||
|
||||
|
|
|
|||
|
|
@ -98,6 +98,8 @@ export function buildOpenRouterCatalog(
|
|||
input: input.length > 0 ? input : ["text"],
|
||||
cost: openRouterCost(model.pricing),
|
||||
contextWindow: model.top_provider?.context_length || model.context_length || 4096,
|
||||
// OpenRouter answers TypeSafe `assets` with 200 but drops the images.
|
||||
compat: { imageInput: "state-image-urls", sendImageOptInHeader: false },
|
||||
});
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -23,10 +23,18 @@ function cloudflareErrorMessage(errors: unknown): string {
|
|||
* Cloudflare-hosted models such as `@cf/cloudflare/clef` return the output directly:
|
||||
* `{ success, result: { model, answers, usage } }`.
|
||||
* https://developers.cloudflare.com/workers-ai/models/clef/
|
||||
*
|
||||
* Clef models take images as an `images` list of data URLs. TypeSafe's models take them as
|
||||
* System One `assets`, without TypeSafe's opt-in header. The catalog sets `compat.imageInput`;
|
||||
* the model ID check only covers models without it.
|
||||
*/
|
||||
const transport: SystemOneTransport = {
|
||||
api: "cloudflare-workers-ai-system-one",
|
||||
label: LABEL,
|
||||
detectCompat: (model) => ({
|
||||
imageInput: model.id.startsWith("@cf/cloudflare/clef") ? "images" : "assets",
|
||||
sendImageOptInHeader: false,
|
||||
}),
|
||||
url: (model) => new URL("run", `${model.baseUrl.replace(/\/+$/u, "")}/`),
|
||||
payload: (model, request) => ({ model: model.id, input: request }),
|
||||
output: (body) => {
|
||||
|
|
|
|||
|
|
@ -5,22 +5,62 @@ import type {
|
|||
ClassifierModel,
|
||||
ClassifierOptions,
|
||||
ClassifierResult,
|
||||
ImageContent,
|
||||
SystemOneCompat,
|
||||
} from "../types.ts";
|
||||
import { formatProviderError, normalizeProviderError } from "../utils/error-body.ts";
|
||||
import { isRecord, parseClassifierUsage, postClassifierRequest, requiredNumber } from "./classifier-shared.ts";
|
||||
|
||||
/** A System One image asset: base64 file bytes, referenced by its name from the state. */
|
||||
interface SystemOneImageAsset {
|
||||
type: "image";
|
||||
format: "png" | "jpeg";
|
||||
data_base64: string;
|
||||
}
|
||||
|
||||
/** An image in the state, OpenRouter's form. */
|
||||
interface SystemOneImageUrlPart {
|
||||
type: "image_url";
|
||||
image_url: { url: string };
|
||||
}
|
||||
|
||||
/** TypeSafe System One request body without the transport-specific envelope. */
|
||||
export interface SystemOneWireRequest {
|
||||
state: ClassifierContext["state"];
|
||||
/** The context state, or with images a list of the images or their asset names followed by the state. */
|
||||
state: unknown;
|
||||
questions: Record<string, unknown>;
|
||||
/** Images as System One assets, keyed by the names the state references. */
|
||||
assets?: Record<string, SystemOneImageAsset>;
|
||||
/** Images as data URLs, Clef's extension to System One. */
|
||||
images?: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* How a service takes images (`SystemOneCompat.imageInput`):
|
||||
* - `assets`: TypeSafe's `assets` map, `{ "<image-1>": { type: "image", format, data_base64 } }`,
|
||||
* with the names placed before the state: `state: ["<image-1>", state]`. PNG and JPEG only.
|
||||
* - `images`: Clef's `images` list of data URLs, which the model reads before the state.
|
||||
* PNG, JPEG, and WebP, at most four.
|
||||
* - `state-image-urls`: OpenRouter's form, data URL parts placed before the state:
|
||||
* `state: [{ type: "image_url", image_url: { url } }, state]`. OpenRouter accepts this for every
|
||||
* model, and drops `assets` silently.
|
||||
*/
|
||||
export type SystemOneImageInput = NonNullable<SystemOneCompat["imageInput"]>;
|
||||
|
||||
/** System One compatibility settings with every field resolved. */
|
||||
export type ResolvedSystemOneCompat = Required<SystemOneCompat>;
|
||||
|
||||
/** TypeSafe's opt-in header for its experimental image input. */
|
||||
const IMAGE_OPT_IN_HEADERS = { "TypeSafe-Danger-Zone-Images": "true" };
|
||||
|
||||
/** Differences between services that serve System One models. */
|
||||
export interface SystemOneTransport {
|
||||
/** Classifier API implemented by this transport. */
|
||||
api: ClassifierApi;
|
||||
/** Service name used in error messages. */
|
||||
label: string;
|
||||
/** Compatibility settings for models that do not set them in `compat`. */
|
||||
detectCompat(model: ClassifierModel<ClassifierApi>): ResolvedSystemOneCompat;
|
||||
/** Absolute request URL. */
|
||||
url(model: ClassifierModel<ClassifierApi>): URL;
|
||||
/** Wraps the System One request in the service's request envelope. */
|
||||
|
|
@ -82,6 +122,56 @@ function parseAnswers(label: string, value: unknown, context: ClassifierContext)
|
|||
return Object.fromEntries(answers);
|
||||
}
|
||||
|
||||
const ASSET_FORMATS: Record<string, SystemOneImageAsset["format"]> = { "image/png": "png", "image/jpeg": "jpeg" };
|
||||
const DATA_URL_TYPES = new Set(["image/png", "image/jpeg", "image/webp"]);
|
||||
/** Clef accepts at most this many images per request. */
|
||||
const MAX_DATA_URL_IMAGES = 4;
|
||||
|
||||
/** Adds the context images to `request` in the service's form. */
|
||||
function addImages(
|
||||
label: string,
|
||||
model: ClassifierModel<ClassifierApi>,
|
||||
input: SystemOneImageInput,
|
||||
images: ImageContent[],
|
||||
request: SystemOneWireRequest,
|
||||
): void {
|
||||
if (input === "images") {
|
||||
if (images.length > MAX_DATA_URL_IMAGES) {
|
||||
throw new Error(
|
||||
`${label} accepts at most ${MAX_DATA_URL_IMAGES} images for ${model.id}, got ${images.length}`,
|
||||
);
|
||||
}
|
||||
request.images = images.map((image) => {
|
||||
if (!DATA_URL_TYPES.has(image.mimeType)) {
|
||||
throw new Error(`${label} accepts PNG, JPEG, and WebP images for ${model.id}, got ${image.mimeType}`);
|
||||
}
|
||||
return `data:${image.mimeType};base64,${image.data}`;
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (input === "state-image-urls") {
|
||||
// OpenRouter validates the images against the model that serves the request.
|
||||
const parts = images.map(
|
||||
(image): SystemOneImageUrlPart => ({
|
||||
type: "image_url",
|
||||
image_url: { url: `data:${image.mimeType};base64,${image.data}` },
|
||||
}),
|
||||
);
|
||||
request.state = [...parts, request.state];
|
||||
return;
|
||||
}
|
||||
const assets: Record<string, SystemOneImageAsset> = {};
|
||||
images.forEach((image, index) => {
|
||||
const format = ASSET_FORMATS[image.mimeType];
|
||||
if (!format) throw new Error(`${label} accepts PNG and JPEG images for ${model.id}, got ${image.mimeType}`);
|
||||
assets[`<image-${index + 1}>`] = { type: "image", format, data_base64: image.data };
|
||||
});
|
||||
request.assets = assets;
|
||||
// Every asset must be referenced from the request. The names go before the state, where Clef
|
||||
// also places its images.
|
||||
request.state = [...Object.keys(assets), request.state];
|
||||
}
|
||||
|
||||
/** Maps public `bool` questions to TypeSafe's wire-level `noul` type. */
|
||||
function wireRequest(context: ClassifierContext): SystemOneWireRequest {
|
||||
return {
|
||||
|
|
@ -113,13 +203,25 @@ export async function classifySystemOne(
|
|||
|
||||
try {
|
||||
if (model.api !== transport.api) throw new Error(`Unsupported classifier API: ${model.api}`);
|
||||
if (context.images?.length) throw new Error(`${transport.label} does not support image input`);
|
||||
const request = wireRequest(context);
|
||||
let requestOptions = options;
|
||||
if (context.images?.length) {
|
||||
if (!model.input.includes("image")) {
|
||||
throw new Error(`${transport.label} does not support image input for ${model.id}`);
|
||||
}
|
||||
const detected = transport.detectCompat(model);
|
||||
const imageInput = model.compat?.imageInput ?? detected.imageInput;
|
||||
addImages(transport.label, model, imageInput, context.images, request);
|
||||
if (model.compat?.sendImageOptInHeader ?? detected.sendImageOptInHeader) {
|
||||
requestOptions = { ...options, headers: { ...IMAGE_OPT_IN_HEADERS, ...options?.headers } };
|
||||
}
|
||||
}
|
||||
const body = await postClassifierRequest(
|
||||
transport.label,
|
||||
transport.url(model),
|
||||
model,
|
||||
transport.payload(model, wireRequest(context)),
|
||||
options,
|
||||
transport.payload(model, request),
|
||||
requestOptions,
|
||||
);
|
||||
const result = transport.output(body);
|
||||
// Set before parsing answers: a request with malformed answers was still billed.
|
||||
|
|
|
|||
|
|
@ -5,10 +5,21 @@ import { classifySystemOne, type SystemOneTransport } from "./system-one-shared.
|
|||
/**
|
||||
* TypeSafe's native System One protocol. OpenRouter serves the same protocol,
|
||||
* so both providers use this API with different base URLs.
|
||||
*
|
||||
* TypeSafe takes images as `assets`. They are experimental and need the
|
||||
* `TypeSafe-Danger-Zone-Images` opt-in header; without it the API answers 400.
|
||||
* OpenRouter drops `assets` silently and takes images as `image_url` parts in the state instead.
|
||||
* Models behind a proxy that hides both the provider ID and the OpenRouter host must set
|
||||
* `compat.imageInput`.
|
||||
*/
|
||||
const transport: SystemOneTransport = {
|
||||
api: "typesafe-system-one",
|
||||
label: "System One API",
|
||||
// OpenRouter by provider ID, which survives baseUrl overrides, or by host, which covers regional endpoints.
|
||||
detectCompat: (model) =>
|
||||
model.provider === "openrouter" || model.baseUrl.includes("openrouter.ai")
|
||||
? { imageInput: "state-image-urls", sendImageOptInHeader: false }
|
||||
: { imageInput: "assets", sendImageOptInHeader: true },
|
||||
url: (model) => new URL("systemone", `${model.baseUrl.replace(/\/+$/u, "")}/`),
|
||||
payload: (model, request) => ({ model: model.id, ...request }),
|
||||
output: (body) => {
|
||||
|
|
|
|||
|
|
@ -521,6 +521,22 @@ export const MistralConversationsCompatSchema = Type.Object(
|
|||
{ description: "Compatibility settings for the Mistral chat API.", additionalProperties: true },
|
||||
);
|
||||
|
||||
export const SystemOneCompatSchema = Type.Object(
|
||||
{
|
||||
imageInput: Type.Optional(
|
||||
Type.Union([Type.Literal("assets"), Type.Literal("images"), Type.Literal("state-image-urls")], {
|
||||
description:
|
||||
'How the service takes classifier images. "assets" sends TypeSafe\'s assets map (PNG and JPEG) and references the images from the state, as TypeSafe and Jev on Workers AI expect. "images" sends Clef\'s images list of data URLs (PNG, JPEG, and WebP, at most four), as Clef on Workers AI expects. "state-image-urls" places image_url parts before the state, as OpenRouter expects; OpenRouter drops assets without an error. Auto-detected from the provider, baseUrl, and model ID. Classifier models read it from the provider-level compat; models and modelOverrides entries apply to chat models only.',
|
||||
}),
|
||||
),
|
||||
sendImageOptInHeader: optionalCompatBoolean({
|
||||
description:
|
||||
"Whether requests with images send the TypeSafe-Danger-Zone-Images: true header, which TypeSafe's own API requires for its experimental image input. Defaults to true for the typesafe-system-one API unless the model is detected as OpenRouter.",
|
||||
}),
|
||||
},
|
||||
{ description: "Compatibility settings for System One classifier APIs.", additionalProperties: true },
|
||||
);
|
||||
|
||||
const ProviderCompatPropertyOverrides = {
|
||||
supportsDeveloperRole: optionalCompatBoolean({
|
||||
description: "Whether the provider supports the developer role instead of system. Defaults are API-specific.",
|
||||
|
|
@ -609,6 +625,8 @@ export const CompatSchemasByApi = {
|
|||
"anthropic-messages": AnthropicMessagesCompatSchema,
|
||||
"bedrock-converse-stream": BedrockCompatSchema,
|
||||
"mistral-conversations": MistralConversationsCompatSchema,
|
||||
"typesafe-system-one": SystemOneCompatSchema,
|
||||
"cloudflare-workers-ai-system-one": SystemOneCompatSchema,
|
||||
} as const;
|
||||
|
||||
const ProviderCompatProperties = mergeCompatProperties(
|
||||
|
|
@ -642,3 +660,5 @@ export interface AnthropicMessagesCompat extends Static<typeof AnthropicMessages
|
|||
export interface BedrockCompat extends Static<typeof BedrockCompatSchema> {}
|
||||
/** Compatibility settings for the Mistral chat API. */
|
||||
export interface MistralConversationsCompat extends Static<typeof MistralConversationsCompatSchema> {}
|
||||
/** Compatibility settings for System One classifier APIs. */
|
||||
export interface SystemOneCompat extends Static<typeof SystemOneCompatSchema> {}
|
||||
|
|
|
|||
|
|
@ -15,6 +15,7 @@ import type {
|
|||
MistralConversationsCompat,
|
||||
OpenAICompletionsCompat,
|
||||
OpenAIResponsesCompat,
|
||||
SystemOneCompat,
|
||||
} from "./providers/compat-schema.ts";
|
||||
import type {
|
||||
CacheRetention,
|
||||
|
|
@ -40,6 +41,7 @@ export type {
|
|||
OpenAIResponsesCompat,
|
||||
OpenRouterRouting,
|
||||
SessionAffinityFormat,
|
||||
SystemOneCompat,
|
||||
ThinkingTokenBudgetField,
|
||||
VercelGatewayRouting,
|
||||
} from "./providers/compat-schema.ts";
|
||||
|
|
@ -888,6 +890,8 @@ export interface ImageModel<TApi extends ImageApi> extends BaseModel<TApi> {
|
|||
export interface ClassifierModel<TApi extends ClassifierApi> extends BaseModel<TApi> {
|
||||
type: "classifier";
|
||||
contextWindow: number;
|
||||
/** Compatibility overrides for System One APIs. If not set, auto-detected from provider, baseUrl, and id. */
|
||||
compat?: TApi extends "typesafe-system-one" | "cloudflare-workers-ai-system-one" ? SystemOneCompat : never;
|
||||
}
|
||||
|
||||
/** Model shape for each model type. */
|
||||
|
|
|
|||
86
packages/ai/test/classifier-images-live.test.ts
Normal file
86
packages/ai/test/classifier-images-live.test.ts
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
/**
|
||||
* Live image classification through System One services. Each provider's block runs only when its
|
||||
* credentials are set. Each case sends a solid red and a solid blue PNG in separate requests, so a
|
||||
* service that drops the image cannot pass by chance.
|
||||
*/
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { createModels, type ModelsClassifierOptions } from "../src/models.ts";
|
||||
import { cloudflareWorkersAIProvider } from "../src/providers/cloudflare-workers-ai.ts";
|
||||
import { openrouterProvider } from "../src/providers/openrouter.ts";
|
||||
import { typesafeProvider } from "../src/providers/typesafe.ts";
|
||||
import type { ClassifierContext } from "../src/types.ts";
|
||||
import { hasCloudflareWorkersAICredentials } from "./cloudflare-utils.ts";
|
||||
|
||||
// 512x512 solid PNGs from the TypeSafe SDK's live image tests.
|
||||
const RED =
|
||||
"iVBORw0KGgoAAAANSUhEUgAAAgAAAAIAAQMAAADOtka5AAAABlBMVEX/AAAAAABBoxIDAAAANklEQVR42u3BAQEAAACCIP+vbkhAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB8G4IAAAFjdVCkAAAAAElFTkSuQmCC";
|
||||
const BLUE =
|
||||
"iVBORw0KGgoAAAANSUhEUgAAAgAAAAIAAQMAAADOtka5AAAABlBMVEUAAP8AAAB7Yr8+AAAANklEQVR42u3BAQEAAACCIP+vbkhAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB8G4IAAAFjdVCkAAAAAElFTkSuQmCC";
|
||||
|
||||
function colorContext(data: string): ClassifierContext {
|
||||
return {
|
||||
state: { task: "Name the color of the attached image" },
|
||||
images: [{ type: "image", data, mimeType: "image/png" }],
|
||||
questions: {
|
||||
color: {
|
||||
type: "choice",
|
||||
instructions: "What color is the attached image?",
|
||||
criteria: { red: "Red", blue: "Blue", green: "Green" },
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
const models = createModels();
|
||||
models.setProvider(typesafeProvider());
|
||||
models.setProvider(cloudflareWorkersAIProvider());
|
||||
models.setProvider(openrouterProvider());
|
||||
|
||||
async function expectColors(provider: string, id: string, options: ModelsClassifierOptions): Promise<void> {
|
||||
const model = models.getModelOfType("classifier", provider, id);
|
||||
if (!model) throw new Error(`missing ${provider}/${id}`);
|
||||
expect(model.input).toContain("image");
|
||||
for (const [data, color] of [
|
||||
[RED, "red"],
|
||||
[BLUE, "blue"],
|
||||
] as const) {
|
||||
const result = await models.classify(model, colorContext(data), options);
|
||||
expect(result.errorMessage).toBeUndefined();
|
||||
expect(result.answers.color).toMatchObject({ type: "choice", choice: color });
|
||||
}
|
||||
}
|
||||
|
||||
describe.skipIf(!process.env.TYPESAFE_API_KEY)("TypeSafe image classification", () => {
|
||||
it("classifies images with jev-latest", async () => {
|
||||
await expectColors("typesafe", "jev-latest", { apiKey: process.env.TYPESAFE_API_KEY });
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
describe.skipIf(!hasCloudflareWorkersAICredentials())("Cloudflare Workers AI image classification", () => {
|
||||
it.each([
|
||||
"@cf/cloudflare/clef",
|
||||
"@cf/cloudflare/clef-flash",
|
||||
"@cf/cloudflare/clef-omni",
|
||||
"typesafe/jev",
|
||||
"typesafe/jev-1.14.0",
|
||||
])(
|
||||
"classifies images with %s",
|
||||
async (id) => {
|
||||
await expectColors("cloudflare-workers-ai", id, {
|
||||
apiKey: process.env.CLOUDFLARE_API_KEY,
|
||||
env: { CLOUDFLARE_ACCOUNT_ID: process.env.CLOUDFLARE_ACCOUNT_ID! },
|
||||
});
|
||||
},
|
||||
60_000,
|
||||
);
|
||||
});
|
||||
|
||||
describe.skipIf(!process.env.OPENROUTER_API_KEY)("OpenRouter image classification", () => {
|
||||
it.each(["typesafe/jev-1.14", "cloudflare/clef-flash"])(
|
||||
"classifies images with %s",
|
||||
async (id) => {
|
||||
await expectColors("openrouter", id, { apiKey: process.env.OPENROUTER_API_KEY });
|
||||
},
|
||||
60_000,
|
||||
);
|
||||
});
|
||||
|
|
@ -248,7 +248,12 @@ describe("Models with classifier models", () => {
|
|||
it("routes OpenRouter classifier models through the System One API", () => {
|
||||
const models = builtinModels();
|
||||
for (const model of getBuiltinClassifierModels("openrouter")) {
|
||||
expect(model).toMatchObject({ api: "typesafe-system-one", baseUrl: "https://openrouter.ai/api/v1" });
|
||||
expect(model).toMatchObject({
|
||||
api: "typesafe-system-one",
|
||||
baseUrl: "https://openrouter.ai/api/v1",
|
||||
// Explicit, so images keep OpenRouter's form when the baseUrl is overridden.
|
||||
compat: { imageInput: "state-image-urls", sendImageOptInHeader: false },
|
||||
});
|
||||
expect(models.getModel("openrouter", model.id)).toBeUndefined();
|
||||
expect(models.getModelOfType("classifier", "openrouter", model.id)).toEqual(model);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -105,7 +105,8 @@ describe("Cloudflare Workers AI System One", () => {
|
|||
|
||||
it.each([
|
||||
["@cf/cloudflare/clef", 0.24],
|
||||
["@cf/cloudflare/clef-flash", 0.09],
|
||||
["@cf/cloudflare/clef-flash", 0.038],
|
||||
["@cf/cloudflare/clef-omni", 0.15],
|
||||
])("runs %s through /ai/run and parses its direct output", async (id, inputPrice) => {
|
||||
const { models } = setup();
|
||||
const clef = models.getModelOfType("classifier", "cloudflare-workers-ai", id);
|
||||
|
|
@ -131,6 +132,104 @@ describe("Cloudflare Workers AI System One", () => {
|
|||
expect(result.usage?.cost.input).toBeCloseTo((222 * inputPrice) / 1_000_000);
|
||||
});
|
||||
|
||||
it("sends images to Clef as data URLs", async () => {
|
||||
const { models } = setup();
|
||||
const clef = models.getModelOfType("classifier", "cloudflare-workers-ai", "@cf/cloudflare/clef");
|
||||
if (!clef) throw new Error("missing Cloudflare Clef model");
|
||||
const bodies: Array<{ model: string; input: Record<string, unknown> }> = [];
|
||||
const fetch = vi.fn(async (_input: string | URL | Request, init?: RequestInit) => {
|
||||
bodies.push(JSON.parse(String(init?.body)) as { model: string; input: Record<string, unknown> });
|
||||
return Response.json({ result: clefOutput, success: true, errors: [], messages: [] });
|
||||
});
|
||||
|
||||
const result = await models.classify(
|
||||
clef,
|
||||
{
|
||||
...context,
|
||||
images: [
|
||||
{ type: "image", data: "cG5n", mimeType: "image/png" },
|
||||
{ type: "image", data: "d2VicA==", mimeType: "image/webp" },
|
||||
],
|
||||
},
|
||||
{ ...auth, fetch },
|
||||
);
|
||||
|
||||
expect(result.stopReason).toBe("stop");
|
||||
// Clef rejects `assets` with a 422, so the state stays as is.
|
||||
expect(bodies[0]?.input).toEqual({
|
||||
state: context.state,
|
||||
questions: expect.any(Object),
|
||||
images: ["data:image/png;base64,cG5n", "data:image/webp;base64,d2VicA=="],
|
||||
});
|
||||
expect(new Headers(fetch.mock.calls[0]?.[1]?.headers).has("typesafe-danger-zone-images")).toBe(false);
|
||||
});
|
||||
|
||||
it("reads the image form from compat before the model ID", async () => {
|
||||
const { models, jev } = setup();
|
||||
const clef = models.getModelOfType("classifier", "cloudflare-workers-ai", "@cf/cloudflare/clef");
|
||||
expect(clef?.compat).toEqual({ imageInput: "images" });
|
||||
expect(jev.compat).toEqual({ imageInput: "assets" });
|
||||
const bodies: Array<{ input: Record<string, unknown> }> = [];
|
||||
const fetch = vi.fn(async (_input: string | URL | Request, init?: RequestInit) => {
|
||||
bodies.push(JSON.parse(String(init?.body)) as { input: Record<string, unknown> });
|
||||
return Response.json({ result: clefOutput, success: true, errors: [], messages: [] });
|
||||
});
|
||||
|
||||
// A Clef deployment under an ID the prefix check does not recognize.
|
||||
await models.classify(
|
||||
{ ...jev, id: "@cf/acme/clef-finetune", input: ["text", "image"], compat: { imageInput: "images" } },
|
||||
{ ...context, images: [{ type: "image", data: "cG5n", mimeType: "image/png" }] },
|
||||
{ ...auth, fetch },
|
||||
);
|
||||
|
||||
expect(bodies[0]?.input.images).toEqual(["data:image/png;base64,cG5n"]);
|
||||
expect(bodies[0]?.input).not.toHaveProperty("assets");
|
||||
});
|
||||
|
||||
it("rejects more than four images for Clef before sending", async () => {
|
||||
const { models } = setup();
|
||||
const clef = models.getModelOfType("classifier", "cloudflare-workers-ai", "@cf/cloudflare/clef-flash");
|
||||
if (!clef) throw new Error("missing Cloudflare Clef Flash model");
|
||||
const fetch = vi.fn(async () => Response.json({ result: clefOutput, success: true }));
|
||||
const image = { type: "image" as const, data: "cG5n", mimeType: "image/png" };
|
||||
|
||||
const result = await models.classify(clef, { ...context, images: Array(5).fill(image) }, { ...auth, fetch });
|
||||
|
||||
expect(fetch).not.toHaveBeenCalled();
|
||||
expect(result.stopReason).toBe("error");
|
||||
expect(result.errorMessage).toBe(
|
||||
"Cloudflare Workers AI accepts at most 4 images for @cf/cloudflare/clef-flash, got 5",
|
||||
);
|
||||
});
|
||||
|
||||
it.each(["typesafe/jev", "typesafe/jev-1.14.0"])("sends images to %s as System One assets", async (id) => {
|
||||
const { models } = setup();
|
||||
const jev = models.getModelOfType("classifier", "cloudflare-workers-ai", id);
|
||||
if (!jev) throw new Error(`missing Cloudflare ${id} model`);
|
||||
const bodies: Array<{ model: string; input: Record<string, unknown> }> = [];
|
||||
const fetch = vi.fn(async (_input: string | URL | Request, init?: RequestInit) => {
|
||||
bodies.push(JSON.parse(String(init?.body)) as { model: string; input: Record<string, unknown> });
|
||||
return Response.json(restResponse("Completed"));
|
||||
});
|
||||
|
||||
const result = await models.classify(
|
||||
jev,
|
||||
{ ...context, images: [{ type: "image", data: "cG5n", mimeType: "image/png" }] },
|
||||
{ ...auth, fetch },
|
||||
);
|
||||
|
||||
expect(result.stopReason).toBe("stop");
|
||||
expect(bodies[0]).toMatchObject({
|
||||
model: id,
|
||||
input: {
|
||||
state: ["<image-1>", context.state],
|
||||
assets: { "<image-1>": { type: "image", format: "png", data_base64: "cG5n" } },
|
||||
},
|
||||
});
|
||||
// Workers AI needs no TypeSafe opt-in header.
|
||||
expect(new Headers(fetch.mock.calls[0]?.[1]?.headers).has("typesafe-danger-zone-images")).toBe(false);
|
||||
});
|
||||
|
||||
it("reports runs that did not complete", async () => {
|
||||
const { models, jev } = setup();
|
||||
const result = await models.classify(jev, context, {
|
||||
|
|
|
|||
|
|
@ -120,17 +120,160 @@ describe("TypeSafe System One", () => {
|
|||
expect(result.errorMessage).toContain("Unsupported classifier API: cloudflare-workers-ai-system-one");
|
||||
});
|
||||
|
||||
it("rejects image input before sending", async () => {
|
||||
it("rejects image input for text-only models before sending", async () => {
|
||||
const fetch = vi.fn(async () => Response.json({ answers: wireAnswers }));
|
||||
const result = await classify(
|
||||
{ ...model, input: ["text", "image"] },
|
||||
model,
|
||||
{ ...context, images: [{ type: "image", data: "aW1hZ2U=", mimeType: "image/png" }] },
|
||||
{ apiKey: "secret", fetch },
|
||||
);
|
||||
|
||||
expect(fetch).not.toHaveBeenCalled();
|
||||
expect(result.stopReason).toBe("error");
|
||||
expect(result.errorMessage).toBe("System One API does not support image input");
|
||||
expect(result.errorMessage).toBe("System One API does not support image input for jev-latest");
|
||||
});
|
||||
|
||||
it("sends images to TypeSafe as assets with the opt-in header", async () => {
|
||||
const requests: Array<{ body: Record<string, unknown>; headers: Headers }> = [];
|
||||
const fetch = vi.fn(async (_input: string | URL | Request, init?: RequestInit) => {
|
||||
requests.push({
|
||||
body: JSON.parse(String(init?.body)) as Record<string, unknown>,
|
||||
headers: new Headers(init?.headers),
|
||||
});
|
||||
return Response.json({ answers: wireAnswers });
|
||||
});
|
||||
const imageModel = { ...model, input: ["text", "image"] as ("text" | "image")[] };
|
||||
|
||||
const result = await classify(
|
||||
imageModel,
|
||||
{
|
||||
...context,
|
||||
images: [
|
||||
{ type: "image", data: "cG5n", mimeType: "image/png" },
|
||||
{ type: "image", data: "anBlZw==", mimeType: "image/jpeg" },
|
||||
],
|
||||
},
|
||||
{ apiKey: "secret", fetch },
|
||||
);
|
||||
const withoutImages = await classify(imageModel, context, { apiKey: "secret", fetch });
|
||||
|
||||
expect(result.stopReason).toBe("stop");
|
||||
expect(requests[0]?.body).toMatchObject({
|
||||
state: ["<image-1>", "<image-2>", context.state],
|
||||
assets: {
|
||||
"<image-1>": { type: "image", format: "png", data_base64: "cG5n" },
|
||||
"<image-2>": { type: "image", format: "jpeg", data_base64: "anBlZw==" },
|
||||
},
|
||||
});
|
||||
expect(requests[0]?.headers.get("typesafe-danger-zone-images")).toBe("true");
|
||||
// Requests without images keep the plain state and send no opt-in header.
|
||||
expect(withoutImages.stopReason).toBe("stop");
|
||||
expect(requests[1]?.body.state).toEqual(context.state);
|
||||
expect(requests[1]?.body).not.toHaveProperty("assets");
|
||||
expect(requests[1]?.headers.has("typesafe-danger-zone-images")).toBe(false);
|
||||
});
|
||||
|
||||
it("rejects image formats TypeSafe does not accept before sending", async () => {
|
||||
const fetch = vi.fn(async () => Response.json({ answers: wireAnswers }));
|
||||
const result = await classify(
|
||||
{ ...model, input: ["text", "image"] },
|
||||
{ ...context, images: [{ type: "image", data: "d2VicA==", mimeType: "image/webp" }] },
|
||||
{ apiKey: "secret", fetch },
|
||||
);
|
||||
|
||||
expect(fetch).not.toHaveBeenCalled();
|
||||
expect(result.stopReason).toBe("error");
|
||||
expect(result.errorMessage).toBe("System One API accepts PNG and JPEG images for jev-latest, got image/webp");
|
||||
});
|
||||
|
||||
it("sends images to OpenRouter as image_url parts in the state", async () => {
|
||||
const requests: Array<{ body: Record<string, unknown>; headers: Headers }> = [];
|
||||
const fetch = vi.fn(async (_input: string | URL | Request, init?: RequestInit) => {
|
||||
requests.push({
|
||||
body: JSON.parse(String(init?.body)) as Record<string, unknown>,
|
||||
headers: new Headers(init?.headers),
|
||||
});
|
||||
return Response.json({ answers: wireAnswers });
|
||||
});
|
||||
const openRouterModel: ClassifierModel<"typesafe-system-one"> = {
|
||||
...model,
|
||||
id: "typesafe/jev-1.14",
|
||||
provider: "openrouter",
|
||||
baseUrl: "https://openrouter.ai/api/v1",
|
||||
input: ["text", "image"],
|
||||
};
|
||||
|
||||
const result = await classify(
|
||||
openRouterModel,
|
||||
{ ...context, images: [{ type: "image", data: "cG5n", mimeType: "image/png" }] },
|
||||
{ apiKey: "secret", fetch },
|
||||
);
|
||||
|
||||
expect(result.stopReason).toBe("stop");
|
||||
// OpenRouter answers 200 for `assets` but drops them, and rejects a top-level `images` list.
|
||||
expect(requests[0]?.body).toEqual({
|
||||
model: "typesafe/jev-1.14",
|
||||
state: [{ type: "image_url", image_url: { url: "data:image/png;base64,cG5n" } }, context.state],
|
||||
questions: expect.any(Object),
|
||||
});
|
||||
expect(requests[0]?.headers.has("typesafe-danger-zone-images")).toBe(false);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["OpenRouter behind a proxy", { provider: "openrouter", baseUrl: "https://proxy.example.com/v1" }],
|
||||
["a regional OpenRouter endpoint", { provider: "custom", baseUrl: "https://us.openrouter.ai/api/v1" }],
|
||||
[
|
||||
"a proxy declared through compat",
|
||||
{
|
||||
provider: "custom",
|
||||
baseUrl: "https://proxy.example.com/v1",
|
||||
compat: { imageInput: "state-image-urls" as const, sendImageOptInHeader: false },
|
||||
},
|
||||
],
|
||||
])("sends OpenRouter's image form to %s", async (_name, overrides) => {
|
||||
const requests: Array<{ body: Record<string, unknown>; headers: Headers }> = [];
|
||||
const fetch = vi.fn(async (_input: string | URL | Request, init?: RequestInit) => {
|
||||
requests.push({
|
||||
body: JSON.parse(String(init?.body)) as Record<string, unknown>,
|
||||
headers: new Headers(init?.headers),
|
||||
});
|
||||
return Response.json({ answers: wireAnswers });
|
||||
});
|
||||
|
||||
const result = await classify(
|
||||
{ ...model, id: "typesafe/jev-1.14", input: ["text", "image"], ...overrides },
|
||||
{ ...context, images: [{ type: "image", data: "cG5n", mimeType: "image/png" }] },
|
||||
{ apiKey: "secret", fetch },
|
||||
);
|
||||
|
||||
expect(result.stopReason).toBe("stop");
|
||||
expect(requests[0]?.body.state).toEqual([
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,cG5n" } },
|
||||
context.state,
|
||||
]);
|
||||
expect(requests[0]?.body).not.toHaveProperty("assets");
|
||||
expect(requests[0]?.headers.has("typesafe-danger-zone-images")).toBe(false);
|
||||
});
|
||||
|
||||
it("lets compat override detected image settings per field", async () => {
|
||||
const requests: Array<{ body: Record<string, unknown>; headers: Headers }> = [];
|
||||
const fetch = vi.fn(async (_input: string | URL | Request, init?: RequestInit) => {
|
||||
requests.push({
|
||||
body: JSON.parse(String(init?.body)) as Record<string, unknown>,
|
||||
headers: new Headers(init?.headers),
|
||||
});
|
||||
return Response.json({ answers: wireAnswers });
|
||||
});
|
||||
|
||||
// Only the header is overridden; the detected `assets` form stays.
|
||||
await classify(
|
||||
{ ...model, input: ["text", "image"], compat: { sendImageOptInHeader: false } },
|
||||
{ ...context, images: [{ type: "image", data: "cG5n", mimeType: "image/png" }] },
|
||||
{ apiKey: "secret", fetch },
|
||||
);
|
||||
|
||||
expect(requests[0]?.body).toHaveProperty("assets");
|
||||
expect(requests[0]?.headers.has("typesafe-danger-zone-images")).toBe(false);
|
||||
});
|
||||
|
||||
it("merges headers case-insensitively and supports null suppression", async () => {
|
||||
|
|
|
|||
|
|
@ -5,6 +5,8 @@
|
|||
### Added
|
||||
|
||||
- Added a Models Configuration reference generated from the `models.json` schema, listing all provider, model, model override, and compatibility fields ([#10790](https://github.com/earendil-works/pi/issues/10790))
|
||||
- Codemode `models.classify()` now accepts `images` for TypeSafe's Jev and Cloudflare's Clef models on `typesafe`, `cloudflare-workers-ai`, and OpenRouter, and lists Clef Omni and Jev 1.14 on Workers AI.
|
||||
- Provider-level `compat` in `models.json` now also applies to classifier models, with the System One fields `imageInput` and `sendImageOptInHeader` for proxies that serve another service's classifiers. Extension classifier models accept `compat` too.
|
||||
|
||||
### Changed
|
||||
|
||||
|
|
|
|||
|
|
@ -185,3 +185,12 @@ Also applies to `azure-openai-responses` and `openai-codex-responses`.
|
|||
| Field | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `supportsMidConvoSystemMessages` | boolean | `false` | Whether the exact model accepts system messages after the conversation has started. When false, later system messages are folded into the leading system message. |
|
||||
|
||||
### `typesafe-system-one`
|
||||
|
||||
Also applies to `cloudflare-workers-ai-system-one`.
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `imageInput` | `"assets" \| "images" \| "state-image-urls"` | | How the service takes classifier images. "assets" sends TypeSafe's assets map (PNG and JPEG) and references the images from the state, as TypeSafe and Jev on Workers AI expect. "images" sends Clef's images list of data URLs (PNG, JPEG, and WebP, at most four), as Clef on Workers AI expects. "state-image-urls" places image_url parts before the state, as OpenRouter expects; OpenRouter drops assets without an error. Auto-detected from the provider, baseUrl, and model ID. Classifier models read it from the provider-level compat; models and modelOverrides entries apply to chat models only. |
|
||||
| `sendImageOptInHeader` | boolean | | Whether requests with images send the TypeSafe-Danger-Zone-Images: true header, which TypeSafe's own API requires for its experimental image input. Defaults to true for the typesafe-system-one API unless the model is detected as OpenRouter. |
|
||||
|
|
|
|||
|
|
@ -141,20 +141,20 @@ Compatibility settings should describe verified differences in the endpoint's re
|
|||
|
||||
## Use classifier models
|
||||
|
||||
Classifier models do not chat. They answer typed questions about JSON state: pick one of several choices, answer yes or no, or give a score, each with probabilities. Pi includes TypeSafe's Jev model from these providers, Cloudflare's Clef and Clef Flash models from Workers AI, and OpenAI's GPT-6 Luna through the [Decisions API](https://developers.openai.com/api/docs/guides/decisions):
|
||||
Classifier models do not chat. They answer typed questions about JSON state: pick one of several choices, answer yes or no, or give a score, each with probabilities. Pi includes TypeSafe's Jev model from these providers, Cloudflare's Clef, Clef Flash, and Clef Omni models from Workers AI and OpenRouter, and OpenAI's GPT-6 Luna through the [Decisions API](https://developers.openai.com/api/docs/guides/decisions):
|
||||
|
||||
| Provider | Model IDs | Authentication |
|
||||
|---|---|---|
|
||||
| `typesafe` | `jev-latest` | `TYPESAFE_API_KEY` |
|
||||
| `openrouter` | `typesafe/jev-1.13`, `~typesafe/jev-latest` | `OPENROUTER_API_KEY` or `/login` |
|
||||
| `cloudflare-workers-ai` | `typesafe/jev`, `@cf/cloudflare/clef`, `@cf/cloudflare/clef-flash` | `CLOUDFLARE_API_KEY` and `CLOUDFLARE_ACCOUNT_ID` |
|
||||
| `openrouter` | `typesafe/jev-1.13`, `typesafe/jev-1.14`, `~typesafe/jev-latest`, `cloudflare/clef`, `cloudflare/clef-flash`, `cloudflare/clef-omni` | `OPENROUTER_API_KEY` or `/login` |
|
||||
| `cloudflare-workers-ai` | `typesafe/jev`, `typesafe/jev-1.14.0`, `@cf/cloudflare/clef`, `@cf/cloudflare/clef-flash`, `@cf/cloudflare/clef-omni` | `CLOUDFLARE_API_KEY` and `CLOUDFLARE_ACCOUNT_ID` |
|
||||
| `vercel-ai-gateway` | `typesafe-ai/jev` | `AI_GATEWAY_API_KEY` |
|
||||
| `opencode` | `jev-1.13`, `jev-1.13-free` | `OPENCODE_API_KEY` |
|
||||
| `openai` | `gpt-6-luna` | `OPENAI_API_KEY` |
|
||||
|
||||
Chat models on a [llama.cpp router](llama-cpp.md#classification) are also listed as classifier models.
|
||||
|
||||
OpenAI's Decisions API needs an API key. Sign in with ChatGPT credentials do not work with it, so `gpt-6-luna` is not listed as available while `openai` is logged in through `/login`, even when `OPENAI_API_KEY` is set; log out of `openai` to use the key. GPT-6 Luna also judges images passed in `images` (see [Codemode](codemode.md#classify)); other classifier models return an error for them. The API rejects inputs above 922K tokens, but requests that run longer than about five seconds, currently above roughly 600K input tokens, fail with a gateway timeout.
|
||||
OpenAI's Decisions API needs an API key. Sign in with ChatGPT credentials do not work with it, so `gpt-6-luna` is not listed as available while `openai` is logged in through `/login`, even when `OPENAI_API_KEY` is set; log out of `openai` to use the key. The API rejects inputs above 922K tokens, but requests that run longer than about five seconds, currently above roughly 600K input tokens, fail with a gateway timeout.
|
||||
|
||||
Classifier models do not appear in `/model`. The model reaches them through the [`codemode`](cli.md#enable-codemode) tool, which is off unless an MCP server turned it on. Enable it with `"defaultTools": ["+codemode"]` in [settings](settings.md#tools). Scripts then list classifier models with `models.getAvailableOfType("classifier")` and call `models.classify(model, { state, questions })`:
|
||||
|
||||
|
|
@ -175,6 +175,10 @@ return result.answers;
|
|||
|
||||
[Codemode](codemode.md#classify) describes the question and answer types.
|
||||
|
||||
Classifier models whose `input` includes `"image"` also judge images passed in `images`; other classifier models return an error for them. Image input works with GPT-6 Luna, with Jev from `typesafe` and `cloudflare-workers-ai`, with Clef from `cloudflare-workers-ai`, and with the OpenRouter models whose catalog lists image input. Jev takes PNG and JPEG images, and image input is experimental on TypeSafe's own API. Clef takes PNG, JPEG, and WebP images, at most four per request. Jev from `vercel-ai-gateway` and `opencode` takes text only.
|
||||
|
||||
Each System One service takes images in its own request format, set by the model's `compat.imageInput`. Built-in models set it, so overriding a provider's `baseUrl` keeps it. A custom provider that proxies another service's classifiers sets it in its provider-level `compat` in `models.json` (see [System One compatibility](models-configuration.md#typesafe-system-one)). OpenRouter in particular answers requests in TypeSafe's format without an error but drops the images.
|
||||
|
||||
When the service reports token counts, as all System One services do, `result.usage` carries them with their cost. Pi adds the usage of a script's classifier calls to the `codemode` tool result, so it counts toward the session cost in the footer and `/session`. The cost uses the model's catalog price; models without one, such as TypeSafe's direct `jev-latest`, report tokens at no cost.
|
||||
|
||||
Extensions call classifiers through `ctx.modelRegistry.classify()`, without codemode. [Virtual models](virtual-models.md#route-requests) can use them to route requests; see the `jev-router.ts` example.
|
||||
|
|
|
|||
|
|
@ -1107,6 +1107,27 @@
|
|||
},
|
||||
"maxItems": 3,
|
||||
"description": "Models Anthropic accepts for server-side refusal fallback, with local pricing metadata for returned fallback responses. When absent or empty, callers must omit fallbacks; Anthropic rejects the field for models with no permitted fallback targets."
|
||||
},
|
||||
"imageInput": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string",
|
||||
"const": "assets"
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"const": "images"
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"const": "state-image-urls"
|
||||
}
|
||||
],
|
||||
"description": "How the service takes classifier images. \"assets\" sends TypeSafe's assets map (PNG and JPEG) and references the images from the state, as TypeSafe and Jev on Workers AI expect. \"images\" sends Clef's images list of data URLs (PNG, JPEG, and WebP, at most four), as Clef on Workers AI expects. \"state-image-urls\" places image_url parts before the state, as OpenRouter expects; OpenRouter drops assets without an error. Auto-detected from the provider, baseUrl, and model ID. Classifier models read it from the provider-level compat; models and modelOverrides entries apply to chat models only."
|
||||
},
|
||||
"sendImageOptInHeader": {
|
||||
"type": "boolean",
|
||||
"description": "Whether requests with images send the TypeSafe-Danger-Zone-Images: true header, which TypeSafe's own API requires for its experimental image input. Defaults to true for the typesafe-system-one API unless the model is detected as OpenRouter."
|
||||
}
|
||||
},
|
||||
"additionalProperties": true
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ import {
|
|||
type AuthInteraction,
|
||||
type AuthResult,
|
||||
type ClassifierApi,
|
||||
type ClassifierModel,
|
||||
type Credential,
|
||||
type ImageApi,
|
||||
isModelType,
|
||||
|
|
@ -83,6 +84,7 @@ export interface ProviderClassifierModelConfig extends ProviderModelConfigBase {
|
|||
type: "classifier";
|
||||
api?: ClassifierApi;
|
||||
contextWindow: number;
|
||||
compat?: ClassifierModel<ClassifierApi>["compat"];
|
||||
}
|
||||
|
||||
export type ProviderModelConfig = ProviderChatModelConfig | ProviderImageModelConfig | ProviderClassifierModelConfig;
|
||||
|
|
@ -324,9 +326,12 @@ function applyModelsJson(
|
|||
|
||||
const models: AnyModel[] = baseModels.map((model) => {
|
||||
const baseUrl = config.oauth === "radius" ? model.baseUrl : (config.baseUrl ?? model.baseUrl);
|
||||
return isModelType(model, "chat")
|
||||
? { ...model, baseUrl, compat: mergeCompat(model.compat, config.compat) }
|
||||
: { ...model, baseUrl };
|
||||
if (isModelType(model, "chat")) return { ...model, baseUrl, compat: mergeCompat(model.compat, config.compat) };
|
||||
// Classifier APIs read their own compat fields, such as how a System One service takes images.
|
||||
if (isModelType(model, "classifier") && config.compat) {
|
||||
return { ...model, baseUrl, compat: { ...model.compat, ...config.compat } };
|
||||
}
|
||||
return { ...model, baseUrl };
|
||||
});
|
||||
for (const definition of config.models ?? []) {
|
||||
const existingIndex = models.findIndex((model) => isModelType(model, "chat") && model.id === definition.id);
|
||||
|
|
|
|||
|
|
@ -1,5 +1,8 @@
|
|||
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { InMemoryModelsStore } from "@earendil-works/pi-ai";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { AuthStorage } from "../src/core/auth-storage.ts";
|
||||
import { ModelRuntime } from "../src/core/model-runtime.ts";
|
||||
|
||||
|
|
@ -14,14 +17,56 @@ const context = {
|
|||
},
|
||||
};
|
||||
|
||||
const image = { type: "image" as const, data: "cG5n", mimeType: "image/png" };
|
||||
|
||||
describe("ModelRuntime classifiers", () => {
|
||||
it("lists Jev separately and classifies with runtime-resolved auth", async () => {
|
||||
const runtime = await ModelRuntime.create({
|
||||
const tempDirs: string[] = [];
|
||||
afterEach(() => {
|
||||
for (const dir of tempDirs.splice(0)) rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function createRuntime(modelsJson?: object): Promise<ModelRuntime> {
|
||||
let modelsPath: string | null = null;
|
||||
if (modelsJson) {
|
||||
const dir = mkdtempSync(join(tmpdir(), "pi-classifiers-"));
|
||||
tempDirs.push(dir);
|
||||
modelsPath = join(dir, "models.json");
|
||||
writeFileSync(modelsPath, JSON.stringify(modelsJson));
|
||||
}
|
||||
return ModelRuntime.create({
|
||||
credentials: AuthStorage.inMemory(),
|
||||
modelsStore: new InMemoryModelsStore(),
|
||||
modelsPath: null,
|
||||
modelsPath,
|
||||
allowModelNetwork: false,
|
||||
});
|
||||
}
|
||||
|
||||
/** Classifies one image and returns the request body and headers the service received. */
|
||||
async function captureImageRequest(runtime: ModelRuntime, provider: string, id: string) {
|
||||
await runtime.setRuntimeApiKey(provider, "secret");
|
||||
const model = runtime.getModelOfType("classifier", provider, id)!;
|
||||
const requests: Array<{ url: string; body: Record<string, unknown>; headers: Headers }> = [];
|
||||
const result = await runtime.classify(
|
||||
model,
|
||||
{ ...context, images: [image] },
|
||||
{
|
||||
fetch: async (input, init) => {
|
||||
requests.push({
|
||||
url: String(input),
|
||||
body: JSON.parse(String(init?.body)) as Record<string, unknown>,
|
||||
headers: new Headers(init?.headers),
|
||||
});
|
||||
return Response.json({ answers: { approved: { type: "noul", noul: 0.8 } } });
|
||||
},
|
||||
},
|
||||
);
|
||||
expect(result.stopReason).toBe("stop");
|
||||
expect(requests).toHaveLength(1);
|
||||
return requests[0]!;
|
||||
}
|
||||
|
||||
it("lists Jev separately and classifies with runtime-resolved auth", async () => {
|
||||
const runtime = await createRuntime();
|
||||
const jev = runtime.getModelOfType("classifier", "typesafe", "jev-latest")!;
|
||||
expect(jev.type).toBe("classifier");
|
||||
expect(runtime.getModel("typesafe", "jev-latest")).toBeUndefined();
|
||||
|
|
@ -39,4 +84,39 @@ describe("ModelRuntime classifiers", () => {
|
|||
const result = await runtime.classify(jev, context, { fetch });
|
||||
expect(result.answers.approved).toEqual({ type: "bool", probability: 0.8 });
|
||||
});
|
||||
|
||||
it("keeps OpenRouter's image form when models.json routes OpenRouter through a proxy", async () => {
|
||||
const runtime = await createRuntime({ providers: { openrouter: { baseUrl: "https://proxy.example.com/v1" } } });
|
||||
|
||||
const request = await captureImageRequest(runtime, "openrouter", "typesafe/jev-1.14");
|
||||
|
||||
// OpenRouter answers TypeSafe `assets` with 200 but drops the images.
|
||||
expect(request.url).toBe("https://proxy.example.com/v1/systemone");
|
||||
expect(request.body.state).toEqual([
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,cG5n" } },
|
||||
context.state,
|
||||
]);
|
||||
expect(request.body).not.toHaveProperty("assets");
|
||||
expect(request.headers.has("typesafe-danger-zone-images")).toBe(false);
|
||||
});
|
||||
|
||||
it("applies provider compat from models.json to classifier models", async () => {
|
||||
const runtime = await createRuntime({
|
||||
providers: {
|
||||
typesafe: {
|
||||
baseUrl: "https://proxy.example.com/v1",
|
||||
compat: { imageInput: "state-image-urls", sendImageOptInHeader: false },
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
const request = await captureImageRequest(runtime, "typesafe", "jev-latest");
|
||||
|
||||
expect(request.url).toBe("https://proxy.example.com/v1/systemone");
|
||||
expect(request.body.state).toEqual([
|
||||
{ type: "image_url", image_url: { url: "data:image/png;base64,cG5n" } },
|
||||
context.state,
|
||||
]);
|
||||
expect(request.headers.has("typesafe-danger-zone-images")).toBe(false);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue