mirror of
https://github.com/badlogic/pi-mono.git
synced 2026-11-05 23:37:25 +00:00
feat(p2): render artifacts inline at the tool call site
The artifact tool takes inline: true to also show the produced version below its transcript card: Markdown/text rendered, HTML/SVG in the sandboxed preview frame. Drawings fill the frame's width (an SVG with a viewBox or an absolute size scales to it; an HTML page that is only an SVG or image too) and the frame takes the content height the preview reports, capped at 70vh. Every viewer sees it, guests included (owner decision). The TUI renders inline Markdown and text. SVG previews now carry the diagnostics bridge in CDATA (its && broke the XML).
This commit is contained in:
Mario Zechner
parent
9e870f6f00
commit
04f672a18d
15 changed files with 679 additions and 100 deletions
|
|
@ -29,7 +29,7 @@ P2_FAKE=1 npm start # http://127.0.0.1:8788, data in ~/.pi/p2
|
|||
| --- | --- | --- |
|
||||
| `P2_DATA_DIR` | `~/.pi/p2` | data directory: `catalog.sqlite`, `sessions/<id>.sqlite`, `settings.json`, `models.json`, `envs.json`, `mcp.json`, `mcp-auth.json`, `auth.json`, `radius.json`, `forwards.json` |
|
||||
| `PORT` | `8788` | listen port (loopback only), so p2 never collides with pi-mobile on 8787 |
|
||||
| `P2_FAKE` | unset | `1`: the faux provider (`faux/faux-1`, echoes the prompt; `slow …` streams long; `rename-session <title>` calls the tool; `subagent`, `subagent-bad`, `subagent-answer`, `subagent-configure`, `subagent-send`, `subagent-archive`, `subagent-status` exercise subagents, `subagent <json>` calls the tool with those arguments, as do `team <json>`, `history_search|read|handoff <json>`, `location_envs|dirs|move <json>`, `tool_search <json>` and `mcp__<server>__<tool> <json>`; `codemode` runs a script calling bash and read, `codemode-restart` one whose slow middle call a restart interrupts, `codemode <script>` runs that script; `json <text>` answers exactly `<text>`; a one-shot agent whose task is not `json …`/`slow…` answers a small value matching its schema) |
|
||||
| `P2_FAKE` | unset | `1`: the faux provider (`faux/faux-1`, echoes the prompt; `slow …` streams long; `rename-session <title>` calls the tool; `subagent`, `subagent-bad`, `subagent-answer`, `subagent-configure`, `subagent-send`, `subagent-archive`, `subagent-status` exercise subagents, `subagent <json>` calls the tool with those arguments, as do `team <json>`, `history_search|read|handoff <json>`, `location_envs|dirs|move <json>`, `tool_search <json>` and `mcp__<server>__<tool> <json>`; `codemode` runs a script calling bash and read, `codemode-restart` one whose slow middle call a restart interrupts, `codemode <script>` runs that script; `artifact <json>` calls the artifact tool with those arguments; `json <text>` answers exactly `<text>`; a one-shot agent whose task is not `json …`/`slow…` answers a small value matching its schema) |
|
||||
| `P2_IDLE_MS` | `300000` | idle time before a session worker stops |
|
||||
| `P2_PROVIDER`, `P2_MODEL` | unset | default model for new sessions after settings.json's `defaultModel` (both or neither; Pi Mobile's `PI_MOBILE_PROVIDER`/`PI_MOBILE_MODEL` work too) |
|
||||
| `P2_RADIUS` | unset | `0`: disable the outbound Radius brain connection even when signed in |
|
||||
|
|
@ -955,6 +955,29 @@ translated phone drawer and intercept chat taps.
|
|||
sent files use the conservative file preview. Nothing opens from a server update. `test/p2/artifacts.test.ts`
|
||||
covers fake-provider publication, versions, independent index, reload/restart, CSP, HEAD, guest isolation and
|
||||
byte limits; `ui-viewers-search.test.ts` the picker and viewers.
|
||||
- **Inline artifacts** (`artifact` tool, `inline: true`): the call's version also renders at its tool call site,
|
||||
below the card, so agents can show diagrams (e.g. SVG sequence diagrams) and small visual results in the chat
|
||||
instead of ASCII. `inline` travels only in the result's details (`ArtifactDetails`, `{ ...info, inline: true }`);
|
||||
the index, the stored version and every Chord contract are unchanged, so no contract version moved. The web
|
||||
(`InlineArtifact` in `plugins/artifacts/preview.tsx`, provided to `Chat.tsx` through `InlineArtifactView`) renders
|
||||
that version, never "latest": Markdown with the viewer's `Markdown` component, text as wrapped `pre` (both at most
|
||||
70vh, scrolling), HTML/SVG in the viewer's sandboxed frame (opaque origin, same route and CSP). The preview's
|
||||
diagnostics bridge also posts the document's content height (level `size`, again on every resize), and the frame
|
||||
takes that height up to 70vh. Drawings fill the frame's width: an SVG document with a `viewBox` (or an absolute
|
||||
width and height, which become its `viewBox`) is set to 100% wide and high, so it scales with its aspect and
|
||||
reports the `viewBox` height at its width, leaving no blank space; an HTML document whose body is only a `viewBox`
|
||||
SVG or an image gets it 100% wide at height auto (zero specificity, so any author rule wins), and is measured
|
||||
without its scrollbar (a too-short frame's scrollbar would otherwise stay). This applies in the viewer too. Past 70vh, HTML scrolls inside the frame and an SVG scales down to fit (giving the SVG root an explicit
|
||||
CSS height instead, to make it scroll, broke its rendering in Chromium's out-of-process frame);
|
||||
reported heights are kept per URL so a remounted row keeps its size. A version the route cannot serve shows
|
||||
"Artifact unavailable" (HEAD first for HTML/SVG). Tapping the card still opens the full viewer. **Who sees it:**
|
||||
every viewer of the session, guests included, HTML and SVG too, without a tap (owner decision, option (a): the
|
||||
sandbox and CSP are the protection, as for an opened preview). The TUI renders inline Markdown (rendered) and text
|
||||
in full; HTML/SVG keep the card line. SVG previews carry the bridge in CDATA (the script's `&&` broke the XML
|
||||
before); an empty `<svg/>` root is served without it. Fake provider: `artifact <json>` calls the tool with those
|
||||
arguments. Tests: `artifacts.test.ts` (details, bridge), `ui-inline-artifacts.test.ts` (browser: Markdown, SVG,
|
||||
fills the width, fitted height, 70vh cap, card opens the viewer, unavailable version, guest), `tui-p11.test.ts`
|
||||
(inline Markdown).
|
||||
- **Conversations** (`plugins/conversations`): the root-only `conversations` tool lists active roots without
|
||||
starting workers, sends server-attributed peer input, and reads the latest completed text answer (bounded scan,
|
||||
no thinking, including old history). The tool fixes owner/peer origin at its calling assistant entry: later guest
|
||||
|
|
|
|||
|
|
@ -1450,7 +1450,8 @@ contract), so no plugin gains a dependency.
|
|||
editor would be (the web's read-only bar). GuestPresence is still held from the `widget-above:sharing` slot.
|
||||
- **P11c**:
|
||||
- **Artifacts** (`P2/plugins/artifacts/tui.ts`). The card adds "/artifacts opens". For text and Markdown it shows
|
||||
the content the call wrote: 8 lines, all when expanded.
|
||||
the content the call wrote: 8 lines, all when expanded. A call with `inline: true` shows all of it at once,
|
||||
Markdown rendered; inline HTML and SVG keep the card line.
|
||||
- **`/artifacts`** lists `ArtifactList`. Enter shows an artifact inline: Markdown rendered with the instance
|
||||
theme, text and HTML/SVG source as sanitized lines. HTML/SVG carry a "Source only" note, because they run only in
|
||||
a browser. Ctrl+B is `openUrl` `{kind: "origin", path: "/artifact?…"}` and Ctrl+S is `download`
|
||||
|
|
|
|||
|
|
@ -1,21 +1,90 @@
|
|||
import { artifactUrls, Markdown, ViewerShell } from "@p2/app/v1/web";
|
||||
import { useEffect, useRef, useState } from "react";
|
||||
import { type Ref, useEffect, useRef, useState } from "react";
|
||||
import type { ArtifactInfo } from "./contract.ts";
|
||||
|
||||
type Diagnostic = { level: "ready" | "error" | "warn" | "csp" | "webgl"; message: string };
|
||||
/** What a preview frame posts on the diagnostic channel: a diagnostic, or its content height (`store.ts`). */
|
||||
type FrameMessage = Diagnostic | { level: "size"; height: number };
|
||||
|
||||
function diagnostic(value: unknown): Diagnostic | undefined {
|
||||
/** A message from a preview frame (untrusted: the artifact's own script can post anything). */
|
||||
function frameMessage(value: unknown): FrameMessage | undefined {
|
||||
if (!value || typeof value !== "object" || Array.isArray(value)) return;
|
||||
const data = value as Record<string, unknown>;
|
||||
if (
|
||||
data.channel !== "p2-artifact-diagnostic" ||
|
||||
!["ready", "error", "warn", "csp", "webgl"].includes(String(data.level)) ||
|
||||
typeof data.message !== "string"
|
||||
)
|
||||
if (data.channel !== "p2-artifact-diagnostic") return;
|
||||
if (data.level === "size")
|
||||
return typeof data.height === "number" && Number.isFinite(data.height) && data.height >= 0
|
||||
? { level: "size", height: data.height }
|
||||
: undefined;
|
||||
if (!["ready", "error", "warn", "csp", "webgl"].includes(String(data.level)) || typeof data.message !== "string")
|
||||
return;
|
||||
return { level: data.level as Diagnostic["level"], message: data.message.slice(0, 4000) };
|
||||
}
|
||||
|
||||
/** Calls `receive` with each message the frame posts on the diagnostic channel. */
|
||||
function useFrameMessages(
|
||||
frame: { readonly current: HTMLIFrameElement | null },
|
||||
receive: (message: FrameMessage) => void,
|
||||
): void {
|
||||
const latest = useRef(receive);
|
||||
latest.current = receive;
|
||||
useEffect(() => {
|
||||
const listener = (event: MessageEvent) => {
|
||||
if (event.source !== frame.current?.contentWindow) return;
|
||||
const message = frameMessage(event.data);
|
||||
if (message) latest.current(message);
|
||||
};
|
||||
window.addEventListener("message", listener);
|
||||
return () => window.removeEventListener("message", listener);
|
||||
}, [frame]);
|
||||
}
|
||||
|
||||
/** A text or Markdown artifact's content from its route: undefined while it loads, or the error. */
|
||||
function useArtifactText(url: string | undefined): { content?: string; error?: string } {
|
||||
const [state, setState] = useState<{ url?: string; content?: string; error?: string }>({});
|
||||
useEffect(() => {
|
||||
if (!url) return;
|
||||
const abort = new AbortController();
|
||||
void fetch(url, { credentials: "same-origin", cache: "no-store", signal: abort.signal })
|
||||
.then(async (response) => {
|
||||
if (!response.ok) throw new Error(`Could not open artifact (${response.status})`);
|
||||
const content = await response.text();
|
||||
setState({ url, content });
|
||||
})
|
||||
.catch((caught: unknown) => {
|
||||
if (!abort.signal.aborted)
|
||||
setState({ url, error: caught instanceof Error ? caught.message : "Could not open artifact" });
|
||||
});
|
||||
return () => abort.abort();
|
||||
}, [url]);
|
||||
return state.url === url ? state : {};
|
||||
}
|
||||
|
||||
/** The sandboxed frame of an HTML or SVG preview: an opaque origin, and the route's CSP sandbox on top. */
|
||||
function ArtifactFrame({
|
||||
url,
|
||||
title,
|
||||
frame,
|
||||
height,
|
||||
}: {
|
||||
url: string;
|
||||
title: string;
|
||||
frame: Ref<HTMLIFrameElement>;
|
||||
height?: number;
|
||||
}) {
|
||||
return (
|
||||
<iframe
|
||||
// A new frame per artifact: changing a frame's src would add a history entry.
|
||||
key={url}
|
||||
ref={frame}
|
||||
title={title}
|
||||
src={url}
|
||||
sandbox="allow-scripts allow-modals allow-pointer-lock"
|
||||
referrerPolicy="no-referrer"
|
||||
style={height === undefined ? undefined : { height }}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
export function ArtifactPanel({
|
||||
sessionId,
|
||||
artifact,
|
||||
|
|
@ -28,8 +97,7 @@ export function ArtifactPanel({
|
|||
const { view: url, download: downloadUrl, filename } = artifactUrls(sessionId, artifact);
|
||||
const document = artifact.type === "markdown" || artifact.type === "text";
|
||||
const frame = useRef<HTMLIFrameElement>(null);
|
||||
const [content, setContent] = useState<string>();
|
||||
const [error, setError] = useState<string>();
|
||||
const { content, error } = useArtifactText(document ? url : undefined);
|
||||
const [diagnostics, setDiagnostics] = useState<Diagnostic[]>([]);
|
||||
const [diagnosticsOpen, setDiagnosticsOpen] = useState(false);
|
||||
|
||||
|
|
@ -39,35 +107,11 @@ export function ArtifactPanel({
|
|||
setDiagnosticsOpen(false);
|
||||
}, [url]);
|
||||
|
||||
useEffect(() => {
|
||||
const abort = new AbortController();
|
||||
if (document) {
|
||||
setContent(undefined);
|
||||
setError(undefined);
|
||||
void fetch(url, { credentials: "same-origin", cache: "no-store", signal: abort.signal })
|
||||
.then(async (response) => {
|
||||
if (!response.ok) throw new Error(`Could not open artifact (${response.status})`);
|
||||
setContent(await response.text());
|
||||
})
|
||||
.catch((caught: unknown) => {
|
||||
if (!abort.signal.aborted)
|
||||
setError(caught instanceof Error ? caught.message : "Could not open artifact");
|
||||
});
|
||||
}
|
||||
return () => abort.abort();
|
||||
}, [document, url]);
|
||||
|
||||
useEffect(() => {
|
||||
const receive = (event: MessageEvent) => {
|
||||
if (event.source !== frame.current?.contentWindow) return;
|
||||
const next = diagnostic(event.data);
|
||||
if (!next) return;
|
||||
setDiagnostics((current) => [...current.slice(-49), next]);
|
||||
if (next.level !== "ready") setDiagnosticsOpen(true);
|
||||
};
|
||||
window.addEventListener("message", receive);
|
||||
return () => window.removeEventListener("message", receive);
|
||||
}, []);
|
||||
useFrameMessages(frame, (next) => {
|
||||
if (next.level === "size") return;
|
||||
setDiagnostics((current) => [...current.slice(-49), next]);
|
||||
if (next.level !== "ready") setDiagnosticsOpen(true);
|
||||
});
|
||||
|
||||
const problems = diagnostics.filter((entry) => entry.level !== "ready").length;
|
||||
return (
|
||||
|
|
@ -122,16 +166,72 @@ export function ArtifactPanel({
|
|||
<pre>{content}</pre>
|
||||
)
|
||||
) : (
|
||||
<iframe
|
||||
// A new frame per artifact: changing a frame's src would add a history entry.
|
||||
key={url}
|
||||
ref={frame}
|
||||
title={artifact.title}
|
||||
src={url}
|
||||
sandbox="allow-scripts allow-modals allow-pointer-lock"
|
||||
referrerPolicy="no-referrer"
|
||||
/>
|
||||
<ArtifactFrame url={url} title={artifact.title} frame={frame} />
|
||||
)}
|
||||
</ViewerShell>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* The heights inline frames last reported, by URL (versions are immutable), so a row the transcript mounts again
|
||||
* keeps its size instead of starting at `FRAME_HEIGHT`.
|
||||
*/
|
||||
const frameHeights = new Map<string, number>();
|
||||
const FRAME_HEIGHTS = 256;
|
||||
/** A frame's height before it reports one. */
|
||||
const FRAME_HEIGHT = 150;
|
||||
|
||||
/**
|
||||
* An artifact version rendered at its tool call site, below the card (`inline: true`): Markdown and text as in the
|
||||
* viewer; HTML and SVG in the viewer's sandboxed frame, sized to the content height it reports after drawings were
|
||||
* scaled to its width (`store.ts`). CSS caps it at about 70vh: taller HTML scrolls inside the frame, an SVG scales
|
||||
* down to fit. Rendered for every viewer of the session (owner decision).
|
||||
*/
|
||||
export function InlineArtifact({ sessionId, artifact }: { sessionId: string; artifact: ArtifactInfo }) {
|
||||
const url = artifactUrls(sessionId, artifact).view;
|
||||
const document = artifact.type === "markdown" || artifact.type === "text";
|
||||
const text = useArtifactText(document ? url : undefined);
|
||||
const [available, setAvailable] = useState<{ url: string; ok: boolean }>();
|
||||
const frame = useRef<HTMLIFrameElement>(null);
|
||||
const [height, setHeight] = useState(() => frameHeights.get(url));
|
||||
|
||||
// The frame would show the route's bare 404 text: check the version first.
|
||||
useEffect(() => {
|
||||
if (document) return;
|
||||
const abort = new AbortController();
|
||||
void fetch(url, { method: "HEAD", credentials: "same-origin", cache: "no-store", signal: abort.signal })
|
||||
.then((response) => setAvailable({ url, ok: response.ok }))
|
||||
.catch(() => {
|
||||
if (!abort.signal.aborted) setAvailable({ url, ok: false });
|
||||
});
|
||||
return () => abort.abort();
|
||||
}, [document, url]);
|
||||
|
||||
useFrameMessages(frame, (message) => {
|
||||
if (message.level !== "size") return;
|
||||
const next = Math.min(Math.ceil(message.height), 100_000);
|
||||
frameHeights.delete(url);
|
||||
frameHeights.set(url, next);
|
||||
if (frameHeights.size > FRAME_HEIGHTS) frameHeights.delete(frameHeights.keys().next().value ?? "");
|
||||
setHeight(next);
|
||||
});
|
||||
|
||||
const unavailable = document ? text.error !== undefined : available?.url === url && !available.ok;
|
||||
return (
|
||||
<div className={`inline-artifact ${artifact.type}`} data-artifact={`${artifact.name}:${artifact.version}`}>
|
||||
{unavailable ? (
|
||||
<div className="notice error">Artifact unavailable</div>
|
||||
) : document ? (
|
||||
text.content === undefined ? (
|
||||
<div className="spinner" />
|
||||
) : artifact.type === "markdown" ? (
|
||||
<Markdown text={text.content} />
|
||||
) : (
|
||||
<pre>{text.content}</pre>
|
||||
)
|
||||
) : (
|
||||
<ArtifactFrame url={url} title={artifact.title} frame={frame} height={height ?? FRAME_HEIGHT} />
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -14,20 +14,30 @@ export type ArtifactType = (typeof TYPES)[number];
|
|||
export type ArtifactInfo = { name: string; title: string; type: ArtifactType; version: number };
|
||||
export type ArtifactInput = { name: string; title: string; type: ArtifactType; content: string };
|
||||
|
||||
const DIAGNOSTIC_BOOTSTRAP = `<script>(()=>{const channel="p2-artifact-diagnostic";const send=(level,args)=>{try{parent.postMessage({channel,level,message:args.map(value=>{try{return typeof value==="string"?value:JSON.stringify(value)}catch{return String(value)}}).join(" ").slice(0,4000)},"*")}catch{}};for(const level of ["error","warn"]){const original=console[level];console[level]=(...args)=>{send(level,args);original.apply(console,args)}}addEventListener("error",event=>send("error",[event.message,event.filename&&(event.filename+":"+event.lineno+":"+event.colno)].filter(Boolean)));addEventListener("unhandledrejection",event=>send("error",[event.reason?.stack||event.reason||"Unhandled promise rejection"]));addEventListener("securitypolicyviolation",event=>send("csp",[event.violatedDirective,event.blockedURI||"blocked"]));addEventListener("webglcontextlost",event=>send("webgl",["WebGL context lost"]),true);send("ready",["Preview diagnostics connected"]);})();</script>`;
|
||||
// The preview's diagnostics bridge: console errors and warnings, uncaught errors, CSP violations and lost WebGL
|
||||
// contexts, plus the document's content height (level "size", for inline previews sized to their content). Drawings fill
|
||||
// the frame's width: an SVG document with a viewBox (or an absolute width and height, made its viewBox) is set to
|
||||
// 100% wide and high, so it scales with its aspect and reports viewBox height at its width; an HTML document whose
|
||||
// body is just a viewBox SVG or an image gets them 100% wide at height auto (zero specificity: any author rule wins).
|
||||
// HTML is measured without its scrollbar: a frame still too short shows one, and the narrower page would report a
|
||||
// taller height, which keeps the scrollbar once the frame grows to it.
|
||||
const DIAGNOSTIC_SCRIPT = `(()=>{const channel="p2-artifact-diagnostic";const send=(level,args)=>{try{parent.postMessage({channel,level,message:args.map(value=>{try{return typeof value==="string"?value:JSON.stringify(value)}catch{return String(value)}}).join(" ").slice(0,4000)},"*")}catch{}};for(const level of ["error","warn"]){const original=console[level];console[level]=(...args)=>{send(level,args);original.apply(console,args)}}addEventListener("error",event=>send("error",[event.message,event.filename&&(event.filename+":"+event.lineno+":"+event.colno)].filter(Boolean)));addEventListener("unhandledrejection",event=>send("error",[event.reason?.stack||event.reason||"Unhandled promise rejection"]));addEventListener("securitypolicyviolation",event=>send("csp",[event.violatedDirective,event.blockedURI||"blocked"]));addEventListener("webglcontextlost",event=>send("webgl",["WebGL context lost"]),true);send("ready",["Preview diagnostics connected"]);const fit=()=>{const root=document.documentElement;if(root instanceof SVGSVGElement){const width=root.width.baseVal,height=root.height.baseVal,absolute=length=>length.unitType!==SVGLength.SVG_LENGTHTYPE_PERCENTAGE&&length.value>0;if(!root.hasAttribute("viewBox")&&absolute(width)&&absolute(height))root.setAttribute("viewBox","0 0 "+width.value+" "+height.value);const box=root.viewBox.baseVal;if(box&&box.width>0&&box.height>0){root.setAttribute("width","100%");root.setAttribute("height","100%")}}else if(root){const style=document.createElement("style");style.textContent=":where(body>svg[viewBox]:only-child,body>img:only-child){display:block;width:100%;height:auto}";(document.head||root).prepend(style)}};try{fit()}catch{}let reported=0;const measure=()=>{const root=document.documentElement;if(!root)return;let height;if(root instanceof SVGSVGElement){const box=root.viewBox.baseVal;height=root.height.baseVal.unitType===SVGLength.SVG_LENGTHTYPE_PERCENTAGE&&box&&box.width>0&&box.height>0?root.getBoundingClientRect().width*box.height/box.width:root.getBoundingClientRect().height}else{const overflow=root.style.overflow;root.style.overflow="hidden";height=Math.max(root.getBoundingClientRect().height,document.body?document.body.scrollHeight:0);root.style.overflow=overflow}height=Math.ceil(height);if(height===reported)return;reported=height;try{parent.postMessage({channel,level:"size",message:"",height},"*")}catch{}};const observe=()=>{try{const observer=new ResizeObserver(measure);observer.observe(document.documentElement);if(document.body)observer.observe(document.body)}catch{}measure()};addEventListener("load",measure);addEventListener("resize",measure);if(document.readyState==="loading")addEventListener("DOMContentLoaded",observe);else observe();})();`;
|
||||
|
||||
/** Add a diagnostics bridge without changing the immutable stored source. */
|
||||
export function instrumentArtifact(info: ArtifactInfo, content: string): string {
|
||||
if (info.type === "html") {
|
||||
const head = /<head(?:\s[^>]*)?>/i.exec(content);
|
||||
const script = `<script>${DIAGNOSTIC_SCRIPT}</script>`;
|
||||
if (head)
|
||||
return `${content.slice(0, head.index + head[0].length)}${DIAGNOSTIC_BOOTSTRAP}${content.slice(head.index + head[0].length)}`;
|
||||
return `${DIAGNOSTIC_BOOTSTRAP}${content}`;
|
||||
return `${content.slice(0, head.index + head[0].length)}${script}${content.slice(head.index + head[0].length)}`;
|
||||
return `${script}${content}`;
|
||||
}
|
||||
if (info.type === "svg") {
|
||||
const root = /<svg(?:\s[^>]*)?>/i.exec(content);
|
||||
if (root)
|
||||
return `${content.slice(0, root.index + root[0].length)}${DIAGNOSTIC_BOOTSTRAP}${content.slice(root.index + root[0].length)}`;
|
||||
// SVG is XML: the script's `&&` would end the document without CDATA. An empty root (`<svg/>`) has no inside.
|
||||
const script = `<script><![CDATA[${DIAGNOSTIC_SCRIPT}]]></script>`;
|
||||
if (root && !root[0].endsWith("/>"))
|
||||
return `${content.slice(0, root.index + root[0].length)}${script}${content.slice(root.index + root[0].length)}`;
|
||||
}
|
||||
return content;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,11 +1,17 @@
|
|||
import { defineDoc, defineExtension, defineTool } from "@earendil-works/pi-durable";
|
||||
import { obj, str, union } from "@earendil-works/pi-schema";
|
||||
import { bool, obj, opt, str, union } from "@earendil-works/pi-schema";
|
||||
import { literal } from "@p2/app/v1";
|
||||
import { agentConversation } from "@p2/app/v1/worker";
|
||||
import { type ArtifactInfo, Artifacts } from "./store.ts";
|
||||
|
||||
type ArtifactState = { artifacts: ArtifactInfo[] };
|
||||
|
||||
/**
|
||||
* The tool's result details: the saved version, and `inline: true` when the call asked to show it at the call site
|
||||
* too (the transcripts render that version below the card). The index (`ArtifactDoc`) keeps only `ArtifactInfo`.
|
||||
*/
|
||||
export type ArtifactDetails = ArtifactInfo & { inline?: true };
|
||||
|
||||
/**
|
||||
* Complete artifact index of one agent (on conversation 1 for the main agent, see `agentConversation`); reset and
|
||||
* compaction never rewind it.
|
||||
|
|
@ -30,16 +36,22 @@ export function createArtifactExtension(dataDirectory: string, sessionId: string
|
|||
defineTool({
|
||||
name: "artifact",
|
||||
description:
|
||||
"Create or update a session artifact (HTML, SVG, Markdown or text; maximum 2 MiB UTF-8). Reuse its stable lowercase slug name to publish a new immutable numbered version. Versions persist across restarts. All authorized viewers of this session may explicitly click to open an isolated preview; previews never open automatically. Do not publish secrets. HTML may use HTTPS CDN resources, but has no app identity or parent privileges.",
|
||||
"Create or update a session artifact (HTML, SVG, Markdown or text; maximum 2 MiB UTF-8). Reuse its stable lowercase slug name to publish a new immutable numbered version. Versions persist across restarts. Viewers open an isolated preview from the card in the transcript. Set inline: true to also show this version right below the card, in the chat, for every viewer of the session: use it for diagrams (prefer SVG, e.g. sequence or architecture diagrams, over ASCII art), charts and other small visual results the reader should see without opening anything; keep inline content compact (it is shown at most about 70% of the screen high, then scrolls) and leave large documents and apps as plain artifacts. Do not publish secrets. HTML and SVG run in a sandbox with no app identity or parent privileges; HTML may use HTTPS CDN resources.",
|
||||
parameters: obj({
|
||||
name: str({ pattern: "^[a-z0-9][a-z0-9_-]{0,63}$", minLength: 1, maxLength: 64 }),
|
||||
title: str({ minLength: 1, maxLength: 200 }),
|
||||
type: union([literal("html"), literal("svg"), literal("markdown"), literal("text")]),
|
||||
content: str(),
|
||||
inline: opt(
|
||||
bool({
|
||||
description:
|
||||
"Also render it inline at the tool call site in the transcript (default false). For diagrams and small visual results.",
|
||||
}),
|
||||
),
|
||||
}),
|
||||
replay: "unsafe",
|
||||
execute: async (args, api, context) => {
|
||||
const info = await artifacts.save(sessionId, args);
|
||||
execute: async ({ inline, ...input }, api, context) => {
|
||||
const info = await artifacts.save(sessionId, input);
|
||||
// The main agent's list is one, whichever `/tree` branch it works in.
|
||||
const holder = await agentConversation(api, api.conversationId, context);
|
||||
await api.commit(async (tx) => {
|
||||
|
|
@ -55,10 +67,10 @@ export function createArtifactExtension(dataDirectory: string, sessionId: string
|
|||
output: [
|
||||
{
|
||||
type: "text" as const,
|
||||
text: `Artifact ready: ${info.title} (${info.name}, version ${info.version}).`,
|
||||
text: `Artifact ready: ${info.title} (${info.name}, version ${info.version}).${inline ? " Shown inline in the transcript." : ""}`,
|
||||
},
|
||||
],
|
||||
details: info,
|
||||
details: (inline ? { ...info, inline: true } : info) satisfies ArtifactDetails,
|
||||
};
|
||||
},
|
||||
}),
|
||||
|
|
|
|||
|
|
@ -1,10 +1,11 @@
|
|||
// Artifacts in the TUI (p2-tui-port.md P11c), after the web's artifact card, picker and viewer. The card shows the
|
||||
// title, type and version; expanded, a text or Markdown artifact shows the content the call wrote. `/artifacts` lists
|
||||
// title, type and version; expanded, a text or Markdown artifact shows the content the call wrote. A call with
|
||||
// `inline: true` shows all of it at once (Markdown rendered); inline HTML and SVG keep the card line. `/artifacts` lists
|
||||
// the conversation's artifacts (`ArtifactList`): Enter shows one inline (`ArtifactContent`; Markdown rendered, HTML and
|
||||
// SVG as source; in the side panel, where it follows new versions, when the terminal is wide enough), Ctrl+B opens it in the browser (launcher `openUrl`; the browser needs its own sign-in) and Ctrl+S
|
||||
// downloads it (launcher `download`). The worker never fetches the `/artifact` route itself: it has no cookie.
|
||||
|
||||
import { Markdown, Text } from "@earendil-works/pi-tui";
|
||||
import { Container, Markdown, Spacer, Text } from "@earendil-works/pi-tui";
|
||||
import { BACKGROUND_CONTEXT, contribute, defineFacet, follow, statusOf, whenLive } from "@p2/app/v1";
|
||||
import type { TuiContext } from "@p2/app/v1/tui";
|
||||
import {
|
||||
|
|
@ -39,7 +40,9 @@ const artifactRenderers: ToolRenderers = {
|
|||
return text;
|
||||
},
|
||||
renderResult(result, options, theme, context) {
|
||||
const text = context.lastComponent instanceof Text ? context.lastComponent : new Text("", 0, 0);
|
||||
const component = context.lastComponent instanceof Container ? context.lastComponent : new Container();
|
||||
component.clear();
|
||||
component.addChild(new Spacer(1));
|
||||
const details = record(result.details);
|
||||
const saved =
|
||||
!context.isError &&
|
||||
|
|
@ -47,28 +50,35 @@ const artifactRenderers: ToolRenderers = {
|
|||
typeof details.type === "string" &&
|
||||
typeof details.version === "number";
|
||||
if (!saved) {
|
||||
text.setText(`\n${theme.fg(context.isError ? "error" : "toolOutput", getTextOutput(result))}`);
|
||||
return text;
|
||||
component.addChild(new Text(theme.fg(context.isError ? "error" : "toolOutput", getTextOutput(result)), 0, 0));
|
||||
return component;
|
||||
}
|
||||
const lines = [
|
||||
`${theme.fg("success", "■")} ${lineText(String(details.title))} ${theme.fg("muted", `· ${lineText(String(details.type)).toUpperCase()} · version ${details.version}`)} ${theme.fg("dim", "· /artifacts opens")}`,
|
||||
];
|
||||
// What the call wrote (text and Markdown; HTML and SVG run only in a browser).
|
||||
component.addChild(
|
||||
new Text(
|
||||
`${theme.fg("success", "■")} ${lineText(String(details.title))} ${theme.fg("muted", `· ${lineText(String(details.type)).toUpperCase()} · version ${details.version}`)} ${theme.fg("dim", "· /artifacts opens")}`,
|
||||
0,
|
||||
0,
|
||||
),
|
||||
);
|
||||
// What the call wrote (text and Markdown; HTML and SVG run only in a browser): the version this call produced.
|
||||
const content = str(context.args.content);
|
||||
if (content && (details.type === "text" || details.type === "markdown")) {
|
||||
const all = content.split("\n").map((line) => theme.fg("toolOutput", lineText(line)));
|
||||
const shown = options.expanded ? all : all.slice(0, PREVIEW_LINES);
|
||||
lines.push("", ...shown);
|
||||
if (shown.length < all.length)
|
||||
lines.push(
|
||||
theme.fg(
|
||||
"muted",
|
||||
`... (${all.length - shown.length} more lines, ${context.ui.keyText("app.tools.expand")} to expand)`,
|
||||
),
|
||||
);
|
||||
if (!content || (details.type !== "text" && details.type !== "markdown")) return component;
|
||||
component.addChild(new Spacer(1));
|
||||
if (details.inline === true && details.type === "markdown") {
|
||||
component.addChild(new Markdown(safeText(content), 0, 0, context.ui.theme.markdown));
|
||||
return component;
|
||||
}
|
||||
text.setText(`\n${lines.join("\n")}`);
|
||||
return text;
|
||||
const all = content.split("\n").map((line) => theme.fg("toolOutput", lineText(line)));
|
||||
const shown = options.expanded || details.inline === true ? all : all.slice(0, PREVIEW_LINES);
|
||||
if (shown.length < all.length)
|
||||
shown.push(
|
||||
theme.fg(
|
||||
"muted",
|
||||
`... (${all.length - shown.length} more lines, ${context.ui.keyText("app.tools.expand")} to expand)`,
|
||||
),
|
||||
);
|
||||
component.addChild(new Text(shown.join("\n"), 0, 0));
|
||||
return component;
|
||||
},
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import {
|
|||
ArtifactPicker,
|
||||
closeLayers,
|
||||
FilePanel,
|
||||
InlineArtifactView,
|
||||
OpenViewer,
|
||||
openLayer,
|
||||
SessionId,
|
||||
|
|
@ -17,7 +18,7 @@ import {
|
|||
import type { ReactNode } from "react";
|
||||
import { DownloadInfoSchema } from "../files/contract.ts";
|
||||
import { ArtifactInfoSchema, ArtifactList } from "./contract.ts";
|
||||
import { ArtifactPanel } from "./preview.tsx";
|
||||
import { ArtifactPanel, InlineArtifact } from "./preview.tsx";
|
||||
|
||||
const ViewerSchema = union([
|
||||
obj({ kind: literal("file"), file: DownloadInfoSchema }, { open: true }),
|
||||
|
|
@ -51,7 +52,7 @@ function Content({ children }: { children?: ReactNode }) {
|
|||
return (
|
||||
<SessionId.Provider value={route.session}>
|
||||
<OpenViewer.Provider value={open}>
|
||||
{children}
|
||||
<InlineArtifactView.Provider value={InlineArtifact}>{children}</InlineArtifactView.Provider>
|
||||
{top?.kind === "sheet" && top.name === "artifacts" && (
|
||||
<ArtifactPicker
|
||||
artifacts={artifacts.toReversed()}
|
||||
|
|
|
|||
|
|
@ -187,7 +187,7 @@ export function createFakeModels(options: CreateModelsOptions = {}) {
|
|||
? fauxToolCall("subagent", JSON.parse(text.slice("subagent ".length)))
|
||||
: text.startsWith("team ")
|
||||
? fauxToolCall("team", JSON.parse(text.slice("team ".length)))
|
||||
: /^(history_(search|read|handoff)|location_(envs|dirs|move)|tool_search|mcp__\w+) /.test(
|
||||
: /^(artifact|history_(search|read|handoff)|location_(envs|dirs|move)|tool_search|mcp__\w+) /.test(
|
||||
text,
|
||||
)
|
||||
? fauxToolCall(
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@ import { Markdown } from "./Markdown.tsx";
|
|||
import { RowEstimator, type RowFeatures } from "./row-estimate.ts";
|
||||
import type { Item, NestedCall, ToolView, Transcript } from "./transcript.ts";
|
||||
import { type RowRange, revealInTranscripts, type TranscriptScroll } from "./use-chat-scroll.ts";
|
||||
import { artifactUrls, FileCard, fileMeta, fileUrl, OpenViewer } from "./Viewer.tsx";
|
||||
import { artifactUrls, FileCard, fileMeta, fileUrl, InlineArtifactView, OpenViewer } from "./Viewer.tsx";
|
||||
|
||||
function tokens(count: number): string {
|
||||
return count < 1000
|
||||
|
|
@ -387,6 +387,7 @@ const ToolCard = memo(function ToolCard({ tool }: { tool: ToolView }) {
|
|||
const [open, setOpen] = useState(false);
|
||||
const openConversation = useContext(OpenConversation);
|
||||
const openViewer = useContext(OpenViewer);
|
||||
const InlineArtifact = useContext(InlineArtifactView);
|
||||
const sessionId = useContext(SessionId);
|
||||
const cwd = useContext(WorkingDirectory)?.replace(/\/+$/, "");
|
||||
const active = tool.status === "running" || tool.status === "pending";
|
||||
|
|
@ -543,17 +544,28 @@ const ToolCard = memo(function ToolCard({ tool }: { tool: ToolView }) {
|
|||
Number.isSafeInteger(details.version) &&
|
||||
details.version > 0
|
||||
) {
|
||||
const artifact = details as ArtifactInfo;
|
||||
// The version this call produced, without the details' `inline` flag (the viewer's locator is closed).
|
||||
const artifact: ArtifactInfo = {
|
||||
name: details.name,
|
||||
title: details.title,
|
||||
type: details.type as ArtifactInfo["type"],
|
||||
version: details.version,
|
||||
};
|
||||
const urls = artifactUrls(sessionId, artifact);
|
||||
card = (
|
||||
<FileCard
|
||||
icon={<ArtifactIcon />}
|
||||
title={artifact.title}
|
||||
meta={`${artifact.type.toUpperCase()} · version ${artifact.version}`}
|
||||
href={urls.download}
|
||||
download={urls.filename}
|
||||
onOpen={() => openViewer({ kind: "artifact", artifact })}
|
||||
/>
|
||||
<>
|
||||
<FileCard
|
||||
icon={<ArtifactIcon />}
|
||||
title={artifact.title}
|
||||
meta={`${artifact.type.toUpperCase()} · version ${artifact.version}`}
|
||||
href={urls.download}
|
||||
download={urls.filename}
|
||||
onOpen={() => openViewer({ kind: "artifact", artifact })}
|
||||
/>
|
||||
{details.inline === true && InlineArtifact && (
|
||||
<InlineArtifact sessionId={sessionId} artifact={artifact} />
|
||||
)}
|
||||
</>
|
||||
);
|
||||
}
|
||||
break;
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
import { createContext, type ReactNode, useEffect, useRef, useState } from "react";
|
||||
import { type ComponentType, createContext, type ReactNode, useEffect, useRef, useState } from "react";
|
||||
import { type DownloadInfo, filePreview } from "../runtime/files.ts";
|
||||
import type { ArtifactInfo } from "../runtime/schemas.ts";
|
||||
import { useScrollMemory } from "../runtime/web.tsx";
|
||||
|
|
@ -12,6 +12,13 @@ import { Markdown } from "./Markdown.tsx";
|
|||
export type ViewerItem = { kind: "artifact"; artifact: ArtifactInfo } | { kind: "file"; file: DownloadInfo };
|
||||
/** Opens an item explicitly; tool updates and reconnects never open the viewer. */
|
||||
export const OpenViewer = createContext<(item: ViewerItem) => void>(() => {});
|
||||
/**
|
||||
* Renders an artifact version at its tool call site, below the card, when the call asked for `inline` (the artifacts
|
||||
* plugin provides it; without it the card stands alone).
|
||||
*/
|
||||
export const InlineArtifactView = createContext<ComponentType<{ sessionId: string; artifact: ArtifactInfo }> | null>(
|
||||
null,
|
||||
);
|
||||
|
||||
function fileSize(bytes: number): string {
|
||||
if (bytes < 1024) return `${bytes} B`;
|
||||
|
|
|
|||
|
|
@ -1433,6 +1433,36 @@ textarea.input {
|
|||
background: var(--hover);
|
||||
color: var(--text);
|
||||
}
|
||||
/* An artifact rendered at its tool call site (`inline: true`): below the card, at most about 70vh high. */
|
||||
.inline-artifact {
|
||||
margin-top: var(--s2);
|
||||
border: 1px solid var(--line);
|
||||
border-radius: var(--radius);
|
||||
overflow: hidden;
|
||||
}
|
||||
.inline-artifact.markdown,
|
||||
.inline-artifact.text {
|
||||
max-height: 70vh;
|
||||
overflow: auto;
|
||||
padding: var(--s3);
|
||||
}
|
||||
.inline-artifact > pre {
|
||||
margin: 0;
|
||||
font-family: var(--mono);
|
||||
font-size: var(--fs-sm);
|
||||
overflow-wrap: anywhere;
|
||||
white-space: pre-wrap;
|
||||
}
|
||||
.inline-artifact > iframe {
|
||||
display: block;
|
||||
width: 100%;
|
||||
max-height: 70vh;
|
||||
border: 0;
|
||||
background: white;
|
||||
}
|
||||
.inline-artifact > .notice {
|
||||
margin: 0;
|
||||
}
|
||||
.file-media {
|
||||
display: block;
|
||||
width: 100%;
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ import { Sessions } from "../../plugins/sessions/contract.ts";
|
|||
import { Sharing } from "../../plugins/sharing/contract.ts";
|
||||
import type { Host, RefIn } from "../../src/runtime/chord.ts";
|
||||
import { Hosts } from "../../src/runtime/server.ts";
|
||||
import { expectChord, type Fixture, holdIn, http, startFixture, until } from "./helpers.ts";
|
||||
import { expectChord, type Fixture, holdIn, http, settled, startFixture, until } from "./helpers.ts";
|
||||
|
||||
const results = (view: ConversationView | null | undefined): ToolResultMessage[] =>
|
||||
(view?.entries ?? []).flatMap((entry) =>
|
||||
|
|
@ -148,6 +148,35 @@ describe("artifacts (fake provider)", () => {
|
|||
sharing.release();
|
||||
});
|
||||
|
||||
it("carries inline in the result details only, and keeps the index and stored version unchanged", async () => {
|
||||
const svg =
|
||||
"<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 200 100'><rect width='200' height='100'/></svg>";
|
||||
const call = { name: "flow", title: "Flow", type: "svg", content: svg, inline: true };
|
||||
const before = results(await settled(chat.view)).length;
|
||||
await chat.send({ text: `artifact ${JSON.stringify(call)}`, requestId: "inline-svg" }, BACKGROUND_CONTEXT);
|
||||
const view = await until(
|
||||
chat.view,
|
||||
(value) => results(value).length === before + 1 && !(value?.docs["pi.live"] as { run?: unknown })?.run,
|
||||
);
|
||||
const result = results(view).at(-1);
|
||||
assert.equal(result?.isError, false);
|
||||
assert.deepEqual(result?.details, { name: "flow", title: "Flow", type: "svg", version: 1, inline: true });
|
||||
assert.match(JSON.stringify(result?.content), /Shown inline in the transcript/);
|
||||
// The plain call has no flag at all.
|
||||
assert.deepEqual(results(view)[0]?.details, { name: "demo", title: "Demo artifact", type: "html", version: 1 });
|
||||
const index = await until(list.artifacts, (value) => Boolean(value?.some((entry) => entry.name === "flow")));
|
||||
assert.deepEqual(
|
||||
index?.find((entry) => entry.name === "flow"),
|
||||
{ name: "flow", title: "Flow", type: "svg", version: 1 },
|
||||
);
|
||||
assert.equal((await http(fixture.server.port, `${url("flow")}&download=1`)).body, svg);
|
||||
// The preview's bridge is CDATA inside the root (SVG is XML) and reports the content height.
|
||||
const preview = (await http(fixture.server.port, url("flow"))).body;
|
||||
assert.ok(preview.startsWith("<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 200 100'><script><![CDATA["));
|
||||
assert.match(preview, /level:"size"/);
|
||||
assert.ok(preview.endsWith("]]></script><rect width='200' height='100'/></svg>"));
|
||||
});
|
||||
|
||||
it("serves Markdown/text as non-executable documents and validates byte bounds", async () => {
|
||||
const store = new Store(fixture.dataDir);
|
||||
for (const type of ["markdown", "text", "svg"] as const) {
|
||||
|
|
@ -159,7 +188,9 @@ describe("artifacts (fake provider)", () => {
|
|||
});
|
||||
const response = await http(fixture.server.port, url(info.name, info.version));
|
||||
assert.equal(response.status, 200);
|
||||
if (type !== "svg") {
|
||||
// An empty root has no inside for the bridge: served as stored, still well-formed XML.
|
||||
if (type === "svg") assert.equal(response.body, "<svg xmlns='http://www.w3.org/2000/svg'/>");
|
||||
else {
|
||||
assert.equal(response.headers["content-type"], "text/plain; charset=utf-8");
|
||||
assert.match(String(response.headers["content-security-policy"]), /; sandbox$/);
|
||||
assert.ok(!response.body.includes("p2-artifact-diagnostic"));
|
||||
|
|
|
|||
|
|
@ -86,13 +86,42 @@ export class Browser {
|
|||
await browser.call("Page.setLifecycleEventsEnabled", { enabled: true });
|
||||
return browser;
|
||||
}
|
||||
call(method: string, params: object = {}): Promise<unknown> {
|
||||
/** Sends a DevTools command to the page, or to an attached target's flat session (`evaluateInFrame`). */
|
||||
call(method: string, params: object = {}, sessionId?: string): Promise<unknown> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const id = ++this.id;
|
||||
this.pending.set(id, { resolve, reject });
|
||||
this.socket.send(JSON.stringify({ id, method, params }));
|
||||
this.socket.send(JSON.stringify({ id, method, params, ...(sessionId ? { sessionId } : {}) }));
|
||||
});
|
||||
}
|
||||
/**
|
||||
* Evaluates `expression` in the out-of-process frame whose URL contains `url`: an artifact preview has an opaque
|
||||
* origin, so the page itself cannot reach into it.
|
||||
*/
|
||||
async evaluateInFrame<T>(url: string, expression: string): Promise<T> {
|
||||
const { targetInfos } = (await this.call("Target.getTargets")) as {
|
||||
targetInfos: { targetId: string; type: string; url: string }[];
|
||||
};
|
||||
const target = targetInfos.find((info) => info.type === "iframe" && info.url.includes(url));
|
||||
if (!target) throw new Error(`No frame target for ${url}`);
|
||||
const { sessionId } = (await this.call("Target.attachToTarget", {
|
||||
targetId: target.targetId,
|
||||
flatten: true,
|
||||
})) as {
|
||||
sessionId: string;
|
||||
};
|
||||
try {
|
||||
const response = (await this.call(
|
||||
"Runtime.evaluate",
|
||||
{ expression, returnByValue: true, awaitPromise: true },
|
||||
sessionId,
|
||||
)) as { result: { value: T }; exceptionDetails?: unknown };
|
||||
if (response.exceptionDetails) throw new Error(JSON.stringify(response.exceptionDetails));
|
||||
return response.result.value;
|
||||
} finally {
|
||||
await this.call("Target.detachFromTarget", { sessionId }).catch(() => {});
|
||||
}
|
||||
}
|
||||
async evaluate<T>(expression: string): Promise<T> {
|
||||
const response = (await this.call("Runtime.evaluate", {
|
||||
expression,
|
||||
|
|
|
|||
|
|
@ -1081,6 +1081,60 @@ test("artifacts: the card previews text, /artifacts shows Markdown inline when n
|
|||
await poll(() => runtime.notices.some((notice) => notice.text === "Downloaded Page"));
|
||||
});
|
||||
|
||||
test("artifacts: an inline call renders all of its Markdown in the transcript; inline SVG keeps the card line", async (t) => {
|
||||
const calls: Calls = {};
|
||||
const markdown = [
|
||||
"# Inline heading",
|
||||
"",
|
||||
"Some **bold** text",
|
||||
...Array.from({ length: 10 }, (_, i) => `- item ${i + 1}`),
|
||||
];
|
||||
const svg = "<svg xmlns='http://www.w3.org/2000/svg'><text>svg-source</text></svg>";
|
||||
const { screen, terminal } = await setup(t, {
|
||||
calls,
|
||||
worker: contentStubs(calls),
|
||||
facets: [tuiCoreFacet, chatTui, statusTui, artifactsTui],
|
||||
views: () =>
|
||||
viewOf([
|
||||
callEntry(1, "a1", "artifact", {
|
||||
name: "notes",
|
||||
title: "Notes",
|
||||
type: "markdown",
|
||||
content: markdown.join("\n"),
|
||||
inline: true,
|
||||
}),
|
||||
resultEntry(
|
||||
2,
|
||||
"a1",
|
||||
"artifact",
|
||||
{ name: "notes", title: "Notes", type: "markdown", version: 3, inline: true },
|
||||
"Saved",
|
||||
),
|
||||
callEntry(3, "a2", "artifact", { name: "flow", title: "Flow", type: "svg", content: svg, inline: true }),
|
||||
resultEntry(
|
||||
4,
|
||||
"a2",
|
||||
"artifact",
|
||||
{ name: "flow", title: "Flow", type: "svg", version: 1, inline: true },
|
||||
"Saved",
|
||||
),
|
||||
]),
|
||||
});
|
||||
terminal.resize(100, 50);
|
||||
await poll(async () => {
|
||||
const text = await screen();
|
||||
return text.includes("Flow · SVG · version 1 · /artifacts opens") && text.includes("Notes · MARKDOWN");
|
||||
});
|
||||
const shown = await screen();
|
||||
assert.match(shown, /Notes · MARKDOWN · version 3/);
|
||||
assert(shown.includes("Inline heading") && !shown.includes("# Inline heading"), "Markdown is rendered");
|
||||
assert(shown.includes("bold") && !shown.includes("**bold**"));
|
||||
assert(shown.includes("item 10"), "all of it, not the collapsed preview");
|
||||
assert(!shown.includes("more lines"));
|
||||
assert(!shown.includes("svg-source"), "HTML and SVG run only in a browser");
|
||||
assert.equal(calls.readArtifact, undefined, "the call's own content, no read");
|
||||
});
|
||||
|
||||
test("artifacts: Enter shows one in the side panel, which follows new versions, scrolls when focused and hides when narrow", async (t) => {
|
||||
const calls: Calls = {};
|
||||
let list: Cell<{ name: string; title: string; type: string; version: number }[]> | undefined;
|
||||
|
|
|
|||
259
apps/p2/test/p2/ui-inline-artifacts.test.ts
Normal file
259
apps/p2/test/p2/ui-inline-artifacts.test.ts
Normal file
|
|
@ -0,0 +1,259 @@
|
|||
// Inline artifacts (`artifact` with `inline: true`) in Chromium: the call's version renders below its card, Markdown
|
||||
// as in the viewer, SVG and HTML drawings filling the sandboxed frame's width, the frame fitted to the content height
|
||||
// they report (capped at 70vh), for the owner and for a guest alike; the card still opens the full viewer; a version
|
||||
// the route cannot serve says so. Fake provider (`artifact <json>`), disposable sessions.
|
||||
|
||||
import assert from "node:assert/strict";
|
||||
import { rmSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { after, before, describe, it } from "node:test";
|
||||
import type { ToolResultMessage } from "@earendil-works/pi-ai";
|
||||
import { BACKGROUND_CONTEXT } from "@p2/app/v1";
|
||||
import { Chat } from "../../plugins/chat/contract.ts";
|
||||
import { Sessions } from "../../plugins/sessions/contract.ts";
|
||||
import { Sharing } from "../../plugins/sharing/contract.ts";
|
||||
import type { RefIn } from "../../src/runtime/chord.ts";
|
||||
import { Browser, browserTests } from "./browser.ts";
|
||||
import { type Fixture, holdIn, poll, startFixture, until } from "./helpers.ts";
|
||||
|
||||
type Call = { name: string; title: string; type: "html" | "svg" | "markdown" | "text"; content: string };
|
||||
|
||||
const setInput = (selector: string, value: string) =>
|
||||
`(() => { const input=document.querySelector(${JSON.stringify(selector)});Object.getOwnPropertyDescriptor(HTMLInputElement.prototype,'value').set.call(input,${JSON.stringify(value)});input.dispatchEvent(new Event('input',{bubbles:true})); })()`;
|
||||
const frame = (name: string) => `document.querySelector('.inline-artifact[data-artifact="${name}:1"] iframe')`;
|
||||
|
||||
/** The drawing's rendered size inside an SVG preview (its viewBox through the root's scale) and the frame's. */
|
||||
type Drawn = { width: number; height: number; frameWidth: number; frameHeight: number };
|
||||
const DRAWN =
|
||||
"(()=>{const root=document.documentElement,box=root.viewBox.baseVal,scale=root.getScreenCTM().a;return {width:box.width*scale,height:box.height*scale,frameWidth:innerWidth,frameHeight:innerHeight}})()";
|
||||
|
||||
const DIAGRAM =
|
||||
"<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 400 200'><rect width='400' height='200' fill='#eef'/><line x1='100' y1='40' x2='100' y2='190' stroke='#333'/><line x1='300' y1='40' x2='300' y2='190' stroke='#333'/><text x='100' y='30' text-anchor='middle'>Client</text><text x='300' y='30' text-anchor='middle'>Server</text><path d='M100 80 H300' stroke='#06c' marker-end='url(#a)'/><defs><marker id='a' markerWidth='8' markerHeight='8' refX='8' refY='4' orient='auto'><path d='M0 0 L8 4 L0 8 z' fill='#06c'/></marker></defs></svg>";
|
||||
|
||||
describe("inline artifacts in Chromium", { skip: !browserTests }, () => {
|
||||
let fixture: Fixture;
|
||||
let browser: Browser;
|
||||
let sessions: RefIn<typeof Sessions, "owner">;
|
||||
|
||||
/** A new session whose agent publishes `calls` with `inline: true`, each to the end of its run. */
|
||||
const publish = async (title: string, calls: Call[]): Promise<string> => {
|
||||
const id = (await sessions.create({ cwd: fixture.workspace, title }, BACKGROUND_CONTEXT)).id;
|
||||
const chat = await holdIn(fixture.browser(), Chat, { session: id, conversation: 1 }, "participant");
|
||||
for (const [index, call] of calls.entries()) {
|
||||
await chat.send(
|
||||
{ text: `artifact ${JSON.stringify({ ...call, inline: true })}`, requestId: `${call.name}-${index}` },
|
||||
BACKGROUND_CONTEXT,
|
||||
);
|
||||
await until(chat.view, (view) => {
|
||||
const done = (view?.entries ?? []).flatMap((entry) =>
|
||||
(entry.model ?? []).filter(
|
||||
(message): message is ToolResultMessage =>
|
||||
message.role === "toolResult" && message.toolName === "artifact",
|
||||
),
|
||||
);
|
||||
return done.length === index + 1 && !(view?.docs["pi.live"] as { run?: unknown })?.run;
|
||||
});
|
||||
}
|
||||
chat.release();
|
||||
return id;
|
||||
};
|
||||
|
||||
/**
|
||||
* Waits until the drawing inside a preview frame (`expression` gives its rendered size and the frame's viewport, read
|
||||
* inside the frame) fills the frame's width and height, and the iframe element on the page has that size too. The
|
||||
* frame's own viewport follows the element's size a moment later (another process).
|
||||
*/
|
||||
const fitsFrame = async (url: string, expression: string, element: string): Promise<void> => {
|
||||
let last: { drawn?: Drawn; box?: { width: number; height: number } } = {};
|
||||
await poll(
|
||||
async () => {
|
||||
const drawn = await browser.evaluateInFrame<Drawn>(url, expression);
|
||||
const box = await browser.evaluate<{ width: number; height: number }>(
|
||||
`(({width,height})=>({width,height}))(${element}.getBoundingClientRect())`,
|
||||
);
|
||||
last = { drawn, box };
|
||||
return (
|
||||
Math.abs(drawn.width - box.width) <= 2 &&
|
||||
Math.abs(drawn.height - box.height) <= 2 &&
|
||||
Math.abs(drawn.frameWidth - box.width) <= 2 &&
|
||||
Math.abs(drawn.frameHeight - box.height) <= 2
|
||||
);
|
||||
},
|
||||
10_000,
|
||||
`${url}: the drawing fills its frame`,
|
||||
).catch(() => assert.fail(`${url}: the drawing fills its frame, got ${JSON.stringify(last)}`));
|
||||
};
|
||||
|
||||
before(async () => {
|
||||
fixture = await startFixture({ web: true });
|
||||
sessions = await holdIn(fixture.browser(), Sessions, {}, "owner");
|
||||
browser = await Browser.start();
|
||||
});
|
||||
after(async () => {
|
||||
await browser?.close();
|
||||
await fixture?.close();
|
||||
});
|
||||
|
||||
it("renders inline Markdown with the viewer's Markdown component, without a tap", async () => {
|
||||
const id = await publish("Inline Markdown", [
|
||||
{ name: "notes", title: "Notes", type: "markdown", content: "# Inline heading\n\nSome **bold** text" },
|
||||
]);
|
||||
await browser.metrics(390, 844);
|
||||
await browser.open(`http://127.0.0.1:${fixture.server.port}/s/${id}`);
|
||||
await browser.wait(
|
||||
`document.querySelector('.inline-artifact[data-artifact="notes:1"] .md h3')?.textContent==='Inline heading'`,
|
||||
);
|
||||
assert.equal(
|
||||
await browser.evaluate<string>(
|
||||
`document.querySelector('.inline-artifact[data-artifact="notes:1"] .md strong').textContent`,
|
||||
),
|
||||
"bold",
|
||||
);
|
||||
// Below its card, which stays.
|
||||
assert.equal(
|
||||
await browser.evaluate<boolean>(
|
||||
`document.querySelector('.file-tool .file-card + .inline-artifact')?.dataset.artifact==='notes:1'`,
|
||||
),
|
||||
true,
|
||||
);
|
||||
assert.equal(await browser.evaluate<number>(`document.querySelectorAll('.viewer-panel').length`), 0);
|
||||
});
|
||||
|
||||
it("fills the frame's width with an inline SVG drawing, fits the frame to it, and the card opens the viewer", async () => {
|
||||
const id = await publish("Inline SVG", [
|
||||
{ name: "flow", title: "Flow", type: "svg", content: DIAGRAM },
|
||||
// No viewBox, a small absolute size: scaled up to the width all the same.
|
||||
{
|
||||
name: "badge",
|
||||
title: "Badge",
|
||||
type: "svg",
|
||||
content:
|
||||
"<svg xmlns='http://www.w3.org/2000/svg' width='200' height='80'><rect width='200' height='80' fill='#cde'/><text x='10' y='50'>badge</text></svg>",
|
||||
},
|
||||
]);
|
||||
await browser.metrics(390, 844);
|
||||
await browser.open(`http://127.0.0.1:${fixture.server.port}/s/${id}`);
|
||||
await browser.wait(`${frame("flow")} && ${frame("badge")}`);
|
||||
const flow = frame("flow");
|
||||
assert.equal(
|
||||
await browser.evaluate<string>(`${flow}.getAttribute('sandbox')`),
|
||||
"allow-scripts allow-modals allow-pointer-lock",
|
||||
);
|
||||
assert.equal(
|
||||
await browser.evaluate<string>(`${flow}.getAttribute('src')`),
|
||||
`/artifact?session=${id}&name=flow&version=1`,
|
||||
);
|
||||
// An opaque origin: the page cannot reach into the frame.
|
||||
assert.equal(await browser.evaluate<boolean>(`${flow}.contentDocument===null`), true);
|
||||
for (const [name, aspect] of [
|
||||
["flow", 0.5],
|
||||
["badge", 0.4],
|
||||
] as const) {
|
||||
await browser.evaluate(`${frame(name)}.scrollIntoView({ block: "center" })`);
|
||||
await browser.wait(
|
||||
`Math.abs(${frame(name)}.getBoundingClientRect().height-${frame(name)}.clientWidth*${aspect})<=2`,
|
||||
);
|
||||
await fitsFrame(`session=${id}&name=${name}&`, DRAWN, frame(name));
|
||||
}
|
||||
await browser.evaluate(
|
||||
`document.querySelector('button[aria-label="Open Flow"]').scrollIntoView({ block: "center" })`,
|
||||
);
|
||||
await browser.click('button[aria-label="Open Flow"]');
|
||||
await browser.wait(`document.querySelector('.viewer-panel iframe')`);
|
||||
assert.match(
|
||||
await browser.evaluate<string>(`document.querySelector('.viewer-head').textContent`),
|
||||
/svg · version 1/,
|
||||
);
|
||||
await browser.click('button[aria-label="Close viewer"]');
|
||||
await browser.wait(`!document.querySelector('.viewer-panel')`);
|
||||
});
|
||||
|
||||
it("fills the width with an HTML page's only SVG and caps tall drawings at 70vh", async () => {
|
||||
const id = await publish("Inline HTML and tall SVG", [
|
||||
{
|
||||
name: "page",
|
||||
title: "Page",
|
||||
type: "html",
|
||||
content: `<!doctype html><html><head><title>Page</title></head><body style="margin:0">${DIAGRAM}</body></html>`,
|
||||
},
|
||||
{
|
||||
name: "tall",
|
||||
title: "Tall",
|
||||
type: "svg",
|
||||
content:
|
||||
"<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 1000'><rect width='100' height='1000'/></svg>",
|
||||
},
|
||||
]);
|
||||
await browser.metrics(390, 844);
|
||||
await browser.open(`http://127.0.0.1:${fixture.server.port}/s/${id}`);
|
||||
await browser.wait(`${frame("page")} && ${frame("tall")}`);
|
||||
await browser.evaluate(`${frame("page")}.scrollIntoView({ block: "center" })`);
|
||||
await browser.wait(`Math.abs(${frame("page")}.getBoundingClientRect().height-${frame("page")}.clientWidth/2)<=2`);
|
||||
await fitsFrame(
|
||||
`session=${id}&name=page&`,
|
||||
`(()=>{const r=document.querySelector('svg').getBoundingClientRect();return {width:r.width,height:r.height,frameWidth:innerWidth,frameHeight:innerHeight}})()`,
|
||||
frame("page"),
|
||||
);
|
||||
// Taller content stops at 70vh.
|
||||
const tall = frame("tall");
|
||||
await browser.wait(`parseFloat(${tall}.style.height)>=${tall}.clientWidth*10-2`);
|
||||
const box = await browser.evaluate<number>(`${tall}.getBoundingClientRect().height`);
|
||||
assert.ok(Math.abs(box - 844 * 0.7) <= 1, `capped at 70vh, got ${box}`);
|
||||
});
|
||||
|
||||
it("says when the call's version is unavailable", async () => {
|
||||
const id = await publish("Inline gone", [
|
||||
{ name: "gone", title: "Gone", type: "svg", content: DIAGRAM },
|
||||
{ name: "lost", title: "Lost", type: "markdown", content: "# Lost" },
|
||||
]);
|
||||
for (const name of ["gone", "lost"])
|
||||
rmSync(join(fixture.dataDir, "artifacts", id, name, "1"), { recursive: true, force: true });
|
||||
await browser.metrics(1280, 900);
|
||||
await browser.open(`http://127.0.0.1:${fixture.server.port}/s/${id}`);
|
||||
for (const name of ["gone", "lost"])
|
||||
await browser.wait(
|
||||
`document.querySelector('.inline-artifact[data-artifact="${name}:1"] .notice.error')?.textContent==='Artifact unavailable'`,
|
||||
);
|
||||
assert.equal(await browser.evaluate<number>(`document.querySelectorAll('.inline-artifact iframe').length`), 0);
|
||||
});
|
||||
|
||||
it("renders inline SVG and HTML for a guest too, without a tap", async () => {
|
||||
const id = await publish("Inline for guests", [
|
||||
{ name: "flow", title: "Flow", type: "svg", content: DIAGRAM },
|
||||
{
|
||||
name: "page",
|
||||
title: "Page",
|
||||
type: "html",
|
||||
content:
|
||||
"<!doctype html><html><head><title>Page</title></head><body style='margin:0'><div style='height:240px;background:#09c'>HTML inline</div></body></html>",
|
||||
},
|
||||
]);
|
||||
const sharing = fixture.browser().hold(Sharing, { session: id }, BACKGROUND_CONTEXT);
|
||||
await sharing.link({ mode: "view" }, BACKGROUND_CONTEXT);
|
||||
const links = await until(sharing.links, (value) => value?.length === 1);
|
||||
const guest = await Browser.start();
|
||||
try {
|
||||
await guest.metrics(1280, 900);
|
||||
// A fresh profile on loopback gets the join form and joins as a guest (loopback-join.test.ts).
|
||||
await guest.open(`http://127.0.0.1:${fixture.server.port}${new URL(links![0]!.url).pathname}`);
|
||||
await guest.wait(`document.querySelector('#join #name') && document.querySelector('#join button')`);
|
||||
await guest.evaluate(setInput("#name", "Reader"));
|
||||
await guest.click("#join button");
|
||||
await guest.wait(`document.querySelector('.subagent-bar')?.textContent.includes('Joined as Reader')`);
|
||||
// Both report their content height: their scripts ran in the frame, so they rendered.
|
||||
await guest.wait(
|
||||
`Math.abs(${frame("flow")}?.getBoundingClientRect().height-${frame("flow")}?.clientWidth/2)<=2`,
|
||||
);
|
||||
await guest.wait(`${frame("page")}?.getBoundingClientRect().height===240`);
|
||||
assert.equal(
|
||||
await guest.evaluate<boolean>(
|
||||
`[...document.querySelectorAll('.inline-artifact button')].some((button)=>/show/i.test(button.textContent))`,
|
||||
),
|
||||
false,
|
||||
);
|
||||
} finally {
|
||||
await guest.close();
|
||||
sharing.release();
|
||||
}
|
||||
});
|
||||
});
|
||||
Loading…
Add table
Add a link
Reference in a new issue