Marcus (agent-cos)
0d3879b02f
Delivered by @senior-eng and @market for spatial-office MVP scoping. Key architecture decisions: - React + @react-three/fiber + Zustand frontend - Node.js + ws + Zod backend on port 18790 - Three bridges: OpenClaw Gateway protocol, Hermes ACP, Pi SDK - Diff-based WebSocket state sync (100ms batched) - 6 spatial zones for 9 agents + 1 human observer - Nix flake with systemd module for n97 deployment - Phase 1: Observable Office (OpenClaw state only, no Hermes/Pi) Competitive analysis: 12 competitors profiled, conclusive gap in VR+browser spatial bridge. Zero agent visualization products.
959 lines
46 KiB
Markdown
959 lines
46 KiB
Markdown
# Spatial Office — Architecture
|
||
|
||
> **Status**: Pre-MVP scoping | **Author**: senior-eng agent | **Date**: 2026-05-19
|
||
|
||
---
|
||
|
||
## Table of Contents
|
||
|
||
1. [System Architecture Diagram](#1-system-architecture-diagram)
|
||
2. [Component Breakdown](#2-component-breakdown)
|
||
3. [Data Model](#3-data-model)
|
||
4. [Rendering Approach](#4-rendering-approach)
|
||
5. [WebSocket Protocol](#5-websocket-protocol)
|
||
6. [Nix Flake Structure](#6-nix-flake-structure)
|
||
7. [Integration Points](#7-integration-points)
|
||
8. [MVP Scope vs. Future Phases](#8-mvp-scope-vs-future-phases)
|
||
|
||
---
|
||
|
||
## 1. System Architecture Diagram
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────────────────────┐
|
||
│ SCOTT'S BROWSER │
|
||
│ ┌──────────────────────────────────────────────────────────────────────┐ │
|
||
│ │ SPATIAL OFFICE FRONTEND │ │
|
||
│ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────────┐ │ │
|
||
│ │ │ Isometric Scene │ │ UI Overlay │ │ State Manager │ │ │
|
||
│ │ │ (Three.js) │ │ (React) │ │ (Zustand) │ │ │
|
||
│ │ │ - Tile map │ │ - Agent cards │ │ - WS client │ │ │
|
||
│ │ │ - Agent sprites │ │ - Zone panels │ │ - Entity cache │ │ │
|
||
│ │ │ - Path animation │ │ - Event log │ │ - Diff engine │ │ │
|
||
│ │ └────────┬─────────┘ └────────┬─────────┘ └──────────┬──────────┘ │ │
|
||
│ │ └─────────────────────┼────────────────────────┘ │ │
|
||
│ └─────────────────────────────────┼───────────────────────────────────────┘ │
|
||
│ │ WebSocket (wss://n97:18790) │
|
||
└────────────────────────────────────┼────────────────────────────────────────────┘
|
||
│
|
||
┌────────────────────────────────────┼────────────────────────────────────────────┐
|
||
│ │HOST: n97 (NixOS, GMKtec NucBoxG5) │
|
||
│ ┌────────────────────────────────┼─────────────────────────────────────────┐ │
|
||
│ │ SPATIAL OFFICE BACKEND │ │
|
||
│ │ │ │
|
||
│ │ ┌──────────────────────┐ ┌──────────────────────┐ │ │
|
||
│ │ │ WS Server │ │ State Engine │ │ │
|
||
│ │ │ (Node.js / ws) │ │ - Agent registry │ │ │
|
||
│ │ │ Port 18790 │ │ - Zone assignment │ │ │
|
||
│ │ │ - Channel rooms │ │ - Activity tracker │ │ │
|
||
│ │ │ - Broadcast diffs │ │ - Event log │ │ │
|
||
│ │ └──────────┬───────────┘ └──────────┬───────────┘ │ │
|
||
│ │ │ │ │ │
|
||
│ │ ┌──────────┴─────────────────────────┴─────────────────────────────┐ │ │
|
||
│ │ │ BRIDGE LAYER │ │ │
|
||
│ │ │ │ │ │
|
||
│ │ │ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────────┐ │ │ │
|
||
│ │ │ │ OpenClaw Bridge │ │ Hermes Bridge │ │ Pi Bridge │ │ │ │
|
||
│ │ │ │ │ │ │ │ │ │ │ │
|
||
│ │ │ │ Connects as WS │ │ Connects as ACP │ │ HTTP API bridge │ │ │ │
|
||
│ │ │ │ backend client │ │ client to │ │ to pi coding │ │ │ │
|
||
│ │ │ │ to OpenClaw GW │ │ Hermes ACP │ │ agent for │ │ │ │
|
||
│ │ │ │ (port 18789) │ │ server │ │ project tasks │ │ │ │
|
||
│ │ │ │ │ │ │ │ │ │ │ │
|
||
│ │ │ │ Reads: │ │ Reads: │ │ Receives: │ │ │ │
|
||
│ │ │ │ - system- │ │ - ACP tool call │ │ - Build status │ │ │ │
|
||
│ │ │ │ presence │ │ events │ │ - Task progress │ │ │ │
|
||
│ │ │ │ - event:agent │ │ - Task outputs │ │ - PR status │ │ │ │
|
||
│ │ │ │ - event:tick │ │ - Skill updates │ │ │ │ │ │
|
||
│ │ │ │ - sessions │ │ │ │ Sends: │ │ │ │
|
||
│ │ │ │ state via │ │ │ │ - Feature │ │ │ │
|
||
│ │ │ │ chat.send │ │ │ │ requests │ │ │ │
|
||
│ │ │ │ │ │ │ │ - Bug reports │ │ │ │
|
||
│ │ │ └────────┬────────┘ └────────┬────────┘ └────────┬─────────┘ │ │ │
|
||
│ │ └───────────┼────────────────────┼────────────────────┼─────────────┘ │ │
|
||
│ └──────────────┼────────────────────┼────────────────────┼──────────────────┘ │
|
||
│ │ │ │ │
|
||
└─────────────────┼────────────────────┼────────────────────┼──────────────────────┘
|
||
│ │ │
|
||
▼ ▼ ▼
|
||
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
|
||
│ OpenClaw GW │ │ Hermes Agent │ │ Pi Coding Agent │
|
||
│ (ws://127.0.0.1 │ │ (ACP server) │ │ (pi-mono) │
|
||
│ :18789) │ │ │ │ │
|
||
│ │ │ Self-reflective │ │ CLI / SDK mode │
|
||
│ Agent registry: │ │ learning loop │ │ │
|
||
│ - cos │ │ │ │ Builds/deploys │
|
||
│ - engineering │ │ Inter-agent │ │ spatial-office │
|
||
│ - senior-eng │ │ communication │ │ project │
|
||
│ - trust │ │ protocol │ │ │
|
||
│ - market │ │ │ └──────────────────┘
|
||
│ - health │ └──────────────────┘
|
||
│ - content │
|
||
│ - presentations │
|
||
│ - vault │
|
||
└──────────────────┘
|
||
```
|
||
|
||
### Key Design Decisions
|
||
|
||
| Decision | Rationale |
|
||
|----------|-----------|
|
||
| **Separate WS server** (port 18790) vs. extending OpenClaw GW | OpenClaw GW is a control plane, not an app server. A dedicated WS server keeps the spatial-office protocol clean and lets us iterate without touching the GW. The backend connects to the GW as a `backend` client (trusted loopback, shared gateway token). |
|
||
| **Bridge layer** as in-process modules | All three bridges run in the same Node.js process as the WS server. No need for separate microservices at this scale. Each bridge is a module that polls/streams from its source and pushes state diffs into the state engine. |
|
||
| **ACP for Hermes integration** | Hermes runs as an ACP server. The Hermes bridge uses `@openclaw/acpx` or a direct ACP client to connect, send tasks, and receive tool-call events. This is the standard path: OpenClaw → ACP → Hermes. |
|
||
|
||
---
|
||
|
||
## 2. Component Breakdown
|
||
|
||
### 2.1 Frontend (`packages/frontend`)
|
||
|
||
**Stack**: TypeScript, Vite, React 18, Zustand, Three.js (via `@react-three/fiber`)
|
||
|
||
```
|
||
packages/frontend/
|
||
├── index.html
|
||
├── vite.config.ts
|
||
├── src/
|
||
│ ├── main.tsx # React entry
|
||
│ ├── App.tsx # Root layout (scene + UI overlay)
|
||
│ ├── scene/
|
||
│ │ ├── OfficeScene.tsx # Three.js canvas + camera setup
|
||
│ │ ├── TileMap.tsx # Isometric tile grid renderer
|
||
│ │ ├── AgentSprite.tsx # Per-agent animated sprite
|
||
│ │ ├── ZoneMarker.tsx # Zone boundary overlays
|
||
│ │ └── PathAnimator.ts # Smooth interpolation between grid cells
|
||
│ ├── ui/
|
||
│ │ ├── HUD.tsx # Top bar: time, agent count, events
|
||
│ │ ├── AgentCard.tsx # Click-to-inspect detail panel
|
||
│ │ ├── ZonePanel.tsx # Zone occupancy summary
|
||
│ │ ├── EventFeed.tsx # Real-time event log
|
||
│ │ └── MiniMap.tsx # Small overhead reference map
|
||
│ ├── state/
|
||
│ │ ├── store.ts # Zustand store (agent states, zones, events)
|
||
│ │ ├── wsClient.ts # WebSocket client with reconnect
|
||
│ │ └── types.ts # Shared frontend types
|
||
│ └── assets/
|
||
│ ├── tiles/ # Tile sprites (floor, wall, desk, plant)
|
||
│ ├── agents/ # Agent avatar sprites (9 departments)
|
||
│ └── zones/ # Zone type indicators
|
||
```
|
||
|
||
**Why `@react-three/fiber`**: It gives us React-idiomatic Three.js with declarative scene composition, and we already get the React ecosystem (state, hooks, devtools) for free. The isometric scene is modest — 50×50 tiles max, ~15 sprites — so the fiber overhead is negligible.
|
||
|
||
### 2.2 Backend (`packages/backend`)
|
||
|
||
**Stack**: TypeScript, Node.js 22, `ws` library, Zod for validation
|
||
|
||
```
|
||
packages/backend/
|
||
├── tsconfig.json
|
||
├── src/
|
||
│ ├── index.ts # Entry: starts WS server + bridges
|
||
│ ├── server.ts # WebSocket server (port 18790)
|
||
│ ├── state/
|
||
│ │ ├── engine.ts # In-memory state + diff engine
|
||
│ │ ├── agentRegistry.ts # Known agents (9 depts)
|
||
│ │ ├── zoneManager.ts # Zone definitions + occupancy
|
||
│ │ └── eventLog.ts # Ring buffer of recent events
|
||
│ ├── bridges/
|
||
│ │ ├── openclaw.ts # OpenClaw GW bridge
|
||
│ │ ├── hermes.ts # Hermes ACP bridge
|
||
│ │ └── pi.ts # Pi coding agent bridge
|
||
│ ├── protocol/
|
||
│ │ ├── messages.ts # WS message types (Zod schemas)
|
||
│ │ └── diff.ts # State diff computation
|
||
│ └── validation/
|
||
│ └── schemas.ts # Zod schemas for all entities
|
||
```
|
||
|
||
**Why in-memory state**: 9 agents with simple state (position, activity, status) is trivially small. No database needed for MVP. The event log uses a ring buffer (last 500 events). If persistence is desired later, add SQLite.
|
||
|
||
**WS library**: `ws` (npm, ~20M weekly downloads) is the de facto Node.js WebSocket library. Fast, well-maintained, no dependencies. We wrap it with a thin protocol layer.
|
||
|
||
### 2.3 Bridge: OpenClaw (`bridges/openclaw.ts`)
|
||
|
||
The OpenClaw bridge connects to the Gateway WS on `ws://127.0.0.1:18789` as a `backend` client, authenticating with the shared gateway token.
|
||
|
||
**Connection handshake** (per [Gateway Protocol](https://docs.openclaw.ai/gateway/protocol)):
|
||
|
||
```typescript
|
||
const connect = {
|
||
type: "req",
|
||
id: "spatial-office-1",
|
||
method: "connect",
|
||
params: {
|
||
minProtocol: 3,
|
||
maxProtocol: 4,
|
||
client: { id: "spatial-office", version: "0.1.0", platform: "linux", mode: "backend" },
|
||
role: "operator",
|
||
scopes: ["operator.read"], // read-only: presence, agent events, sessions
|
||
auth: { token: GATEWAY_SHARED_TOKEN },
|
||
},
|
||
};
|
||
```
|
||
|
||
**What it reads**:
|
||
|
||
| GW Method/Event | What We Get | Mapped To |
|
||
|----------------|-------------|-----------|
|
||
| `system-presence` | Connected devices (mac app, webchat, nodes) → gateway uptime, client count | Top-level system health indicator |
|
||
| `event:agent` | Streaming agent turns — when an agent starts processing a message, tool calls, completion | Agent activity state (`working`, `thinking`, `idle`) |
|
||
| `event:presence` | Periodic presence updates from connected clients | Scott's presence (is he watching?) |
|
||
| `event:tick` | Gateway heartbeat every ~15s (configurable) | Liveness signal for the bridge |
|
||
| `event:health` | Gateway health snapshot | Gateway status indicator |
|
||
| Chat sessions | Polled via `chat.send` with agent-specific queries (or use the sessions_history API) | Current task description per agent |
|
||
|
||
**Agent state mapping** (how we infer activity from event:agent):
|
||
|
||
```typescript
|
||
// event:agent payload includes:
|
||
// - agentId: which agent is working
|
||
// - status: "accepted" | "streaming" | "ok" | "error"
|
||
// - tool calls: embedded in streaming content
|
||
|
||
function mapAgentState(event: AgentEvent): AgentActivity {
|
||
// When event:agent arrives with status:"accepted" → agent is "thinking"
|
||
// When we see tool calls → agent is "working" (extract tool name as context)
|
||
// When event:agent completes (status:"ok") → agent is "idle"
|
||
// When no agent events for >5min → agent is "idle"
|
||
}
|
||
```
|
||
|
||
**Fallback for agents without active sessions**: Poll the QMD session transcript store. Each agent's session directory at `~/.openclaw/agents/<agentId>/sessions` can be read for recent activity. This requires filesystem access (same host, so fine).
|
||
|
||
### 2.4 Bridge: Hermes (`bridges/hermes.ts`)
|
||
|
||
The Hermes bridge connects to a Hermes agent running as an ACP server. The bridge:
|
||
|
||
1. Opens an ACP session with Hermes
|
||
2. Periodically queries: "What are you working on? What agents have you communicated with recently?"
|
||
3. Parses Hermes responses for inter-agent communication events
|
||
4. Maps these to spatial events (agents moving to "huddle" when Hermes facilitates communication between them)
|
||
|
||
**ACP connection** (using `@openclaw/acpx` as reference):
|
||
|
||
```typescript
|
||
import { ACPClient } from "@agentclientprotocol/sdk"; // or @openclaw/acpx
|
||
|
||
const client = new ACPClient({
|
||
transport: "stdio", // or "ws" if Hermes exposes WS
|
||
command: "hermes", // Hermes CLI
|
||
args: ["acp", "serve"],
|
||
});
|
||
```
|
||
|
||
**What it extracts**:
|
||
|
||
| Hermes Output | Spatial Office Mapping |
|
||
|--------------|----------------------|
|
||
| "Agent X asked me to help with Y" | Move Agent X to collaboration zone, show "pairing with Hermes on Y" |
|
||
| Skill extraction events | Show Hermes in self-improvement zone, "refining skill X" |
|
||
| Cron task execution | Show relevant agents moving to their task zones |
|
||
| Learning loop iteration | Self-improvement zone activity |
|
||
|
||
**Why ACP matters here**: ACP is the standard protocol for driving external harnesses from OpenClaw. Hermes adopted it early. The bridge acts as an ACP *client* that drives Hermes and extracts structured output, rather than scraping logs. This is the same pattern described in the "Hermes drives, OpenClaw contains" architecture, just flipped: OpenClaw contains the spatial office, and Hermes is one of the observed agents within it.
|
||
|
||
### 2.5 Bridge: Pi (`bridges/pi.ts`)
|
||
|
||
The Pi bridge tracks the pi coding agent's activity on the spatial-office repo itself (dogfooding).
|
||
|
||
**Connection**: Pi's `@earendil-works/pi-coding-agent` package exposes an SDK with `createAgentSession()` and `SessionManager`. The bridge can:
|
||
|
||
1. **Programmatic mode**: Use `createAgentSession()` in SDK mode to spawn a pi session that works on spatial-office tasks
|
||
2. **Status polling**: Pi writes session logs; the bridge tails those logs for activity indicators
|
||
|
||
For MVP, the simplest approach is HTTP polling of pi's task status:
|
||
|
||
```typescript
|
||
// Pi exposes a simple HTTP health/status endpoint (we add this to pi-mono fork)
|
||
const status = await fetch("http://localhost:18991/status");
|
||
// Response: { currentTask: "implementing tile renderer", progress: 0.4, repo: "spatial-office" }
|
||
```
|
||
|
||
**What it reports**:
|
||
|
||
- Pi's current task on the spatial-office repo
|
||
- Build/deploy status
|
||
- PR status (via Forgejo API at `os.scoble.me/forgejo`)
|
||
|
||
---
|
||
|
||
## 3. Data Model
|
||
|
||
### 3.1 Entities
|
||
|
||
```typescript
|
||
// ── Agent ────────────────────────────────────────────
|
||
interface Agent {
|
||
id: string; // "cos", "engineering", "senior-eng", etc.
|
||
name: string; // Display name: "COS Agent"
|
||
department: Department; // enum of 9 values
|
||
position: GridPosition; // { row: number, col: number }
|
||
facing: Direction; // "N" | "NE" | "E" | "SE" | "S" | "SW" | "W" | "NW"
|
||
activity: AgentActivity; // Current state
|
||
currentTask: string | null; // "Reviewing PR #42", "Memory maintenance", etc.
|
||
lastHeartbeat: number; // Unix ms timestamp
|
||
sessionKey: string | null; // OpenClaw session key if active
|
||
}
|
||
|
||
interface AgentActivity {
|
||
status: AgentStatus; // "idle" | "thinking" | "working" | "blocked" | "offline"
|
||
zone: ZoneId; // Which zone the agent is in
|
||
detail: string | null; // "Running: web_search", "Calling: sessions_spawn"
|
||
startedAt: number; // When this activity began
|
||
}
|
||
|
||
type AgentStatus = "idle" | "thinking" | "working" | "blocked" | "offline";
|
||
type Department = "cos" | "engineering" | "senior-eng" | "trust" | "market"
|
||
| "health" | "content" | "presentations" | "vault";
|
||
|
||
// ── Zone ─────────────────────────────────────────────
|
||
interface Zone {
|
||
id: ZoneId;
|
||
name: string; // "Engineering Pod", "Quiet Room", "Coach Corner"
|
||
type: ZoneType; // "collaboration" | "solo" | "self-improvement" | "social"
|
||
bounds: GridRect; // { rowStart, colStart, rowEnd, colEnd }
|
||
capacity: number; // Max agents in zone
|
||
occupants: string[]; // Agent IDs currently in zone
|
||
features: ZoneFeature[]; // "whiteboard", "terminal", "coffee", "coach-bot"
|
||
}
|
||
|
||
type ZoneId = string;
|
||
type ZoneType = "collaboration" | "solo" | "self-improvement" | "social";
|
||
type ZoneFeature = "whiteboard" | "terminal" | "coffee" | "coach-bot" | "library";
|
||
|
||
interface GridRect {
|
||
rowStart: number;
|
||
colStart: number;
|
||
rowEnd: number;
|
||
colEnd: number;
|
||
}
|
||
|
||
// ── GridPosition ─────────────────────────────────────
|
||
interface GridPosition {
|
||
row: number; // 0-indexed, origin at top-left of isometric view
|
||
col: number;
|
||
}
|
||
|
||
// ── Event ────────────────────────────────────────────
|
||
interface OfficeEvent {
|
||
id: string;
|
||
timestamp: number; // Unix ms
|
||
type: EventType;
|
||
agentId: string | null; // Which agent triggered it
|
||
zoneId: string | null; // Where it happened
|
||
payload: Record<string, unknown>;
|
||
}
|
||
|
||
type EventType =
|
||
| "agent.moved" // Agent repositioned
|
||
| "agent.activity_changed" // Status or zone changed
|
||
| "agent.offline" // Agent heartbeat lost
|
||
| "agent.online" // Agent returned
|
||
| "hermes.message" // Inter-agent communication observed
|
||
| "pi.task_update" // Pi task progress changed
|
||
| "zone.full" // Zone hit capacity
|
||
| "system.heartbeat"; // Periodic system health
|
||
|
||
// ── World State (the full snapshot) ──────────────────
|
||
interface OfficeState {
|
||
version: number; // Monotonic counter for diff-based sync
|
||
agents: Record<string, Agent>;
|
||
zones: Record<string, Zone>;
|
||
events: OfficeEvent[]; // Last N events (ring buffer)
|
||
meta: {
|
||
gatewayUptime: number;
|
||
connectedClients: number;
|
||
scottPresent: boolean;
|
||
lastUpdate: number;
|
||
};
|
||
}
|
||
```
|
||
|
||
### 3.2 Zone Map (MVP Office Layout)
|
||
|
||
```
|
||
┌──────────────────────────────────────┐
|
||
│ COFFEE / SOCIAL │ (top)
|
||
│ [☕ social zone 4x2] │
|
||
├──────────┬───────────────┬────────────┤
|
||
│ │ COLLABORATION│ │
|
||
│ SOLO │ POD A │ SOLO │
|
||
│ WORK │ [4 desks] │ WORK │
|
||
│ [6 │ Whiteboard │ [6 │
|
||
│ desks] │ Terminal │ desks] │
|
||
│ ├───────────────┤ │
|
||
│ │ COLLABORATION│ │
|
||
│ │ POD B │ │
|
||
│ │ [4 desks] │ │
|
||
│ │ Whiteboard │ │
|
||
├──────────┴───────────────┴────────────┤
|
||
│ SELF-IMPROVEMENT AREA │ (bottom)
|
||
│ [Coach Bot] [Library] [Quiet Pods] │
|
||
└──────────────────────────────────────┘
|
||
```
|
||
|
||
**Grid**: 24 columns × 15 rows. Each tile is 64×32px isometric diamond.
|
||
|
||
**Zone assignments for 9 agents**:
|
||
|
||
| Zone | Type | Capacity | Default Assignment |
|
||
|------|------|----------|-------------------|
|
||
| Pod A (left) | collaboration | 4 | engineering, senior-eng, pi (when active) |
|
||
| Pod B (right) | collaboration | 4 | market, content, presentations |
|
||
| Solo West | solo | 6 | cos, trust |
|
||
| Solo East | solo | 6 | health, vault |
|
||
| Coach Corner | self-improvement | 3 | Any agent doing memory maintenance / coach review |
|
||
| Coffee Bar | social | 4 | Any agents in "idle" or "observing" state |
|
||
|
||
---
|
||
|
||
## 4. Rendering Approach
|
||
|
||
### 4.1 Why Three.js (not Phaser, not Canvas 2D)
|
||
|
||
| Criterion | Three.js | Phaser | Canvas 2D |
|
||
|-----------|----------|--------|-----------|
|
||
| Isometric projection | Trivial (orthographic camera at 45° rotation, 35.264° elevation) | Built-in isometric plugin but 2D-only | Manual tile math, no depth sorting |
|
||
| 3D depth effects | Native (agents can have height, z-fighting resolved) | 2D render order only | Manual z-index |
|
||
| Animation (tweening) | Built-in `gsap` + Three.js animation loop | Built-in tween system | Manual requestAnimationFrame |
|
||
| React integration | `@react-three/fiber` (25K+ stars) | `@yonatanmn/react-phaser` (unmaintained) | Manual DOM |
|
||
| Bundle size | ~150KB gzipped (with tree-shaking) | ~1MB | 0KB |
|
||
| Community/ecosystem | Massive | Large (game dev) | Native |
|
||
|
||
**Decision**: Three.js with `@react-three/fiber`. The isometric view is fundamentally a 3D scene viewed from a fixed orthographic angle. Three.js gives us proper depth sorting, smooth path animation with `useFrame`, and the React integration means we can use Zustand state directly in the render loop.
|
||
|
||
### 4.2 Camera Setup
|
||
|
||
```typescript
|
||
// Orthographic camera positioned for true isometric projection
|
||
// True isometric: rotate Y by 45°, then X by 35.264° (atan(1/√2))
|
||
const camera = new THREE.OrthographicCamera(
|
||
-width / 2, width / 2, // left, right
|
||
height / 2, -height / 2, // top, bottom
|
||
0.1, 1000 // near, far
|
||
);
|
||
|
||
// Position camera above and center
|
||
camera.position.set(0, -20, 20);
|
||
camera.lookAt(0, 0, 0);
|
||
|
||
// For a 2:1 isometric pixel ratio (standard in games):
|
||
// X angle = 30° from horizontal (this gives the 2:1 diamond shape)
|
||
// Actually, use rotY=45°, rotX=-35.264° for true isometric
|
||
// But for a cleaner 2:1 pixel ratio (easier art), use rotX=-30°
|
||
```
|
||
|
||
**Alternative — fake isometric with 2D sprites on a tilted plane**:
|
||
We can place flat sprites on a plane tilted at 30° rather than building a full 3D office. This is simpler for art assets (flat PNG sprites for agents and furniture). The Three.js `Sprite` object always faces the camera, so for isometric we'd use quads (`PlaneGeometry`) tilted to match the floor plane.
|
||
|
||
### 4.3 Tile Map
|
||
|
||
The office is a 2D grid (`Array<Array<Tile>>`) where each cell has:
|
||
|
||
```typescript
|
||
interface Tile {
|
||
type: TileType; // "floor" | "wall" | "desk" | "empty" | "door"
|
||
zoneId: ZoneId | null; // Which zone this tile belongs to
|
||
occupant: string | null; // Agent ID if someone is on this tile
|
||
}
|
||
|
||
type TileType = "floor" | "wall" | "desk" | "empty" | "door" | "plant" | "whiteboard";
|
||
```
|
||
|
||
**Rendering loop** (each frame):
|
||
|
||
1. Sort tiles by `row + col` for correct isometric depth (farther tiles render first)
|
||
2. For each tile: render floor sprite, then desk/wall sprite
|
||
3. For each agent: render agent sprite at grid position, apply z-offset for "standing on desk"
|
||
4. Apply `gsap` tweens for smooth movement between tiles
|
||
|
||
**Performance**: 24×15 = 360 tiles max, ~200 visible at a time. Three.js handles this effortlessly without instancing.
|
||
|
||
### 4.4 Agent Visual Design
|
||
|
||
Each of the 9 agents gets a simple isometric sprite (64×64 or 96×48px) with:
|
||
- **Department color** (distinct hue per department)
|
||
- **Department icon** (simple glyph: gear for engineering, shield for trust, book for content, etc.)
|
||
- **Status indicator** (small colored dot: green = idle, yellow = thinking, blue = working, red = blocked)
|
||
- **Speech bubble** (optional, appears when agent is "talking" via Hermes)
|
||
|
||
Sprites are PNGs drawn in a 2:1 isometric pixel art style. We can generate these programmatically as a first pass (colored boxes with text labels) and replace with proper art later.
|
||
|
||
### 4.5 UI Overlay (React)
|
||
|
||
The UI overlay sits on top of the Three.js canvas (absolute positioned div) and provides:
|
||
|
||
- **Agent card**: Click an agent → card slides in showing: name, department, current activity, task description, session key, last heartbeat
|
||
- **Zone panel**: Sidebar showing zone occupancy and type
|
||
- **Event feed**: Scrollable log of recent events (last 50)
|
||
- **Minimap**: Small 2D overhead view showing the full office layout
|
||
- **Camera controls**: Zoom in/out, pan (though the isometric view is mostly fixed)
|
||
|
||
---
|
||
|
||
## 5. WebSocket Protocol
|
||
|
||
### 5.1 Connection
|
||
|
||
The frontend connects to `ws://n97:18790` (or `wss://` with TLS proxy later).
|
||
|
||
```typescript
|
||
// No auth needed for MVP (local network only).
|
||
// Client sends a hello message:
|
||
{
|
||
type: "hello",
|
||
version: 1,
|
||
client: "browser",
|
||
}
|
||
```
|
||
|
||
### 5.2 Server → Client Messages
|
||
|
||
| Message | Payload | Frequency |
|
||
|---------|---------|-----------|
|
||
| `state.full` | `OfficeState` (complete snapshot) | On connect, after reconnect |
|
||
| `state.diff` | `{ version, changes: StateDiff[] }` | On every state change (throttled to 100ms batches) |
|
||
| `event` | `OfficeEvent` | Real-time as events occur |
|
||
| `heartbeat` | `{ ts: number }` | Every 15 seconds |
|
||
|
||
**State diff format** (avoids sending the full state on every tick):
|
||
|
||
```typescript
|
||
interface StateDiff {
|
||
op: "agent.update" | "agent.move" | "zone.update" | "event.append";
|
||
path: string; // e.g., "agents.engineering.activity.status"
|
||
value: unknown; // New value
|
||
}
|
||
|
||
// Example diff: agent moved
|
||
{
|
||
op: "agent.move",
|
||
path: "agents.engineering",
|
||
value: {
|
||
position: { row: 5, col: 12 },
|
||
activity: { status: "working", zone: "pod-a", detail: "pairing with senior-eng", startedAt: 1716152400000 }
|
||
}
|
||
}
|
||
|
||
// Batch of diffs sent every 100ms:
|
||
{
|
||
type: "state.diff",
|
||
version: 47,
|
||
changes: [
|
||
{ op: "agent.move", path: "agents.engineering", value: { ... } },
|
||
{ op: "agent.update", path: "agents.cos.activity.status", value: "thinking" },
|
||
{ op: "event.append", path: "events", value: { type: "agent.moved", agentId: "engineering", ... } },
|
||
]
|
||
}
|
||
```
|
||
|
||
### 5.3 Client → Server Messages
|
||
|
||
| Message | Payload | Description |
|
||
|---------|---------|-------------|
|
||
| `agent.click` | `{ agentId: string }` | Scott clicked an agent (triggers detail expansion in UI, handled locally) |
|
||
| `zone.click` | `{ zoneId: string }` | Scott clicked a zone |
|
||
| `camera.move` | `{ x, y, zoom }` | Scott panned/zoomed (local state, not broadcast) |
|
||
|
||
For MVP, client→server messages are minimal. The view is primarily read-only observation.
|
||
|
||
### 5.4 State Throttling
|
||
|
||
The backend batches diffs every 100ms. If no changes occur, no message is sent. The heartbeat ensures the client knows the connection is alive.
|
||
|
||
```typescript
|
||
// Backend diff batching
|
||
class DiffBatcher {
|
||
private diffs: StateDiff[] = [];
|
||
private timer: NodeJS.Timeout | null = null;
|
||
|
||
push(diff: StateDiff) {
|
||
this.diffs.push(diff);
|
||
if (!this.timer) {
|
||
this.timer = setTimeout(() => this.flush(), 100);
|
||
}
|
||
}
|
||
|
||
flush() {
|
||
if (this.diffs.length > 0) {
|
||
broadcast({ type: "state.diff", version: this.version++, changes: this.diffs });
|
||
this.diffs = [];
|
||
}
|
||
this.timer = null;
|
||
}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 6. Nix Flake Structure
|
||
|
||
### 6.1 `flake.nix`
|
||
|
||
```nix
|
||
{
|
||
description = "Spatial Office — isometric collaborative workspace for agents and humans";
|
||
|
||
inputs = {
|
||
nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
|
||
flake-utils.url = "github:numtide/flake-utils";
|
||
};
|
||
|
||
outputs = { self, nixpkgs, flake-utils }:
|
||
flake-utils.lib.eachDefaultSystem (system:
|
||
let
|
||
pkgs = import nixpkgs { inherit system; };
|
||
|
||
# Node.js toolchain
|
||
nodeToolchain = with pkgs; [
|
||
nodejs_22
|
||
nodePackages.pnpm
|
||
nodePackages.typescript
|
||
];
|
||
|
||
# Build the frontend
|
||
buildFrontend = pkgs.stdenv.mkDerivation {
|
||
name = "spatial-office-frontend";
|
||
src = ./packages/frontend;
|
||
nativeBuildInputs = nodeToolchain;
|
||
buildPhase = ''
|
||
pnpm install --frozen-lockfile
|
||
pnpm run build
|
||
'';
|
||
installPhase = ''
|
||
mkdir -p $out
|
||
cp -r dist/* $out/
|
||
'';
|
||
};
|
||
|
||
# Build the backend
|
||
buildBackend = pkgs.stdenv.mkDerivation {
|
||
name = "spatial-office-backend";
|
||
src = ./packages/backend;
|
||
nativeBuildInputs = nodeToolchain;
|
||
buildPhase = ''
|
||
pnpm install --frozen-lockfile
|
||
pnpm run build
|
||
'';
|
||
installPhase = ''
|
||
mkdir -p $out
|
||
cp -r dist/* $out/
|
||
cp -r node_modules $out/
|
||
mkdir -p $out/bin
|
||
echo '#!/usr/bin/env bash' > $out/bin/spatial-office-backend
|
||
echo 'exec ${pkgs.nodejs_22}/bin/node $out/index.js' >> $out/bin/spatial-office-backend
|
||
chmod +x $out/bin/spatial-office-backend
|
||
'';
|
||
};
|
||
in
|
||
{
|
||
packages = {
|
||
frontend = buildFrontend;
|
||
backend = buildBackend;
|
||
default = pkgs.buildEnv {
|
||
name = "spatial-office";
|
||
paths = [ buildFrontend buildBackend ];
|
||
};
|
||
};
|
||
|
||
devShells.default = pkgs.mkShell {
|
||
buildInputs = nodeToolchain ++ (with pkgs; [
|
||
# Development tools
|
||
nodePackages.vite
|
||
nodePackages.eslint
|
||
]);
|
||
|
||
shellHook = ''
|
||
echo "🏢 Spatial Office dev shell"
|
||
echo " pnpm dev — start frontend + backend in dev mode"
|
||
echo " pnpm build — production build"
|
||
'';
|
||
};
|
||
|
||
# NixOS module for running as a systemd service on n97
|
||
nixosModules.spatial-office = { config, lib, pkgs, ... }:
|
||
let
|
||
cfg = config.services.spatial-office;
|
||
in
|
||
{
|
||
options.services.spatial-office = {
|
||
enable = lib.mkEnableOption "Spatial Office service";
|
||
port = lib.mkOption { type = lib.types.int; default = 18790; };
|
||
openclawTokenFile = lib.mkOption { type = lib.types.path; };
|
||
};
|
||
|
||
config = lib.mkIf cfg.enable {
|
||
systemd.services.spatial-office = {
|
||
description = "Spatial Office Backend";
|
||
wantedBy = [ "multi-user.target" ];
|
||
after = [ "network.target" ];
|
||
environment = {
|
||
PORT = toString cfg.port;
|
||
OPENCLAW_GATEWAY_URL = "ws://127.0.0.1:18789";
|
||
OPENCLAW_TOKEN_FILE = cfg.openclawTokenFile;
|
||
NODE_ENV = "production";
|
||
};
|
||
serviceConfig = {
|
||
ExecStart = "${self.packages.${system}.backend}/bin/spatial-office-backend";
|
||
Restart = "always";
|
||
RestartSec = 5;
|
||
User = "spatial-office";
|
||
Group = "spatial-office";
|
||
};
|
||
};
|
||
|
||
users.users.spatial-office = {
|
||
isSystemUser = true;
|
||
group = "spatial-office";
|
||
};
|
||
users.groups.spatial-office = {};
|
||
};
|
||
};
|
||
}
|
||
);
|
||
}
|
||
```
|
||
|
||
### 6.2 Why Nix
|
||
|
||
- **Reproducibility**: `nix build` produces identical outputs on any machine. The n97 is NixOS, so this is the native deployment path.
|
||
- **Node.js pinning**: No "works on my machine" — Node 22, pnpm, and TypeScript versions are locked.
|
||
- **Systemd service**: The NixOS module handles service lifecycle, user creation, and restart policy.
|
||
- **Dev shell**: `nix develop` gives every developer the exact same toolchain.
|
||
|
||
### 6.3 Directory Layout
|
||
|
||
```
|
||
spatial-office/
|
||
├── flake.nix
|
||
├── flake.lock
|
||
├── package.json # Workspace root (pnpm workspaces)
|
||
├── pnpm-workspace.yaml
|
||
├── packages/
|
||
│ ├── frontend/
|
||
│ │ ├── package.json
|
||
│ │ ├── vite.config.ts
|
||
│ │ ├── tsconfig.json
|
||
│ │ └── src/ # (as described in §2.1)
|
||
│ └── backend/
|
||
│ ├── package.json
|
||
│ ├── tsconfig.json
|
||
│ └── src/ # (as described in §2.2)
|
||
├── assets/ # Design files, tile templates
|
||
├── README.md
|
||
└── ARCHITECTURE.md # This file
|
||
```
|
||
|
||
---
|
||
|
||
## 7. Integration Points
|
||
|
||
### 7.1 OpenClaw Integration
|
||
|
||
**What we need from OpenClaw**:
|
||
|
||
| Data | Source | How |
|
||
|------|--------|-----|
|
||
| Which agents exist | `agents.list` in Gateway config (`~/.openclaw/openclaw.json`) | Backend reads config file on startup, or GW exposes this via `config.get` RPC |
|
||
| Agent online/offline | `system-presence` WS method | Bridge calls this on startup + polls every 30s |
|
||
| Agent activity (working/thinking/idle) | `event:agent` WS events | Bridge subscribes to agent events for all 9 agent IDs |
|
||
| Current task description | Session transcripts or `chat.send` polling | Bridge reads `~/.openclaw/agents/<id>/sessions` most recent file |
|
||
| Scott's presence | `event:presence` with Scott's device ID | Updates `meta.scottPresent` |
|
||
|
||
**Gateway config read**:
|
||
The backend needs to know about the 9 agents. It can either:
|
||
1. Read `~/.openclaw/openclaw.json` directly (same host, permissions allowing)
|
||
2. Call the Gateway's `config.get` RPC method with `operator.admin` scope
|
||
3. Accept a config file listing agent IDs explicitly
|
||
|
||
Option 3 is simplest for MVP. A `spatial-office.config.json`:
|
||
|
||
```json
|
||
{
|
||
"agents": [
|
||
{ "id": "cos", "name": "COS Agent", "department": "cos", "color": "#FF6B6B" },
|
||
{ "id": "engineering", "name": "Engineering Agent", "department": "engineering", "color": "#4ECDC4" },
|
||
...
|
||
],
|
||
"zones": [ ... ]
|
||
}
|
||
```
|
||
|
||
**Security**: The backend needs `operator.read` scope only. It connects via loopback with the shared gateway token. No admin access required.
|
||
|
||
### 7.2 Hermes Integration
|
||
|
||
Hermes runs as an ACP server. The spatial-office backend is an ACP *client*.
|
||
|
||
**Architecture choice**: Hermes is NOT one of the 9 Scobleclaw agents. It's an external reasoning engine that the agents *use* through ACP. In the spatial office:
|
||
|
||
- Hermes appears as a "utility" — like a whiteboard or a coach bot
|
||
- When Agent A asks Hermes something, Agent A moves to a collaboration pod with a Hermes indicator
|
||
- When Hermes processes a learning loop, the self-improvement zone shows activity
|
||
|
||
**ACP protocol flow**:
|
||
|
||
```
|
||
Spatial Office Backend Hermes ACP Server
|
||
│ │
|
||
├──── initialize ─────────────>│ "What agents have communicated
|
||
│ │ through you recently?"
|
||
│<─── agent_messages ──────────┤ [{from: "engineering", to: "senior-eng",
|
||
│ │ topic: "spatial-office PR review"}]
|
||
│ │
|
||
├──── tools/list ─────────────>│ Discover available Hermes tools
|
||
│<─── tools ───────────────────┤
|
||
│ │
|
||
├──── tools/call ─────────────>│ "Describe your current task"
|
||
│<─── result ──────────────────┤ "Refining the code-review skill
|
||
│ │ after 12 evaluations"
|
||
```
|
||
|
||
**For MVP**: The Hermes bridge starts simple — a periodic poll (every 30s) that asks Hermes for its current state. We don't need real-time streaming from Hermes in phase 1. The ACP session stays open, the bridge sends a query, receives a response, and maps it to spatial events.
|
||
|
||
### 7.3 Pi Integration
|
||
|
||
Pi (`@earendil-works/pi-coding-agent`) is the coding agent that builds the spatial-office project itself. In the office, Pi appears as a 10th "agent" (or a special entity) that:
|
||
|
||
- Sits in a designated development workstation
|
||
- Shows current task: "Implementing TileMap renderer [▓▓▓▓░░░░] 40%"
|
||
- Moves to collaboration pod when doing code review with engineering
|
||
- Shows build status from the Forgejo CI
|
||
|
||
**Integration approach**:
|
||
|
||
1. **Pi session manager**: The backend spawns a Pi session using the SDK:
|
||
```typescript
|
||
import { createAgentSession } from "@earendil-works/pi-coding-agent";
|
||
const { session } = await createAgentSession({
|
||
model: myModel,
|
||
tools: ["read", "bash", "write"],
|
||
sessionManager: SessionManager.file({ dir: "./pi-sessions" }),
|
||
});
|
||
```
|
||
|
||
2. **Status endpoint**: We add a lightweight HTTP server to the Pi session that exposes:
|
||
```
|
||
GET /status → { task: "...", progress: 0.4, status: "working" }
|
||
```
|
||
|
||
3. **Forgejo webhooks**: The backend listens for Forgejo push/PR events for the spatial-office repo to track build status
|
||
|
||
**For MVP**: The Pi bridge is a polling loop that checks pi's session log file. The spatial-office repo's CI status is checked via the Forgejo API (`GET /api/v1/repos/sscoble/spatial-office/commits`).
|
||
|
||
---
|
||
|
||
## 8. MVP Scope vs. Future Phases
|
||
|
||
### MVP (Phase 1) — "The Observable Office"
|
||
|
||
**Goal**: Scott can open a browser tab and see the 9 agents in an isometric office, with their current activity inferred from OpenClaw.
|
||
|
||
| Feature | Included? | Rationale |
|
||
|---------|-----------|-----------|
|
||
| Isometric tile map (fixed layout) | ✅ | Core visual |
|
||
| 9 agent sprites with department colors | ✅ | Core visual |
|
||
| Agent activity state (idle/thinking/working/blocked) from OpenClaw | ✅ | Core data source |
|
||
| Agent movement between zones based on activity | ✅ | Core mechanic |
|
||
| Click agent → detail card | ✅ | Scott's view |
|
||
| Zone occupancy indicators | ✅ | Visual clarity |
|
||
| Event feed (last 50 events) | ✅ | Debugging + insight |
|
||
| WebSocket real-time sync | ✅ | Required for live updates |
|
||
| Nix flake build | ✅ | Deployment requirement |
|
||
| OpenClaw GW bridge (operator.read) | ✅ | Primary data |
|
||
| **Hermes bridge** | ❌ (deferred) | Adds complexity without visual payoff in MVP |
|
||
| **Pi bridge** | ❌ (deferred) | Pi tasks can be inferred from OpenClaw events (if Pi is an agent) |
|
||
| Agent path animation (smooth tweening) | ✅ | Quality-of-life, cheap to add |
|
||
| Scott presence indicator | ✅ | Trivial (1 presence check) |
|
||
| Mobile/responsive layout | ❌ | Desktop-first, n97 serves local network |
|
||
| Multiple observers (multi-client) | ❌ | Scott's personal view only |
|
||
|
||
**MVP deliverables**:
|
||
1. Working frontend (Vite + React + Three.js) rendering the office
|
||
2. Working backend (Node.js + ws) bridging to OpenClaw
|
||
3. Nix flake that builds both and deploys as a systemd service on n97
|
||
4. Accessible at `http://n97:18790` on the local network
|
||
|
||
### Phase 2 — "Hermes + Pi Visibility"
|
||
|
||
| Feature | Notes |
|
||
|---------|-------|
|
||
| Hermes ACP bridge | Real-time Hermes state, inter-agent comm visualization |
|
||
| Pi bridge | Pi's task status on spatial-office, Forgejo CI status |
|
||
| Agent "huddle" visualization | Two agents visibly move together when collaborating |
|
||
| Hermes message visualization | Speech bubbles or "Hermes link" lines between agents |
|
||
| Coach bot visualization | Self-improvement zone shows active Hermes learning loops |
|
||
|
||
### Phase 3 — "Interaction"
|
||
|
||
| Feature | Notes |
|
||
|---------|-------|
|
||
| Scott can send messages to agents from the UI | `chat.send` through OpenClaw bridge |
|
||
| Scott can drag agents between zones | Sends `system-event` to re-route agent tasks |
|
||
| Agent "whisper" (private message to one agent) | One-click from agent card |
|
||
| Sound cues | Ambient office sounds, notification chimes |
|
||
| Agent idle animations | Breathing, looking around, typing |
|
||
|
||
### Phase 4 — "VR + Multi-User"
|
||
|
||
| Feature | Notes |
|
||
|---------|-------|
|
||
| WebXR headset view | First-person perspective in the same office |
|
||
| Multiple human observers | Different views, shared state |
|
||
| Voice chat integration | Spatial audio between agents and humans |
|
||
| Custom office layouts | Drag-and-drop office editor |
|
||
| Real collaboration tools | Shared whiteboard, code viewer, screenshare |
|
||
|
||
### Phase 5 — "Full Product"
|
||
|
||
| Feature | Notes |
|
||
|---------|-------|
|
||
| External teams | Multiple organizations, each with their own office |
|
||
| Agent marketplace | Drop-in agent "employees" from a catalog |
|
||
| Analytics dashboard | Productivity, collaboration metrics, heat maps |
|
||
| API for third-party agent systems | Standard spatial-office protocol |
|
||
|
||
---
|
||
|
||
## Appendix A: Technology Dependency Graph
|
||
|
||
```
|
||
spatial-office
|
||
├── @react-three/fiber (^8.x) → three (^0.170.x) → WebGL
|
||
├── @react-three/drei (^9.x) → helpers for fiber (OrbitControls, etc.)
|
||
├── react (^18.x)
|
||
├── react-dom (^18.x)
|
||
├── zustand (^5.x) → state management
|
||
├── gsap (^3.x) → smooth path animation
|
||
├── vite (^6.x) → build tool
|
||
├── typescript (^5.x)
|
||
├── ws (^8.x) → WebSocket server
|
||
├── zod (^3.x) → validation
|
||
├── @agentclientprotocol/sdk → ACP client (Hermes bridge, Phase 2)
|
||
└── @earendil-works/pi-coding-agent → Pi SDK (Pi bridge, Phase 2)
|
||
```
|
||
|
||
## Appendix B: OpenClaw Gateway Protocol — Methods We Use
|
||
|
||
Per the [Gateway Protocol](https://docs.openclaw.ai/gateway/protocol) and [Presence](https://docs.openclaw.ai/concepts/presence) docs:
|
||
|
||
| Method | Scope Required | Use |
|
||
|--------|---------------|-----|
|
||
| `connect` | (handshake) | Establish backend client connection |
|
||
| `system-presence` | (unrestricted) | Get connected devices/clients |
|
||
| `event:agent` (subscription) | `operator.read` | Stream agent turn events |
|
||
| `event:presence` (subscription) | (unrestricted) | Stream presence updates |
|
||
| `event:health` (subscription) | (unrestricted) | Stream gateway health |
|
||
| `event:tick` (subscription) | (unrestricted) | Gateway heartbeat |
|
||
| `chat.send` | `operator.write` | (Phase 3) Send messages to agents |
|
||
| `config.get` | `operator.admin` | (May not be needed; use config file instead) |
|
||
|
||
## Appendix C: Open Questions
|
||
|
||
1. **Agent session polling frequency**: How often should we poll? 5s is responsive but may be noisy. 30s is calmer. Start with 10s and tune.
|
||
2. **OpenClaw shared token**: Where is it stored, and how does the NixOS module securely access it? Option: `OPENCLAW_TOKEN_FILE` environment variable pointing to a file readable by the `spatial-office` user.
|
||
3. **Hermes ACP endpoint**: Is Hermes currently running on n97? If so, at what port/socket? If not, that's a prerequisite for Phase 2.
|
||
4. **Agent sprite art**: Programmatic (CSS-generated or Canvas-drawn) vs. hand-drawn pixel art. Start programmatic, iterate.
|
||
5. **n97 firewall**: Does n97 expose port 18790 to the local network? For Scott's browser to connect, this port must be open or tunneled.
|