diff --git a/coach-pattern-analysis.md b/coach-pattern-analysis.md new file mode 100644 index 0000000..7420624 --- /dev/null +++ b/coach-pattern-analysis.md @@ -0,0 +1,114 @@ +# Coach Critique Analysis: Recurring 5/10 "Repetitive Behavior Pattern" + +**Issue:** marcus/scobleclaw #9 +**Date:** 2026-05-23 +**Analyzed by:** agent-cos + +## Summary of Findings + +Three consecutive coach critiques (coach #24, #25, #26) all scored **5/10 overall** with the same root complaint: **agents looping/repeating instead of completing work**. After thorough analysis, these critiques are **valid signals**, not noise. Three distinct agents (trust, engineering, vault) all exhibit the same failure mode. + +--- + +## Critiques Analyzed + +### #24 — Trust Agent (May 7) +- **Scores:** Accuracy 7, Efficiency 4, Tools 6, Skills 5, Overall 5 +- **Pattern:** Repeated identical "trust skip" entries at 14:23 and 15:23 PT with no new analysis; yields on all three ventures repeatedly +- **Root cause:** When all ventures are blocked on human, agent enters yield loop with no escalation mechanism + +### #25 — Engineering Agent (May 10) +- **Scores:** Accuracy 7, Efficiency 2, Tools 6, Skills 4, Overall 5 +- **Pattern:** Repeatedly states inability to write files, yet continues planning without executing; zero tangible progress +- **Root cause:** Environment constraints (file write restrictions during flush) cause agent to stall in planning loop + +### #26 — Vault Agent (May 20) +- **Scores:** Accuracy 8, Efficiency 3, Tools 7, Skills 4, Overall 5 +- **Pattern:** Runs memory promotion cron 14+ consecutive times with zero net-new content +- **Root cause:** No staleness detection in cron job; runs despite exhausted candidate pool + +--- + +## Root Cause Analysis + +### 1. Model: `deepseek-v4-flash:cloud` is the common denominator + +| Agent | Model Used | +|---|---| +| All agents (trust, engineering, vault, market, health, etc.) | `deepseek-v4-flash:cloud` | +| Coach sidecar (critiquing agent) | `deepseek-v4-flash:cloud` (same model!) | + +**Problem:** `deepseek-v4-flash:cloud` is a fast/cheap model optimized for speed. It lacks the depth to: +- Detect when it's in a loop and break out +- Propose creative alternatives when blocked +- Self-assess whether it's making progress vs. spinning + +The coach itself uses the same model to critique agents — meaning **the model is grading itself**. + +### 2. No skill guardrails for loop detection + +The agent config (`openclaw.json`) shows: +- `timeoutSeconds: 600` (hard timeout only — no semantic loop detection) +- `subagents.runTimeoutSeconds: 600` +- **No skills defined** for any agent (skill lists are empty) + +No agent has a "loop breaker" skill, a "staleness detector," or a "maximum-yield" guard. + +### 3. Three specific failure patterns (all with the same fix direction) + +| Pattern | Agent(s) | Fix | +|---|---|---| +| **Yield loop** — blocked on human, keeps yielding | Trust | Add max yield counter + escalation | +| **Planning loop** — can't write files, replans instead of escalating | Engineering | Add env-constraint detection + file-write alternative path | +| **Cron loop** — runs job despite stale/no candidates | Vault | Add staleness detection before running cron | + +### 4. Coach architecture limitations + +The sidecar (`sidecar.mjs`) at `/home/sscoble/.openclaw/plugins/scobleclaw-agent-coach/sidecar.mjs`: +- Has **48-hour dedup window** — prevents spam but can mask patterns +- Uses `temperature: 0.05` with `response_format: json_object` — produces consistent but shallow critiques +- The `ISSUE_THRESHOLD = 5` means anything 5 or below gets reported; **all 3 critiques hit exactly 5/10**, which is suspiciously consistent + +--- + +## Proposed Improvements + +### Immediate (config/guardrails — can implement today) + +1. **Add loop-breaker skill to agent prompts** + - "After N consecutive identical actions (yield, no-op, re-plan), stop and escalate to human with context" + - "When all ventures/tasks are blocked, generate summary report instead of looping" + +2. **Add cron staleness detection** (already partially acknowledged in #26) + - Skip promotion runs if no new entries since last successful run + - Require minimum number of unique candidates before running + +3. **Add environmental constraint handling** + - When file writes fail, write to alternate path or queue instead of re-planning + - Detect permission issues proactively + +### Medium-term (dispatch/coach improvements) + +4. **Use different model for agent vs. coach** + - Coach should use a stronger model (e.g. `deepseek-v4-pro:cloud` or `deepseek-v3.2:cloud`) for more nuanced critiques + - Currently both agent and coach use `deepseek-v4-flash:cloud` — the critiquer is the same model being critiqued + +5. **Raise coach threshold or reduce dedup window** + - ISSUE_THRESHOLD of 5 with 48-hour dedup means we only see the worst cases. Consider threshold=6 with 24-hour dedup for earlier pattern detection. + +6. **Add "repetitive loop" as a specific tracked metric in coach critiques** + - Track consecutive identical actions, yield counts, and planning iterations + +### Long-term (architecture) + +7. **Consider model tiering** — use `deepseek-v4-flash:cloud` for simple/known tasks, upgrade to `deepseek-v4-pro:cloud` or `deepseek-v3.2:cloud` for complex/problem-solving tasks +8. **Add iteration budget** — max N iterations per task before forced escalation +9. **Coach should use a different judge model** to avoid self-referential grading + +--- + +## Verdict + +**These critiques are VALID signals.** They identify a real systemic issue: all agents share the same shallow model (`deepseek-v4-flash:cloud`) and lack loop-detection guardrails. The pattern is real — it's not coach noise. + +The fixes are relatively tractable: add guardrail skills to agent configs, improve staleness detection, and tier model selection. The critical first step is adding "loop breaker" prompts and yield limits to agents.