sscoble/scobleclaw
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Coach pattern analysis: root cause analysis of recurring 5/10 repetitive behavior critiques

Browse source 
Analyzes coach issues #24 (trust), #25 (engineering), and #26 (vault) — all
scored 5/10 for repetitive behavior patterns. Identifies common root cause:
all agents share deepseek-v4-flash:cloud model with no loop-detection
guardrails or iteration budgets.

Key findings:
- All 3 critiques are VALID — real systemic issue, not coach noise
- Root cause: single shallow model across all agents + no loop-breaker skills
- Coach uses same model to critique itself (self-referential grading)
- Three distinct failure patterns: yield loops, planning loops, cron loops
- Recommended fixes: guardrail skills, staleness detection, model tiering

Closes #9
This commit is contained in:
Scott Scoble 2026-05-23 19:27:14 -07:00
parent d30c47ebab
commit a9e0031721
1 changed files with 114 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

114
coach-pattern-analysis.md Normal file
View file

@ -0,0 +1,114 @@
# Coach Critique Analysis: Recurring 5/10 "Repetitive Behavior Pattern"
**Issue:** marcus/scobleclaw #9
**Date:** 2026-05-23
**Analyzed by:** agent-cos
## Summary of Findings
Three consecutive coach critiques (coach #24, #25, #26) all scored **5/10 overall** with the same root complaint: **agents looping/repeating instead of completing work**. After thorough analysis, these critiques are **valid signals**, not noise. Three distinct agents (trust, engineering, vault) all exhibit the same failure mode.
---
## Critiques Analyzed
### #24 — Trust Agent (May 7)
- **Scores:** Accuracy 7, Efficiency 4, Tools 6, Skills 5, Overall 5
- **Pattern:** Repeated identical "trust skip" entries at 14:23 and 15:23 PT with no new analysis; yields on all three ventures repeatedly
- **Root cause:** When all ventures are blocked on human, agent enters yield loop with no escalation mechanism
### #25 — Engineering Agent (May 10)
- **Scores:** Accuracy 7, Efficiency 2, Tools 6, Skills 4, Overall 5
- **Pattern:** Repeatedly states inability to write files, yet continues planning without executing; zero tangible progress
- **Root cause:** Environment constraints (file write restrictions during flush) cause agent to stall in planning loop
### #26 — Vault Agent (May 20)
- **Scores:** Accuracy 8, Efficiency 3, Tools 7, Skills 4, Overall 5
- **Pattern:** Runs memory promotion cron 14+ consecutive times with zero net-new content
- **Root cause:** No staleness detection in cron job; runs despite exhausted candidate pool
---
## Root Cause Analysis
### 1. Model: `deepseek-v4-flash:cloud` is the common denominator
| Agent | Model Used |
|---|---|
| All agents (trust, engineering, vault, market, health, etc.) | `deepseek-v4-flash:cloud` |
| Coach sidecar (critiquing agent) | `deepseek-v4-flash:cloud` (same model!) |
**Problem:** `deepseek-v4-flash:cloud` is a fast/cheap model optimized for speed. It lacks the depth to:
- Detect when it's in a loop and break out
- Propose creative alternatives when blocked
- Self-assess whether it's making progress vs. spinning
The coach itself uses the same model to critique agents — meaning **the model is grading itself**.
### 2. No skill guardrails for loop detection
The agent config (`openclaw.json`) shows:
- `timeoutSeconds: 600` (hard timeout only — no semantic loop detection)
- `subagents.runTimeoutSeconds: 600`
- **No skills defined** for any agent (skill lists are empty)
No agent has a "loop breaker" skill, a "staleness detector," or a "maximum-yield" guard.
### 3. Three specific failure patterns (all with the same fix direction)
| Pattern | Agent(s) | Fix |
|---|---|---|
| **Yield loop** — blocked on human, keeps yielding | Trust | Add max yield counter + escalation |
| **Planning loop** — can't write files, replans instead of escalating | Engineering | Add env-constraint detection + file-write alternative path |
| **Cron loop** — runs job despite stale/no candidates | Vault | Add staleness detection before running cron |
### 4. Coach architecture limitations
The sidecar (`sidecar.mjs`) at `/home/sscoble/.openclaw/plugins/scobleclaw-agent-coach/sidecar.mjs`:
- Has **48-hour dedup window** — prevents spam but can mask patterns
- Uses `temperature: 0.05` with `response_format: json_object` — produces consistent but shallow critiques
- The `ISSUE_THRESHOLD = 5` means anything 5 or below gets reported; **all 3 critiques hit exactly 5/10**, which is suspiciously consistent
---
## Proposed Improvements
### Immediate (config/guardrails — can implement today)
1. **Add loop-breaker skill to agent prompts**
- "After N consecutive identical actions (yield, no-op, re-plan), stop and escalate to human with context"
- "When all ventures/tasks are blocked, generate summary report instead of looping"
2. **Add cron staleness detection** (already partially acknowledged in #26)
- Skip promotion runs if no new entries since last successful run
- Require minimum number of unique candidates before running
3. **Add environmental constraint handling**
- When file writes fail, write to alternate path or queue instead of re-planning
- Detect permission issues proactively
### Medium-term (dispatch/coach improvements)
4. **Use different model for agent vs. coach**
- Coach should use a stronger model (e.g. `deepseek-v4-pro:cloud` or `deepseek-v3.2:cloud`) for more nuanced critiques
- Currently both agent and coach use `deepseek-v4-flash:cloud` — the critiquer is the same model being critiqued
5. **Raise coach threshold or reduce dedup window**
- ISSUE_THRESHOLD of 5 with 48-hour dedup means we only see the worst cases. Consider threshold=6 with 24-hour dedup for earlier pattern detection.
6. **Add "repetitive loop" as a specific tracked metric in coach critiques**
- Track consecutive identical actions, yield counts, and planning iterations
### Long-term (architecture)
7. **Consider model tiering** — use `deepseek-v4-flash:cloud` for simple/known tasks, upgrade to `deepseek-v4-pro:cloud` or `deepseek-v3.2:cloud` for complex/problem-solving tasks
8. **Add iteration budget** — max N iterations per task before forced escalation
9. **Coach should use a different judge model** to avoid self-referential grading
---
## Verdict
**These critiques are VALID signals.** They identify a real systemic issue: all agents share the same shallow model (`deepseek-v4-flash:cloud`) and lack loop-detection guardrails. The pattern is real — it's not coach noise.
The fixes are relatively tractable: add guardrail skills to agent configs, improve staleness detection, and tier model selection. The critical first step is adding "loop breaker" prompts and yield limits to agents.