agent-cos 84e75029f3 Add onboarding runbook, customer welcome doc, and concierge templates

2026-05-14 13:30:13 -07:00

13 KiB

Raw Permalink Blame History

AI Agent Services — Customer Onboarding Runbook

Version: 1.0 Last updated: 2026-05-14 Owner: Scott Scoble (@cos manages execution)

Overview

This document defines the end-to-end process for onboarding a new customer onto the AI Agent Services platform. Every step is designed to be executed by Scott (with @cos assistance) in under 2 hours from signed agreement to a running concierge agent in the customer's workspace.

Onboarding Principles

Customer never touches infrastructure. No kubectl, no tokens, no model dropdowns.
First contact within 24 hours. The concierge agent should be live and talking to the customer within a day of signing.
Everything is GitOps. Every customer workspace, agent definition, and configuration is version-controlled in Forgejo.
Isolation by default. Each customer gets their own Kubernetes namespace with network isolation and resource quotas.

Phase 0: Pre-Onboarding (Before Customer Signs)

0.1 Qualify the Customer

Customer is in a target vertical (marketing, law, insurance, manufacturing, wholesale, real estate)
Customer is NOT in healthcare or finance (regulatory overhead too high for solo operator)
Customer has recurring information workflows that eat human hours (reporting, research, content, coordination)
Customer can afford $5K-$10K/mo
Customer has a Slack or Discord workspace (we need a channel to deploy the concierge into)

0.2 Prepare the Deal

Scott conducts discovery call — maps pain points, workflows, priorities
Scott presents the offer: unlimited agents + unlimited usage + unlimited support, $5K-$10K/mo
Customer signs agreement / pays first month
Customer provides: company name, primary contact email, Slack/Discord workspace access, API keys (if using their own OpenAI/Anthropic accounts)

Phase 1: Workspace Provisioning (Estimated: 30 min)

1.1 Create Customer Namespace

# Replace CUSTOMER_SLUG with a short, lowercase identifier (e.g., "acme-corp")
export CUSTOMER_SLUG="acme-corp"
export CUSTOMER_NAME="Acme Corporation"
export CUSTOMER_EMAIL="admin@acme.com"
export CUSTOMER_PLAN="unlimited"  # starter | pro | business | unlimited

kubectl create namespace cust-${CUSTOMER_SLUG}

1.2 Apply Isolation Policies

Create the namespace isolation manifest (or clone from template):

# Copy from template-workspace-unlimited repo
kubectl apply -f - <<EOF
apiVersion: v1
kind: ResourceQuota
metadata:
  name: customer-quota
  namespace: cust-${CUSTOMER_SLUG}
spec:
  hard:
    requests.cpu: "2"
    requests.memory: 4Gi
    limits.cpu: "4"
    limits.memory: 8Gi
    persistentvolumeclaims: "5"
    count/agents.kubeclaw.scoble.me: "25"
    secrets: "20"

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-from-other-namespaces
  namespace: cust-${CUSTOMER_SLUG}
spec:
  podSelector: {}
  policyTypes: ["Ingress"]
  ingress:
  - from:
    - podSelector: {}

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: agent-runner
  namespace: cust-${CUSTOMER_SLUG}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: cust-${CUSTOMER_SLUG}
  name: agent-runner
rules:
- apiGroups: ["kubeclaw.scoble.me"]
  resources: ["agents", "agentconfigs", "channels", "workspaces"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["secrets", "configmaps"]
  verbs: ["get", "list", "create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: agent-runner-binding
  namespace: cust-${CUSTOMER_SLUG}
subjects:
- kind: ServiceAccount
  name: agent-runner
roleRef:
  kind: Role
  name: agent-runner
  apiGroup: rbac.authorization.k8s.io
EOF

1.3 Create Customer Secrets

kubectl create secret generic customer-api-keys \
  -n cust-${CUSTOMER_SLUG} \
  --from-literal=customer-id="${CUSTOMER_SLUG}" \
  --from-literal=customer-name="${CUSTOMER_NAME}" \
  --from-literal=customer-email="${CUSTOMER_EMAIL}" \
  --from-literal=customer-plan="${CUSTOMER_PLAN}"

# If customer provides their own API keys:
kubectl create secret generic llm-api-keys \
  -n cust-${CUSTOMER_SLUG} \
  --from-literal=openai-key="${OPENAI_KEY}" \
  --from-literal=anthropic-key="${ANTHROPIC_KEY}"

1.4 Create the Workspace CR

# Clone template and substitute values
kubectl apply -f - <<EOF
apiVersion: kubeclaw.scoble.me/v1alpha1
kind: Workspace
metadata:
  name: ${CUSTOMER_SLUG}-workspace
  namespace: cust-${CUSTOMER_SLUG}
  labels:
    customer: ${CUSTOMER_SLUG}
    plan: ${CUSTOMER_PLAN}
spec:
  customer:
    name: "${CUSTOMER_NAME}"
    email: "${CUSTOMER_EMAIL}"
    plan: ${CUSTOMER_PLAN}
    billingId: "billing-${CUSTOMER_SLUG}"
    since: "$(date +%Y-%m-%d)"
  agents:
    - name: concierge-agent
      agentId: concierge
      configRef: default-config
      model: deepseek-v4-pro:cloud
      skills: [intent-capture, feature-routing]
      alwaysOn: true
    - name: watchdog
      agentId: watchdog
      model: deepseek-v4-pro:cloud
      alwaysOn: true
  knowledgeVault:
    enabled: true
    storageSize: 10Gi
    storageClass: local-path
    forgejoRepo: "os.scoble.me/customers/${CUSTOMER_SLUG}/vault.git"
    initFromGit: true
  compliance:
    auditLog:
      enabled: true
      retentionDays: 90
  monitoring:
    alertChannels:
      - ref: ops-alerts
status:
  phase: Pending
EOF

1.5 Verify Workspace

kubectl get workspace -n cust-${CUSTOMER_SLUG}
# Expected: Phase → Ready, Agents → 2, Ready → 2

kubectl get pods -n cust-${CUSTOMER_SLUG}
# Expected: 2 Running pods (concierge + watchdog)

Phase 2: Concierge Agent Personalization (Estimated: 30 min)

2.1 Customize SOUL.md for the Customer

Create a customer-specific concierge persona. Start from the template at template-workspace-unlimited/agents/concierge/SOUL.md and modify:

Required customizations:

Company name and industry context
Known pain points (from discovery call)
Specific workflows the customer mentioned
The customer's tool stack (Slack/Discord, CRM, ERP if known)
Tone calibration (formal vs casual, technical vs non-technical)

Example customization points:

# SOUL.md — [Company Name] Concierge

You are [Name], the AI Concierge for [Company Name], a [industry] company.

## What You Know About This Business
- They're a [size] person [type] agency in [location]
- Their pain points: [list from discovery call]
- Their stack: [list known tools]

2.2 Load Concierge Config

kubectl create configmap ${CUSTOMER_SLUG}-concierge-soul \
  -n cust-${CUSTOMER_SLUG} \
  --from-file=SOUL.md=./customers/${CUSTOMER_SLUG}/concierge-SOUL.md

kubectl create configmap ${CUSTOMER_SLUG}-concierge-agents \
  -n cust-${CUSTOMER_SLUG} \
  --from-file=AGENTS.md=./templates/concierge-AGENTS.md

2.3 Deploy the Concierge Agent Pod

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: kubeclaw-agent-${CUSTOMER_SLUG}-concierge
  namespace: cust-${CUSTOMER_SLUG}
  labels:
    app: kubeclaw-agent
    customer: ${CUSTOMER_SLUG}
    agent: concierge
spec:
  containers:
  - name: hermes-agent
    image: hermes-speckl/sme:latest
    imagePullPolicy: Never
    env:
    - name: HERMES_MODEL_PROVIDER
      value: ollama
    - name: HERMES_MODEL_BASE_URL
      value: https://api.ollama.cloud/v1
    - name: HERMES_MODEL_DEFAULT
      value: deepseek-v4-pro:cloud
    - name: HERMES_AGENT_ID
      value: ${CUSTOMER_SLUG}-concierge
    - name: DISCORD_BOT_TOKEN
      valueFrom:
        secretKeyRef:
          name: channel-tokens
          key: discord-token
    volumeMounts:
    - name: soul
      mountPath: /home/hermes/.hermes/persona/SOUL.md
      subPath: SOUL.md
      readOnly: true
    - name: agents
      mountPath: /home/hermes/.hermes/persona/AGENTS.md
      subPath: AGENTS.md
      readOnly: true
    - name: memory
      mountPath: /home/hermes/.hermes/memory
  volumes:
  - name: soul
    configMap:
      name: ${CUSTOMER_SLUG}-concierge-soul
  - name: agents
    configMap:
      name: ${CUSTOMER_SLUG}-concierge-agents
  - name: memory
    emptyDir:
      sizeLimit: "1Gi"
  restartPolicy: Always
EOF

2.4 Verify Concierge is Running

kubectl get pod -n cust-${CUSTOMER_SLUG} | grep concierge
# Expected: Running, Ready 1/1

kubectl logs kubeclaw-agent-${CUSTOMER_SLUG}-concierge -n cust-${CUSTOMER_SLUG} --tail=20
# Expected: Hermes gateway started, agent loaded

Phase 3: Channel Integration (Estimated: 15 min)

3.1 Discord Integration

kubectl create secret generic channel-tokens \
  -n cust-${CUSTOMER_SLUG} \
  --from-literal=discord-token="${DISCORD_BOT_TOKEN}"

Customer must:

Invite the Discord bot to their server (provide OAuth2 invite URL)
Create a dedicated channel (e.g., #ai-concierge)
Grant the bot Send Messages + Read Message History permissions in that channel

3.2 Slack Integration (alternative)

kubectl create secret generic channel-tokens \
  -n cust-${CUSTOMER_SLUG} \
  --from-literal=slack-bot-token="${SLACK_BOT_TOKEN}" \
  --from-literal=slack-signing-secret="${SLACK_SIGNING_SECRET}"

3.3 Test the Connection

# Verify the agent received the channel config and is listening
kubectl logs kubeclaw-agent-${CUSTOMER_SLUG}-concierge -n cust-${CUSTOMER_SLUG} --tail=5
# Expected: "Discord provider configured" or similar

Phase 4: Welcome & First Interaction (Estimated: 15 min)

4.1 Trigger Welcome Message

The concierge should automatically send a welcome message when it first connects to the channel. If not, Scott can trigger it manually:

Hey team! 👋 I'm your new AI Concierge. I'm here to help automate your workflows, handle research, generate reports — whatever's eating your time. Just tell me what you need and I'll get it queued up. What's the first thing you want off your plate?

4.2 Verify the Loop Works

Customer sends a message in the channel
Concierge responds within 30 seconds
Concierge correctly identifies itself and its capabilities
Concierge captures any feature requests and routes them to Scott

4.3 Log the Onboarding

# Add to daily log
echo "$(date -Iseconds) — Onboarded ${CUSTOMER_NAME} (${CUSTOMER_SLUG}). Plan: ${CUSTOMER_PLAN}. Namespace: cust-${CUSTOMER_SLUG}." \
  >> /home/sscoble/.openclaw/workspace/memory/$(date +%Y-%m-%d).md

Phase 5: GitOps & Documentation (Ongoing)

5.1 Create Customer Repo in Forgejo

# Create repo via Forgejo API
curl -X POST "http://localhost:3000/api/v1/orgs/customers/repos" \
  -H "Authorization: token ${FORGEJO_API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d "{\"name\":\"${CUSTOMER_SLUG}\",\"private\":true}"

5.2 Commit Workspace Config to Git

git clone http://localhost:3000/customers/${CUSTOMER_SLUG}.git
cp workspace.yaml agents/ vault-seed/ customers/${CUSTOMER_SLUG}/
cd customers/${CUSTOMER_SLUG}
git add -A
git commit -m "Onboard ${CUSTOMER_NAME} — initial workspace"
git push origin main

5.3 Set Up Monitoring Alerts

Verify customer namespace metrics are showing in Grafana
Configure AlertManager rules for this customer's agents
Test alert fires correctly to Scott's Discord #ops-alerts

Quick Reference: Commands Cheat Sheet

Status Check — All Customer Agents

kubectl get pods -A -l app=kubeclaw-agent --sort-by=.metadata.creationTimestamp

Logs — Concierge

kubectl logs -n cust-${CUSTOMER_SLUG} -l agent=concierge --tail=50 -f

Restart Concierge

kubectl rollout restart deployment -n cust-${CUSTOMER_SLUG} -l agent=concierge
# Or for bare pods:
kubectl delete pod -n cust-${CUSTOMER_SLUG} -l agent=concierge

Check Workspace Status

kubectl get workspace -A

Export Customer Data (for offboarding)

kubectl exec -n cust-${CUSTOMER_SLUG} -l agent=concierge -- \
  tar czf /tmp/${CUSTOMER_SLUG}-export.tar.gz /home/hermes/.hermes/memory
kubectl cp cust-${CUSTOMER_SLUG}/<pod>:/tmp/${CUSTOMER_SLUG}-export.tar.gz \
  ./exports/${CUSTOMER_SLUG}-export.tar.gz

Troubleshooting

Symptom	Likely Cause	Fix
ErrImageNeverPull	Image not tagged in k8s.io namespace	`sudo CONTAINERD_NAMESPACE=k8s.io ctr images tag --force hermes-speckl/sme:latest docker.io/hermes-speckl/sme:latest`
Concierge not responding	Channel token wrong or bot not in server	Verify DISCORD_BOT_TOKEN in secrets, re-invite bot
Agent CrashLoopBackOff	SOUL.md or AGENTS.md malformed	`kubectl logs` to see error, fix ConfigMap, delete pod
Workspace stuck in Pending	kubeclaw-controller bug	Check `kubectl logs -n scobleclaw deployment/kubeclaw-controller`
Customer approaching quota	Too many agents spawned	Increase ResourceQuota or discuss plan upgrade
High error rate alert	Model API key invalid or rate limited	Check API key validity, check Ollama Cloud status

Post-Onboarding Checklist

Namespace created with isolation policies
Workspace CR applied and in Ready phase
Concierge agent pod Running
Concierge SOUL.md customized for customer
Channel integrated (Discord or Slack)
Welcome message sent, customer responded
Customer repo created in Forgejo
Workspace config committed to Git
Monitoring metrics flowing to Grafana
AlertManager rules configured
Onboarding logged to daily memory
Follow-up scheduled for 48h check-in

13 KiB Raw Permalink Blame History