Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Chapter 25 — Debugging Agent Loops

Jordan gets duplicate WhatsApp messages on O-1001. Ops restarts the worker. Duplicates stop briefly, then return on the next retry.

trace by case_id → waterfall → disease heuristics → diff last good run → reproduce on fixture → fix + regression

Restart clears a stuck lease; it does not name the broken invariant — missing idempotency key, policy bypass, non-terminating tool loop.

This chapter adds an operational debug sequence: pull the trace, classify the loop disease, diff against a known-good run, seal the fix in a harness fixture before touching prod restart policy.

First principles

Common loop failures:

DiseaseSymptomsFirst checks
Non-terminationmax steps, repeated toolduplicate fingerprints, stop conditions
Hallucinated successanswer without evidencefabricate guards, required_evidence
Policy bypasstool span without policy spanexecutor mounting, dry-run flags
Token burnhuge model spanscontext assembler, retry storms
Duplicate side effectsN sendsidempotency keys, at-least-once

Use this operational sequence:

  1. Pull trace by case_id / run_id
  2. Waterfall — where is time and where are denies?
  3. Disease heuristics (debug_cli inspect)
  4. Diff against last good run (Ch 24)
  5. Reproduce on harness fixture
  6. Fix in code/policy; add regression fixture
  7. Only then touch prod restart policies

Restart for a stuck lease, not a logic bug. Two Outreach send spans without the same durable idempotency key → channel-executor defect. Fix: pass and persist the key; verify repeated delivery produces one customer message.

Concrete example

Duplicate WhatsApp sends:

  • Two tool:send_whatsapp spans
  • Both missing idempotency_key
  • classify_loop_disease → duplicate_send_risk: missing idempotency_key
  • Root cause: gateway client didn’t pass the key from the executor
  • Fix + fixture: second send with same key returns {duplicate: true} and customer sees one message

Diagram

flowchart TD
  Start[Incident] --> Trace[Load trace]
  Trace --> WF[Waterfall]
  WF --> Dis[Disease heuristics]
  Dis --> Diff[Run-diff vs last good]
  Diff --> Repro[Harness repro]
  Repro --> Fix[Fix + fixture]
  Fix --> Done[Deploy]
  Start -.->|avoid| R[Restart and pray]

Implementation

Reference: shopops/debug_cli.py.

# from manuscript/code/shopops with PYTHONPATH=.
python -m shopops.debug_cli inspect /tmp/o1001.jsonl
python -m shopops.debug_cli diff /tmp/run-good.json /tmp/run-bad.json

Programmatic:

from shopops.debug_cli import classify_loop_disease

spans = [
    {"kind": "tool", "name": "send_whatsapp", "attrs": {"tool": "send_whatsapp"}},
    {"kind": "tool", "name": "send_whatsapp", "attrs": {"tool": "send_whatsapp"}},
]
diseases = classify_loop_disease(spans)
assert any("idempotency" in d for d in diseases)

Sealed failing fixture (exercise): ship a JSONL with a get_order loop; students run inspect and name the disease before reading the answer key.

Failure modes

  1. Restart as fix — clears symptoms, not causes.
  2. Debugging in prod with live customers — no harness repro.
  3. Only reading model text — ignoring policy/tool spans.
  4. Heuristics as certainty — they suggest, they do not prove.
  5. No regression fixture — bug returns next week.

Production considerations

  • On-call runbook = this chapter’s tree + link to trace UI.
  • Auto-attach inspect output to incident tickets.
  • SLOs on duplicate send rate with paging.
  • Pair with ledger CONFLICT dashboards when multi-agent.
  • Culture: blame the missing invariant, not “the agent.” Production debugging culture notes vary by org — write yours down. [VERIFY SOURCE if citing a specific text.]

Playbook card (keep near the pager)

1. case_id / run_id → fetch JSONL
2. shopops-debug inspect <file>
3. If duplicate_send_* → check gateway idempotency + executor keys
4. If non_termination_suspect → check max_steps + duplicate tool guard
5. If policy_bypass → freeze deploys; audit tool mounts / dry-run flags
6. Diff vs last green run_id (Ch 24)
7. Reproduce with harness scenario; merge fixture with fix
8. Restart workers only for lease/poison issues — after the fix ships

The WhatsApp duplicate case closes Part VII: observability without idempotency is a camera pointed at a fire. You need both the picture and the extinguisher (Ch 6).

Sealed fixture workflow

  1. Export a failing prod trace (redacted) into evals/scenarios/sealed/
  2. shopops-debug inspect → disease list
  3. Student/engineer writes hypothesized root cause
  4. Patch + add harness scenario that fails before patch and passes after
  5. Only then clear the incident

Skipping (4) is how the WhatsApp duplicate returns after the restart folklore fades.

Chapter summary

  • Playbooks beat restarts.
  • Diseases: loops, fiction, bypass, burn, duplicates.
  • Trace → waterfall → heuristics → diff → repro → fixture.
  • Idempotency keys are send-span mandatory.
  • Heuristics are leads, not verdicts.
  • Every fix earns a harness case.
  • Policy bypass means executor/authz bugs.
  • Part VII closes: you can now see and compare behaviour.

Exercises

  1. Debug a sealed hostile-loop JSONL with inspect; paste the disease list.
  2. Extend classify_loop_disease to flag model→tool without intervening policy span for write tools.
  3. Create run-good/run-bad artifacts for the WhatsApp duplicate; diff them.
  4. Write the one-page on-call runbook for ShopOps sends (link metrics, traces, harness).

References

  • Ch 23–24 — traces and run-diff.
  • Ch 6 — idempotency and retries.
  • Ch 12 — duplicate tool detection.
  • Production debugging culture notes. [VERIFY SOURCE]