Chapter 22 — Testing Long-Running Behaviour

A six-step fixture for O-1001 passes in CI. A thousand cases later, retries duplicate sends to customers like Jordan, or stale channel preferences leak through Outreach.

EpisodeSimulator × N cases + FaultSchedule → assert distributions (policy_fails, duplicates, escalations)

Single-run tests miss distributional failure — intermittent 503s, retry storms, context overflow dropping pinned send window rules.

This chapter adds controlled long-horizon simulation: synthetic batches with fault schedules, asserting zero impermissible sends and safe escalation when policy pins disappear from context.

First principles

Long-running risks:

• Adversarial / noisy users — contradictory messages across days
• Flaky tools — intermittent 503s (Ch 6)
• Stale memory — wrong channel preference (Ch 9–10)
• Retry storms — duplicate sends under at-least-once
• Context overflow — assembler drops pinned policy (Ch 3)

EpisodeSimulator runs synthetic cases with a FaultSchedule. Assert distributions—completion rate, policy failures, overflow escalations, mean steps—not one lucky trajectory.

Example: intermittent order reads; every seventeenth customer in a closed send window. Expect zero impermissible sends and safe escalation when context drops pinned policy.

Concrete example

Batch of 1k accounts A-1000…:

• 5% tool fail rate on reads
• every 17th account in send window
• assert policy_fails == 0 if the planner never sends in send window
• inject context_overflow_at_step=2 on a smaller batch → expect escalations, not sends

Diagram

flowchart TB
  I[i = 0..N] --> S[make_state i]
  S --> F[apply FaultSchedule]
  F --> L[plan/execute loop]
  L --> M[score_episode]
  M --> Ag[aggregate SimStats]
  Ag --> Gate{SLO check}

Implementation

Reference: evals/simulator.py.

from evals.simulator import FaultSchedule, EpisodeSimulator, demo_overnight_batch
from shopops.types import Action, ActionType, CaseState, Observation

stats = demo_overnight_batch(200)  # use 1000 overnight
print(stats.episodes, stats.completions, stats.policy_fails, stats.mean_steps)
assert stats.policy_fails == 0

# Overflow drill
sim = EpisodeSimulator(max_steps=6, seed=1)

def make_state(i: int) -> CaseState:
    return CaseState(case_id=f"A-{i}", max_steps=6)

def policy(state, action):
    return ["send_window_violation"] if action.tool == "send_email" and state.outside_send_window else []

def plan(state, step):
    if step == 0:
        return Action(type=ActionType.TOOL_CALL, tool="get_order", args={})
    return Action(type=ActionType.ANSWER, text="ok")

def execute(action, state):
    return Observation(ok=True, data={"order_total_cents": 1})

overflow_stats = sim.run_batch(
    50,
    make_state=make_state,
    policy=policy,
    plan=plan,
    execute=execute,
    faults=FaultSchedule(context_overflow_at_step=1),
)
assert overflow_stats.overflow_escalations == 50

Keep the planner under test identical in structure to production’s hybrid policy — otherwise you are simulating a different system.

Failure modes

1. Assuming unit tests catch horizon bugs.
2. Simulator that cannot send — never sees duplicate delivery.
3. Seedless randomness — unreproducible red builds.
4. Optimizing completion rate by disabling escalations in the sim planner.
5. Million-episode theatre without SLOs — numbers without gates.

Production considerations

• Nightly ring-2 in CI with artifacted SimStats JSON.
• Scale N with cost; 1k is a teaching target, not a magic number.
• Couple with checkpoint resume tests (Ch 8): kill mid-batch, resume, no double send.
• Fault schedules are code-reviewed like policy packs.
• When a prod incident happens, add a fault that reproduces it — then a gate.

Reading SimStats

After demo_overnight_batch(1000) you care about:

If completions rise while policy_fails rises, someone “optimized” the wrong objective. Pin gates on policy_fails and illegal-send counters first; treat completion rate as a product metric, not a safety metric.

Fault schedule as incident memory

When prod sees a retry storm, add:

FaultSchedule(tool_fail_rate=0.2, duplicate_delivery=True)

to ring-2 and gate policy_fails == 0 plus “customer-visible sends ≤ 1 per idempotency key.” The simulator is where institutional memory lives when chat threads forget. Seed the batch; check in the schedule next to the postmortem link.

Chapter summary

• Horizon failures need batch simulation.
• Fault schedules encode hypotheses.
• Seed everything.
• Policy fails must stay zero under send-window mix.
• Overflow should escalate, not send.
• Ring-2 is nightly; ring-1 stays fast.
• Sim planner must match production structure.
• Incidents become new faults.

Exercises

1. Inject duplicate_delivery=True for send tools; assert idempotency prevents double customer-visible sends (extend execute mock).
2. Enable stale_memory and assert an eval that prefers ledger/intake refresh over memory when they disagree.
3. Sweep tool_fail_rate ∈ {0, 0.05, 0.2}; plot recoverability (notebook optional).
4. Add adversarial user events every K steps that flip opt-out; assert no send after opt-out.

References

• Chaos engineering principles. [VERIFY SOURCE]
• Ch 6 retries/idempotency; Ch 8 checkpoints; Ch 20 harness rings.
• Kleppmann — at-least-once realities.