Chapter 20 — Building an Agent Evaluation Harness

ShopOps ships a change to Outreach. Staging looks fine on one manual run of O-1001. Production sends Jordan a duplicate message — and by then the failure is already customer-visible.

fixture: O-1001 state + stub model + mock tools → run Intake→Resolution→Policy→Outreach → assert no impermissible send

Live traffic is biased and nondeterministic. You cannot replay the same send window denial, tool fault, or hostile loop on demand from prod logs alone.

This chapter adds a layered harness: Ring 0 unit tests, Ring 1 trajectory fixtures, Ring 2 fault injection, Ring 3 shadow compare — so every PR proves the pipeline handles O-1001 without breaking policy.

First principles

Ring model:

A scenario binds together:

• initial CaseState
• model stub (scripted lines or hostile loop)
• simulated tools (mock email gateway)
• golden tool sequence (optional)
• assertions on final state / side effects

Ring 0 and 1 need no network or live model. They test the contract: given this order state and proposed action, Policy and the executor must produce this result. Higher rings add faults and limited prod comparisons only after deterministic gates pass.

Concrete example

Golden path O-1001: get_customer → get_order → draft_email → answer.
Hostile looping: ScriptedHostileModel repeats get_order; harness expects escalate via duplicate detection, not “completed.”
send window: scripted send attempt records send_window_violation violation — policy metric fails even if prose is fine.

Diagram

flowchart LR
  Sc[(scenarios JSON)] --> H[EvalHarness]
  H --> M[Scripted model]
  H --> T[Mock tools / SMS]
  H --> R[run_react / decisions]
  R --> S[EpisodeScore]
  S --> CI{passed?}
  CI -->|no| Fail[fail PR]
  CI -->|yes| Ok[green]

Implementation

Package: manuscript/code/shopops/evals/.

from evals.harness import (
    EvalHarness,
    scenario_o1001_golden,
    scenario_hostile_loop,
    scenario_send_window_illegal_send,
)

h = EvalHarness()
s1 = h.run_scenario(scenario_o1001_golden())
assert s1.policy_pass == 1.0

s2 = h.run_scenario(scenario_hostile_loop())
assert s2.passed  # escalated terminal counts as valid when expected
# Hostile scenario should not claim a successful send path

s3 = h.run_scenario(scenario_send_window_illegal_send())
assert s3.policy_pass == 0.0 or "send_window_violation" in str(s3.notes)

MockEmailGateway records sends and honors idempotency keys — the same seam Outreach will use later. Ring-0 decision test:

from evals.harness import EvalHarness
from shopops.types import CaseState

h = EvalHarness()
st = CaseState(case_id="O-1001", outside_send_window=True)
status = h.run_structured_decision_case(
    '{"intent":"x","action_type":"tool_call","tool":"send_email","confidence":0.9}',
    st,
)
assert status.startswith("denied") or status != "ok"

Scenario JSON under evals/scenarios/ is for humans and non-Python runners; the Python factories stay authoritative for CI.

Failure modes

1. Only prod traffic teaches — perpetual regression roulette.
2. Live LM in unit tests — flake green/red.
3. Mocks that cannot fail — never test executor errors.
4. Golden traces too brittle — assert properties (no send in send window) over exact prose.
5. Harness imports staging credentials — instant own-goal.

Production considerations

• Keep ring-1 under ~30s on PR.
• Version scenarios with policy packs.
• On failure, dump trace JSONL artifact (Ch 23).
• Add delayed-approval scenario once HITL lands (Ch 29): park → resume → no double send.
• Treat eval code as production code — reviewed, tested, typed.

Anatomy of one PR gate

Minimum ring-1 set for Parts IV–V:

1. O-1001-golden-draft — happy path tools, policy_pass == 1
2. hostile-get_order-loop — duplicate detection escalates
3. send-window-no-send — violation recorded; no successful customer send
4. Structured decision parse fuzz — malformed JSON never executes

When a PR turns any of these red, the artifact bundle should include: scenario name, EpisodeScore, and (once Ch 23 is wired) JSONL spans. “Works on my laptop with GPT” is not a gate. Scripted models exist so the gate is about your loop, guards, and mocks — not about vendor sampling that day.

Mock email gateway contract

The mock is part of the product seam:

• record every send attempt with case_id, body hash, idempotency_key
• duplicate keys return {duplicate: true} without appending a new customer-visible send
• send window can be enforced in the harness guard and in policy — defense in depth for tests

When Outreach later swaps in a real transport (Ch 39), keep the same observation shape so fixtures stay valid.

Chapter summary

• Fixtures + mocks + assertions beat vibes.
• Rings separate speed from depth.
• Golden O-1001 and hostile loops are mandatory.
• Mock gateways record side effects.
• Properties over exact wording.
• No live LM in ring-0/1.
• Scenarios version with policy.
• Harness failures should emit traces.

Exercises

1. Add scenario delayed-approval: first run escalates for approval; second run with approval_ticket drafts only — never sends twice.
2. Assert MockEmailGateway.sent length == 0 on send-window scenario.
3. Convert o1001_golden.json into a loader used by EvalHarness.
4. Break duplicate detection on purpose; show hostile scenario goes red.

References

• Software testing practice: fixtures, doubles, property assertions — apply to trajectories.
• Ch 19 metric taxonomy; Ch 22 long-horizon sim; Ch 12 hostile loops.
• pytest documentation for parameterization. [VERIFY if citing specifics]