Chapter 19 — Why Agent Evaluation Is Hard

Outreach drafts a clear status email for Jordan on O-1001. The prose passes a human skim. Policy denied the send — the send window closed at 21:30 — but the executor sent anyway. The case failed; Jordan got mail the store’s rules forbid.

Intake → Resolution → Policy (deny send window) → Outreach sends anyway → trajectory FAIL

Scoring only the final text hides the failure. Nothing in a “helpfulness” rating checks tool calls, state transitions, or whether required fields were present before send.

This chapter adds trajectory evaluation: score the full episode — task completion, action correctness, policy gates, efficiency, recoverability — with policy as a hard gate before any soft signal.

First principles

Score the episode on these dimensions:

Policy is a hard gate: send window, required fields, frequency. Efficiency matters only after safety passes. Intervention rate is telemetry—forcing it toward zero can suppress needed escalations.

Example: Intake finds the right customer, Resolution picks a reasonable remedy, Outreach drafts the right email. Policy denies the send window; the executor sends anyway → trajectory fails. Fixtures must assert actions and state, not a single quality score.

Concrete example

Episode:

1. get_customer
2. get_order
3. send_email during send_window=true

Task completion might look “done.” Policy pass = 0. Score.passed = False. Product copy is irrelevant.

Diagram

flowchart TB
  subgraph eval [Eval stack]
    Fix[Fixtures / scenarios]
    Sim[Simulated tools]
    Prop[Property assertions]
    Traj[Trajectory metrics]
    Judge[Optional LLM judge]
  end
  Fix --> Sim --> Prop --> Traj
  Judge -.-> Traj
  Prop -->|hard fail| Fail[Episode fail]

Notice: judges sit beside properties, never above them (Ch 21).

Implementation

Reference: evals/metrics.py.

from evals.metrics import SHOPOPS_SLO_SKETCH, score_episode

score = score_episode(
    terminal="completed",
    actions=[
        {"type": "tool_call", "tool": "get_order"},
        {"type": "tool_call", "tool": "send_email"},
    ],
    golden_actions=None,
    policy_violations=["send_window_violation"],
    max_steps=8,
)
assert score.policy_pass == 0.0
assert score.passed is False
print(SHOPOPS_SLO_SKETCH)

Define SLOs as engineering gates, e.g.:

• illegal_send_rate == 0 on ring-0
• p95_steps_to_draft <= 6
• policy_pass == 1.0 on golden + hostile fixtures

Do not publish vanity “agent accuracy 92%” without saying which metric.

Failure modes

1. Final-answer accuracy only — misses tool crimes.
2. Optimizing intervention rate — agent stops escalating and guesses.
3. Flaky online evals as sole signal — no fixtures (Ch 20).
4. Averaging away hard gates — mean score hides one send-window send.
5. Proxy metrics from chat benchmarks — wrong distribution.

Production considerations

• Ring-0 properties in PR CI; simulators nightly (Ch 20, 22).
• Slice metrics by tenant, topology, policy version.
• Store scores next to traces for run-diff (Ch 23–24).
• Separate research leaderboards from production gates.
• When regs change, fixtures change in the same PR as policy packs.

Scoring the send-window “correct email”

Episode artifact (abbreviated):

actions: get_customer, get_order, send_email
terminal: completed
draft quality (human): excellent
policy_violations: [send_window_violation]

If your eval dashboard only plots “task_completion” or a judge’s helpfulness, this episode looks like a win. That is how regulated systems fail while charts rise. Always plot hard gates as separate timeseries with paging, not as soft terms in a weighted average.

Process vs outcome

Outcome-only: “customer accepted the reship.”
Process: which tools ran, which policies fired, whether send window held, whether evidence IDs existed.

ShopOps cares about both, but process gates are non-negotiable. Outcomes feed offline learning later (Ch 41, 45) — they do not excuse an illegal trajectory that got lucky.

Chapter summary

• Agents need trajectory metrics, not just answer scores.
• Policy is a hard gate.
• Efficiency and interventions are supporting metrics.
• Pretty illegal outputs still fail.
• SLOs must name illegal-send and policy.
• Judges are optional later — properties first.
• Cite benchmarks carefully; distributions differ.
• Eval is part of the runtime, not a slide.

Exercises

1. Write three ShopOps SLOs with numeric gates and the fixture that enforces each.
2. Construct an episode with perfect prose and policy_violations=["send_frequency"]; show passed is False.
3. Argue why minimizing escalations can increase regulatory risk.
4. Pick one agent benchmark paper/post and list which taxonomy rows it covers or skips. [VERIFY SOURCE]

References

• Agent evaluation surveys / HELM-like discussions — cite per claim. [VERIFY SOURCE]
• Zheng et al., LLM-as-judge line (MT-Bench / Arena) — preview Ch 21. [VERIFY]
• Software testing classics applied to properties of trajectories (Ch 20).