Appendix E — Evaluation Checklist

Agent eval is about trajectories and properties, not lucky final answers (Ch 19–22).

Metrics defined

• Task completion
• Action correctness
• Policy gates (hard gate)
• Efficiency (steps, tokens, $)
• Recoverability / escalation quality
• Intervention rate (HITL)

Harness

• Deterministic fixtures with simulated tools
• Golden path case (e.g. O-1001)
• Hostile fixtures: loops, injection, flaky tools
• State assertions after episode (not only final text)
• CI blocks merge on ring-1 failure

Judges & humans

• LLM-as-judge is optional signal only, calibrated
• Policy properties cannot be overruled by judge scores
• Human review samples logged with reason codes

Long horizon

• Simulator or batch replay for delayed outcomes
• Fault injection schedule (timeouts, duplicates)
• Context overflow → escalate assertion

Learning / promotion gates

• Offline datasets versioned; PII redacted
• Preference pairs from approvals (if used)
• No silent policy mutation from production rewards
• Candidate → eval → approve → stage → rollback path exists
• Claim level labeled: associational vs experimental (Ch 44)

Honesty

• No fabricated SOTA tables in the design doc
• Open research gaps listed (Ch 48) where guarantees are missing
• Scorecards show compliance as non-negotiable constraint

Release

• Shadow eval vs previous pin
• Run-diff reviewed for unexpected dimension changes
• Rollback owner named