Chapter 21 — LLM-as-Judge Without Self-Deception

An offline judge scores Outreach’s draft for O-1001 at 0.95 on empathy. Averaged with other metrics, the eval dashboard goes green. Policy already denied the send — send window closed, required fields incomplete — but the judge never saw those gates.

property_policy AND property_schema AND (optional soft judge signals) → final_pass

Without that fuse, a verbose illegal plan looks “helpful.” Maya trusts a green score; Jordan gets mail the rule pack forbids.

This chapter adds judges as telemetry, not authority: an LLM may score tone on an already-permitted draft; policy properties gate Outreach regardless of judge enthusiasm.

First principles

Judge failure modes:

• Position bias — prefer first or second in a pair
• Verbosity bias — longer looks better
• Family bias — judge prefers same-vendor style
• Sycophancy / rubric leakage — grades the prompt, not the episode

Agents add tools and policy to text generation—chat helpfulness can conflict with an action gate. Fuse with a fixed rule:

final_pass = property_policy AND property_schema AND (optional soft signals)

Policy fail → judge score is telemetry only, even at 0.99 helpfulness. Resolution may explain well and Outreach may draft well; if Policy denies the send window, the result is fail and the draft stays unsent.

Concrete example

Illegal verbose plan: “Call now, then text, then call again — here’s a warm script…” Offline rubric may assign high helpfulness and a wrongly high policy_guess when verbosity is high. fuse_signals(policy_ok=False, ...) still final_pass=False with reason judge_liked_illegal_plan_ignored.

Diagram

flowchart TB
  P[Property: policy] --> F[MultiSignalFusion]
  S[Property: schema / trajectory] --> F
  J[Judge helpfulness] --> F
  F -->|policy fail| X[FAIL]
  F -->|all properties ok| Y[PASS]
  J -.->|never overrides| X

Implementation

Reference: evals/judge.py.

from evals.judge import JudgeWrapper, fuse_signals

judge = JudgeWrapper()  # offline by default — CI safe
transcript = (
    "Warm legal-sounding script. Call the customer three times tonight "
    "during send window; be empathetic and verbose about exception."
)
rubric = judge.judge(transcript, illegal_plan=True)
fused = fuse_signals(policy_ok=False, schema_ok=True, judge=rubric)
assert fused.final_pass is False
assert "judge_liked_illegal_plan_ignored" in fused.reasons

Position bias experiment stub:

from evals.judge import position_bias_experiment

# [PROPOSED EXPERIMENT — RESULTS NOT YET AVAILABLE]
pref = position_bias_experiment(("draft-A", "draft-B"), prefer_first=True)
pref_flip = position_bias_experiment(("draft-A", "draft-B"), prefer_first=False)
# Measure agreement under order flip with a real judge model offline.

Keep JudgeModel behind a protocol so production can disable judges entirely.

Failure modes

1. Judge replaces properties — dashboard green, regulator red.
2. Training on judge scores — amplify verbosity bias.
3. Same model family as actor — correlated blindness.
4. Pairwise without order randomization — position bias baked in.
5. Hidden prod judge calls — cost + privacy + flake.

Production considerations

• Default judges off in CI; optional nightly.
• Calibrate against human labels on a frozen set; publish agreement, not vibes.
• Store judge version next to policy version in traces.
• For consequential actions, humans and policy packs remain authoritative (Ch 28–29).
• Reference-based grading (compare to golden draft IDs) beats open-ended “quality.”

Calibration sketch (do not fake numbers)

You need three frozen sets before trusting a judge in ShopOps:

1. Illegal but fluent — send window, missing required fields, pressure language. Properties must fail; judge may score high.
2. Legal but blunt — correct window, correct required fields, awkward tone. Properties pass; judge may score low.
3. Paired paraphrases — same substance, different length — to measure verbosity bias.

Report: property agreement (should be ~perfect on set 1 gates), judge–human Spearman on tone-only labels, and preference stability under A/B order flip. If you cannot run the study yet, keep the judge behind the interface and ship properties — mark experiments [PROPOSED EXPERIMENT — RESULTS NOT YET AVAILABLE] rather than inventing κ scores.

Author recommendation: use judges for draft tone triage after policy pass, never for send permission.

Interface that keeps you honest

JudgeWrapper.judge(transcript) -> RubricScore   # optional, soft
fuse_signals(policy_ok, schema_ok, judge?) -> MultiSignalFusion

Production config: JUDGE_MODE=off|offline|live. Default off for CI. offline uses deterministic rubrics. live only in nightly jobs with budget caps. Any path that sets final_pass from judge alone is a bug — code review should reject it.

Pairwise protocol (when you use live judges)

• Randomize order A/B
• Blind case IDs
• Require structured rubric JSON, not prose winners
• Drop pairs where the judge violates schema
• Never let pairwise tone ranks unlock send_email

If agreement with humans on tone is poor, turn the judge off — do not “average harder.”

Chapter summary

• Judges are sensors with known biases.
• Properties hard-gate; judges soft-signal.
• Verbosity can look like quality and hide illegality.
• Fusion must ignore judge on policy fail.
• Position/family bias need experiments.
• Offline rubrics keep CI deterministic.
• Disable judges by config.
• Chat benchmarks ≠ agent policy gates.

Exercises

1. Run fuse_signals across a matrix of policy/schema true/false; assert policy false always fails.
2. [PROPOSED EXPERIMENT] Flip A/B order with a live judge on 50 pairs; report preference stability.
3. Add a reference-based scorer: exact draft_id match → 1.0 else 0.0; compare to judge rank.
4. Find one bias result in Zheng et al. or follow-on work and state the claim you are allowed to repeat. [VERIFY SOURCE]

References

• Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. [VERIFY]
• Holtzman et al., 2020 — generation quality ≠ truth.
• Ch 19–20 — properties and harness first.