Chapter 24 — Why Did the Agent Behave Differently?

Friday: O-1001 escalates — Outreach refuses to send outside the send window. Monday: a similar stalled-shipment case proposes send_email. The team says “the model changed.” That is speculation until you diff the artifacts.

run A artifacts ↔ run B artifacts → ranked candidates (policy → prompt → model → tools → memory → input)

Text-only comparison hides what moved — prompt version, policy pack, send window flag in input state. You fix the wrong layer.

This chapter adds run diff: compare two closed executions of O-1001 across versioned inputs and produce a ranked candidate list before anyone opens a model rollback.

First principles

Compare two closed runs across:

Prioritize candidates: policy → prompt → model → tools → memory → input → observations. If actions diverge with no artifact diff, inspect sampling and recording gaps—that is still an engineering finding, not a guess.

Example: same order, model, and policy; prompt v2 vs v3. Outreach flips from escalation to send proposal → leading candidate is prompt_version, not “model rolled.”

Concrete example

Prompt v2 vs v3 send-window regression fixture in run_diff.py: same model, same policy, same outside_send_window=True input; actions flip from escalate to send_email. Diff report’s top candidate: prompt_version changed. That is enough to stop the “model rolled” thread and open a prompt PR revert.

Hidden memory write exercise: two runs identical except memory_snapshot.preferred_channel; preferred channel flips — find it in the report.

Diagram

flowchart LR
  A[Run A artifact] --> D[RunDiff]
  B[Run B artifact] --> D
  D --> C[candidate_causes]
  C --> P[policy/prompt/model/...]

Dimensions:  model  prompt  policy  tools  memory  state  obs  actions
Changed:       .      ≠       .       .      .       .     .     ≠
Candidates:  prompt_version changed

Implementation

Reference: shopops/run_diff.py.

from shopops.run_diff import diff_runs, send_window_regression_fixture

v2, v3 = send_window_regression_fixture()
report = diff_runs(v2, v3)
print("\n".join(report.summary_lines()))
assert any(c.startswith("prompt_version") for c in report.candidate_causes)
action_dim = next(d for d in report.dims if d.dimension == "actions")
assert action_dim.changed

Wire traces into run artifacts: TraceEmitter.to_dict() plus input_state and memory_snapshot at run start. Without those snapshots, memory diffs are invisible — which is itself a design bug.

Failure modes

1. “Model rolled” as default explanation.
2. Diffing prose only — missing versions.
3. Unsnapshot memory — ghost behaviour.
4. Claiming causality from a candidate list (wait for Ch 44).
5. Comparing live unsaved runs — no artifact, no science.

Production considerations

• Store run artifacts immutably keyed by run_id.
• Auto-diff on eval regressions in CI.
• Alert when policy_version differs across canary vs control.
• Experimentation practices (config diffs, feature flags) apply; agents are not exempt. [VERIFY SOURCE for any specific experimentation guide.]
• Redact PII inside artifacts the same as traces.

Reading a diff report without fooling yourself

Example summary_lines():

RunDiff run-v2 → run-v3
  ≠ prompt_version: 'outreach-v2' → 'outreach-v3'
  ≠ actions: [{'type': 'escalate', ...}] → [{'type': 'tool_call', 'tool': 'send_email'}]
Candidate causes (engineering, not proven causality):
  - prompt_version changed

What you may say in the incident channel: “Actions diverged; prompt_version is the leading config diff; rolling back outreach-v3 while we inspect assembler pins.”

What you may not say: “Prompt v3 caused the send” as a settled causal claim — unless you also show the input state, policy version, and a controlled replay (Ch 44). Candidate ≠ verdict. Still: candidates beat “model rolled.”

Hidden memory drill: clone run-v2, alter only memory_snapshot={"outside_send_window": false} while wall-clock send window are true. If Outreach trusts memory over clock, actions flip and the diff’s top line is memory_snapshot changed. That is why snapshots are mandatory at run start.

Artifact schema (minimum)

Persist per run_id:

{
  "run_id": "...",
  "meta": {"model_version": "...", "prompt_version": "...", "policy_version": "..."},
  "input_state": {"outside_send_window": true, "case_id": "O-1001"},
  "memory_snapshot": {},
  "tool_schemas": ["get_order", "draft_email", "send_email"],
  "actions": [],
  "stop_reason": "escalate",
  "spans": []
}

If memory_snapshot is missing, your diff tool must say so loudly — absence is itself a finding.

Sampling residual

When every config dimension matches and actions still differ, the honest candidate is sampling / nondeterminism. Mitigations: lower temperature on tool choice, structured outputs, seed if the vendor supports it, and golden scripted models in CI so your code paths stay deterministic even when prod samples.

Chapter summary

• Behaviour change → diff versions and state first.
• Candidate causes are ranked engineering hypotheses.
• Prompt/policy often beat “model rolled.”
• Memory snapshots are mandatory for fair diffs.
• Fixtures encode known regressions.
• Do not overclaim causality.
• CI can auto-diff failing evals.
• Artifacts beat Slack memory.

Exercises

1. Build two run dicts that differ only in memory_snapshot; assert candidate list leads with memory.
2. Add tool_schemas change that introduces send_whatsapp; show actions diff.
3. Integrate diff_runs into the eval harness when passed flips vs last golden.
4. Sealed puzzle: given two artifacts (create them), find the hidden memory write.

References

• Experimentation / config diff practices. [VERIFY SOURCE]
• Ch 23 traces; Ch 35 cost when model version changes.
• Ch 44 causal reasoning — upgrade path from candidates to interventions.