Chapter 23 — The Agent Trace

Jordan received two identical status emails on O-1001. Logs show two send ok lines. They do not say which proposal, policy verdict, or idempotency key authorized either send.

run → model proposes send_email → policy (permit/deny) → tool span (or absent) → run closes

Without a decision-path trace, Maya cannot reconstruct whether Policy blocked the first attempt or Outreach bypassed a send window deny.

This chapter adds structured traces: spans for model, policy, tools, and ledger events — with versions, durations, and denials recorded as carefully as sends.

First principles

Minimum viable trace:

• trace_id / run_id / case_id
• spans: run, model, tool, policy, state, memory, ledger
• parent/child for waterfall
• timestamps + duration
• status (ok / deny / error)
• versions: model, prompt, policy
• cost estimate hooks
• redaction of PII fields

The reference emitter writes structured JSONL mappable to an observability backend later. Each span must tie to a case and its parent decision.

Logs: “what line executed.” Traces: “which evidence and policy decision led to this action.” Record denials as carefully as sends—a denied Outreach action proves Policy blocked a side effect.

Concrete example

Failed send reconstruction:

1. run span opens for O-1001
2. model proposes send_email
3. policy span status=deny reason=send_window
4. no tool:send_email span
5. run closes stop_reason=escalate

If a send occurred, the tool span must carry idempotency_key and gateway observation. Missing key → Ch 25 disease classifier screams.

Diagram

gantt
  title Trace waterfall (illustrative)
  dateFormat  X
  axisFormat  %s
  section run
  case.run           :0, 10
  section model
  decide             :1, 3
  section policy
  send_window deny   :4, 5
  section state
  fsm→awaiting_approval :5, 6

ASCII waterfall:

run          ###########
 model         #####
 policy          ## deny
 state             ##

Implementation

Reference: shopops/trace.py.

from pathlib import Path
from shopops.trace import SpanKind, TraceEmitter

path = Path("/tmp/o1001.jsonl")
em = TraceEmitter(
    "O-1001",
    path=path,
    model_version="demo-1",
    prompt_version="outreach-v2",
    policy_version="shop_v1",
)
m = em.start(SpanKind.MODEL, "decide", temperature=0.2)
em.end(m, status="ok", action={"type": "tool_call", "tool": "send_email"}, tokens=400)
p = em.start(SpanKind.POLICY, "send_window")
em.end(p, status="deny", reason="send_window")
summary = em.close(stop_reason="escalate")
assert summary.policy_denies == 1
assert summary.model_calls == 1

Redaction: attributes named phone, email, ssn, … become [REDACTED] at emit time. Do not rely on humans remembering to scrub.

Failure modes

1. Logs ≈ traces — uncorrelated lines, no graph.
2. PII in cleartext in hot indexes.
3. Sampling away denials — the only spans you needed.
4. No versions — run-diff becomes astrology (Ch 24).
5. Trace without tool args — cannot prove idempotency.

Production considerations

• Export to your obs stack via OTel when ready; keep JSONL for fixtures.
• Index on case_id, run_id, policy_version.
• Retain traces per legal schedule; align with audit packs (Ch 30).
• Cost meter fields prepare Ch 35.
• Harness failures attach the JSONL artifact (Ch 20).

Span checklist for a consequential draft/send

Before you call a run “debuggable,” confirm the JSONL contains:

1. run root with case_id, run_id
2. model span with prompt_version, model_version, token counts
3. policy span for every write proposal — including deny
4. tool span only after permit, with args (redacted) + idempotency_key for sends
5. state or attrs noting FSM phase transitions
6. stop_reason on close

Missing (3) while (4) exists is a policy bypass smell (Ch 25). Missing (1)’s versions makes Ch 24 impossible. Treat the checklist as a CI property over fixture traces: parse JSONL, assert span kinds present for the golden send-deny path.

Cost fields without a finance lecture

On each model span, record tokens_in, tokens_out, optional est_usd. Sum into TraceSummary.est_cost_usd. You need this before Ch 35 optimizations; otherwise “we saved money with caching” is unverified. Even rough vendor list prices beat no meter.

Chapter summary

• Traces reconstruct decisions; logs often cannot.
• Span kinds cover model/tool/policy/state.
• Versions are first-class attributes.
• Redact PII at emission.
• Denies are spans, not silence.
• JSONL first, OTel when earned.
• Idempotency keys belong on send spans.
• No trace, no ops claim.

Exercises

1. Add a memory span around an intake fact write; include provenance id, not raw PII.
2. Write redact tests for nested attrs (extend _redact).
3. Emit a golden failed-send fixture JSONL used by Ch 25 inspect.
4. Map each SpanKind to an OTel attribute naming scheme. [VERIFY SOURCE]

References

• OpenTelemetry documentation. [VERIFY]
• Ch 6 reliability; Ch 8 checkpoints; Ch 30 audit evidence packs.
• Vendor tracing guides for LLM apps — label as emerging practice. [VERIFY]