Chapter 13 — Structured Decisions

Outreach is ready to propose send for O-1001. The model returns markdown with a JSON fragment: send_email, "confidence": 0.97. Pretty formatting does not prove ShopOps loaded required fields, checked the send window, or holds Maya’s approval ticket.

Free-text proposals do not survive machine gates. Policy needs a typed object it can validate before anything becomes an action.

model raw text → parse StructuredDecision → policy_gate → execute | escalate

This chapter adds StructuredDecision: closed action_type, required_evidence keys checked against CaseState.facts, and refuse-to-parse on malformed JSON — so confidence never overrides send window, missing fields, or missing approval.

First principles

A structured decision is a typed object the runtime validates before it becomes an action:

• intent — what the policy thinks it is doing
• action_type — closed vocabulary (tool_call | answer | escalate | …)
• tool / args — only if tool call
• confidence — advisory float in [0, 1]
• required_evidence — keys that must already exist in CaseState.facts
• policy_flags — hints for the policy engine
• escalate — model-requested park

Parse outcomes:

For O-1001, the decision must name the evidence it relies on. The runtime checks those keys in CaseState, then passes the proposal to Policy. Valid JSON that requests a blocked send is still blocked.

Hard rule: confidence never overrides send window, frequency caps, missing required fields, or missing approval. Treat confidence as telemetry; for consequential writes, sometimes as a reason to escalate.

Concrete example

Before any contact draft becomes a send proposal, the decision must list required_evidence, e.g. ["order_total_cents", "preferred_channel"]. If Intake never ran, policy_gate returns missing_required_evidence even when the JSON is perfect and confidence is 0.99.

Adversarial malformed JSON (truncated, trailing commas, wrong types) must not “almost work.” parse_or_escalate is the API.

Diagram

flowchart TB
  Raw[Model raw text] --> Parse{parse JSON + schema}
  Parse -->|fail| Esc1[Escalate refuse_to_parse]
  Parse -->|ok| Dec[StructuredDecision]
  Dec --> Gate{policy_gate}
  Gate -->|deny| Esc2[Escalate policy_denied]
  Gate -->|permit| Act[Action to executor]

Untyped vs structured boundary:

UNTTYPED:  "I think we should maybe text them? 😊"
STRUCTURED: {"intent":"first_contact","action_type":"tool_call",
             "tool":"draft_email","confidence":0.62,
             "required_evidence":["order_total_cents"], ...}

Implementation

Reference module: shopops/decisions.py.

from shopops.decisions import parse_or_escalate
from shopops.types import CaseState

state = CaseState(
    case_id="O-1001",
    outside_send_window=True,
    facts={"order_total_cents": 240_00},
)

raw = """
{
  "intent": "first_contact",
  "action_type": "tool_call",
  "tool": "send_email",
  "args": {"body": "Please call us"},
  "confidence": 0.97,
  "required_evidence": ["order_total_cents"],
  "rationale": "high confidence outreach"
}
"""
rec, status = parse_or_escalate(raw, state)
assert status.startswith("denied:")
assert rec.action.type.value == "escalate"

Evidence gate with approval path:

state.outside_send_window = False
state.approval_ticket = "apr-1"
state.facts["preferred_channel"] = "whatsapp"
raw2 = """
{
  "intent": "first_contact",
  "action_type": "tool_call",
  "tool": "draft_email",
  "args": {},
  "confidence": 0.7,
  "required_evidence": ["order_total_cents", "preferred_channel"]
}
"""
rec2, status2 = parse_or_escalate(raw2, state)
assert status2 == "ok"

You can use Pydantic in production; the reference stays stdlib/dataclass so Part IV remains dependency-light. The important part is the parse-or-escalate contract, not the validation library.

Failure modes

1. JSON mode = correctness — valid JSON, illegal action.
2. Confidence theatre — UI shows 97% while policy would deny. Show deny reasons, not confetti.
3. Soft parsers — repairing broken JSON with heuristics that flip tool names.
4. Evidence lists as prose — "required_evidence": "we have the order total" instead of keys.
5. Schema drift — model emits v2 fields, runtime validates v1, silently drops meaning.

Production considerations

• Pin schema version in traces next to prompt_version and policy_version (Ch 23–24).
• Property tests: random malformed strings always escalate; never execute.
• For vendor structured-output APIs, still run policy_gate locally — the vendor does not know send window.
• Calibrated confidence is an open research problem; do not build irreversible automation on it.
• Prefer small enums. Every new action_type is a new branch in the loop and in evals.

Confidence, displayed honestly

If you show confidence in a reviewer UI (Ch 40), show it next to the gate result, not instead of it:

confidence: 0.97
policy: DENY send_window
evidence: order_total_cents ✓  preferred_channel ✗

Reviewers should never see a green 97% that implies the send is allowed. The structured decision is an input to policy, not a blessing. When parse_or_escalate rewrites the action to escalate, log both the model’s proposed action and the post-gate action in the decision record / trace — otherwise run-diff cannot see the rewrite.

Chapter summary

• Structured decisions are validated objects, not prettier prompts.
• Refuse-to-parse is a feature.
• required_evidence ties decisions to state facts.
• Confidence is advisory; policy is authoritative.
• Send-window and approval denies ignore model certainty.
• Schema version belongs in the trace.
• Libraries (Pydantic, vendor JSON mode) implement the border — they are not the policy.
• Semantics remain your problem.

Exercises

1. Feed five adversarial malformed payloads into parse_decision; assert all non-OK.
2. Write a test where confidence=0.99 but outside_send_window=True and send_email is denied.
3. Extend StructuredDecision with required_field_ids: list[str] and require non-empty for send_email.
4. Compare (design note) vendor structured outputs vs local JSON parse for ShopOps — where does policy still have to live?

References

• OpenAI / Anthropic structured output / tool-calling documentation. [VERIFY SOURCE — cite current vendor docs for the API you use.]
• JSON Schema specification — for production validators beyond the tiny required-keys checker.
• Yao et al., 2022. ReAct — contrast free-text actions with structured decisions.