41. Closing the Feedback Loop

Outreach sent O-1001’s update by email. Jordan prefers WhatsApp — Intake memory still says email. Delivery fails. Three days later a human fixes the preference by hand. Nothing in the runtime learns. The same miss hits the next case.

Traces (Ch 23), evals (Ch 19–20), and memory controllers (Ch 10) tell you what happened. They do not, by themselves, turn outcomes into safe updates.

outcome event → ingest → memory | eval candidate | scorecard → human/CI gate → (never silent policy rewrite)

Wire production outcomes straight into policy packs or prompts and you get silent drift — send-window language softens, required fields drop out, and the system calls it “learning.”

This chapter adds outcome ingestion with gated channels: what may update typed memory, what becomes an eval fixture candidate, and what must never auto-mutate Policy.

First principles

Outcome learning for agents is offline dataset construction plus gated updates, not online RL.

Separate four channels:

1. Operational state — case FSM, checkpoints (must stay correct tonight).
2. Typed memory — customer facts with provenance and confidence (Ch 9–10).
3. Eval corpus — frozen scenarios and golden trajectories (Ch 20).
4. Policy packs — versioned permit/deny code (Ch 28).

Outcomes may update (2) under a memory controller, append to (3) as candidates, and propose changes to (4) — never apply (4) without review and regression.

Invariant:

[ \text{policy}{v+1} \neq G{\text{silent}}(\text{production rewards}) ]

Silent policy mutation from live rewards is how regulated systems launder failures into “the agent adapted.”

Concrete example

Fictional account O-1001:

Resolution scorecards aggregate offline: contact success, agreed-resolution rate, policy violations (must stay at zero), human override rate. They inform humans and model-routing experiments — not an automatic rewriter of PolicyEngine.

Diagram

flowchart LR
  outcome[Outcome event] --> ingest[Outcome ingestion]
  ingest --> mem[Memory controller]
  ingest --> evalAppend[Eval set candidate]
  ingest --> score[Resolution scorecard]
  mem --> profile[Typed customer facts]
  evalAppend --> review[Human / CI gate]
  review --> harness[Eval harness]
  score --> humans[Ops review]
  humans --> candidate[Candidate policy / prompt / router]
  candidate --> review

Caption: Outcomes fan into memory, eval candidates, and scorecards. Policy and prompts move only through a gate.

Implementation

Minimal outcome ingestion that forbids silent policy writes:

# shopops/outcomes.py (teaching sketch)
from __future__ import annotations

from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Any, Literal


UpdateKind = Literal["memory", "eval_candidate", "scorecard"]


@dataclass(frozen=True)
class OutcomeEvent:
    case_id: str
    tenant_id: str
    kind: str  # e.g. email_failed, resolution_kept, approval_rejected
    payload: dict[str, Any]
    at: str  # ISO timestamp


@dataclass
class OutcomeWrite:
    kind: UpdateKind
    target: str
    data: dict[str, Any]


class PolicyMutationForbidden(RuntimeError):
    pass


def ingest_outcome(event: OutcomeEvent) -> list[OutcomeWrite]:
    writes: list[OutcomeWrite] = []

    if event.kind == "email_failed":
        writes.append(
            OutcomeWrite(
                kind="memory",
                target="memory.preferred_channel",
                data={
                    "value": "whatsapp",
                    "confidence": 0.7,
                    "provenance": f"outcome:{event.kind}",
                    "case_id": event.case_id,
                },
            )
        )
        writes.append(
            OutcomeWrite(
                kind="eval_candidate",
                target="channel_preference_stale",
                data={"seed_case": event.case_id, "event": event.payload},
            )
        )

    if event.kind == "resolution_kept":
        writes.append(
            OutcomeWrite(
                kind="scorecard",
                target="resolution.kept",
                data={"case_id": event.case_id, "at": event.at},
            )
        )

    # Hard guard: nothing here may target policy packs.
    for w in writes:
        if w.target.startswith("policy.") or w.kind == "policy":  # type: ignore[comparison-overlap]
            raise PolicyMutationForbidden(w.target)

    return writes


def apply_writes(writes: list[OutcomeWrite], *, memory, eval_queue, scorecard) -> None:
    for w in writes:
        if w.kind == "memory":
            memory.propose_update(w.target, w.data)  # controller accepts/rejects
        elif w.kind == "eval_candidate":
            eval_queue.append(w.data)
        elif w.kind == "scorecard":
            scorecard.record(w.target, w.data)
        else:
            raise PolicyMutationForbidden(str(w))

Wire this after channel adapters emit terminal events. Memory still goes through MemoryController (Ch 10): contradictory preferences supersede; they do not stack forever.

Failure modes

Production considerations

• Emit outcomes as first-class events on the ledger (Ch 16), with correlation to case_id and trace IDs.
• Redact PII before eval candidates leave the tenant boundary.
• Scorecards are ops artifacts; page on policy-pass rate, not on model self-grades.
• Preview for Ch 45–46: offline preference data from approvals; bounded improvement pipelines with rollback.

Chapter summary

• Close the loop with offline datasets, memory updates, and scorecards — not silent online RL.
• Outcomes may update typed memory and propose eval fixtures; they must not rewrite policy packs alone.
• Instrument channel and payment events; do not trust free-text “success” claims.
• Resolution scorecards inform humans and gated experiments.
• Poisoning, overfitting, and reward hacking are design constraints, not edge cases.

Exercises

1. Mechanical: Add an outcome kind opt_out that writes a permanent memory flag and an eval candidate asserting Outreach never sends.
2. Property: Write a unit test that ingest_outcome raises if any write target starts with policy..
3. Design: Sketch a weekly scorecard for ShopOps with three leading and three lagging metrics; mark which may never be auto-optimized.
4. Hostile: An outcome stream marks every denied send as user_unhappy. Show how a naive learner would weaken policy — and where your gates stop it.

References

• Sutton & Barto, Reinforcement Learning: An Introduction — offline vs online distinction (selected sections). [VERIFY edition]
• Board of Governors of the Federal Reserve System, SR 11-7: Guidance on Model Risk Management (2011) — change control intuition for model-adjacent systems.
• Chapter cross-links: Ch 10 (memory controller), Ch 20 (eval harness), Ch 28 (policy-as-code), Ch 45–46 (learning and self-improvement).