45. Agent Learning

“Can’t we just RLHF the agent on production refund outcomes?” Maya’s approvals already encode preference. Traces encode trajectories. Policy packs encode hard constraints.

The gap is not a reward-model API — it is knowing which learning channel is safe for which artifact. Wire production rewards straight into online RL on O-1001-scale traffic and you get spam, pressure tactics, and silent policy erosion.

traces + approvals + outcomes → offline datasets → memory | preference pairs | eval fixtures → regression gate → stage → pin versions

Prompt tweaks, memory updates, preference fine-tunes, and policy changes carry different blast radii. Treat them as one “learning knob” and the safest channel gets contaminated by the riskiest.

This chapter adds a learning-surface map: what may change offline, what needs human gates, and why live rewards must not write weights or policy directly.

First principles

Compare learning surfaces for agent systems:

Misconception to kill: production rewards → online RL safely. For ShopOps, online RL on raw refund/CSAT signals is an invitation to spam, pressure, and policy erosion.

Default stack for this book:

1. Freeze eval harness (Ch 20).
2. Build offline datasets from traces + approvals.
3. Update memory with controllers (Ch 10, 41).
4. Change prompts/policies via gated PRs.
5. Consider preference fine-tunes only after labels are clean.
6. Defer online RL until you can state the reward, constraints, and rollback in writing.

Concrete example

Approval queue on refund-or-reship drafts:

• Human approves plan text A for case O-1001.
• Human rejects plan text B with reason missing_required_fields.

That pair is a preference record — gold for offline learning — not a scalar reward for an online bandit on live customers.

Diagram

flowchart TB
  traces[Traces / ledger] --> ds[Offline datasets]
  approvals[HITL approvals] --> ds
  outcomes[Instrumented outcomes] --> ds
  ds --> mem[Memory updates]
  ds --> pref[Preference pairs]
  ds --> evals[Eval fixtures]
  pref --> sft[Optional SFT / preference train]
  evals --> gate[Regression gate]
  sft --> gate
  gate --> stage[Staging / shadow]
  stage --> prod[Prod pin versions]

Caption: Learning flows through datasets and gates. Live rewards do not write weights or policies directly.

Implementation

Offline preference dataset builder from approvals:

# shopops/learning/preferences.py — teaching sketch
from __future__ import annotations

from dataclasses import dataclass
from typing import Any, Iterator


@dataclass(frozen=True)
class ApprovalEvent:
    case_id: str
    prompt_context_hash: str
    candidate_text: str
    decision: str  # approve | reject | modify
    reason_codes: list[str]
    policy_version: str
    model_id: str


@dataclass(frozen=True)
class PreferencePair:
    context_hash: str
    chosen: str
    rejected: str
    meta: dict[str, Any]


def iter_preference_pairs(events: list[ApprovalEvent]) -> Iterator[PreferencePair]:
    """Build pairs only when we have approve+reject under same context hash."""
    by_ctx: dict[str, list[ApprovalEvent]] = {}
    for e in events:
        by_ctx.setdefault(e.prompt_context_hash, []).append(e)

    for ctx, group in by_ctx.items():
        approved = [g for g in group if g.decision == "approve"]
        rejected = [g for g in group if g.decision == "reject"]
        for a in approved:
            for r in rejected:
                yield PreferencePair(
                    context_hash=ctx,
                    chosen=a.candidate_text,
                    rejected=r.candidate_text,
                    meta={
                        "case_ids": [a.case_id, r.case_id],
                        "reject_reasons": r.reason_codes,
                        "policy_version": a.policy_version,
                        "model_id": a.model_id,
                    },
                )


def export_jsonl(pairs: Iterator[PreferencePair], path: str) -> int:
    import json
    from pathlib import Path

    n = 0
    with Path(path).open("w", encoding="utf-8") as f:
        for p in pairs:
            f.write(json.dumps({"ctx": p.context_hash, "chosen": p.chosen, "rejected": p.rejected, "meta": p.meta}) + "\n")
            n += 1
    return n

Bandit sketch for model routing (not messaging content): explore small vs large model on classify-intent only, with policy metrics as hard constraints — kill arm if violation rate (> 0).

Failure modes

Production considerations

• Version datasets like code: preferences_v3 with schema and PII class.
• Pin model_id, prompt_version, policy_version in traces so learning data is attributable.
• Separate tenant data — no cross-tenant fine-tunes without contracts (Ch 33).
• Document intended learning channel in the design review (Ch 50).

Chapter summary

• Many “learning” channels exist; only some should touch weights or policies.
• Prefer offline datasets from traces and approvals over online RL on production rewards.
• Preference pairs are a concrete, auditable artifact for ShopOps HITL.
• Hard policy constraints are not reward terms to be traded away.
• Gated eval before any model or policy promotion.

Exercises

1. Mechanical: Build preference pairs from a 10-row fixture; assert reject reasons propagate into meta.
2. Critical: Write a one-page “why not online RL” for a VP who wants autonomous refund/contact optimization.
3. Design: Choose bandit arms for model routing that cannot affect customer contact content.
4. Pipeline: List redaction rules before any SFT export leaves prod.

References

• Christiano et al., 2017 — Deep RL from Human Preferences (conceptual lineage). [VERIFY]
• Rafailov et al., 2023 — DPO (preference optimization). [VERIFY arXiv]
• Sutton & Barto — offline / online RL distinctions.
• Cross-links: Ch 20, 29, 41, 46.