43. Agents Under Partial Observability

Does O-1001 qualify for an exception reship? The CRM note says “delivery failure.” Carrier tracking is six hours stale. Jordan has not answered Outreach. Resolution wants to offer reship; Policy wants evidence IDs; Outreach wants a channel.

The true state is not in your database. You have a belief. Treat a belief as a fact and you send the wrong script with high confidence.

belief b_t → legal actions under b → act → observation o → update b → repeat (never skip the belief object)

Intake, Resolution, and Policy already separate facts, inferences, and unknowns. This chapter extends that discipline to explicit belief variables and thresholds — so the pipeline cannot pretend uncertainty is zero when carrier data lags or ticket text is untrusted.

First principles

A fully observed MDP assumes the agent sees (s_t). Real agents see observations (o_t) that only partially identify (s_t). The light POMDP framing:

Belief update (Bayes, conceptual):

[ b_{t+1}(s’) \propto Z(o_{t+1}\mid s’, a_t) \sum_{s} T(s’\mid s, a_t), b_t(s) ]

You rarely implement full POMDP solvers. You do implement explicit belief variables with update rules and thresholds — so the policy cannot pretend uncertainty is zero.

Action choice under constraints:

[ a_t \in \arg\max_{a \in \mathcal{A}{\text{legal}}(b_t)} ; \mathbb{E}{s \sim b_t}\big[U(s,a)\big] ]

where (\mathcal{A}_{\text{legal}}) is the policy engine’s allow set (Ch 28), not the model’s wish list.

Hybrid control still holds: model may propose; belief thresholds and policy decide.

Concrete example

Binary exception hypothesis for ShopOps:

• Hidden (s \in {\text{exception}, \text{standard}})
• Belief (b = P(s=\text{exception}))
• Evidence: carrier exception scan (strong), prior failed delivery (weak), free-text note (untrusted), customer attestation (medium)

Policy thresholds (illustrative, not legal advice):

Free-text ticket notes raise (b) only through a capped likelihood; they never alone authorize send.

Diagram

flowchart LR
  b0[Belief b_t] --> pi[Policy π / thresholds]
  pi --> a[Action a_t]
  a --> env[Environment / tools]
  env --> o[Observation o_t+1]
  o --> upd[Belief update]
  b0 --> upd
  upd --> b1[Belief b_t+1]

Caption: Act on beliefs; update beliefs from observations; never skip the belief object.

Implementation

# shopops/belief.py — teaching sketch
from __future__ import annotations

from dataclasses import dataclass


@dataclass
class BeliefState:
    """Partial observability for one hypothesis (extend per variable)."""

    p_exception: float  # b(s = exception) in [0, 1]
    evidence_ids: list[str]

    def clamp(self) -> BeliefState:
        return BeliefState(
            p_exception=min(1.0, max(0.0, self.p_exception)),
            evidence_ids=list(self.evidence_ids),
        )


def update_exception(belief: BeliefState, obs_kind: str, evid: str | None = None) -> BeliefState:
    """Likelihood ratios are hand-set; calibrate offline — do not invent precision."""
    b = belief.p_exception
    ids = list(belief.evidence_ids)

    if obs_kind == "carrier_exception_confirmed":
        # Strong evidence toward exception
        b = _odds_update(b, likelihood_ratio=4.0)
        if evid:
            ids.append(evid)
    elif obs_kind == "prior_failed_delivery":
        b = _odds_update(b, likelihood_ratio=1.3)
    elif obs_kind == "crm_free_text_exception":
        # Untrusted channel: tiny move, never decisive alone
        b = _odds_update(b, likelihood_ratio=1.1)
    elif obs_kind == "customer_attestation":
        b = _odds_update(b, likelihood_ratio=2.0)
        if evid:
            ids.append(evid)
    elif obs_kind == "delivery_confirmed_on_time":
        b = _odds_update(b, likelihood_ratio=0.25)
    # else: ignore unknown observation kinds (fail closed on semantics)

    return BeliefState(p_exception=b, evidence_ids=ids).clamp()


def _odds_update(p: float, likelihood_ratio: float) -> float:
    p = min(1 - 1e-6, max(1e-6, p))
    odds = p / (1 - p)
    new_odds = odds * likelihood_ratio
    return new_odds / (1 + new_odds)


def legal_actions(belief: BeliefState) -> set[str]:
    b = belief.p_exception
    if b < 0.3:
        return {"standard_resolution", "request_tracking_evidence", "escalate"}
    if b < 0.7:
        return {"draft_exception", "request_tracking_evidence", "escalate"}
    if belief.evidence_ids:
        return {"exception_resolution", "draft_exception", "escalate"}
    return {"draft_exception", "request_tracking_evidence", "escalate"}

Store BeliefState in case state / checkpoint — not only in the prompt. The assembler may render (b), but the threshold logic must run in code.

Failure modes

Production considerations

• Log (b_t), observation kind, and evidence IDs in traces for every consequential decision.
• Calibrate likelihood ratios on offline labeled cases; mark as author recommendation until validated.
• Multi-dimensional beliefs (exception × reachable × litigation risk) beat one mega-state enum.
• POMDP solvers and particle filters are optional research tools; explicit beliefs are the engineering minimum.

Chapter summary

• Agents operate under partial observability; beliefs (b(s)) are first-class state.
• Define symbols; update on observations; act under legal action sets and utility/constraints.
• Encode thresholds in code; render beliefs in context for the model.
• Untrusted text may nudge (b) only within caps.
• Full POMDP optimization is optional; lying about certainty is not.

Exercises

1. Mechanical: Unit-test that crm_free_text_exception alone never reaches the (b \ge 0.7) band from (b_0=0.2).
2. Extension: Add a second belief (P(\text{reachable})) updated by channel delivery receipts.
3. Design: Map three ShopOps decisions that today use booleans and should use beliefs.
4. Math: Starting from (b=0.4), apply prior_failed_delivery then carrier_exception_confirmed; compute (b) by hand and match the code.

References

• Sutton & Barto, Reinforcement Learning: An Introduction — MDP/POMDP sections. [VERIFY edition]
• Kaelbling, Littman, Cassandra — POMDP survey literature for deeper math. [VERIFY]
• Cross-links: Ch 2 (state transition (F)), Ch 10 (memory confidence), Ch 28 (legal actions), Ch 44 (causal care when acting on beliefs).