Chapter 31 — Designing the Agent Runtime

One Python file can demo a feature on O-1001. It cannot carry authentication, lease recovery, policy gates, HITL pause, trace export, and offline eval for a production case. When those boundaries blur, Maya cannot tell whether Policy, Outreach, or the orchestrator failed.

OMS event → API → orchestrator → queue/worker → model gateway + policy + tools → HITL → audit + trace

Without explicit components, every bug becomes “the agent misbehaved” and nothing is testable in isolation.

This chapter adds the runtime topology: containers and interfaces for API, orchestrator, workers, model gateway, policy engine, channel adapters, and audit — one readable path for Jordan’s case from Intake through Outreach.

First principles

1. A runtime is a topology, not a prompt file.
2. Sync vs async paths differ. Intake may be sync; case work is async with leases (Ch 32).
3. Hybrid control is structural: model gateway proposes; policy/authz/executor dispose.
4. Boundaries are interfaces. Packages and services exist so you can test and replace parts.
5. Each inbound order event should have one readable path through the diagram.

Concrete example — support ticket / exception event → path

1. Store OMS POSTs /cases with fictional order O-1001.
2. API authenticates tenant, writes case row, enqueues intake_build.
3. Orchestrator assigns topology Intake → Resolution → Policy → Outreach.
4. Worker leases work, loads checkpoint, calls model gateway with assembled context.
5. Policy engine gates tools; memory controller may write intake facts.
6. Amount threshold → HITL queue; case parks.
7. Reviewer approves; worker resumes; Outreach sends via channel adapter.
8. Audit event + trace spans land; eval harness later scores the trajectory offline.

Diagram — C4-style containers

flowchart TB
  Client[Store OMSs  / Review UI] --> API[API layer]
  API --> Orch[Orchestrator]
  Orch --> Q[Work queue]
  Q --> Worker[Case workers]
  Worker --> GW[Model gateway]
  Worker --> Pol[Policy engine]
  Worker --> Tools[Tool service]
  Worker --> Mem[Memory service]
  Worker --> CP[Checkpoint store]
  Worker --> HITL[Approval service]
  Worker --> Ledger[Ledger / bus]
  Worker --> Trace[Trace + metrics]
  Tools --> Ext[CRM / channels]
  Eval[Eval harness] -.offline.-> Trace

 client/API → orchestrator → case worker(s) → model gateway
                  │                │              │
                  │                ├─ context assembler
                  │                ├─ policy engine
                  │                ├─ tool executor ──► systems
                  │                ├─ memory controller
                  │                └─ checkpoint store
                  ├─ event bus / ledger
                  ├─ trace/obs pipeline
                  └─ eval harness

Caption: Notice the model gateway is a peer of policy and tools — not the center of the universe.

Implementation — interface boundaries (skeleton)

# Conceptual packages under shopops/ (filled across chapters)
# api/           — authn, case intake, review APIs
# orchestrator   — assign, budgets, topologies (Ch 18)
# worker/        — lease, tick FSM (Ch 32)
# router.py      — model selection (Ch 34)
# policy/        — packs + engine (Ch 28)
# tools/         — registry + executor (Ch 4–6)
# memory/        — store + controller (Ch 9–10)
# checkpoint     — resume (Ch 8)
# audit.py       — evidence (Ch 30)
# authz.py       — scopes (Ch 27)
# agents/        — intake/resolution/policy/outreach (Ch 36–39)

class CaseAPI:
    def create_case(self, tenant_id: str, order_id: str) -> str: ...

class Orchestrator:
    def assign(self, case_id: str, topology: str) -> None: ...

class ModelGateway:
    def complete(self, model_id: str, messages: list[dict], tools: list) -> dict: ...

You do not need microservices on day one. You do need module boundaries that could become services without rewriting the control ideas.

Failure modes

Sync vs async paths (detail)

Rule of thumb: if a human or an external SLA longer than a few seconds is on the critical path, it is async + checkpointed.

Production considerations

• Document sync SLAs (intake) vs async SLAs (time-to-first-contact).
• Per-service dashboards: queue depth, deny rates, approval age, $/case (Ch 35).
• Feature flags for topologies and packs, not for “let the model decide architecture.”
• Map this diagram onto your org: who owns policy packs vs channel adapters vs model gateway.
• Keep a living architecture.md in the repo; chapters implement slices against it, not against folklore.

Chapter summary

• Runtime = named services + flows, not a single script.
• Model gateway is one component among many.
• Sync intake, async case work.
• Interfaces first; distribution later if needed.
• Closed orchestrator topologies.
• HITL, audit, eval are runtime citizens.
• One event → one narratable path.
• Draw your company’s mapping onto this topology.

Exercises

1. Mechanical. Label each ShopOps component in the diagram with the chapter that introduces it.
2. Mapping. Redraw the topology for your workplace agent; mark what you currently collapse into one process.
3. Design. Specify the API contract for POST /cases and POST /approvals/{id}/decide.

References

• 12-factor / service boundary heuristics. [VERIFY]
• Manuscript architecture.md — evolving target diagram.
• Chapters 18, 27–30, 32–35