Evolving Reference Architecture — ShopOps Runtime

This document is the target architecture the book builds toward. Early chapters implement slices; Part IX–X assemble the whole.

Thesis in one diagram

                    ┌──────────────── human review ────────────────┐
                    ▼                                              │
 client/API → orchestrator → case worker(s) → model gateway        │
                    │                │              │              │
                    │                ├─ context assembler           │
                    │                ├─ policy engine ──────────────┤
                    │                ├─ tool executor ──► systems   │
                    │                ├─ memory controller           │
                    │                └─ checkpoint store            │
                    │                                              │
                    ├─ event bus / ledger                          │
                    ├─ trace/obs pipeline                          │
                    └─ eval harness (offline + gated online)       │

Invariant: Model proposes actions; policy + authz + executor dispose; ledger/trace/audit remember.

Logical components

Component	Responsibility	Introduced
API layer	AuthN, case intake, review UI APIs	31, 40, 42
Orchestrator	Task graph, budgets, assign, pause	18, 31
Case worker	Runs control loop / FSM ticks	1→8, 32
Model gateway	Provider I/O, routing, timeouts	34–35
Context assembler	Budgeted prompt sections	3, 35
Tool service / registry	Schemas, execution	4–6
Policy engine	Deterministic permit/deny	5, 28
Memory service	Typed memory + controller	9–10
Checkpoint store	Resume / HITL	8
Event bus + ledger	Coordination & shared facts	16–17
Approval / HITL service	Queues, timeouts	29, 40
Trace + metrics	Debug, cost, SLOs	23, 35
Eval service	Fixtures, regression, judges	19–22
Channel adapters	SMS/email/WhatsApp/voice mocks→real	39

Control flow (happy path)

support ticket / exception event creates case (checkpointed).
Orchestrator schedules Intake work.
Worker assembles context (policies pinned), model returns structured Decision.
Policy engine permits/denies each tool.
Reads execute; writes may require approval.
Resolution → Policy → Outreach as separate steps or agents with SoD.
Consequential sends append audit evidence pack.
Outcomes feed offline eval datasets (no silent online RL).

State model (conceptual)

CaseState {
  case_id, tenant_id,
  fsm_status,           # idle|planning|awaiting_tool|awaiting_approval|...
  profile,              # typed facts/inferences/unknowns
  strategy,             # proposed actions + evidence refs
  compliance_result,
  approvals[],
  memory_refs[],
  ledger_seq,
  checkpoint_version
}

Transition: (s_{t+1} = F(s_t, a_t, o_{t+1})) with (a_t) filtered by policy.

Trust boundaries

[untrusted] user messages, ticket free text, retrieved web/docs, tool raw bodies
[structured] tool JSON after validation
[trusted config] policy packs, tool allowlists, model route tables
[secrets] never in prompts — only in executor sidecars

Injection defenses live at boundaries (Ch 26), not in “please ignore” wording.

Multi-agent topology (ShopOps)

Agent / module	Can read	Can write	Notes
Intake	OMS/CRM, order/shipment history	memory (intake facts)	No customer contact
Resolution	profile, policies	resolution proposals	No send
Policy	proposals, rule pack	policy verdicts	Prefer deterministic
Outreach	approved actions	channel APIs	Least privilege tokens
Orchestrator	all metadata	assignments, budgets	Not an LLM free-for-all

Shared beliefs that matter to audit go to the ledger, not cross-chats.

Data stores

Store	Contents
Primary DB	cases, checkpoints, memory cells, tenants
Ledger log	append-only events (hash-chained per case)
Object store	large traces, evidence packs
Queue	work leases by `case_id` partition
Optional vector index	retrieval corpus — not system of record

Caching & performance (where concepts live)

Mechanism	Role in ShopOps
KV cache	Decode efficiency inside a generation
Prefix / prompt cache	Stable policy+tool preamble across turns
HTTP idempotency + result cache	Safe retries to channels
Retrieval cache	Hot policy docs — careful with staleness

Assembler stability (Ch 3) is what makes prefix caching real (Ch 35).

Framework Rosetta (late)

After manual builds, map:

Our concept	Typical framework feature
FSM / graph	LangGraph nodes/edges
Tools	OpenAI/Anthropic tools, MCP servers
Checkpoints	LangGraph checkpointers, Temporal
Workers	Celery/Temporal/cloud queues
Policies	Custom; rarely first-class

Frameworks are implementations of these boxes — not replacements for understanding them.

Non-goals

Unrestricted autonomous refunds/contact
Online RL from raw production rewards without gates
“Multi-agent debate” as a truth engine

Keyboard shortcuts

Notes on Agents