42. Deploying the System

ShopOps works on your laptop: one process, SQLite, mock email. Maya approves O-1001; you kill the worker mid-send. The case vanishes. A colleague clones the repo and cannot reproduce your topology. Someone ships a policy pack with no CI gate.

Deployment is when the architecture from Chapters 31–32 stops being a diagram and starts failing in ways a laptop hides — lost checkpoints, double sends, schema drift.

POST /cases (O-1001) → worker lease → Intake → Resolution → Policy → park on approval → Maya decides → resume → Outreach dry-run → trace + audit export

Without leases and partitions, two workers tick the same case. Without idempotency at Outreach, resume double-sends. Without migrations and health checks, “it worked locally” is not a deployment story.

This chapter adds the smallest production topology — API, worker, DB, queue — that forces those failures visible before customers hit them.

First principles

A production-shaped agent runtime is a small set of services with explicit contracts:

Twelve-factor habits still apply: config in env, disposable workers, backing services attached by URL. Agents do not get a special exemption from migrations, health checks, or rollback plans.

Local Compose is not “fake production.” It is the smallest topology that forces you to confront leases, double delivery, and process boundaries.

Concrete example

Fresh clone path for fictional store tenant demo:

1. docker compose up — API, worker, Postgres, Redis.
2. POST /cases with support ticket / exception event for O-1001.
3. Worker claims lease, runs Intake → Resolution → Policy; parks on approval.
4. Reviewer approves via API; worker resumes from checkpoint; Outreach dry-runs email.
5. Trace JSONL shows the full path; audit pack exportable.

If step 3 dies after draft, resume must not double-send (idempotency keys from Ch 6).

Diagram

flowchart TB
  subgraph clients
    UI[Review UI / curl]
  end
  UI --> API[shopops-api]
  API --> DB[(Postgres)]
  API --> Q[(Redis queue)]
  Q --> W[shopops-worker]
  W --> DB
  W --> MG[Model gateway]
  W --> Tools[Tool sidecars / mocks]
  W --> Trace[Trace volume]

Caption: API enqueues; workers lease; DB holds checkpoints; tools and model stay outside the request thread when work is long.

Implementation

Compose sketch (see manuscript/code/shopops/docker-compose.yml):

services:
  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: shopops
      POSTGRES_PASSWORD: shopops
      POSTGRES_DB: shopops
    ports: ["5432:5432"]

  queue:
    image: redis:7-alpine
    ports: ["6379:6379"]

  api:
    build: .
    command: python -m shopops.api
    environment:
      DATABASE_URL: postgres://shopops:shopops@db:5432/shopops
      REDIS_URL: redis://queue:6379/0
    ports: ["8080:8080"]
    depends_on: [db, queue]

  worker:
    build: .
    command: python -m shopops.worker
    environment:
      DATABASE_URL: postgres://shopops:shopops@db:5432/shopops
      REDIS_URL: redis://queue:6379/0
      MODEL_API_KEY: ${MODEL_API_KEY:-}
    depends_on: [db, queue]

CI sketch (conceptual):

# .github/workflows/shopops.yml (emerging practice)
# - pytest ring-0/1
# - docker compose build
# - smoke: create case O-1001, assert checkpoint row

Worker lease reminder (Ch 32): process at-least-once; tools must be idempotent; poison messages go to DLQ, not infinite retry.

Failure modes

Production considerations

• Environments: dev (Compose), staging (prod-like data scrubbed), prod.
• Rollout: policy pack and prompt versions are deploy artifacts; pin in traces.
• Observability: ship traces from day one (Ch 23); dashboards for queue depth, lease expiries, approval lag, $/case.
• Multi-tenant: tenant_id on every row before second customer (Ch 33).
• Checklist appendix: Deployment checklist.

Chapter summary

• Deploy API + worker + DB + queue; do not ship a laptop monolith as “the agent.”
• At-least-once delivery demands idempotent tools and durable checkpoints.
• Config and secrets via environment; migrations are part of the agent runtime.
• CI should exercise fixtures and a smoke case, not only unit tests of prompts.
• Shadow / dry-run before consequential channels.

Exercises

1. Mechanical: From a clean tree, bring Compose up and create one case; document the exact commands in the repo README.
2. Chaos: Kill the worker after awaiting_approval; approve; prove resume does not double-send.
3. Design: Draw your company’s real topology onto the Part IX boxes; mark which boxes are missing.
4. CI: Add a job that fails if send-window policy fixtures are red.

References

• Kleppmann, Designing Data-Intensive Applications — queues, retries, derived state.
• Temporal docs (concepts: workflows, activities) — durable execution as emerging practice. [VERIFY]
• 12-factor app methodology — config and backing services. [VERIFY URL]
• Cross-links: Ch 8 (checkpoints), Ch 31–32 (runtime / distributed execution), Appendix C (deployment checklist).