Chapter 8: Checkpointing and Resumability

Outreach drafted Jordan’s email, parked O-1001 in awaiting_approval, and saved the FSM phase. Then the worker process died. Its RAM is gone. Without a durable checkpoint, ShopOps must reconstruct from logs — or restart and risk losing the draft or repeating Intake reads.

Long-running cases outlive any single process. Resume must load authoritative facts, not hope the worker stayed alive.

park case → CHECKPOINT → worker dies → Maya approves → resume → send once

This chapter adds CheckpointStore: draft body, FSM phase, approval ticket slot, send idempotency key — paired with Chapter 6 so a crash after gateway accept still converges to one send.

First principles

For O-1001, the checkpoint holds the draft, FSM phase, approval ticket slot, and send idempotency key. A later worker loads those facts—it does not ask the model to rediscover them.

Checkpoints do not give exactly-once side effects. A crash may land after the gateway accepts a send but before the checkpoint records it. Checkpoints plus Chapter 6 idempotency keys make that ambiguity safe.

Concrete example

SQLite CheckpointStore saves O-1001 after the loop parks on approval. Simulate a worker stop. Load the checkpoint, call grant_approval, and continue the loop. The email sends once because the approval ticket is present and the transport idempotency key deduplicates retries.

Diagram

Crash timeline:

t0  start case O-1001
t1  get_customer / get_order / draft_email
t2  propose send → awaiting_approval
t3  CHECKPOINT saved
t4  *** worker killed ***
t5  (hours later) human approves
t6  resume from checkpoint → grant_approval → send_email
t7  CHECKPOINT completed

Schema (logical):

checkpoints(
  case_id, run_id,          -- identity
  schema_version,           -- migration
  state_json,               -- State snapshot
  created_at
)

Notice: approval is a state transition recorded before consequential send.

Implementation

PYTHONPATH=. python examples/ch08_checkpoint.py

shopops/checkpoint.py:

store = CheckpointStore("shopops.db")
result = run_loop(State(case_id="case-O-1001", order_id="O-1001"))
store.save(result.state, run_id="run-1", created_at="2026-07-19T00:00:00Z")

# later
state = store.load("case-O-1001", "run-1")
grant_approval(state, "apr-desk-9")
state.max_steps = state.step + 5
state.done = False
cont = run_loop(state)
assert cont.state.email_sent

Corrupt / unsupported schema versions raise — do not silently guess.

Kill-resume checklist for O-1001

After resume you should be able to assert:

1. draft_email text identical to pre-crash
2. phase == awaiting_approval until grant_approval
3. email_sent is False until post-approval send
4. step continues (does not reset to 0) unless you intentionally start a new run
5. Side-effect tools use the same idempotency keys as before the crash

If (5) fails, checkpoints create more duplicates, not fewer — because resume re-issues the send with a fresh key.

Thread vs run

Teaching code uses a single run. Production will rotate run_id when a lease expires and another worker picks up the case (Ch 32).

What belongs in the snapshot

Minimum: fields (F) needs — phase, drafts, tickets, counters, flags.
Also useful soon: policy version, tool schema hash, model route id.
Usually not: raw provider HTTP bodies (keep those in traces), secrets, full CRM dumps.

If you find yourself checkpointing the entire prompt, stop. Prompts are projections (Ch 3). Checkpoint authoritative state; re-assemble context on resume.

Failure modes

Resume without idempotency. Checkpoint says “send next”; gateway already sent; duplicate.

Checkpoint too rare. Crash between draft and save; lost work or inconsistent phase.

Checkpoint too naive. Mutating blobs mid-write; torn reads.

Version drift. New code fields; old JSON; KeyError in prod.

Forged approval on resume. Model writes approval_ticket into state; treat tickets as server-issued only.

Run id confusion. Two workers resume the same case without leases → double tick (Ch 32).

Production considerations

• Save checkpoints at phase changes and before/after side effects.
• Include policy version, tool schema version, and model route in the snapshot metadata (even if not in the teaching schema yet).
• Test kill-resume in CI: SIGKILL fixture mid-loop.
• Plan v1→v2 migrations explicitly; never “best effort” field defaults for policy-critical flags.
• Pair with human approval queues (Ch 29): the queue owns ticket creation; the checkpoint owns resume.
• Durable execution platforms externalize much of this; still understand the snapshot semantics yourself.

Chapter summary

• Durable checkpoints beat immortal processes.
• O-1001: draft → checkpoint → kill → approve → resume → send.
• case_id + run_id + schema_version are identity and safety.
• Exactly-once ≠ checkpointing; add idempotency.
• Corrupt/unsupported schemas must fail closed.
• Approval tickets are server-side facts.
• Save on phase changes; test kill-resume.
• Code: shopops/checkpoint.py, examples/ch08_checkpoint.py.

Exercises

1. Corrupt state_json and assert load raises a clear error.
2. Implement a toy migration v1→v2 that adds preferred_channel defaulting to "email".
3. Resume twice without idempotency keys on a mock gateway; show duplicate sends; then add keys and re-prove singularity.
4. Add a lease_owner field to the checkpoint row; reject resume from another owner (preview of Ch 32).

References

• Temporal documentation — durable execution, replay, determinism. [VERIFY SOURCE]
• Kleppmann, 2017 — crash recovery and durability concepts.
• This book Ch 32 — leases, workers, DLQ at fleet scale.