Chapter 6: Reliability Around External Systems

Maya approved the O-1001 draft. Outreach calls the email gateway to send. The gateway returns 503. Did it reject the request, or send the email and lose the response? The caller cannot know. A naive retry turns one approved message into two emails to Jordan.

Models cannot infer delivery from an ambiguous timeout. Retries, deadlines, backoff, and idempotency belong in transport code — applied consistently on every consequential write.

approve → send attempt → 503 → retry with idempotency key → one recorded send (or escalate)

This chapter wraps external calls in ReliableTransport: transient vs permanent errors, step deadlines, circuit breakers, and client idempotency keys so ShopOps survives real gateway behavior without duplicating customer mail.

First principles

For O-1001, the safety objective is not “never retry.” It is “retry transient failures without duplicating a consequential write.” The transport needs a stable idempotency key, a bounded retry policy, and a deadline shared with the rest of the case step.

The model is not exactly-once middleware. It should receive a structured outcome such as deadline_exceeded or circuit_open, then the workflow can retry later or escalate.

Concrete example

ReliableTransport wraps a mock email gateway. Chaos fail rate injects TransientError. Retries with exponential backoff. An idempotency key ensures three logical retries produce one recorded send.

Diagram

Retry state machine:

stateDiagram-v2
  [*] --> Attempt
  Attempt --> Success: ok
  Attempt --> RetryWait: transient and attempts left
  Attempt --> Fail: permanent OR budget exhausted
  RetryWait --> Attempt: backoff elapsed
  Success --> [*]
  Fail --> [*]

Timeout budget waterfall:

case step deadline ─────────────────────────────────────────►
  ├─ model call ──────────┤
  ├─ tool validate/policy ┤
  └─ transport attempts ──┬─ attempt1 ─┬─ sleep ─┬─ attempt2 ─┘
                          └──── must fit remaining budget ────┘

Notice: retries without a deadline are how overnight jobs melt gateways.

Implementation

PYTHONPATH=. python examples/ch06_transport.py

shopops/tools/transport.py:

transport = ReliableTransport(config=TransportConfig(max_attempts=6, ...))
result = send_email_reliable(
    transport,
    gateway,
    to="+1-555-0101",          # fictional
    body="...",
    idempotency_key="email-O-1001-1",
)

Idempotency store:

def call(self, fn, *, idempotency_key=None):
    if idempotency_key is not None:
        return self.idempotency.get_or_set(idempotency_key, lambda: self._retrying_call(fn))
    return self._retrying_call(fn)

add_order_note in the registry requires idempotency_key in its schema — make the contract visible to the model and enforce it in transport for the real write path.

Circuit breaker opens after N failures; subsequent calls fail fast with circuit_open until cooldown.

The 503-after-send story, slowly

1. Transport attempt 1: TCP write succeeds; gateway enqueues email; response times out.
2. Transport thinks: transient failure → retry.
3. Without an idempotency key, attempt 2 enqueues a second email.
4. With key email-O-1001-1, the gateway (or your ledger in front of it) returns the first result; customer gets one email.

Your teaching IdempotencyStore is client-side: good for demos and tests. Production prefers server-side idempotency when the vendor supports it, plus your own ledger of “we already requested send for this key” so resume-after-checkpoint (Ch 8) does not depend on vendor memory alone.

Budgets compose

A case step might allow 8 seconds total:

If the model burns 7.5s, the tool should see a tiny remainder and fail fast — not start a four-attempt retry ladder. Pass deadlines down; do not give every layer a fresh full timeout.

What the model should see

When transport fails after exhaustion, the observation should say deadline_exceeded or circuit_open, not a stack trace. The policy can then escalate or park. Hiding infrastructure errors in natural language (“something went wrong”) trains the model to invent success.

Failure modes

Retrying non-idempotent sends without keys → duplicate email.

Infinite retries on permanent 400s → log storms, no progress.

Retry storms across many cases when a region blips → shared jitter and breakers matter.

Partial failure. Resolution note recorded, email failed (or reverse). Need reconciliation or sagas (later), not hope.

Model-driven retry. Scratchpad says “try again” and bypasses transport policy.

Clock ignorance. No deadline; a single case holds a worker forever.

Production considerations

• Prefer server-supported idempotency (Stripe-style keys) over client-only dedupe when the vendor offers it. [VERIFY SOURCE: Stripe idempotency docs]
• Propagate idempotency keys into audit logs (Ch 30).
• Separate timeout budgets for model vs tools; do not let tools eat the whole SLA silently.
• Chaos-test with fault injection in CI/nightly (Part VI simulators).
• Alert on circuit open; do not silently park every case forever without ops visibility.
• At-least-once + idempotency is the honest goal; exactly-once is usually a marketing sentence.

Chapter summary

• Agents inherit distributed failure modes.
• Retry transient errors; fail permanent ones; bound by deadlines.
• Idempotency keys make at-least-once delivery safe for email sends and order notes.
• Backoff, jitter, and circuit breakers protect dependencies.
• Models are not your retry layer.
• Partial failures need explicit reconciliation paths.
• Teach keys in tool schemas, enforce them in transport.
• Code: shopops/tools/transport.py, examples/ch06_transport.py.

Exercises

1. Set fail_rate=0.2 and prove that with a stable idempotency key, len(gateway.sends)==1 across successes.
2. Inject permanent errors; assert no retry (immediate raise / fail observation).
3. Trip the circuit breaker after N failures; assert fast circuit_open.
4. Split a deadline between two tools in one step; show the second tool gets the remainder, not a fresh full budget.

References

• Kleppmann, 2017. Designing Data-Intensive Applications — retries, idempotency, at-least-once semantics.
• Stripe Idempotent Requests documentation. [VERIFY SOURCE for URL]
• Classic distributed systems guidance on backoff and jitter (e.g. AWS Architecture Blog “Exponential Backoff And Jitter”). [VERIFY SOURCE]