Chapter 34 — Model Routing

Intake on O-1001 only needs to classify Jordan’s message and extract order fields. Resolution needs a stronger model to weigh reship vs refund. Running the largest model for every step wastes latency and cost without improving send window compliance or required fields checks.

RouteRequest(task, tokens, tools, region) → model_id (+ sticky per case) → recorded in trace

One default frontier model simplifies a slide deck; it does not simplify ops when P99 latency and token spend dominate the case bill.

This chapter adds routing as policy: task shape drives model choice, sticky routing keeps a case comparable across drafts, explicit fallbacks, and router decisions logged next to token usage.

First principles

1. Task shape drives model choice: complexity, tools, context length, latency, cost, and privacy region.
2. One frontier model does not simplify ops enough to ignore cost and risk.
3. Sticky routing keeps a case on one model when comparing drafts/traces.
4. Fallbacks are explicit, not “whatever the SDK does.”
5. Shadow compare is optional: score a second model offline without affecting customers.
6. Router decisions belong in traces next to token usage (Ch 23, 35).

Concrete example

O-1001 resolution call pins large-reason. Later draft stays sticky if configured, or re-routes to mid-tools for tool schemas — your policy, but record it.

Diagram — routing decision tree

flowchart TD
  R[RouteRequest] --> P{privacy_region set?}
  P -->|yes| Reg[Filter catalog by region]
  P -->|no| All[Full catalog]
  Reg --> T{needs_tools?}
  All --> T
  T -->|yes| Tool[Drop non-tool models]
  T -->|no| Ok[Keep]
  Tool --> C{context fits?}
  Ok --> C
  C -->|no| Bigger[Next larger context]
  C -->|yes| Pref[Task preference]
  Pref --> Sticky[Maybe sticky save]

 classify/extract ──► tiny-fast
 strategy ──────────► large-reason ──► sticky(case)
 draft ─────────────► mid-tools
 privacy=eu ────────► eu-mid only
 miss ──────────────► fallbacks[]

Caption: Notice privacy pins dominate preferences — policy constraints beat cost savings.

Implementation

Module: shopops/router.py.

from shopops.router import ModelRouter, RouteRequest, TaskKind

router = ModelRouter()
d = router.route(
    RouteRequest(
        case_id="O-1001",
        tenant_id="store_northline",
        task=TaskKind.RESOLUTION,
        needs_tools=True,
        approx_tokens=12_000,
    )
)
assert d.model_id == "large-reason"

eu = router.route(
    RouteRequest(
        case_id="O-1001",
        tenant_id="store_eu",
        task=TaskKind.DRAFT,
        needs_tools=True,
        approx_tokens=4_000,
        privacy_region="eu",
        prefer_sticky=False,
    )
)
assert eu.model_id == "eu-mid"

shadow = router.shadow_compare(
    RouteRequest(
        case_id="O-1001",
        tenant_id="store_northline",
        task=TaskKind.DRAFT,
        needs_tools=True,
        approx_tokens=4_000,
        prefer_sticky=False,
    ),
    shadow_model="large-reason",
)

Gateway pseudocode: decision = router.route(req) → gateway.complete(decision.model_id, …) → on provider 5xx, try decision.fallbacks.

Failure modes

Production considerations

• Drive routing changes with eval harness scores (Ch 19–20), not anecdotes.
• Maintain a model catalog with region, cost, context, tool support as data.
• Circuit-break a model id on error rate; fail over to fallbacks.
• Per-tenant allowlists: some tenants forbid certain providers.
• Revisit stickiness when prompt versions change majorly.

Chapter summary

• Route by task, tools, context, latency, cost, privacy.
• Largest model is rarely the default.
• Sticky routing helps continuity; use deliberately.
• Fallbacks are part of the contract.
• Region pins override preferences.
• Shadow compare stays offline unless explicitly designed.
• Trace every routing decision.
• Eval-driven changes beat intuition.

Exercises

1. Mechanical. Force privacy_region="eu" and assert non-EU models never win.
2. Catalog. Add a tiny-eu model; update preferences for CLASSIFY under EU tenants.
3. Design. Write a policy for when to break stickiness after a model incident.

References

• Model gateway products — conceptual routing/fallback. [VERIFY]
• Eval-driven routing practice — emerging. [VERIFY]
• Chapters 31, 35