Chapter 35 — Cost and Latency Engineering

O-1001 crosses four model calls — Intake classify, Resolution remedy, Policy assist, Outreach draft — plus tool reads, retries, and queue wait. A “cheaper model” in one step does not make the case cheap if the assembler shuffles pinned policy every turn and burns prefix cache.

tokens + tools + retrieval + retries + queue → measure per span → stable pin:policy prefix → cache hit

Without measurement and assembly discipline, send window rules cost the same whether they are enforced or ignored — and P99 latency hides in tool fanout, not the model call you optimized.

This chapter adds cost and latency as runtime properties: KV cache vs prefix cache, assembler stability, retry economics, and why batching techniques that help throughput can hurt approval and customer-response latency.

First principles

1. Tokens, tools, retrieval, retries, summarization, parallelism, and caching all affect cost and P99 latency.
2. KV cache makes autoregressive decoding affordable within one generation.
3. Prompt/prefix cache makes repeated stable prefixes affordable across requests.
4. Assembler stability is an economic feature. Shuffle the pinned policy and you burn cache.
5. Largest context is not free — attention and KV memory grow with sequence length.
6. Retries and tool fanout often dominate model $ in agent systems.
7. Batching is not interactive agent work. Throughput techniques can hurt approval and customer-response latency.

KV cache — what is stored per decode step

Transformers compute attention using queries (Q), keys (K), and values (V). For token position (t), the new query must attend to keys/values of positions (1..t). Without caching, every new token would recompute K/V for the entire prefix — (O(n^2)) compute per token in the naive picture.

KV cache stores the K and V tensors already computed for prior tokens so decode step (t+1) only computes K/V for the new token and appends them.

 Decode step t=1          Decode step t=2           Decode step t=3
 ┌─────────────┐          ┌──────────────────┐      ┌────────────────────────┐
 │ tok1: K1 V1 │          │ tok1: K1 V1      │      │ tok1: K1 V1             │
 └─────────────┘          │ tok2: K2 V2 NEW  │      │ tok2: K2 V2             │
                          └──────────────────┘      │ tok3: K3 V3 NEW         │
                                                    └────────────────────────┘
   cache grows ──────────────────────────────────────────────────────────►

Caption: Notice the cache is append-only along the sequence axis for a single generation; length ⇒ memory.

Rough memory (teaching estimate, fp16):

[ \text{bytes} \approx 2 \times L \times H_{kv} \times D \times T \times 2 ]

where (L) = layers, (H_{kv}) = KV heads, (D) = head dim, (T) = sequence length, and the leading 2 is K+V. (Exact layouts vary; multi-query / GQA reduce (H_{kv}).)

from shopops.cost import estimate_kv_memory_bytes

# Illustrative numbers — not a specific production model claim
bytes_ = estimate_kv_memory_bytes(
    num_layers=32, num_kv_heads=8, head_dim=128, seq_len=8_000
)
print(bytes_ / 1e6, "MB (order-of-magnitude)")

Why long generations cost memory: each tool-loop turn that keeps the full transcript in context grows (T). Agents that stuff entire tool JSON histories into the window pay KV memory and attention latency even when the billable API hides it behind a flat token price.

Paging intuition (PagedAttention): serving systems allocate KV in blocks/pages so many concurrent generations can share GPU memory without giant contiguous reservations. You do not implement this in ShopOps — but you feel it when concurrency collapses under long contexts. See Kwon et al., 2023 (vLLM / PagedAttention).

Prompt / prefix cache — reuse across requests

KV cache is usually per generation (per request). Providers also offer prompt caching / prefix caching: if request N+1 begins with the same byte-stable prefix as a recent request, the server can reuse computed prefix state and often charge less for those input tokens.

 Request A                         Request B (same prefix)
 ┌──────────────────────┐          ┌──────────────────────┐
 │ PREFIX (stable)      │          │ PREFIX (stable)      │  ← cache HIT
 │  policy v1.0.0       │          │  policy v1.0.0       │
 │  tool schemas        │          │  tool schemas        │
 ├──────────────────────┤          ├──────────────────────┤
 │ SUFFIX (volatile)    │          │ SUFFIX (volatile)    │  ← computed fresh
 │  case O-1001 hist    │          │  case A-1002 hist    │
 └──────────────────────┘          └──────────────────────┘

 Request C (shuffled policy order)
 ┌──────────────────────┐
 │ PREFIX' ≠ PREFIX     │  ← cache MISS (you re-pay)
 │  tools… policy…      │
 └──────────────────────┘

Caption: Notice hits require identical prefix bytes/tokens — semantic “same rules” is not enough if assembly reorders them.

Assembler stability enables prefix caching

Chapter 3’s assembler is not just about Lost-in-the-Middle. It is the control surface for cache economics:

 GOOD assembly order          BAD assembly order
 ┌─────────────────┐          ┌─────────────────┐
 │ pin:policy      │          │ case history    │  ← pollutes prefix
 │ pin:tools       │          │ pin:policy      │  ← moves every turn
 │ pin:role        │          │ random tool dump│
 ├─────────────────┤          ├─────────────────┤
 │ intake          │          │ pin:tools       │
 │ resolution      │          │ …               │
 │ history         │          └─────────────────┘
 └─────────────────┘          prefix fingerprint changes ⇒ MISS

Rule: never put volatile case data before the pinned prefix. Pin policy constraints at the front; omit or summarize history under budget — you save attention quality and money.

Speculative decoding (brief)

Speculative decoding uses a small draft model to propose several tokens that a large model verifies in parallel — a latency trade-off when verification succeeds often. Emerging serving feature; treat as optional infra, not something your agent loop implements. [VERIFY current vendor support]

Batching vs interactive turns

GPU batching raises throughput for offline scoring. Interactive case workers care about P99 latency for a single lease. Do not blindly batch HITL-path completions with bulk classify jobs; isolate pools (Ch 34 routing + gateway config).

Tool-call fanout costs

One “smart” step that calls five tools serially adds five round trips + five observation injections into context. Parallelize read-only tools when safe; never parallelize sends. Retries without idempotency multiply $ and incident rate (Ch 6, 32).

Cost waterfall (where money goes)

 $/case
  │
  ├─ model input tokens (uncached)
  ├─ model input tokens (cached prefix)   ← cheaper when stable
  ├─ model output tokens
  ├─ tool / channel fees
  ├─ retrieval embeddings
  ├─ retries / redelivery
  └─ human review minutes (often dominant)

Caption: Notice humans and retries can dwarf model $ — still meter the model so you see regressions.

Concrete example — pin policy → lower $/case

Two assembler modes on the same Outreach draft task:

1. Stable: pin:policy + pin:tools fixed string for shop_v1.0.0.
2. Shuffled: policy clauses reshuffled each call (or timestamp injected into system block).

Run N turns; prefix-hit proxy rises only in mode 1. Same quality gates; lower input $ when the vendor discounts cached tokens.

Diagram — hit vs miss + cost meter

flowchart LR
  A[ContextAssembler] -->|stable pin: sections| P[Prefix fingerprint]
  P --> G[Model gateway]
  G -->|hit| C1[Charge cached rate]
  G -->|miss| C2[Charge full input]
  G --> M[CostMeter line]
  M --> T[Trace / case_total]

Implementation

Module: shopops/cost.py.

from shopops.cost import (
    CostMeter,
    TokenUsage,
    experiment_prefix_stability,
    split_prefix_suffix,
)

policy = "SEND_WINDOW 08-21\nMAX_DAY 2\nREQUIRED_FIELDS …"
tools = "draft_email(...)\nsend_email(...)"
history = "case O-1001 … long …"

prefix, suffix = split_prefix_suffix(
    [
        ("pin:policy", policy),
        ("pin:tools", tools),
        ("history", history),
    ]
)
assert prefix.startswith("SEND_WINDOW")

meter = CostMeter()
# First call: miss; second call: hit (proxy)
for i in range(3):
    experiment_prefix_stability(
        meter,
        case_id="O-1001",
        pinned_policy=policy,
        volatile_history=history,
        shuffle_policy=False,
    )
print(meter.case_total("O-1001"))
# prefix_hits should be 2

meter2 = CostMeter()
for i in range(3):
    experiment_prefix_stability(
        meter2,
        case_id="O-1001",
        pinned_policy=policy,
        volatile_history=history,
        shuffle_policy=True,
    )
print(meter2.case_total("O-1001"))
# prefix_hits should be 0 — you broke stability

Wire CostMeter.estimate into the trace emitter (Ch 23) so every model span carries cost_usd, prefix_cache_hit, and prefix_fingerprint.

Failure modes

Production considerations

• Dashboard: $/case, prefix hit rate, p95 latency by TaskKind, retry fraction.
• Alert on hit-rate collapse after prompt deploys — assembler regression.
• Load-test long tool traces; estimate KV pressure on self-hosted gateways.
• Prefer mid models + stable prefixes over giant models + chaos assembly.
• Document vendor cache semantics (TTL, minimum tokens, whether tools break cache). [VERIFY per vendor]

Chapter summary

• Cost and latency are first-class runtime metrics.
• KV cache reuses K/V within a generation; length costs memory.
• Prefix/prompt cache reuses stable prefixes across requests.
• Context assembler stability is what makes prefix caching real.
• Pin policy/tools; keep volatile history in the suffix.
• Shuffling “equivalent” policy text destroys hits.
• Tool fanout and retries often dominate spend.
• Meter $/case with hit/miss fingerprints in traces.

Exercises

1. Break the cache. Inject datetime.utcnow() into the pinned policy section; show prefix_hits fall to zero across repeated calls.
2. KV estimate. Using estimate_kv_memory_bytes, compare 2k vs 32k sequence length; discuss when to summarise tool traces.
3. Waterfall. Instrument a fixture episode; attribute $ to model vs simulated tool fees vs assumed review minutes.
4. Design. Propose an assembler snapshot test that fails CI if pin:* bytes change unexpectedly.

References

• Kwon et al., 2023. Efficient Memory Management for LLM Serving with PagedAttention (vLLM). SOSP.
• Zheng et al., 2024. SGLang: Efficient Execution of Structured Language Model Programs. arXiv:2312.07104 (RadixAttention / prefix sharing — emerging practice).
• Liu et al., 2023. Lost in the Middle. arXiv:2307.03172.
• Vendor docs: prompt/prefix caching (OpenAI, Anthropic, etc.). [VERIFY per vendor]
• Chapters 3, 23, 34