Chapter 33 — Multi-Tenant Agent Systems

Northline and Harbor share one ShopOps deployment. O-1001 belongs to Northline. A buggy retrieval query returns Harbor’s order notes into Intake’s context. Prompts cannot isolate customer data — only schemas, filters, and credentials can.

request → tenant_id on every read/write/cache/queue/tool → authz checks principal.tenant == resource.tenant

Without tenant isolation, one store’s Outreach agent reads another store’s PII. A missing filter is an access-control bug, not a model mistake.

This chapter adds defense in depth for multi-tenancy: tenant_id on the write path and read path, per-tenant credentials, scoped memory and quotas, and negative tests that prove cross-store reads fail.

First principles

1. tenant_id belongs on the write path and the read path. A missing filter is an access-control bug.
2. Credentials are per tenant or per tenant+agent. No shared god token for CRM.
3. Memory, config, quotas, metering are tenant-scoped.
4. Encryption and audit boundaries may be stricter than logical rows (some tenants demand separate keys).
5. Noisy neighbor is real. Quotas protect gateway and workers.
6. Negative tests are part of the product. A cross-store read must fail.

Concrete example

Worker leases item {case_id: O-1001, tenant_id: store_northline}. Intake tool get_customer is called with principal agent:intake scoped to store_northline. A buggy assembler tries to pull “similar cases” and the retrieval layer returns B’s chunk because the embedding index omitted tenant filters. Defense in depth: retrieval query requires tenant_id; memory controller refuses writes without it; authz checks principal.tenant_id == resource.tenant_id.

Diagram — isolation layers

 ┌──────────── tenant edge ─────────────┐
 │  authn → tenant context on request   │
 └─────────────────┬────────────────────┘
                   ▼
 ┌──────────── logical isolation ───────┐
 │  DB: tenant_id column + RLS/forced   │
 │  queue payloads carry tenant_id       │
 │  cache keys: tenant|case|…           │
 │  vector filters: tenant_id=…         │
 └─────────────────┬────────────────────┘
                   ▼
 ┌──────────── crypto / audit ──────────┐
 │  per-tenant keys (optional)          │
 │  audit packs not cross-readable      │
 └─────────────────┬────────────────────┘
                   ▼
 ┌──────────── noisy-neighbor ──────────┐
 │  tokens/min, cases/day, worker slots │
 └──────────────────────────────────────┘

Caption: Notice prompts do not appear as an isolation layer — because they are not one.

flowchart LR
  Req[Request] --> Auth[AuthN + tenant ctx]
  Auth --> Q[Queue item + tenant_id]
  Q --> W[Worker]
  W --> DB[(rows WHERE tenant_id=?)]
  W --> Mem[(memory cells)]
  W --> X[Tools with tenant creds]

Implementation — tenant on all rows + negative test

from dataclasses import dataclass
from shopops.authz import Principal, PrincipalKind


@dataclass
class CaseRow:
    case_id: str
    tenant_id: str
    order_id: str


class CaseStore:
    def __init__(self) -> None:
        self._rows: dict[tuple[str, str], CaseRow] = {}

    def put(self, row: CaseRow) -> None:
        self._rows[(row.tenant_id, row.case_id)] = row

    def get(self, principal: Principal, case_id: str) -> CaseRow:
        row = self._rows.get((principal.tenant_id, case_id))
        if row is None:
            # Do not reveal whether it exists in another tenant.
            raise KeyError("case not found")
        return row


store = CaseStore()
store.put(CaseRow("O-1001", "store_northline", "acc-1"))
store.put(CaseRow("B-2002", "store_harbor", "acc-9"))

alice = Principal(PrincipalKind.AGENT, "agent:intake", "store_northline", frozenset())
assert store.get(alice, "O-1001").order_id == "acc-1"
try:
    store.get(alice, "B-2002")
    raise AssertionError("IDOR")
except KeyError:
    pass

Queue items already carry tenant_id in shopops/worker/queue.py. Propagate that field into every tool call and audit event.

Failure modes

Production considerations

• Prefer database row-level security plus application checks.
• Tenant config: policy pack id, region pin (Ch 34), retention, rate limits.
• Escape hatches for support engineering must be break-glass audited.
• Load tests with two tenants asymmetric traffic.
• Legal: data residency may force region-pinned model routing per tenant.

Chapter summary

• Multi-tenant isolation is schema + credentials + quotas.
• tenant_id everywhere: DB, queue, cache, vectors, audit.
• Negative IDOR tests are mandatory.
• Prompts are not an isolation boundary.
• Per-tenant tooling credentials limit blast radius.
• Noisy-neighbor controls protect shared gateways.
• Encryption/audit may be stricter than row filters.
• Two store brands on one SaaS is the design test.

Exercises

1. Mechanical. Implement CaseStore.get negative test; add a deliberate bug that omits tenant filter; watch the test catch it.
2. Cache. Design Redis key layout for case snapshots; show a cross-tenant miss.
3. Design. Propose quota dimensions for ShopOps ($/day, sends/day, model tokens/min).

References

• Multi-tenant SaaS isolation patterns. [VERIFY]
• Chapters 27, 31, 34