Chapter 17 — Coordination and Conflict

O-1001 is at the Policy stage. Resolution asserts permit_send=true — Jordan needs a status update. Policy asserts permit_send=false — the send window closed at 21:00. Both writes land milliseconds apart. Last-writer-wins picks a winner by timing, not by authority.

Resolution ASSERT permit_send=true → Policy ASSERT permit_send=false → CONFLICT → Outreach blocked until resolved

Without recording the disagreement, Outreach may send on stale belief or a race winner. Maya sees “sent” in one dashboard and “denied” in another.

This chapter adds conflict as data: the ledger keeps both assertions, marks the key disputed, and applies an authority ladder — human reviewer > Policy > Resolution — before Outreach may draft or send.

First principles

Four hazards:

• Race — two writers, one key, undefined reader.
• Duplicate action — two Outreach sends (idempotency, Ch 6).
• Lost update — reader acts on stale belief.
• False consensus — “no one objected in chat.”

Practical controls—no consensus protocol required:

1. Partition by case_id — one writer lane per case (Ch 32).
2. Optimistic concurrency — include ledger seq in decisions; reject stale acts.
3. Locks / leases — short lease for send critical section.
4. Arbitration — RESOLUTION entries with basis and authority rank.

ShopOps authority: human reviewer > Policy > Resolution proposals. Encode the ladder in code; record the basis on every resolution. Eventual consistency is fine for analytics—not for the fact that permits a send. Outreach acts only on an undisputed policy-approved value or a recorded resolution.

Concrete example

Timeline:

t1 resolution ASSERT permit_send=true
t2 policy ASSERT permit_send=false  → CONFLICT
t3 outreach attempts send → denied (disputed key or closed send window)
t4 human RESOLUTION permit_send=false basis=human_reviewer
t5 outreach may draft; send still needs approval + open send window

Resolution’s “send now” never outranks Policy’s send-window clock.

Diagram

sequenceDiagram
  participant S as Resolution
  participant C as Policy
  participant L as Ledger
  participant H as Human
  S->>L: ASSERT permit_send=true
  C->>L: ASSERT permit_send=false
  L-->>L: CONFLICT
  H->>L: RESOLUTION permit_send=false
  Note over L: disputed cleared

Resolution ladder:

human_reviewer
    ↑
policy_agent / policy_engine
    ↑
resolution_agent (proposals only)

Implementation

from shopops.ledger import RESOLUTION_LADDER, EntryType, Ledger

ledger = Ledger("O-1001")
ledger.assert_fact("resolution", "permit_send", True)
ledger.assert_fact("policy", "permit_send", False)
assert ledger.belief("permit_send")["disputed"] is True
assert ledger.conflicts

# Human path
ledger.resolve(
    "human_reviewer",
    "permit_send",
    False,
    basis="human_reviewer",
)
assert ledger.belief("permit_send")["disputed"] is False
assert ledger.belief("permit_send")["value"] is False
assert "human_reviewer" in RESOLUTION_LADDER

Detect conflicts on ASSERT mismatch; never delete history — append CONFLICT and RESOLUTION. Readers that ignore disputed are bugs.

For duplicate sends: coordination ≠ idempotency. Even after conflict clears, channel adapters need idempotency keys (Ch 6). The duplicate WhatsApp case returns in Ch 25.

Failure modes

1. Last writer wins on regulated keys.
2. Resolve by model vote — two LLMs “agree” to break send window.
3. Silent overwrite in a mutable DB row without ledger.
4. Long locks — lease held across human overnight approval; deadlocks. Prefer park FSM state (Ch 7–8).
5. Conflict without owner — disputed forever, cases stuck; SLO on time-to-resolution.

Production considerations

• Single-thread ticks per case_id removes most races cheaply; still keep CONFLICT detection for multi-writer mistakes.
• Emit metrics: ledger_conflicts_total, disputed_age_seconds.
• HITL queues consume CONFLICT entries (Ch 29).
• Document authority: who may resolve which keys.
• Kleppmann’s concurrency chapters are the right mental model; agents do not exempt you.

Optimistic concurrency for Outreach

Before send, Outreach records ledger_seq_seen = len(entries). At execute time, if len(entries) != ledger_seq_seen and any new entry touches permit_send, required_fields, or opt-out keys, abort and re-read. This is ordinary OCC — boring, testable, sufficient for many case-partitioned systems.

If you skip OCC and also skip leases, you rely on luck. Luck is not an SLO. Pair OCC with idempotency keys so a retried send after abort cannot double-deliver when the gateway already accepted the first attempt.

What must never be last-writer-wins

Treat these keys as regulated (illustrative ShopOps list — jurisdiction-specific rules need counsel):

• permit_send / send-window outcome
• required_fields complete
• opt_out
• approved refund-or-reship thresholds

Resolution may propose; it may not silently overwrite Policy. If your store is a mutable row without a ledger, you have already chosen last-writer-wins — change the store, not the prompt.

Chapter summary

• Races are normal once you split writers.
• Conflicts are append-only data.
• Last-writer-wins is unacceptable for send authority.
• Resolution needs basis + ranked authority.
• Case partitioning prevents many races.
• Idempotency still required for side effects.
• Disputed keys block consequential acts.
• Measure conflict rate and age.

Exercises

1. Two writers assert different preferred_channel values; assert CONFLICT; resolve with basis.
2. Implement act_if_not_disputed(key) helper used by Outreach.
3. Add a lease table (case_id, owner, expires_at) and write a test for expiry.
4. Show that flipping ASSERT order still ends in CONFLICT (commutative detection).

References

• Kleppmann, 2017. Designing Data-Intensive Applications — concurrency, logs.
• Lamport, 1978. Time, clocks, ordering.
• Distributed systems primers on optimistic concurrency / leases. [VERIFY SOURCE for any specific text you assign]