Maya asks whether Outreach wording caused O-1001 ’s resolution. Dashboards show correlation. Causal identification under adaptive agent policies is still open. Calling the chart proof is the failure mode: research gaps dressed as KPIs .
Hard agent problems split into two buckets. Some are engineering-hard : known patterns, incomplete execution. Some are research-open : no method with broad, reliable guarantees. Treating either bucket as the other produces fake roadmaps — and fake results in production.
gap → classify (engineering-hard | research-open | hybrid) → ship engineering subset → label research half honestly
ShopOps already has engineering answers for typed memory, policy-as-code, and send-window property tests. It does not have a general solution for semantic run-diff with causal identification or verified LM-in-the-loop behaviour.
This chapter separates the two buckets so roadmap slides cannot smuggle open problems past production gates.
For each gap, classify honestly:
Engineering hard — solvable with today’s primitives if you invest in systems work; quality varies by execution.Research open — missing theory, metrics, or algorithms with broad, reliable guarantees; expect partial heuristics.
Many items are hybrids: ship the engineering subset; do not pretend the research half is solved.
Problem What to write, supersede, forget under privacy and contradiction — at scale, across tenants. Engineering Typed memory, controllers, provenance, TTL, GDPR-style delete paths (Ch 9–10). Research Principled consolidation policies; formal guarantees against poisoning; evaluation of long-term memory quality.
Problem Attribute behaviour change across prompt/model/memory/tools/policy/data with causal care. Engineering Structured traces; config pins; diff reports across known dimensions (Ch 23–24). Research Automated semantic attribution with reliable causal identification — still open.
Problem Prove an agent cannot take illegal actions under all model samples. Engineering Closed action spaces; policy-as-code; property tests; capability tokens. Research End-to-end formal verification of LM-in-the-loop systems; scalable certified bounds.
Problem Judge policies over weeks of interaction with delayed outcomes and distribution shift. Engineering Simulators, fault injection, multi-metric scorecards, golden trajectories (Ch 20, 22, 41). Research External validity of simulators; credit assignment over long horizons without unsafe online RL.
Problem Know whether an agent action caused resolution or harm. Engineering Label associational analyses; run constrained experiments; avoid auto-policy from charts (Ch 44). Research General causal identification from observational agent logs under confounding and adaptive policies.
Problem Concurrent agents with conflicts, races, and authority. Engineering Ledger, locks, arbitration, separation of duties (Ch 16–18). Research Normative protocols for heterogeneous adaptive agents with provable conflict resolution.
Problem Rules change; models change; evidence standards change. Engineering Versioned policy packs; pins in traces; change control boards. Research Algorithms that adapt policies safely under formal regulatory constraints — largely open.
Problem Injection via tools, retrieval, and multi-agent messages. Engineering Trust boundaries, structured channels, least privilege, canaries (Ch 26–27). Research Robust defenses with guarantees against adaptive prompt/tool attackers.
Problem Know which foreign agent to trust for which task (A2A world). Engineering Capability allowlists, separate credentials, contractual scopes (Ch 27, 47). Research Portable reputation with Sybil resistance and privacy — open.
Problem Improve without policy drift or reward hacking. Engineering Bounded pipelines with eval + human gates + rollback (Ch 45–46). Research Autonomous improvement with safety certificates under shifting objectives.
quadrantChart
title Gaps — where effort goes
x-axis Engineering clearer --> Research heavier
y-axis Lower urgency --> Higher urgency
quadrant-1 Prioritize research partnerships
quadrant-2 Ship systems patterns now
quadrant-3 Watch / defer
quadrant-4 Hard hybrid bets
Caption: Use the catalog rows to place your roadmap items; do not invent a point that “solves” a research open with a blog post.
(If your renderer lacks quadrantChart, treat the caption as the lesson and use the tables above.)
There is no “solver code” for open research. The production move is a gap register :
# shopops/research_register.py — tracking sketch
from __future__ import annotations
from dataclasses import dataclass
from typing import Literal
Kind = Literal["engineering", "research", "hybrid"]
@dataclass
class Gap:
id: str
title: str
kind: Kind
engineering_subset: str
research_open: str
our_stance: str # what ShopOps will / will not claim
GAPS = [
Gap(
id="causal-outcomes",
title="Causal attribution of customer outcomes",
kind="hybrid",
engineering_subset="Associational scorecards + labeled experiments",
research_open="Identification under adaptive agent policies",
our_stance="No causal claims from dashboards alone",
),
# ... extend per program
]
Mistake Result
Fake SOTA table Misleading execs; brittle prod “Model will fix it” No boundary engineering Over-formalize early Paralysis without properties Ignore engineering subset Wait for papers while shipping chaos
Put gap IDs in design docs (Ch 50) so scope is honest.
Fund engineering subsets explicitly; partner for research opens.
When a paper ships, re-classify — do not forever-freeze “research.”
Evaluation checklists (Appendix E) catch overclaiming in PRs.
Separate engineering-hard from research-open; many gaps are hybrids.
Memory, verification, long-horizon eval, causality, coordination, robustness, reputation, and safe adaptation remain partially open.
Ship the engineering subset; do not fake results for the research remainder.
Maintain an explicit gap register in the program.
Honesty about unknowns is part of production quality.
Classify: Take three items from your backlog; label engineering / research / hybrid with one sentence each.Scope: Write a non-goal section forbidding causal claims from ShopOps dashboards.Design: Pick one hybrid gap; define the engineering MVP that does not pretend to solve the research half.Review: Red-team a vendor claim that “fully solves” multi-agent trust.
OWASP Top 10 for LLM Applications — adversarial surface framing. [VERIFY edition]
Pearl / causal inference primers — for the shape of the open problem (Ch 44). [VERIFY]
Agent evaluation surveys — cite specifically when claiming a metric is solved; [VERIFY]
Cross-links: Parts VI–VIII, Ch 44–47, 49–50.
No results are reported in this chapter. It is a map of ignorance with engineering footholds.