Research

The question, made falsifiable

A recurring argument in agent design is that a typed memory store (facts, constraints, episodes with criticality) is better than treating retrieval as memory — ranking everything by similarity and filling the context window. I have made that argument in prose before. This article tests it on a specific system I built, memcell, whose `baseline_v0` policy assigns a retrieval mode per cell and enforces a token budget by priority (constraints > criticality > future-utility > score > confidence).

Concretely: in a regulated case, one cell is a critical constraint — `account under fraud review: no outbound transfers until review closes` (criticality 0.95). The agent's query is `review account 456 transactions for fraud indicators`. By construction, that query shares few words with the constraint but many words with wrong distractor cells like `fraud review transactions … review completed, no constraints apply`. The falsifiable question: under a tight budget and growing distractor pressure, does the constraint stay in the selected context?

• Typed system: the real memcell pipeline — `retrieve_cells` (top-k) then `apply_baseline_v0` (criticality-aware budgeting).
• Flat baseline: rank purely by word overlap, greedily fill the same token budget. No types, no criticality.
• Metric: critical-constraint retention rate. Everything deterministic — no LLM, no randomness.

Result: 0% vs 100% retention

Sweeping the number of wrong-but-similar distractors from 0 to 24 (budget = 120 tokens, k = 10), the flat lexical retriever keeps the critical constraint in 0 of 13 conditions — a retention rate of 0.00. The typed pipeline keeps it in all 13 — a retention rate of 1.00. The gap is not marginal; it is total, because the lexical trap is exactly the case typed memory is designed for: the right cell does not look like the query, and the wrong cells do.

A natural objection is that the typed advantage lives only in the final selection step, so a large enough distractor set should push the constraint out of the top-k during retrieval before the policy ever sees it. I tested that by sweeping k from 5 to 30 at 20 distractors. The constraint is retained at every k, including k = 5. The reason — which I only confirmed by reading the scorer — is that memcell's retrieval score itself folds in `0.5 × criticality`, so the 0.95 constraint never falls out of the top-k. Criticality-awareness lives in retrieval and selection together, not only in the final policy.

• flat top-k lexical: constraint retention 0.00
• typed (memcell): constraint retention 1.00 (robust across distractors 0–24 and k 5–30)
• Mechanism: criticality enters the retrieval score, so the constraint survives the k-cutoff that would sink a purely-lexical ranking.

The finding worth keeping: a discontinuity at criticality 0.7

Measuring a system you trust is most useful when it surprises you. In a second experiment I swept the token budget from 10 to 400 with k large enough to disable the retrieval pre-filter, isolating the policy's budgeting. The 0.95 constraint is the last thing the policy gives up: it survives down to a 20-token budget — essentially its own size — while episodes and facts are squeezed out first. That is the intended behavior, and it holds.

But I also planted a second constraint at criticality 0.69 — just below the policy's hard `criticality ≥ 0.7` cutoff for 'constraint mode'. It is dropped much earlier (gone below a 40-token budget), because under the cutoff it no longer gets constraint priority and competes with ordinary cells on raw criticality. Two near-identical constraints — 0.69 vs 0.71 — receive qualitatively different protection. If criticality is ever estimated by a model rather than hand-set, that knife-edge is a latent safety bug: a small estimation error around 0.7 flips whether a safety constraint is protected at all. The fix is to make the priority continuous near the threshold rather than a hard gate.

• critical constraint (0.95): retained down to a 20-token budget (dropped last)
• near-threshold constraint (0.69): dropped below a 40-token budget
• Implication: replace the hard 0.7 gate with a smooth priority, or the policy is brittle to criticality-estimation error.

Honesty notes and limits

The flat baseline is lexical (Jaccard word overlap), not embedding-based. An embedding retriever would handle paraphrase better, but it still has no notion of criticality or type, so the qualitative gap should persist; quantifying it needs an embedding model, which this study does not run. The corpus is a single hand-built case with hand-set criticality, so these results characterize the policy's behavior, not real-world task accuracy. Plugging an LLM judge and a multi-case suite on top is the obvious next step.

What makes this research rather than a demo is that every number here is produced by running the real policy code, the experiments are deterministic, and the data, plots, and regression tests are committed in the repo. Anyone can reproduce them with two commands. The contribution is small but real: a measured comparison plus a specific, fixable fragility in the policy.

The decoder block as a residual dynamical system

A contemporary LLM block is usually pre-normalized: `x_{l+1} = x_l + Attn(RMSNorm(x_l))`, followed by `x_{l+2} = x_{l+1} + MLP(RMSNorm(x_{l+1}))`. Pre-norm keeps the residual stream close to an identity path, which makes very deep stacks trainable because gradients can flow through the skip path without being repeatedly multiplied by unstable layer Jacobians.

RMSNorm replaces full LayerNorm by normalizing with the root mean square of activations, typically without subtracting the mean. The practical gain is fewer operations and a stable scale control mechanism. The mathematical point is that the residual stream becomes a shared coordinate system: attention writes relational information into it, while the MLP performs high-dimensional feature transformation at each token position.

• Attention mixes information across sequence positions.
• The MLP expands and contracts each token representation, often using SwiGLU-style gated activations.
• The residual stream carries state across layers; normalization controls the magnitude before each write.

Self-attention as content-addressed routing

For a hidden state matrix `X in R^{n x d_model}`, a head forms `Q = XW_q`, `K = XW_k`, and `V = XW_v`, with head dimension `d_h`. Causal attention computes `softmax((QK^T / sqrt(d_h)) + M)V`, where `M_ij = -infinity` when token `i` is not allowed to attend to token `j`. This is a differentiable routing operator: queries score keys, then copy and blend value vectors into the current representation.

Multi-head attention gives multiple routing subspaces. Each head owns separate projections, so the dot product implements a different learned similarity metric. The final output projection recombines those subspaces into the residual stream. Attention heads can specialize into patterns such as local syntax, long-range reference, delimiter tracking, code indentation, or retrieval-like copying, although individual-head interpretations should be treated as empirical observations rather than guaranteed semantics.

• Per-head score matrix: `S = QK^T / sqrt(d_h) + M`.
• Per-head output: `A = softmax(S)V`.
• Layer output: concatenate head outputs, then apply the output projection `W_o`.

Position is injected into the bilinear form

Vanilla attention is permutation-invariant; without position, a model cannot distinguish `dog bites man` from `man bites dog` by order alone. RoPE solves this by rotating query and key pairs using position-dependent rotation matrices before the dot product. The resulting attention score depends on relative offsets while keeping the attention kernel structurally simple.

The limitation is that standard RoPE is static: the transformation is a function of token index, not token content. Recent work such as PaTH Attention replaces static rotation with accumulated data-dependent Householder-like transformations, making positional interaction depend on the sequence itself. That is important for state-tracking tasks where the relevant coordinate frame changes as commands, entities, or memory states evolve.

Why GQA and KV-cache shape matter

At inference time, autoregressive decoding stores keys and values for every previous token. Standard multi-head attention caches one K/V pair per attention head; grouped-query attention shares K/V heads across groups of query heads. If a model has `L` layers, context length `T`, KV heads `h_kv`, head dimension `d_h`, and `b` bytes per scalar, the KV cache is approximately `2 * L * T * h_kv * d_h * b` bytes. Reducing `h_kv` is therefore a direct memory-bandwidth intervention.

The bottleneck during decoding is often not raw FLOPs but memory traffic: each generated token repeatedly reads the KV cache. That is why GQA, multi-query attention, paged attention, quantized caches, FlashAttention-style kernels, and latent attention variants matter. They change the system-level feasibility of long contexts.

The converged 2023-2026 stack

Across many major open-weight model families, the stack has crystallized around pre-norm residual blocks, RMSNorm, RoPE or RoPE variants, SwiGLU-style MLPs, GQA/MQA for inference efficiency, and in some cases MoE layers to decouple total parameters from activated parameters. The field still varies more in long-context strategy, expert routing, cache compression, attention variants, and whether recurrence or external memory is introduced.

The engineering lesson is that architecture must be read with hardware. A mathematically elegant attention variant is incomplete until one asks how it trains in parallel, how it affects KV-cache layout, whether kernels exist, and whether it improves perplexity, reasoning, retrieval, and length extrapolation under fixed compute.

Length generalization is a positional problem

A model trained mostly on short sequences can fail when evaluated at longer lengths even if the architecture accepts the tokens. Positional encodings determine whether attention scores remain meaningful outside the training range. RoPE extrapolation, NTK-aware scaling, YaRN-style interpolation, ALiBi, NoPE variants, and data-dependent encodings all attack this problem differently.

The mathematical issue is that position changes the geometry of query-key similarity. In RoPE, query/key channels are rotated at different frequencies; at long lengths, frequency choices and training distribution affect whether relative-position signals remain usable. If rotations alias or if the model never learned long-range dependencies during training, simply increasing `max_position_embeddings` does not create robust long-context reasoning.

Attention cost and cache cost diverge

During prompt prefill, full attention over `n` tokens has quadratic interaction unless optimized by kernels or sparse/block strategies. During decode, the new token attends over cached K/V states, so the main cost becomes reading a cache that grows linearly with context length and layer count.

This distinction matters because a method that improves prefill may not improve decode. Sliding-window attention, block-sparse attention, chunked prefill, paged KV caches, GQA, and latent KV compression should be evaluated against the phase they are meant to optimize.

Forgetting is a feature

Long contexts become noisy. If every old token remains equally eligible for attention, the model can retrieve distractors, stale instructions, or irrelevant examples. Forgetting mechanisms deliberately down-weight old or low-utility information. This is closer to how working memory should behave in an agentic system: retain state, not every observation.

PaTH-FoX-style combinations point to a useful direction: make positional interaction content-aware and allow the network to modulate what should decay. The question becomes not 'how many tokens fit?' but 'which facts remain causally active?'

State streams and recurrence

Stateful transformer proposals add persistent latent state alongside token context. Instead of rebuilding all internal computation from text at each step, the model maintains recurrent vectors across layers or time. A state stream may carry compressed reasoning traces, task state, or latent summaries without appending everything to the visible token sequence.

This is research-stage, not settled practice. Recurrence introduces stability, credit assignment, and parallelization problems, and reported gains can be difficult to separate from training procedure, benchmark choice, or additional inference-time compute. Any stateful mechanism must show that it improves reasoning or long-horizon coherence without becoming an opaque cache that drifts, amplifies errors, or resists inspection.

Retrieval is still necessary

Even excellent long-context models should not ingest entire enterprise corpora on every request. Retrieval supplies authority, access control, recency, and cost discipline. The strongest designs combine long-context capacity with retrieval-aware context assembly: enough window to include rich evidence, enough retrieval discipline to avoid irrelevant memory.

The context window is not a memory system

The analogy people reach for is RAM. A larger context window, on this view, means a model can 'remember' more. But RAM is random-access: any address is read in constant time with the same fidelity. A context window is not. Attention is quadratic in sequence length at prefill, and the probability that any particular past token is attended to decays as the sequence grows. The model is not reading all of its 'memory'; it is running a soft query across a buffer it accesses unequally.

This matters because 'just use a longer context' is frequently offered as the engineering answer to memory problems in agents. The answer is incomplete. If you insert a user preference at position 0 of a million-token context and the model generates from position 999,999, the probability that the preference is attended to with high weight is not guaranteed — and in practice, attention to distant tokens is unreliable across many tasks and model families.

Attention dilution at scale

Attention over long sequences dilutes. Each token's attention score must sum to 1 across all keys. As context grows, the softmax distribution becomes flatter. Relevant tokens at distant positions compete against a much larger set of keys. High-entropy contexts — mixed topics, interleaved documents, long histories — make this worse. The signal from the first 10,000 tokens can be overwhelmed by noise in the next 990,000.

Experiments on needle-in-a-haystack tasks reveal the failure mode clearly: insert a single fact at varying positions in a long document, then ask a question that requires that fact. Retrieval accuracy is not uniform across position. Most models show position bias — better at start and end, weaker at middle positions — which is the exact opposite of a uniform-access memory.

• Softmax attention normalizes across all keys; long contexts spread probability mass.
• Needle-in-a-haystack: accuracy is position-dependent, not uniform.
• Distractor documents between needle and question further degrade retrieval.

The stale instruction problem

Agent systems face a particular pathology: instructions injected early in the context become less reliable as the context grows. A system prompt stating 'never reveal account numbers' is more likely to be followed if the context is short than if 10,000 tokens of tool calls, retrieved documents, and conversation history have accumulated since.

This is the stale instruction problem. It is not a model alignment failure; it is an attention-geometry failure. The instruction is still in context. But attending to it reliably over a long history requires that the model learn positionally robust instruction following — which is harder to train and less reliable in practice than keeping instructions close to the current position.

Retrieval vs memory: different design problems

Retrieval and memory are often conflated. Retrieval is a query-time operation: given a current query, fetch relevant passages from an external store. Memory is a persistence operation: represent important information in a form that can be accessed accurately later, updated incrementally, and integrated with new observations.

These require different engineering. Retrieval needs a vector store, a good embedding model, a query rewriter, and a reranker. Memory needs a schema, a write policy, a read policy, a conflict resolution strategy, and a forgetting mechanism. A context window with no memory architecture is just short-term retrieval on a growing buffer.

• Retrieval: fetch relevant content at query time from an external store.
• Memory: persist, update, and selectively access durable facts across sessions.
• Context window: a temporary buffer; it conflates both and does neither well at scale.

Typed memory objects as a better architecture

A more inspectable architecture uses typed memory objects alongside the context. Instead of appending every observation to the transcript, the agent writes structured updates into a memory store: {entity, fact, confidence, source, expires}. The context includes a compressed representation of active memory objects, not the raw history.

This gives the system the properties that context alone cannot. Memory objects can be updated without re-reading the whole transcript. They can be filtered by relevance, entity, confidence, or recency. They can be audited, versioned, and explained. The model attends to a small, curated set of facts rather than a large, unfiltered buffer.

Open questions

How should write policies be trained? If the agent decides what to store in memory, it must be trained to recognize what is important — a harder task than retrieval.

How do we evaluate long-horizon memory coherence? Needle-in-a-haystack tests one fact; realistic memory needs multi-hop consistency over many turns. When is in-context retrieval sufficient and when does it fail? The threshold likely depends on context length, topic density, and the model's positional robustness.

Why final answers are insufficient

A language model that answers questions gets evaluated on answers. An agent system is different: the agent observes state, decides an action, executes a tool, receives an observation, updates its state, and decides the next action. The final output is not produced by a single forward pass; it is the terminal state of a sequence of decisions.

Evaluating only the final answer masks many failure modes. An agent can produce a correct answer by the wrong reasoning — it might have skipped a tool call that was necessary for safety, used stale data because retrieval failed, or violated a policy in an intermediate step while still arriving at a numerically correct result. Conversely, an agent can fail because a tool returned an error, not because the model reasoned incorrectly. Final answer accuracy conflates these cases.

The anatomy of a trajectory

A trajectory is the ordered sequence of events in an agent session: initial observation, memory state at start, each action the agent chose, each tool result, each memory write, each state update, and the final output. A full trajectory log should be replayable: given the same initial state and a fixed agent version, the same trajectory should be reproducible.

Each element of the trajectory is evaluable. Was the initial intent correctly parsed? Did the first tool call use the right arguments? Did the observation change the plan appropriately? Was a memory write triggered by the right condition? Was an error handled by retrying, rephrasing, or escalating correctly? These are distinct failure points.

• Observation: what the agent sees — user message, document, tool result, memory state.
• Action: what the agent decides — tool call, response, memory write, escalation.
• Tool result: what the environment returns — success, error, data, side effect.
• State transition: how memory or context changed after the action.

Tool calls as first-class evaluation units

Tool calls are often the most consequential part of an agent trajectory. In a regulated system, a tool call may trigger an account update, schedule a communication, pull a financial record, or log an event. Evaluating whether the correct tool was called, with correct arguments, at the right step, under the right permissions — this is a separate evaluation task from evaluating the final answer.

A tool call evaluation should check: argument validity (syntactically and semantically), permission correctness, idempotency where required, appropriate guard conditions, and correct behavior when the tool returns an unexpected result. An agent that calls the right tool with the wrong argument is a different failure from one that calls the wrong tool entirely.

Intermediate state matters

An agent's intermediate state — what it has stored in memory, what it has decided provisionally, what context it has accumulated — affects all future decisions. An error in intermediate state propagates. An agent that stores the wrong entity in working memory will use that entity in later tool calls, even if the final answer appears correct.

This is why trajectory evaluation must inspect state at each step, not only the final output. A useful evaluation framework records a state snapshot after each action and compares it against a reference trajectory's state at the same step. Divergence in state is often the root cause of failure, visible before the final answer reveals it.

Recovery from errors is an evaluable capability

A useful agent must handle tool errors, unexpected results, ambiguous observations, and conflicting evidence. An agent that halts or loops on a tool error is less useful than one that retries with rephrased arguments, falls back to an alternative tool, or escalates to a human.

Trajectory evaluation should test recovery explicitly: inject tool errors at known positions, observe whether the agent retries correctly, and measure whether recovery adds unnecessary steps, introduces bad state, or correctly reaches the task goal. Recovery quality is invisible in final answer accuracy.

Trajectory-level scoring methods

Step-level correctness: label each trajectory step as correct, incorrect, or neutral, and aggregate. This is interpretable but expensive to label.

Reference trajectory matching: compare the agent's trajectory against a reference that solves the same task, using edit distance or action alignment. This requires reference trajectories and must allow for valid alternative paths.

Property-based evaluation: define invariant properties that any correct trajectory must have — 'permission check must precede tool call', 'error must be logged before retry', 'refusal must occur when user lacks access' — and test each trajectory for violations. Property-based tests generalize better than reference matching because they specify what must be true without specifying the exact path.

Why build it?

Reading about transformers and building one are different cognitive tasks. Reading gives you the forward pass as a sequence of matrix operations. Building gives you the failure modes: why your loss spikes after a gradient step, why your attention heads all collapse to the same pattern in the first epoch, why your model generates repetitive text despite low loss, and what happens when you get RMSNorm and pre-norm wrong.

The goal is not to train a competitive model. It is to train a model small enough to fully inspect — attention weights, residual stream norms, loss curves, generated samples — and large enough to learn something non-trivial on a small corpus. A 1–10M parameter model trained on a few hundred MB of text is a good target. Code lives in mini-transformer-lab.

Tokenization

A tokenizer converts strings to integer token IDs. The simplest approach is character-level: every character maps to an integer, vocabulary size is small, sequences are long. A better approach is byte pair encoding (BPE): start with byte-level tokens, greedily merge frequent pairs into subword units until reaching a target vocabulary size.

The important lesson: the model never sees text. It sees integers. The embedding matrix maps integers to vectors. Every evaluation and training interaction is mediated by tokenizer decisions: what byte sequences get merged, how rare words are split, whether numbers are tokenized consistently. Tokenizer bugs are input bugs, and they affect everything downstream.

• Character-level: simple, vocabulary size ≈ 100, sequences are long.
• BPE: subword tokens, vocabulary 4K–50K, better for rare words and numbers.
• Changing the tokenizer requires retraining from scratch.

Embeddings and the residual stream

Each token ID maps to a `d_model`-dimensional embedding vector via a learned lookup table. Positional information is added through separate positional embeddings or by rotating query/key vectors in attention (RoPE).

After embedding, every transformer layer reads from and writes to the same `(sequence_length, d_model)` matrix — the residual stream. Attention adds relational information; the MLP adds position-wise feature transformations. The residual connection means each layer refines the stream rather than replacing it, which is why very deep transformers are trainable.

Causal self-attention

Implement multi-head causal attention: project `X` into `Q`, `K`, `V` with learned matrices, split into heads, compute `softmax((QK^T / sqrt(d_h)) + mask) * V`, project back. The mask is an upper-triangular matrix of `-inf` values preventing token `i` from attending to any token `j > i`.

The key thing to get right is the mask. An off-by-one error in masking produces information leakage: the model sees the next token during training, inflating training accuracy and producing a model that cannot actually generate. Test your masked attention by checking that attention scores are exactly zero for all future positions before training anything.

• Head dimension: `d_h = d_model / n_heads`.
• Causal mask: `mask[i, j] = -inf if j > i else 0`.
• KV cache: at inference, store K and V for each past token rather than recomputing.

RoPE: rotating queries and keys

Vanilla attention is position-unaware: `dog bites man` and `man bites dog` produce the same attention scores if embeddings are position-agnostic. RoPE solves this by rotating query and key vectors using position-dependent rotation matrices before the dot product.

The rotation makes attention scores depend on relative position: token 5 attending to token 3 uses the same rotation as token 100 attending to token 98. Implementing RoPE requires computing sine and cosine frequencies for each head dimension, applying them before the attention computation, and verifying that the resulting attention weights are sensitive to position reordering.

MLP with SwiGLU and the training loop

The MLP expands the representation into a higher-dimensional space, applies a nonlinearity, and projects back. SwiGLU uses two parallel projections where one gates the other: the expansion ratio is typically 4x. The MLP is where factual associations are largely stored; attention handles which tokens to mix, the MLP transforms the resulting representation.

Training loop: tokenize text into fixed-length chunks, use each as both input (tokens 0..n-1) and target (tokens 1..n), compute cross-entropy loss, backpropagate, step with AdamW and a cosine learning rate schedule with warmup. Loss curves tell you a great deal: fast initial decrease (learning token frequencies), then slower descent (syntax), then very slow descent (semantics). Loss that drops perfectly to near-zero on small data means overfitting; the model has memorized, not generalized.

The question

When a language model begins training, what does it learn first? The question has practical implications: it affects how we interpret loss curves, design data curricula, understand overfitting, and evaluate how much a model has actually learned versus memorized.

The short answer from small-model experiments: first token frequencies, then local syntax, then longer-range dependencies, and finally semantic associations. This is not a clean sequential partition — learning overlaps — but the rough ordering is consistent across model sizes and training corpora.

Loss curves as training signal

A training loss curve is not a scalar measurement of how good the model is. It is a time series of the average negative log-likelihood over the training distribution. Different parts of the loss correspond to different aspects of language: predicting the next character in a common word, predicting punctuation, predicting a rare technical term, predicting the verb that a long subject phrase demands.

The loss curve can plateau and then resume declining. A plateau often marks a phase transition: the model has exhausted one class of pattern and is beginning to learn a harder class. For small models on small corpora, these transitions are visible in the curve. For large models on large corpora, they are smoother but still present.

Syntax before semantics

Empirically, small transformer models show improved perplexity on syntactic patterns — article-noun agreement, subject-verb agreement, auxiliary ordering — before showing reliable improvement on semantic associations: correct pronoun resolution, entity tracking, factual recall.

This is consistent with the distributional view of language: syntactic patterns are dense and consistent across the training corpus, so gradient descent reduces loss faster on syntactic signals than on sparse semantic associations. Semantics requires more data or more parameters to encode reliably.

Token frequency effects

The model learns frequent tokens first. Common function words, punctuation, and high-frequency content words converge quickly. Rare tokens — technical jargon, proper nouns, domain-specific terms — converge slowly or not at all on small corpora.

This has implications for domain-specific applications: a model trained on general text will have high uncertainty on rare domain terms. Targeted fine-tuning, domain-specific tokenizer design, and frequency-adjusted data sampling are all partial mitigations.

Memorization vs generalization

A model memorizes training examples when it assigns very high probability to specific training sequences. It generalizes when it assigns high probability to novel sequences with similar structure. These are measurable differently: memorization is detected by repeating training examples verbatim; generalization is detected by evaluating on held-out distributions.

Small models memorize earlier than large models. Models trained for many epochs memorize more. The practical implication: a model with very low training loss but high validation loss has mostly memorized, not generalized. Validation loss is a better indicator of what the model has actually learned.

Grokking: delayed generalization

A striking phenomenon in small models trained on structured tasks: the model first memorizes the training set (training loss near zero, validation loss high), then — after many additional gradient steps — suddenly generalizes. Loss on the validation set drops sharply, often thousands of steps after training loss plateaued. This is called grokking.

Grokking suggests that the generalization solution exists in weight space but takes more gradient steps to reach than the memorization solution. It implies that early stopping based on training loss can miss a model that would eventually generalize. Monitoring validation loss over long training runs is essential.

Why a single label is insufficient

The word hallucination covers too much. A model that invents a citation, a model that repeats a document's claim incorrectly, a model that fails to retrieve the right evidence, and a model that reasons correctly from wrong premises are all called hallucinating — but they are different failure modes requiring different fixes.

A taxonomy of failures is more useful than a single label. It tells you where in the system the failure originated, what evidence you need to diagnose it, and what intervention addresses the root cause rather than the symptom. The following covers the most common failure modes in deployed LLM systems.

Retrieval failure

The model would have answered correctly if it had seen the right document. Retrieval failed because the query was not well-formed, the embedding did not surface the relevant chunk, the chunk was filtered by metadata, or the document was not in the corpus.

Diagnostic signal: the correct answer exists in the corpus; if you inject the correct document manually, the model answers correctly. Mitigation: query rewriting, hybrid dense+sparse retrieval, metadata-aware filtering, reranking, corpus auditing.

Instruction conflict

The system prompt says one thing; a retrieved document says another; a user message says a third. The model must arbitrate between conflicting instructions but receives no explicit guidance on priority.

Diagnostic signal: model behavior is inconsistent across similar requests; injecting a higher-authority instruction resolves the conflict; removing conflicting documents changes the answer. Mitigation: explicit instruction hierarchy in the system prompt; middleware that detects and flags instruction conflicts before the model call.

Tool misuse and planning collapse

Tool misuse: the model calls the right tool with wrong arguments — wrong entity ID, wrong date range, wrong field name. Or it calls the wrong tool. Or it fails to call a required tool entirely. Mitigation: explicit tool schemas with argument validation, constrained decoding for structured tool outputs, fine-tuning on tool-use demonstrations.

Planning collapse: multi-step tasks require tracking what has been done and deciding what to do next. Planning collapse occurs when the model loses track of task state mid-sequence — repeating a completed step, skipping a required step, or generating an action inconsistent with current state. Mitigation: typed state stores, explicit task decomposition in the system prompt, step-level validation middleware.

• Wrong arguments: correct tool, wrong entity, date, or field.
• Wrong tool: retrieved information when a calculation was needed, or vice versa.
• Missing tool call: answered from parametric memory when retrieval was required.

Hidden assumption failure and unsupported synthesis

Hidden assumption failure: the model produces an answer that is correct given an unstated assumption — but the assumption is wrong or unstated in the context. Example: the model assumes the user's account is in a particular currency and answers accordingly, without stating or checking. Mitigation: prompt the model to state assumptions before answering; use structured output contracts that require source citations for every material claim.

Unsupported synthesis: the model synthesizes a conclusion that sounds plausible but is not directly supported by any single retrieved document. It combines fragments from multiple documents in a way that produces a claim none of them actually make. Mitigation: require citation at the claim level, not only the answer level; use a verifier pass that flags synthesized claims.

Overconfident refusal and long-context drift

Overconfident refusal: the model refuses a legitimate request, classifying it as policy-violating when it is not. In regulated environments, unnecessary refusals can also constitute a compliance failure. Mitigation: calibrate refusal classifiers on your actual policy distribution, not generic safety categories; test refusal on a balanced set of legitimate and illegitimate requests.

Long-context drift: over a long context, the model's behavior shifts. Instructions stated early are followed less reliably. Entity resolutions drift. Contradictory facts accumulate. Mitigation: typed state stores for durable facts; context compression and summarization; explicit instruction re-injection at intervals; long-context-specific evaluation sets.