Research

My research interest is in the machinery of modern AI systems: how models learn representations, how they reason over context, how they use memory and tools, and how their behavior can be measured.

Books

Longer writing on agents and related work.

  • Notes on Agents

    Practical notes on building production agent systems — ShopOps from control loop to policy, ledger, eval, and durable execution.

  • The Audit Trail

    Accountability in the Age of AI — keeping agent and model systems inspectable, attributable, and governable.

  • The White Month

    A Kindle ebook by Aditi Chatterji.

All books

Lab notebooks

Code and experiments that go with the research.

  • mini-transformer-lab

    Train a tiny decoder-only transformer from scratch — tokenization, attention, RoPE, KV cache, and long-context failure modes.

  • llm-evals-from-scratch

    Evaluation harnesses for LLM outputs, retrieval quality, long-context behavior, and agent trajectories.

  • stateful-agent-lab

    Small reproducible agents with typed memory, tool calls, planning logs, and trajectory-level scoring.

  • FlowIndex

    Behavior-first repository indexing for AI coding agents — entrypoints, call paths, tests, git history, impact analysis, and MCP-ready context packs.

All repos on GitHub

Articles

Essays and notes. Open any article to read.

  • Typed Memory Beats Flat Retrieval Under Distractor Pressure — Measured

    A small, fully reproducible study on a typed-memory policy I built (memcell). I measure whether a criticality-aware selection policy keeps a critical safety constraint in context when wrong-but-lexically-similar distractors compete for a tight token budget — and compare it against a flat top-k lexical retriever. Flat retrieval keeps the constraint 0% of the time; the typed pipeline keeps it 100%, across 0–24 distractors and k from 5 to 30. A second experiment exposes a discontinuity at the policy's criticality≥0.7 cutoff. All numbers come from running the real code; the data and plots are committed.

    12 min read

  • Transformer Architecture: From Attention Logits to Inference Kernels

    A technical reconstruction of the modern decoder-only transformer stack: token embeddings, pre-normalized residual blocks, causal self-attention, RoPE, grouped-query attention, feed-forward expansion, KV-cache economics, and why many documented 2023-2026 open-weight LLMs converged on a similar set of design choices.

    18 min read

  • Long-Context and Stateful Transformers

    An article on why long-context performance is not solved by increasing the token window: positional extrapolation, cache memory, attention sparsity, recurrence, forgetting, state streams, and retrieval-aware context design.

    15 min read

  • A Million Tokens Is Not Memory

    Why increasing context length does not solve the memory problem for language models and agents: attention dilution, stale instructions, the retrieval-vs-memory distinction, and typed memory objects as a better architecture.

    15 min read

  • The Unit of Evaluation for Agents Is the Trajectory

    Why final answer accuracy is an insufficient metric for agent systems, and what trajectory-level evaluation looks like: tool calls, intermediate state, observations, memory writes, error recovery, and scoring methods.

    14 min read

  • Building a Tiny Transformer from Scratch

    A first-principles walkthrough of building a small decoder-only transformer: tokenizer, embeddings, causal self-attention, RoPE, SwiGLU MLP, residual blocks, training loop, loss curves, KV cache, and what each piece teaches you about modern LLMs.

    22 min read

  • What Does a Language Model Learn First?

    A look at training dynamics in small language models: how loss curves reflect stages of learning, the syntax-before-semantics pattern, token frequency effects, and the boundary between memorization and generalization.

    13 min read

  • A Field Guide to LLM Failure Modes

    A taxonomy of distinct failure modes in language model systems: retrieval failure, instruction conflict, tool misuse, planning collapse, hidden assumption failure, unsupported synthesis, overconfident refusal, and long-context drift — with diagnostics and mitigations.

    16 min read