46. Self-Improving Systems Without Magical Thinking

Nightly prompt mutation is not self-improvement. It is unsupervised drift with a cron job.

ShopOps learns this on Outreach copy for O-1001. Thumbs-up on “friendlier” drafts trims required fields, softens send-window language, and adds urgency. Judge scores rise for a week. Then Policy starts clearing sends that should have waited for Maya. Rollback fails because prompt_version was never pinned.

measure → candidate → eval (send window + required fields + no-send-without-approval) → human approve → stage → prod pin OR rollback

The failure mode is ungoverned auto-apply: you measured something, changed customer-visible behaviour, and skipped the gates.

This chapter adds the bounded improvement loop — every arrow is a gate; drop one and you have a self-modifying blob, not improvement.

First principles

A self-improving system is a bounded loop — not a model that rewrites itself because metrics moved:

[ \text{measure} \rightarrow \text{candidate} \rightarrow \text{eval} \rightarrow \text{approve} \rightarrow \text{stage} \rightarrow \text{rollback} ]

Every arrow is a gate. Drop one and you do not have improvement; you have a self-modifying blob.

What may auto-propose vs auto-apply:

Author recommendation: humans approve anything that changes customer-visible behaviour or policy surface. Machines may auto-apply only within pre-authorized envelopes (e.g. cache warmups, non-semantic router shadow).

Concrete example

Nightly ShopOps improvement job:

1. Measure: scorecards + fixture diffs vs last week.
2. Candidate: new Outreach prompt snippet reducing average tokens 8% on golden set.
3. Eval: ring-1 properties — send window, required-field canaries, no-send-without-approval.
4. Approve: human reviews diff + eval report.
5. Stage: 5% shadow traffic; compare run-diff distributions.
6. Rollback: one-click pin to previous prompt_version.

If step 3 fails, candidates die in the queue. No “best effort” prod push.

Diagram

stateDiagram-v2
  [*] --> Measure
  Measure --> Candidate: metrics / gaps
  Candidate --> Eval: package artifact
  Eval --> Rejected: fixtures red
  Eval --> Approve: fixtures green
  Approve --> Stage: human ACK
  Stage --> Prod: SLOs hold
  Stage --> Rollback: SLOs break
  Prod --> Measure
  Rejected --> Measure
  Rollback --> Measure

Caption: Improvement is a state machine with reject and rollback edges — not a ratchet that only goes “forward.”

Implementation

# shopops/improvement/pipeline.py — teaching sketch
from __future__ import annotations

from dataclasses import dataclass
from enum import Enum
from typing import Callable


class Phase(str, Enum):
    MEASURE = "measure"
    CANDIDATE = "candidate"
    EVAL = "eval"
    APPROVE = "approve"
    STAGE = "stage"
    PROD = "prod"
    ROLLBACK = "rollback"
    REJECTED = "rejected"


@dataclass
class ImprovementCandidate:
    id: str
    kind: str  # prompt | policy | router
    diff_ref: str
    created_by: str  # user or job name


@dataclass
class PipelineResult:
    phase: Phase
    ok: bool
    detail: str


def run_bounded_pipeline(
    cand: ImprovementCandidate,
    *,
    eval_fn: Callable[[ImprovementCandidate], tuple[bool, str]],
    human_approved: bool,
    stage_slos_ok: Callable[[ImprovementCandidate], tuple[bool, str]],
    apply_prod: Callable[[ImprovementCandidate], None],
    rollback: Callable[[ImprovementCandidate], None],
) -> PipelineResult:
    ok, detail = eval_fn(cand)
    if not ok:
        return PipelineResult(Phase.REJECTED, False, detail)

    if not human_approved:
        return PipelineResult(Phase.APPROVE, False, "waiting_for_human")

    sog_ok, sog_detail = stage_slos_ok(cand)
    if not sog_ok:
        rollback(cand)
        return PipelineResult(Phase.ROLLBACK, False, sog_detail)

    apply_prod(cand)
    return PipelineResult(Phase.PROD, True, "pinned")

The important line is not apply_prod — it is that human_approved and eval_fn can hard-stop the machine.

Failure modes

Production considerations

• Store candidates as immutable blobs (s3://.../candidates/{id}) with provenance.
• Page humans on approval backlog aging — stale candidates are a risk.
• Align with MLOps staged rollout patterns: canary, bake, promote. Label as emerging practice when mapping to your platform.
• Ch 48 lists where automation still fails open — do not paper over research gaps with cron jobs.

Chapter summary

• Self-improvement is a gated pipeline: measure → candidate → eval → approve → stage → rollback.
• Auto-propose is not auto-apply; policy-critical artifacts need humans.
• Rollback and version pins are part of the product.
• Multi-metric evaluation beats single reward optimization.
• Magical overnight agents are usually ungoverned prompt mutation.

Exercises

1. Mechanical: Implement eval_fn that fails if a candidate prompt removes the substring REQUIRED_FIELDS_V1.
2. Design: Define SLOs for staging Outreach changes (violation rate, approval lag, escalation rate).
3. Process: Write the on-call runbook for “rollback prompt_version.”
4. Hostile: A candidate improves judge scores but fails send-window fixtures — assert pipeline rejects.

References

• MLOps staged rollout / canary patterns — emerging practice; map to your deploy system. [VERIFY]
• SR 11-7 — model change control intuition.
• Cross-links: Ch 20–21, 41, 45, 48.