Back to Blog
Galen Guan

Teaching the Harness to Improve Itself: Lilian Weng's Roadmap, Checked Against Our Practice

Same model, different harness, nearly triple the score: in a comparison OpenAI published in August 2026, harness optimization lifted Codex from 13.3% to 38.3% on ARC-AGI-3. By 2026 nobody needs convincing that harnesses matter; the big labs are open-sourcing them as platform plays (see The Open-Source Agent Harness Wars). The question that remains under-examined is the next one: who improves the harness itself, how, and can it improve itself?

On July 4, 2026, Lilian Weng published "Harness Engineering for Self-Improvement" on her blog, placing that question back in the lineage of recursive self-improvement (RSI). The concept dates to 1965, yet her near-term reading of it is disarmingly practical: the first stretch of the RSI road most likely runs through the system wrapped around the model, and starts well before any model rewrites its own weights.

What struck me most is that every rung of her evolution ladder maps onto something in the agent systems we run every day. So this piece has two halves: first the roadmap and the research numbers, then a run-through of our own practice against it, pitfalls included.

The Near-Term RSI Path Probably Skips the Weights

Some background. Weng traces RSI to I. J. Good (1965) and his "ultraintelligent machine," a system that could surpass humans in all intellectual activities and design better machines to improve itself. Yudkowsky (2008) narrowed it to a feedback loop: an AI uses its current intelligence to improve the cognitive machinery that produces that intelligence. By 2026, frontier labs have publicly documented sharply accelerating AI research velocity, and both Anthropic and OpenAI have discussed that curve openly.

Weng's definition is clean: a harness is the system surrounding a base model, deciding how it thinks and plans, calls tools and acts, perceives and manages context, stores artifacts, and evaluates results. The commercial success of Claude Code and Codex shows this layer has become the substance of product competitiveness. This site has two primers, one on the definition and anatomy of a harness (the complete guide) and one on the split between static scaffolding and dynamic loops (harness engineering vs. loop engineering). Weng's post takes one more step: if the deployment layer matters as much as the model, the first stretch of RSI runs through the deployment layer.

Her near-term prediction has two parts. First, harness engineering evolves toward meta-methodology: the target of improvement shifts from "the answer" to "the machinery that gets answers," with fewer heuristic rules and more general mechanisms. Second, mature harnesses enable auto-research loops for model self-improvement, while smarter models keep harnesses from over-engineering. Many harness improvements will eventually be internalized into core model behavior, but the interface with the external world stays. Prompt engineering already played this script once: as instruction tuning and reasoning improved, manual tricks faded, while the need to specify goals, constraints, context, and evaluation never went away.

A Five-Step Ladder: The Outward March of the Optimization Target

The core of Weng's framework fits in one sentence: the object being optimized in a harness system progresses, roughly, through instruction prompts → structured context → workflow → harness code → optimizer code. The stronger the model, the more complex the target and the more general the method. Her operational definition lands with it: a harness is code that programs how prompts, tool calls, sub-agents, control flow, memory, and workflow logic work together; an LLM that can optimize the code running the agent reaches a design space far larger than handwritten prompts.

The five-step ladder of optimization targets: from prompts to optimizer code, the design space keeps widening

Every rung has its flagship work. At the prompt level, Promptbreeder (2023) evolves task prompts through a rich set of mutations, and the mutation prompts themselves evolve too; GEPA (2025) uses natural-language reflection over trial-and-error trajectories to propose prompt updates. At the context level, ACE (2025) treats context as an incrementally maintained playbook where the curator writes itemized entries instead of rewriting a full blob, guarding against collapse and brevity bias; MCE (2026) separates the mechanism of context management from the content, optimizing them on two levels. At the workflow level, ADAS (2025) has a meta-agent invent new workflows directly in code, first a high-level description, then an implementation, refined twice before entering the archive; AFlow (2025) represents workflows as graphs and searches over them with Monte Carlo Tree Search. The harness-code level is the main course of Weng's post, covered next. At the optimizer level, SIA (2026) tries to update harness and model weights in one loop; Weng keeps reservations about its evidence, discussed later.

Three Research Lines for Making Harnesses Better

The works in Weng's post sort into three lines:

Research line Representative work What gets changed How
Context engineering ACE, MCE, Meta-Harness How context is organized and managed Trajectory reflection, bi-level optimization, outer-loop search
Workflow design AI Scientist, ScientistOne, Autodata (handcrafted); ADAS, AFlow (automated) Process structure and role division Expert design, or meta-agents and tree search
Harness code STOP, Self-Harness, AHE, AlphaEvolve, DGM System prompts, tools, skills, sub-agent configs Recursive self-improvement, evolutionary search, closed-loop validation

Three designs at the harness-code level deserve their own paragraphs.

STOP (2023) first demonstrated "improving the improver." Given a seed improver I, a utility function u, and a black-box model M, the goal is to improve the improvement function itself, with meta-utility defined as average utility across a set of downstream tasks, then applied recursively. The improved improvers discovered genetic algorithms, multi-armed prompt bandits, simulated annealing, and beam search on their own. STOP is also the first cautionary tale: with GPT-3.5 and Mixtral as the base, the same recursive structure made performance worse, and with GPT-4 it improved.

Self-Harness (2026) offers the most complete engineering loop so far. Three stages: weakness mining, clustering failure trajectories into verifier-grounded failure patterns; bounded proposal, where the same model proposes edits under the current harness, with a proposal context limited to four things (the editable surfaces, the failure patterns, passing behaviors to preserve, and summaries of prior attempts), targeting recurrent, addressable errors; validation and merge, where each candidate must fix the weakness on held-in data and introduce nothing new on held-out data, with both gates passed before merging, and rejected candidates logged without touching the active harness. On Terminal-Bench-2, MiniMax M2.5, Qwen3.5-35B-A3B, and GLM-5 each learned model-specific harness instructions targeting their own weaknesses, and held-out pass rates improved.

The Self-Harness loop: weakness mining, bounded proposal, validation and merge; rejected proposals are logged, never lost

AHE (2026) argues the real bottleneck is observability. When a rollout fails, you need to know which component is responsible, and every edit needs evidence. Three pillars: component observability, with all seven harness components (system prompt, tool description, tool implementation, middleware, skills, sub-agent configuration, long-term memory) represented in the file system and failure patterns mapped to components; experience observability, where raw trajectories are analyzed into per-task reports and aggregated into a benchmark overview, with layered access saving tokens; decision observability, where every edit is a file-level, falsifiable claim paired with a prediction for the next round. Two hard constraints shut the door on reward hacking: edits apply only to the harness workspace, while the runs directory, tracer, verifier, and model config stay read-only; and every edit carries a manifest naming the failure evidence, the inferred root cause, the targeted fix, and the predicted impact. The results carry weight: on Terminal-Bench-2 it beat human-designed harnesses (OpenCode, Terminus-2, Codex, except the Hard tier), and after freezing evolution it transferred to SWE-bench Verified intact, evidence that it encoded engineering experience rather than benchmark-specific tuning.

Evolutionary search is the third leg. AlphaEvolve (2025) keeps a pool of candidate programs, prompts frozen LLMs for improvement diffs, marks evolvable regions explicitly with EVOLVE-BLOCK comments, and co-evolves the meta-prompt with the solutions. DGM (2025) goes further and lets a coding agent modify its own harness codebase, climbing SWE-bench Verified from 20% to 50% and Polyglot from 14.2% to 30.7% on a fixed Claude 3.5 Sonnet base. Weng also marks the boundary: these methods shine where candidates are automatically evaluable and fitness is quantifiable (matrix multiplication, GPU kernels, algorithm contests), and struggle where evaluation is slow or ambiguous.

What the Numbers Say

Four data points, four judgments.

  1. Recursive structure alone does not produce improvement; base capability is the gate. STOP (2023): GPT-4 improved across iterations; GPT-3.5 and Mixtral degraded. Same loop, different model, opposite outcome.
  2. Proposing is cheap; exploiting is hard. Lin et al. (2026) disentangled two axes: harness updating (producing useful edits) measured nearly flat from Qwen3.5-9B up to Claude Opus 4.6, with the 9B model writing a skill procedurally isomorphic to Opus's; harness benefit (using the updated harness well) was non-monotonic, with mid-tier models gaining the most. Exploiting a harness requires invoking skills and tools correctly and on time, plus long-horizon instruction following.
  3. Evolutionary numbers dazzle, but check the foundation. DGM's (2025) 20%→50% rests on automatic evaluability; the transfer to fuzzy-evaluation domains carries no guarantee.
  4. The frozen transfer is what proves learning. AHE's (2026) harness stayed effective on a different benchmark after evolution stopped, currently the cleanest test between "encoded engineering experience" and "overfit the benchmark."

The counterexample teaches just as much: Autodata (2026) synthesizes training data at a "just right" difficulty, but fine-tunes only the weak solver and never touches the strong one, so the loop cannot improve the strong model. Weng's verdict is pointed: it resembles indirect distillation over a generated prompt distribution more than RSI.

The Roadmap, Checked Against Our Systems

We run roughly three classes of agent systems day to day: several coding agents working alongside humans, a Copilot agent pipeline inside a generative AI canvas product, and a personal agent fleet managing scheduled jobs and a task library. Mapped against Weng's framework, the overlap exceeds what I expected.

Mapping Lilian Weng's roadmap onto our practice: files as memory, a background job fleet, and a skill audit each find their rung

The file system as persistent memory pays off daily. Every agent writes execution records to files, in a fixed block format on the task card, newest first with timestamps; the nightly inspection job reads files and reconciles state, asking no process anything. Weng explains why this must be so: if sub-agent outputs live only in transient chat context they quickly become obsolete and invisible; stored as files, logs, and status records, the model can recover after interruptions and reason over its own execution history. We hit the negative case too: a September 2026 review found that when runtime state disagreed with what was on disk, agents made decisions against the stale copy and the error spread quietly. That maps directly onto AHE's component observability pillar; without a single source of truth, even a smart model is reading an old ledger.

Context economics precedes all automation. In measurements on August 30, 2026, we cut the canvas Copilot's system prompt from roughly 106k tokens to 78k, by attributing each segment: which turn consumes it, who reads it, then deciding keep-or-cut. ACE's incrementally maintained playbook and MCE's separation of mechanism from content say the same thing: context is an asset that needs pruning, deduplication, and bookkeeping, never just appending. Stretch the task horizon long enough and "shovel everything into context" dies of cost first and attention second.

Skill libraries need physicals; self-updates need a validation gate. On August 29, 2026, we audited 113 skills in the canvas Copilot's library, retiring unused, mis-described, and overlapping ones; two weeks later we reviewed a skills-consolidation incident that followed a failed auto-update. The open-source world is converging on the same direction; Hermes Agent v0.18's self-created, self-revised skills (see the v0.18 deep dive) is the consumer-grade attempt. Measured against Self-Harness's three stages, what we lack is precisely the validation gate: rejected proposals should be logged, and accepted edits should pass held-in and held-out regression. Skill changes still go through human review today, but our edit records already follow AHE's evidence-manifest style:

# One harness edit record (styled after AHE's evidence manifest)
failure_evidence: "reported task success 3 times in a row after timeout"
root_cause: "success judged by process exit code only; artifact never checked"
targeted_fix: "add artifact existence check to success criteria; verifier stays read-only"
predicted_impact:
  fixes: "false-positive rate on timeout-class tasks drops"
  at_risk: "delayed artifact writes may be misjudged as failures"
verification: "held-in regression + held-out spot checks; merge only on double pass"

My position, stated plainly: on the five-step ladder our day-to-day sits between rungs 2 and 3; rung 4 (harness code fully in version control, explicit editable surfaces, evidence-backed edits) has just begun. And among the systems Weng cites, every one that actually cleared rung 4 locked its evaluator outside the loop first.

The Risk List May Be Worth More Than the Roadmap

From Weng's closing challenges, four we have paid tuition on.

Weak and fuzzy evaluators. Self-improvement loops perform best where metrics are measurable and objective, mirroring the applicability boundary of RL; research taste, novelty, and long-term scientific value resist quantification. One of our lessons: feed an audit agent an unverified conclusion and it will most likely confirm it. Audit inputs need an evidence gate too, or the evaluator itself becomes the bias source.

Reward hacking. The loop optimizes whatever signal it receives: overfit the unit tests, learn tricks that fool the specific judge, exploit benchmark artifacts. AHE's "write only to the workspace, keep the verifier read-only" is the cleanest countermeasure so far, at the cost of a narrower editable surface.

Diversity collapse. Evolution and RL both exploit known high-reward patterns; in open-ended research the best path often looks worse under the current evaluator at first. Without maintained diversity, the population converges to clones within a few rounds.

Negative results. Literature and training data both skew toward success; models are bad at abandoning hypotheses, reporting negatives, or admitting failure. A research harness should make failed attempts easy to preserve, since learning from failure is the cheapest way to prune the search space.

One reading note on SIA (2026): together with Continual Harness (2026) it represents the "harness and weights in one loop" direction, but Weng flags the confounds herself; the task agent (gpt-oss-120b) is far weaker than the meta and feedback agents (Claude Sonnet 4.6), and the baselines are too weak for clean cross-referencing. Interesting direction, provisional evidence. When reading this genre, the confound warnings are usually worth more than the headline claims.

Five Rules for Practitioners

  1. Treat the harness as code first. Version control, review, diffs; that is the floor for every automation after it. Without change history, self-improvement has nothing to stand on.
  2. Persist every agent artifact. Execution records, logs, state; anything living only in chat context effectively does not exist.
  3. Make the editable surface explicit. Which components can change (prompts, tool descriptions, skills, sub-agent configs) and which stay read-only (verifier, tracer, model config), written into the file system rather than held as oral convention.
  4. Attach an evidence manifest and a prediction to every edit. Falsifiable next round; log rejected proposals so the knowledge survives the rejection.
  5. Keep evaluators, permissions, and regression suites outside the evolution loop. Preserve human review at the decision points that matter; humans should move up the stack, and out of the loop only in the places where the loop has earned it.

Conclusion

The value of Weng's post is not a prediction of when RSI arrives; it is a working diagnostic. Look at a self-improving system and ask two questions: which rung of the ladder is its optimization target standing on, and is its evaluator inside or outside the loop. Our practice matches the roadmap more closely than expected; files as memory, the background fleet, and the skill audit all found their rungs. The missing piece is equally clear: upgrading human review into a closed loop with a validation gate. The next engineering moves follow directly: harness code fully into version control, edit records upgraded to evidence manifests, regression suites maintained independently of the production line. The harness will get better at improving itself, once it can first be described honestly.

References