Study: frontier models rewriting their own agent harness gain little on held-out games

An arXiv study on ARC-AGI-3 finds two self-revisions of a model-built harness raise held-out scores by 2.80 points, with an interval crossing zero.

A controlled study posted to arXiv on October 21, 2026 tested whether frontier models can build and then improve their own agent harness, meaning the code that handles perception, memory, tool use, recovery and inference allocation around the model. The answer, under its protocol, is not reliably. If you're tempted to let an agent "self-improve" its scaffold without your own evals, this is a useful data point.

The setup, per the paper: each model uses a fixed coding bootstrap to write a complete harness for ARC-AGI-3, an interactive benchmark where agents explore unfamiliar environments and work out their goals as they go. The model plays five development games through that harness, then revises the entire codebase twice. Every version is scored on twenty held-out games that never feed back into the revisions.

Across 27 lineages, harness performance ranged from zero to near-ceiling. Two revisions raised mean held-out score by only 2.80 points, with a 95% interval of -1.44 to 7.35, so the true effect could be zero. The authors also report that gains on the development games overstated held-out improvement, and that cutting inference cost can come at the expense of task performance. Human-steered systems still show more headroom on the same games.

As @omarsar0 summarized on X, better designs exist; the models just don't find them reliably on their own yet.

The practical takeaway: keep a held-out eval set separate from whatever an agent tunes against, or its "improvements" may not generalize. This is a single study with no independent replication found, and it covers only a two-revision protocol on one benchmark.

Sources