Do AIDE2's improvements transfer to unseen tasks?

The abstract: "These gains generalize to four held-out benchmarks spanning machine learning engineering, heuristic algorithm engineering, and physics-based weather forecasting, the last of which is out of distribution from the selection tasks." The discussion repeats it as "four held-out benchmarks spanning in- and out-of-distribution tasks."

This is the check a self-editing loop most needs. A loop that keeps what scores best on its selection tasks can end up encoding those tasks. Held-out benchmarks separate a better agent from one tuned to the selection set, and the out-of-distribution one goes further: weather forecasting is not a variant of the tasks the rewrites were chosen on. The paper's framing, "transferable gains," rests on this result (How does an AI agent improve its own research code?).

It bears directly on the memorization finding. Do harness edits learn reusable strategies or memorize task fixes? reports that inspected harness edits are mostly information an agent could rediscover in one rollout. Held-out transfer, one of the two controls in How should we measure gains from automatic harness evolution?, is what would show otherwise, and this excerpt reports it. The tension is filed as ops/tensions/AIDE2's rewrites transfer to held-out benchmarks while the vault's harness-evolution notes find evolved edits mostly memorize task-specific fixes — what the edits encode may decide.md.

Limits. The excerpt gives no scores, no per-benchmark results, and no count of how many selection tasks there were. It does not report a matched-budget test-time-search baseline, which is the note's other control. "Generalize" is the paper's word for gains of unstated size on four benchmarks, and the out-of-distribution result is one benchmark.