Upgrading an AI agent's scaffolding helps mid-tier models most — the weakest can't use it, and the strongest ignore it.
Can harness updates benefit agents equally across all model sizes?
This explores whether upgrades to an agent's harness — its prompts, tools, and memory scaffolding — help every model equally, or whether some sizes of model gain more than others.
This explores whether upgrades to an agent's harness — its prompts, tools, and memory scaffolding — help every model equally. The corpus says no, and in a surprising way: the ability to *write* a good harness update is roughly flat across model sizes, but the ability to *benefit* from one follows an inverted-U curve that peaks at mid-tier models Do stronger models always evolve harnesses better?. Weak models can't reliably invoke the harness even when handed a good one; the strongest models struggle to follow the harness's instructions faithfully, often overriding them. So the same edit lands unevenly, and the middle of the range is where it pays off most.
That asymmetry makes more sense once you see the harness as a place to *offload cognitive work* rather than a productivity multiplier applied uniformly. Reliable agents externalize three burdens — memory, skills, and interaction protocols — into a harness layer so the model doesn't re-solve the same problems every run Where does agent reliability actually come from?. A weak model lacks the baseline competence to use those externalized structures; a very strong model already carries much of that competence internally and gains less from the scaffold. This is also why context management helps different agents differently: an external manager should preserve high-fidelity context for strong agents but aggressively compress for weak ones to keep them reliable Can an external manager handle context for frozen agents?.
The most counterintuitive part is that a good harness can *invert* the usual size advantage. Reorganizing a harness repository around runtime behavior let a weaker planner match stronger models' code localization across every precision and recall metric, while cutting tokens Can explicit behavior maps help weaker planners compete with stronger models?. So harness updates aren't just unequal — they can be a substitute for scale, closing gaps that raw model size would otherwise open. That reframes the whole cost question: if small models handle most agentic subtasks at a fraction of the price Can small language models handle most agent tasks?, then investing in the harness rather than the model is often the economically rational move.
There's a catch worth knowing before you assume any harness update is a win. Most evolved harness edits turn out to memorize task-specific fixes rather than distill reusable strategies — they cache shortcuts for already-solvable problems instead of converting failures into successes Do harness edits learn reusable strategies or memorize task fixes?. And because harness evolution is itself a search loop, its reported gains are confounded with sheer search effort; only improvements beyond an equal-budget baseline can be credited to the design How should we measure gains from automatic harness evolution?. So the honest answer is layered: harness updates don't benefit all sizes equally, they benefit mid-tier models most, they can let small models punch above their weight — but only if the update genuinely encodes strategy rather than memorized answers.
Zooming out, this sits inside a broader split in how agents improve at all: a slow parametric loop that rewrites model weights versus a fast non-parametric loop that rewrites prompts, memory, and tools Do self-improving agents really split into two distinct loops?. Harness updates are the fast loop, and its appeal is precisely that scaffold edits are cheap and reversible where weight updates are not. Related work shows agents can even adapt continuously through memory alone, with no parameter changes at all Can agents learn continuously from experience without updating weights? — which is really the same bet as the harness question, just taken to its limit.
Sources 9 notes
Model capability to produce useful harness edits stays constant across tiers, but capacity to actually benefit from those edits follows an inverted U-shape, peaking in mid-tier models. Weak models fail to invoke harnesses; strong models struggle with faithful instruction-following.
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
AdaCoM trains an external RL-based manager to prune and preserve context for frozen agents. The key finding: stronger agents benefit from high-fidelity preservation, while weaker agents need aggressive compression—optimal context management is agent-specific, not task-universal.
A behavior-to-code mapping representation improved win rates by 10–19 points while reducing planner tokens by 8–13%. Weaker planners using this mapping matched stronger models' code localization across all precision and recall metrics.
SLMs handle the repetitive, well-defined language tasks that constitute most agent work at 10–30× lower cost than LLMs, making heterogeneous architectures (SLMs by default, LLMs selective) the economically rational design pattern.
Show all 9 sources
Analysis of evolved harness trajectories shows rational, well-motivated edits across prompt and tool layers, but most persist fixes an agent could rediscover in a single rollout. Gains remain limited because memorized shortcuts cache what's already within reach rather than converting hard failures into successes.
Automatic harness evolution must be compared against task-level test-time search under equal feedback and inference budgets. Only gains beyond what matched search achieves are attributable to the harness design itself, not just more computation.
A survey framework organizes self-improving agents into two update mechanisms: slow parametric loops updating foundation model weights, and fast non-parametric loops updating prompts, memory, and tools. Recent progress concentrates in the fast loop because scaffold updates are cheaper and reversible than weight updates.
AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
- Rethinking the Evaluation of Harness Evolution for Agents
- RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
- Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution
- DarwinX: Evolving Agent Harnesses Through Natural Selection
- Useful Memories Become Faulty When Continuously Updated by LLMs