HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model’s ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability—task success on held-out benchmarks, and efficiency—execution-token cost. The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development.

Introduction. As agents move from research prototypes to deployed tools such as coding assistants [2, 32], data-analysis copilots, browser workers [5], and research pipelines, their capability increasingly depends on software outside the model’s weights. This surrounding execution infrastructure, commonly termed the agent harness [31, 39], manages the execution loop, tool use, context, failure recovery, and result verification that turn model outputs into actions [3]. Its impact is substantial: with identical weights, GPT-5 solves 35.2% of Terminal-Bench 2.1 inside Terminus 2 but 49.6% inside Codex CLI [45]. As agents specialize to more domains, the demand for purpose-built harnesses will continue to grow. Because these systems require continuous development rather than one-time implementation, a practical question is whether LLMs can assist harness engineers—or even take over such a role—in building and continually improving the harness.

Discussion / Conclusion. The results show why harness development should be evaluated directly. Creation performance varies sharply by domain: under Self-Eval, current models match the human reference in writing and exceed it in machine-learning experimentation, but remain far behind in search and research and still trail it in code. Cross-executor comparisons show that some harnesses improve under a stronger executor, while others exhibit creator co-adaptation. Evolution is harder still: useful intermediate updates are often erased by later changes, and more updates do not guarantee a positive final gain. Together, these findings separate the quality of the persistent execution system from the capability of the model running inside it. The fixed-Gemini Evolution ablation sharpens this point: changing only the runtime binding can substantially move H0 and alter which harness changes are useful. HarnessDev moves agent evaluation from whether a model can solve tasks inside a fixed system to whether it can create and maintain the systems that solve future tasks.