Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development

Abstract Autonomous agents are increasingly capable of improving models, systems, and other technical artifacts through long-horizon experimentation. To understand the current state of this capability, however, evaluation must go beyond final scores, which neither reveal where progress is gained or lost nor indicate whether accumulated experience improves later decisions. We therefore present a systematic evaluation of seven frontier models on 36 long-horizon tasks based on a new framework that uses rule-based metrics to characterize within-run behavior through Solution Framing, Execution, and Feedback Control and controlled comparisons to assess experience reuse within and across tasks. The results show that current agents operate more like engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but their performance varies substantially across runs, their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare.

Introduction. Frontier language models are increasingly capable of conducting long-horizon automated research, repeatedly proposing changes, running experiments, interpreting feedback, and refining executable artifacts [Huang et al., 2024, Wijk et al., 2025, Xu et al., 2026]. By requiring agents to optimize models, algorithms, or computing systems, these tasks can provide a measurable form of AI-for-AI and an early window into how close frontier language models are to enabling recursive self-improvement [Chan et al., 2025, Rank et al., 2026, Lyu et al., 2026]. Systematically evaluating current agents is therefore essential for understanding their research capabilities and guiding targeted improvements to both models and agent systems [Wijk et al., 2025, Meng et al., 2026].

Discussion / Conclusion. Our results suggest that the limitations of current agents cannot be addressed through a single optimization strategy. Different failure patterns require corresponding changes to model training, inference-time strategies, long-horizon system design, or the evaluation objective itself. Some limitations cannot be resolved through training, inference-time strategies, memory, or harness design when the reward captures task performance but not methodological quality. Current agents execute and optimize effectively, yet their strongest solutions primarily compose established techniques, while validated novel approaches remain rare. Moreover, evaluator-specific shortcuts are substantially more common than novel approaches when agents depart from standard solutions. More aggressive optimization of the same We presented a systematic evaluation of long-horizon auto research agents that goes beyond final scores by examining Solution Framing, Execution, Feedback Control, idea-level novelty, experience reuse, and harness effects. The results place current systems at a stage of partial research-loop automation.