The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks

LLM agents increasingly work on long-horizon tasks, and the decisions they make along the way, such as which hypothesis to test or which implementation to build on, determine the outcome of the whole run. Making these decisions well is becoming a key capability for both engineering and research agents. We refer to the ability to make good long-horizon decisions as the taste of an agent. While existing benchmarks measure the end-to-end success of agents on long-horizon tasks, none of them measures the taste of an agent. To address this problem, we build Taste-Bench, a benchmark of taste questions constructed automatically from trajectories that agents produced in engineering and research tasks. Each question presents a decision fork, a point in a trajectory where multiple directions are available and one of them leads to a better outcome, and the evaluated model chooses among these directions without seeing what happens after the fork. We mine these forks automatically from parallel attempts at the same task and from detours inside a single trajectory, without needing human annotation. We evaluate frontier models on Taste-Bench and find that the best model answers only 59.7% of the questions correctly.

Introduction. LLM agents increasingly work on long-horizon tasks, and the length of the tasks they can complete keeps increasing [1, 2]. For instance, recent systems conduct machine-learning research from idea to paper [3, 4], evolve large software projects across releases [5], and refine their own scaffolds during deployment [6]. In these tasks, the agent makes many decisions whose influence is not limited to the current step, such as which hypothesis to test, which implementation to build on, or which experiment to run next. Making these decisions well is becoming a key capability for agents [7, 8]. However, a wrong decision often looks reasonable at the moment, and its cost appears only much later, after the agent has spent a large part of its budget. We refer to the ability to make good long-horizon decisions as the taste of an agent. While previous research has measured the end-to-end performance of agents on long-horizon tasks [9–13], these benchmarks only report whether the agent finishes the task and provide no measure of the quality of the decisions made along the way.