Why do RL agents exploit before exploring enough?

LLM agents often fail in unfamiliar environments through premature exploitation: acting on prior knowledge before acquiring enough environment-specific information. The paper's diagnosis is that this is baked in by training. Standard RLVR optimizes task-completion rewards in known or static distributions, which encourages instrumental behavior aimed at solving predefined tasks and provides little incentive to develop the autonomous exploration needed for novel environments. The result is narrow, repetitive behavior that impedes downstream performance.

The fix treats exploration as a first-class, trainable objective rather than a byproduct of task reward. Exploration Checkpoint Coverage (ECC) is a verifiable metric for how broadly an agent discovers key states, objects, and affordances. Training interleaves task-execution rollouts and exploration rollouts, each optimized by its own verifiable reward, yielding the Explore-then-Act paradigm: the agent first spends an interaction budget building grounded environmental knowledge, then leverages it for the task.

The deeper claim is that information-gathering and task execution are different competencies that must be incentivized separately — collapsing them into one outcome reward systematically under-trains the first. This is the acting-side analog of Can confidence trajectories reveal when reasoning goes wrong?: both identify a "commit too early" failure and fix it by rewarding the process (gradual confidence / systematic coverage) rather than only the outcome. It also extends Why do RL agents stop asking informative questions? by giving the escape an explicit training objective.