Does thinking-token overuse actually degrade reasoning accuracy in practice?
This explores whether spending more 'thinking' tokens—the extended chains models generate before answering—actually makes reasoning worse, or whether the relationship is more conditional than a simple 'more is better' or 'more is worse'.
This explores whether spending more 'thinking' tokens actually degrades reasoning accuracy in practice, and the corpus answer is clear: yes, but conditionally. Several notes converge on a non-monotonic curve—accuracy climbs, peaks, then falls. One study watched benchmark accuracy collapse from 87.3% to 70.3% as thinking tokens scaled from ~1,100 to ~16,000 Does more thinking time always improve reasoning accuracy?, and a companion analysis frames the mechanism: models overthink easy problems and underthink hard ones, with extended reasoning inflating output variance and inviting self-revision errors rather than improving the answer When does thinking too much actually hurt reasoning?. So the degradation isn't a quirk—it's a reliable threshold effect.
The more interesting twist is that the optimal length isn't fixed. It follows an inverted-U whose peak shifts: harder tasks want longer chains, but more capable models want shorter ones, and RL training naturally pushes improving models toward brevity—simplicity emerges from the reward signal, not from explicit instruction Why does chain of thought accuracy eventually decline with length?. That reframes 'overuse' as a mismatch between chain length and the model-task pair, not an absolute token budget.
But quantity is only half the story—quality of thinking matters at least as much. Vanilla models use extended thinking counterproductively, talking themselves into self-doubt; RL training redirects the very same mechanism toward productive gap analysis Does extended thinking help or hurt model reasoning?. And within a chain, the value is wildly uneven: only ~20% of tokens are high-entropy 'forking points' that actually steer the outcome Do high-entropy tokens drive reasoning model improvements?, while specific reflection tokens like 'Wait' and 'Therefore' spike in mutual information with the correct answer—suppress those and accuracy drops, suppress equally many random tokens and nothing happens Do reflection tokens carry more information about correct answers?. Overuse, then, often means diluting a few load-bearing tokens with filler.
Here's what you might not have expected: the visible chain may not be where the reasoning lives at all. Models trained with hidden CoT compute the correct answer in their first few layers, then actively overwrite it to emit format-compliant filler Do transformers hide reasoning before producing filler tokens?. Deliberately corrupted or irrelevant traces teach as well as correct ones Do reasoning traces need to be semantically correct?, and invalid traces routinely yield right answers—suggesting traces are stylistic scaffolding, not causal reasoning Do reasoning traces actually cause correct answers?. If much of the visible token stream is decorative, piling on more of it has obvious downside and limited upside—which is exactly why latent-reasoning architectures scale test-time compute through hidden-state iteration without verbalizing anything Can models reason without generating visible thinking tokens?.
There's also a darker reason long chains hurt: more tokens are more chances to go wrong. Longer inputs alone degrade reasoning sharply—accuracy fell from 92% to 68% with just 3,000 tokens of padding, far below the context limit Does reasoning ability actually degrade with longer inputs?—and within long chains, local memorization from immediately preceding tokens drives up to 67% of reasoning errors as a chain drifts off-distribution Where do memorization errors arise in chain-of-thought reasoning?. So overthinking doesn't just waste compute; it lengthens the runway for error accumulation. The practical takeaway is that 'think more' is the wrong knob—what helps is thinking the right amount for the model-task pair, and making the few decisive tokens count.
Sources 12 notes
Increasing thinking tokens from ~1,100 to ~16K reduced benchmark accuracy from 87.3% to 70.3%, revealing a non-monotonic relationship where models overthink easy problems and underthink hard ones.
Empirical studies demonstrate non-monotonic scaling in test-time reasoning: accuracy peaks at a critical thinking-token count, then declines sharply (87.3% to 70.3% as tokens scale from 1,100 to 16,000). Extended thinking inflates output variance and introduces self-revision errors rather than improving solution quality.
Task accuracy peaks at intermediate CoT length, with optimal length increasing alongside task difficulty but decreasing with model capability. RL training naturally gravitates toward shorter chains as models improve, revealing that simplicity emerges from reward signals rather than explicit training.
Vanilla models use thinking mode counterproductively, inducing self-doubt that degrades performance. RL training reverses this, transforming the same mechanism into beneficial gap analysis. Training mediates reasoning quality, not just quantity.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
Specific tokens like "Wait" and "Therefore" show sharp spikes in mutual information with correct answers. Suppressing them harms reasoning while suppressing equal random tokens does not, and representation recycling improves accuracy 20%.
Logit lens analysis shows models trained with hidden CoT tokens compute correct answers in layers 1-3, then actively suppress these representations in final layers to produce format-compliant filler output. The reasoning is fully recoverable from lower-ranked token predictions.
Models trained on systematically irrelevant traces maintain solution accuracy and sometimes improve out-of-distribution generalization, suggesting traces function as computational scaffolding rather than meaningful reasoning steps.
R1's intermediate tokens carry no special execution semantics and are generated identically to other LLM output. Invalid traces frequently produce correct answers, proving traces are not causally necessary—they correlate with answers via learned formatting, not functional reasoning.
Multiple architectures—depth-recurrent models, Heima, and Coconut—demonstrate that test-time compute scales through hidden state iteration rather than token generation. This suggests verbalization is a training artifact, not a reasoning requirement.
FLenQA shows reasoning accuracy drops from 92% to 68% at just 3000 tokens of padding, far below context window capacity. The degradation is task-agnostic, uncorrelated with language modeling performance, and persists even with chain-of-thought prompting.
STIM framework identifies local, mid-range, and long-range memorization sources in CoT reasoning. Local memorization—based on preceding tokens—accounts for up to 67% of reasoning errors, especially as complexity increases and distributional shift occurs.