How do reasoning-invariant tokens dilute learning signals in uniform averaging?
This explores a sharper version of a known training problem: if most tokens in a reasoning chain carry little of the actual learning signal, then averaging the gradient evenly across every token drowns the few that matter — and the corpus has a surprising amount to say about which tokens those are.
This reads the question as: when you train a reasoning model by spreading the update equally over every token, the tokens that don't change the reasoning ('reasoning-invariant' filler, grammar, connective tissue) water down the signal from the handful that do. The corpus turns out to agree with the premise — and then complicates it in interesting ways.
The cleanest evidence comes from work showing the signal lives in a tiny minority. Only about 20% of tokens in a reasoning chain are high-entropy 'forking points' where the model actually decides where the reasoning goes, and training exclusively on those matches or beats updating on everything Do high-entropy tokens drive reasoning model improvements?. That's the dilution thesis stated directly: the other 80% aren't just neutral, they're ballast. A complementary line shows models already rank tokens by functional importance internally — when you prune a chain by what preserves the answer, symbolic-computation tokens survive while grammar and meta-discourse get cut first, and students trained on the pruned chains do *better* Which tokens in reasoning chains actually matter most?.
But the corpus also unsettles the idea that you can cleanly separate 'real reasoning' tokens from filler. Models trained on deliberately corrupted, irrelevant traces stay just as accurate Do reasoning traces need to be semantically correct?, and transformers have been caught computing the answer in their first few layers and then *overwriting* it with format-compliant filler tokens Do transformers hide reasoning before producing filler tokens?. So some of what looks reasoning-invariant on the surface is scaffolding the model needs, and some of what looks like reasoning is theater. Uniform averaging doesn't just dilute — it can reward the theater, since chain-of-thought often works by reproducing familiar reasoning *forms* rather than doing inference Does chain-of-thought reasoning reveal genuine inference or pattern matching?.
This is why several approaches sidestep token-level averaging entirely. If verbalized tokens are partly a training artifact, you can reason in latent space without emitting them Can models reason without generating visible thinking tokens?, or move the unit of reasoning up to whole sentences in an embedding space so the signal isn't smeared across hundreds of low-information sub-word tokens Can reasoning happen at the sentence level instead of tokens?. Others change *what* gets rewarded rather than where: using the model's own answer-span confidence as the signal concentrates credit on traces that lead to calibrated answers Can model confidence work as a reward signal for reasoning?.
The thing you didn't know you wanted to know: the dilution problem and the model's own behavior point the same direction. Models already sparsify their internal activations under hard tasks, behaving as if they're filtering for the few features that matter Do language models sparsify their activations under difficult tasks?. Uniform averaging fights that instinct — it treats every token as equally worth learning from, when the model itself has already decided most of them aren't.
Sources 9 notes
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
Greedy likelihood-preserving pruning reveals six functional token categories; symbolic computation tokens are preferentially preserved while grammar and meta-discourse are pruned first. Student models trained on these pruned chains outperform those trained on frontier-model compression.
Models trained on systematically irrelevant traces maintain solution accuracy and sometimes improve out-of-distribution generalization, suggesting traces function as computational scaffolding rather than meaningful reasoning steps.
Logit lens analysis shows models trained with hidden CoT tokens compute correct answers in layers 1-3, then actively suppress these representations in final layers to produce format-compliant filler output. The reasoning is fully recoverable from lower-ranked token predictions.
CoT works by constraining models to reproduce familiar reasoning patterns from training, not by enabling novel symbolic reasoning. Performance degrades predictably under distribution shifts—the signature of imitation rather than capability emergence.
Multiple architectures—depth-recurrent models, Heima, and Coconut—demonstrate that test-time compute scales through hidden state iteration rather than token generation. This suggests verbalization is a training artifact, not a reasoning requirement.
Meta's Large Concept Model operates on sentence embeddings rather than tokens, reasoning in a language-agnostic space before decoding to any target language. This hierarchical approach with paragraph-level planning produces more coherent output than flat token generation.
RLSF uses answer-span confidence to rank reasoning traces, creating synthetic preferences that strengthen step-by-step reasoning while reversing RLHF's calibration degradation—without requiring human labels or external verifiers.
As task difficulty increases, LLM hidden states become substantially sparser in a localized, systematic way that correlates with task unfamiliarity and reasoning load. This sparsification acts as a selective filter stabilizing performance under OOD shift rather than a failure mode.