Why do large language models outperform fine-tuned models once repeated items are removed?
This explores a memorization-vs-generalization story: the claim that fine-tuned models lean on seeing the same (or near-identical) examples again, so when you strip those repeats away and test on genuinely novel items, a less-specialized large model that generalizes wins.
This explores a memorization-vs-generalization story — the idea that fine-tuning often teaches a model to recognize examples it has effectively already seen, so removing repeated or near-duplicate items exposes how little real reasoning the fine-tuned model installed. The corpus has a surprisingly direct answer to this, and it's less about model size than about what fine-tuning actually does.
The sharpest evidence comes from out-of-distribution stress tests. When researchers build "N-1" variants of problems — same structure, swapped specifics, so the answer can't be retrieved from memory — RL fine-tuned models drop sharply versus their performance on in-distribution problems Do fine-tuned language models actually learn optimization procedures?. The interpretation there is blunt: methods like GRPO sharpen template-matching rather than installing a reasoning procedure. A related finding shows models don't actually execute iterative numerical methods at all; they recognize a problem as template-similar to something memorized and emit plausible-but-wrong values, a failure that persists across scale and training approach Do large language models actually perform iterative optimization?. So when repeated items are present, the fine-tuned model looks strong because it's matching; remove them and the crutch disappears.
The twist the corpus adds is that fine-tuning isn't uniformly a memorization trap — it depends on what signal you give it. Small models fine-tuned with DPO on both correct and incorrect examples outperform plain supervised fine-tuning, precisely because the explicit negative examples target rigid failure modes instead of just reinforcing seen outputs Can small models match large models on function calling?. That's the flip side of the same coin: supervised fine-tuning on repeated correct examples teaches the model to reproduce, while training that contrasts right against wrong pushes toward something more transferable.
There's also a deeper ceiling lurking underneath. Self-improvement in LLMs is formally bounded by a generation-verification gap — a model can't reliably fix itself without an external signal to validate the fix What stops large language models from improving themselves?. Fine-tuning on repeated items is, in a sense, a weak external signal that mostly confirms what the model already does. Once the repeats are gone, there's nothing left to lean on, and the broader, less-overfit base model's general capability re-asserts itself.
What's worth carrying away: the gap you see when duplicates are removed isn't really "big model beats small model." It's "generalization beats memorization," and fine-tuning sits on either side of that line depending on whether it teaches a procedure or just rehearses answers. If you want to feel where this bites in practice, the linguistic blind-spot work shows the same surface-pattern-vs-deep-rule divide from another angle — models capture statistical surface patterns but miss the underlying grammatical structure Why do large language models fail at complex linguistic tasks?.
Sources 5 notes
Even GRPO-trained models show sharp performance drops on out-of-distribution variants (N-1 test sets) compared to in-distribution problems, indicating RL optimizes template-matching rather than genuine problem-solving procedures.
Research shows LLMs cannot perform iterative procedures in latent space. They recognize optimization problems as template-similar and emit plausible-looking but incorrect values, a failure mode that persists across model scale and training approaches.
Small models fine-tuned via DPO on correct and incorrect function-calling examples from a large teacher model achieve high accuracy on logical and mathematical tasks. DPO's explicit negative examples directly target the rigid output format failures where SFT alone underperforms.
Self-improvement in LLMs is formally bounded by the generation-verification gap, meaning every reliable fix requires something external to validate and enforce it. Models cannot escape this constraint through metacognition alone.
Top-tier LLMs like Llama3-70b consistently misidentify embedded clauses, verb phrases, and complex nominals. Performance degrades predictably as syntactic depth increases, revealing that statistical learning captures surface patterns but not deep grammatical rules.