Why do different language models independently converge toward similar outputs in open-ended generation?
This explores why separately-built language models tend to produce the same answers when asked open-ended questions — and what that sameness reveals about how they generate text at all.
This explores why separately-built language models tend to produce the same answers when asked open-ended questions. The headline result comes from INFINITY-CHAT, which ran 26,000 open-ended queries across 70+ models and found an "Artificial Hivemind": models converge on strikingly similar — sometimes identical — responses regardless of who built them Do different AI models actually produce diverse outputs?. The surface explanation is shared inputs: overlapping training data scraped from the same web, plus alignment procedures (RLHF and friends) that push everyone toward the same polite, helpful register. But the corpus suggests something deeper than "they read the same books."
The more interesting cause is mechanical. Token prediction trains a model to continue *toward the training distribution* — to emit the most probable next piece of text, not to branch off into competing or contrarian directions Does LLM generation explore competing claims while producing text?. If every model is independently solving "what's the highest-probability continuation of this prompt," and they're all fitting roughly the same underlying distribution of human text, then convergence isn't a coincidence — it's the objective working as designed. Same target, same math, same answer. One framing puts it sharply: LLMs are best understood as autoregressive probability machines, and that lens actually *predicts* their behavior, including where they'll fail Can we predict where language models will fail?.
There's a subtle wrinkle worth knowing: convergence on the *visible* output doesn't mean the models are identical inside. A model holds a superposition of plausible continuations and *samples* from it at generation time — regenerate the same prompt and you can get different characters, different phrasings, each internally consistent Do large language models actually commit to a single character?. So the hivemind is a convergence of distributions, not of fixed beliefs. The models agree on the *shape* of likely text even while sampling different points from it. And because generation is a smooth probabilistic flow rather than a deliberative search — sequential but atemporal, with no pause to weigh alternatives Does AI text generation unfold through temporal reflection? — nothing in the process actively pushes a model *away* from the consensus continuation toward a novel one.
The thing you didn't know you wanted to know: this convergence quietly breaks a popular assumption — that ensembling diverse models buys you diverse answers. If they all collapse toward the same distribution, the ensemble is closer to one voice repeated than many voices in dialogue Do different AI models actually produce diverse outputs?. It also connects to why models struggle to improve themselves: escaping the consensus requires something *external* to validate a better answer, because a model has no internal signal telling it the popular continuation is the wrong one — the generation-verification gap means novelty can't be bootstrapped from probability alone What stops large language models from improving themselves?. Genuine diversity, on this reading, isn't a tuning knob — it has to be engineered in against the grain of what next-token prediction naturally does.
Sources 6 notes
INFINITY-CHAT analyzed 70+ models across 26K open-ended queries and found an "Artificial Hivemind" effect: models independently generate strikingly similar or identical responses due to overlapping training data and alignment procedures, undermining the diversity benefits of model ensembles.
Token prediction trains models to continue toward the training distribution, not to explore logically related counterpositions. This smoothness in process produces smooth claims that multiply without generating new perspectives.
By framing LLMs as autoregressive probability machines, researchers predicted tasks with low-probability target responses would be systematically harder, even when logically simple. Experiments confirmed predictions like backwards alphabet and letter counting.
Shanahan's 20-questions test shows LLMs maintain a superposition of consistent objects or characters and sample from that distribution at generation time. Regenerating the same response yields different outputs, each consistent with prior context, proving no fixed commitment exists.
Token ordering in LLMs follows probabilistic selection without intervening reflection or revision. Human discourse gains meaning from temporal structure—time spent thinking changes what comes next—but AI text production lacks this duration-in-reflection despite appearing sequentially composed.
Self-improvement in LLMs is formally bounded by the generation-verification gap, meaning every reliable fix requires something external to validate and enforce it. Models cannot escape this constraint through metacognition alone.