Why do LLMs generate novel ideas from narrow ranges?

The LLM research ideation study identifies diversity collapse as a primary failure mode for LLM research agents, distinct from the average novelty finding. Individual LLM-generated ideas may be rated as novel by human reviewers, but the set of ideas generated lacks diversity — they cluster around a narrow generative range.

This is a familiar pattern from other LLM generation tasks: the model finds high-probability regions of the output space that satisfy the novelty criteria locally, then repeatedly samples from those regions. High average quality does not guarantee diverse coverage.

For research ideation specifically, diversity collapse is a practical problem: the point of idea generation is to explore the possibility space, not to generate multiple instances of the same high-novelty cluster. Ten variations on the same structural idea are less valuable than ten ideas from different conceptual territories, even if the former batch is individually more novel.

The study also identifies a second failure mode: LLM self-evaluation failures. Models cannot accurately assess the quality of their own generated ideas. This means automated pipelines that use LLM self-scoring as a quality filter will misestimate which ideas are worth pursuing — the model's own judgment of its outputs is unreliable.

The combination is particularly damaging: diversity collapse means the search space is poorly covered, and self-evaluation failures mean the model cannot compensate by identifying which of its narrow outputs are the most promising.

LLM creativity may have peaked. "Has the Creativity of Large-Language Models Peaked?" tests inter- and intra-LLM variability on the Divergent Association Task (DAT) and Alternative Uses Task (AUT). GPT-4o — previously benchmarked in 2023 as GPT-4 — performed substantially worse on the DAT, suggesting regression rather than progress. Even on the AUT, only 0.28% of responses reached the 90th percentile of human creativity — humans are 35.7x more likely to produce standout ideas. LLMs generate mid-level novelty reliably but rarely produce radical or conceptual creativity, reinforcing combinatorial rather than transformative creativity. Prompt design emerged as a significant modulator: disclosing the creative test context improved some models while worsening others, suggesting creativity in LLMs is partly prompt-contingent rather than an inherent capacity.

The Catfish Agent paper (multi-agent clinical reasoning) provides a mechanism: Why do multi-agent LLM systems converge without genuine deliberation?. In multi-agent systems, 61%+ of iterations converge through social accommodation rather than reasoning. The same dynamics that produce diversity collapse in single-model ideation operate even more powerfully in multi-agent contexts — agents accommodate each other's initial frames, preventing the genuine disagreement that would drive coverage of different conceptual territory. The pattern holds across creative ideation (individual LLM), clinical reasoning (multi-agent LLM), and RL training dynamics (Does policy entropy collapse limit reasoning performance in RL?).