Do rare cultural concepts fail predictably as model scale increases?
This reads the question two ways at once: do models systematically mishandle culturally rare concepts, and does scaling them up make that failure *predictable* rather than random — and the corpus suggests the failure is structural and predictable, but not in the direction 'more scale = more failure.'
This explores whether culturally rare concepts break down in a regular, forecastable way as models grow — and the corpus pulls apart two things the question fuses together: rareness as a *data* problem and rareness as a *representational* problem. On the data side, the encouraging finding is that bigger models actually get *better* at rare cases, and for a predictable mechanical reason. Why do larger models learn rare tasks better? shows large models don't succeed at rare tasks because they can represent something small models can't — they succeed because their spare capacity weakens the gradient pressure from common tasks, so slowly-accumulating rare-task features stop getting overwritten. Rare-concept performance is governed by interference, not expressivity, which means it's tunable by data-mixture design rather than only by scale.
But 'rare cultural concept' isn't just a low-frequency task — and here the failure looks structural and does *not* dissolve with scale. Do LLMs represent low-resource cultures through dominant cultural proxies? uses mechanistic interpretability to show that low-resource cultures (Ethiopia, Algeria) are represented *through* high-resource proxies inside the model's internal states. Crucially, this persists even when the model produces the correct surface answer. That's the predictable failure mode: not a wrong output you can catch, but a flattening baked into the representation pathway, where rare cultures are routed through dominant ones. Scaling the model doesn't unbend that pathway — it's an architectural bias, not a coverage gap.
The social-norms work sharpens what 'predictable failure' means. Can AI systems learn social norms without embodied experience? and Can AI learn social norms better than humans? find frontier models out-predicting *every individual human* on social appropriateness — yet all the models share *identical systematic errors* on unwritten norms. That shared error signature is the tell: the failures aren't random noise that averages out with more scale or more models; they're the same blind spots reproduced everywhere, which is what 'predictable' really looks like. Why do AI systems fail at social and cultural interpretation? names the split: statistical mastery of norms coexists with an inability to do culturally-resonant meaning-making. The model knows the statistics of the culture from the outside without participating in it.
There's a deeper diagnosis lurking under all of this — that the failures are about *familiarity*, not difficulty. Do language models fail at reasoning due to complexity or novelty? finds reasoning breaks at instance-novelty boundaries, not complexity thresholds: models fit instance patterns rather than general algorithms, so anything close to training data succeeds and anything genuinely novel fails — regardless of how 'hard' it is. A rare cultural concept is, almost by definition, an instance the model has seen little of, so this predicts exactly the cultural-flattening result from a different angle. And Can LLMs understand concepts they cannot apply? explains why these failures hide so well: models can correctly *explain* a concept while failing to *apply* it, because explanation and execution run on disconnected pathways. The correct surface answer in the cultural-flattening study is a Potemkin — fluency masking a representation that never had the concept on its own terms.
One caution worth carrying away: be careful what you call a 'scaling law' here. Are LLM emergent abilities real or measurement artifacts? argues that the sharp, scale-dependent capability jumps people report are often artifacts of discontinuous metrics — switch to a continuous measure and the curve is smooth. So if you go looking for a clean threshold where rare cultural concepts 'turn on' or 'break' at a certain model size, you may be measuring your metric, not the model. The predictable failure is real, but it lives in the representation pathway and the familiarity boundary — not in a dramatic kink on a scaling curve.
Sources 8 notes
Larger models succeed at rare tasks not because they can represent solutions smaller models cannot, but because abundant capacity weakens gradients on common tasks, preventing them from overwriting slowly-accumulating rare-task features. Data-mixture design may be cheaper than scaling.
Mechanistic interpretability analysis reveals that low-resource cultures like Ethiopia and Algeria are structurally represented through high-resource cultural proxies in internal model states, not just output. This architectural bias persists even when models can produce correct surface-level answers.
GPT-4.5 predicted appropriateness of 555 social scenarios at the 100th percentile compared to human raters, with Gemini and Claude also exceeding 96% accuracy. However, all models show identical systematic errors, revealing boundaries of pattern-based social understanding that embodied experience may still be necessary to cross.
GPT-4.5 outperformed every individual human at judging social appropriateness across 555 scenarios, challenging the theory that embodied cultural experience is necessary. However, all AI models share identical systematic errors on unwritten norms.
LLMs achieve 100th-percentile performance on norm prediction yet regress on theory-of-mind tasks and cannot generate culturally-resonant interpretations. The pattern shows that statistical competence coexists with absence of actual social understanding and participation.
LRMs don't break at complexity thresholds but at instance-novelty boundaries. Models fit instance-based patterns rather than generalizable algorithms, so any reasoning chain succeeds if trained on similar instances, regardless of length.
Models can explain concepts accurately, fail to apply them, and recognize the failure—a triple pattern incompatible with human cognition. This indicates functionally disconnected explanation and execution pathways rather than simple knowledge gaps.
Sharp, unpredictable capability transitions vanish when using continuous metrics instead of discontinuous ones. The same model outputs show smooth predictable improvement with scale, suggesting emergence is a measurement choice rather than a real behavioral change.