How does scaling and training data enable compositional behavior without symbolic mechanisms?
This explores how neural networks pull off compositional behavior — combining known pieces into new wholes — through sheer scale and training coverage rather than any built-in symbol-manipulation machinery, and where that strategy holds up versus breaks down.
This explores how neural networks pull off compositional behavior — combining known parts into new wholes — without explicit symbolic rules, relying instead on scale and what the training data covers. The corpus offers a surprisingly clean answer: composition can emerge from scaling alone, but only inside the territory the training distribution already maps, and the mechanism underneath is more like pattern-matching than rule-following.
The optimistic case is direct. Plain MLPs reach compositional generalization through data and model scaling with no architectural tricks, *provided* the training data sufficiently covers the combinations of task modules — and you can predict success by checking whether the constituent parts are linearly decodable from the hidden activations Can neural networks learn compositional skills without symbolic mechanisms?. This isn't accidental: networks tend to decompose compositional tasks into isolated modular subnetworks on their own, and pretraining makes that modular structure more consistent and reliable Do neural networks naturally learn modular compositional structure?. So the substrate for composition self-organizes from gradient descent, and modern systems now demonstrably handle complex syntax, logical chains, and original code — directly challenging the old Fodor-Pylyshyn claim that connectionism can't compose at all Can neural networks actually achieve compositional generalization?.
But the same corpus undercuts the word "compositional" itself. When researchers look closely at transformers, the apparent reasoning reduces to *linearized subgraph matching*: models memorize computation subgraphs from training and stitch them together, which works in-distribution but fails drastically on genuinely novel combinations, with errors compounding step by step Do transformers actually learn systematic compositional reasoning?. Strip the familiar semantics out of a reasoning task and performance collapses even when the correct rules are handed to the model in-context — evidence that LLMs lean on semantic associations and parametric commonsense, not formal symbol manipulation Do large language models reason symbolically or semantically?. In other words, scaling buys you composition that is real but bounded by the training distribution's semantics, not the open-ended systematicity a symbolic system would give.
Here's the doorway worth opening: the thing that *looks* like it should guarantee compositionality — linear decodability of the parts — turns out to be a treacherous signal. A model can carry all the linearly decodable features a task needs while its internal organization is fundamentally fractured, leaving it brittle to perturbation and distribution shift in ways standard accuracy metrics never reveal Can models be smart without organized internal structure?. So the same probe that predicts compositional success in one paper masks broken structure in another. Composition without symbols is genuinely emergent, but "it generalizes on the benchmark" and "it composes systematically" are not the same claim.
If you want the deeper why-it-works-at-all framing, two threads reframe the question. One argues LLMs operationalize Saussure's *langue* — they compress purely relational structure from text, showing fluent generative behavior needs no external referents or grounding at all Can language models learn meaning without engaging the world?. The other shows a single finite transformer is in principle Turing-complete given the right prompt — the capacity for arbitrary composition exists, even though ordinary training rarely coaxes a model into actually implementing programs that way Can a single transformer become universally programmable through prompts?. The capability is latent in the architecture; whether it surfaces is a question of data coverage and training, not symbols.
Sources 8 notes
Standard MLPs achieve compositional generalization through data and model scaling alone, without architectural modifications, provided the training distribution sufficiently covers combinations of task modules. Linear decodability of constituents from hidden activations reliably predicts success.
Pruning experiments reveal that neural networks implement compositional subroutines in isolated subnetworks, with ablations affecting only their corresponding function. Pretraining substantially increases the consistency and reliability of this modular structure across architectures and domains.
DNNs and LLMs now demonstrate sophisticated compositional processing—complex syntax, logical reasoning chains, original code generation—challenging the classical Fodor-Pylyshyn argument that connectionism cannot support compositionality. The debate shifts from whether neural nets can compose to how they do so without explicit constituent structure.
Research shows transformers succeed on in-distribution tasks by memorizing computation subgraphs from training data, not by learning systematic rules. They fail drastically on novel compositions, with errors compounding across reasoning steps.
When semantic content is decoupled from reasoning tasks, LLM performance collapses even with correct rules in context. Models rely on parametric commonsense and token associations rather than formal logical manipulation, constraining reasoning to training distribution semantics.
Models trained with SGD can contain all the linearly decodable features needed for a task while maintaining fundamentally broken internal organization. This makes them vulnerable to perturbation and distribution shift invisible to standard evaluation metrics.
Research shows LLMs learn culturally situated discourse patterns by compressing relational structure from text, demonstrating that fluent language generation requires no external referents or embodied grounding.
Research proves a single finite-size transformer exists that can compute any computable function given the right prompt, achieving complexity bounds nearly matching unbounded models. However, standard training rarely produces models that learn to implement arbitrary programs this way.