SYNTHESIS NOTE
Do neural networks naturally learn modular compositional structure?
Explores whether neural networks decompose compositional tasks into distinct subroutines without explicit symbolic design. This challenges the longstanding view that neural networks are fundamentally non-compositional.
Structural compositionality is the extent to which neural networks break down compositional tasks into subroutines and implement them in modular subnetworks. The alternative: matching inputs to learned templates without task decomposition.
The evidence supports compositionality. Using model pruning to isolate subnetworks:
- Subnetworks that implement one subroutine can be identified
- Ablating a subnetwork harms its corresponding subroutine while leaving others largely intact
- This holds across multiple architectures (CNNs, transformers), tasks (vision, language), and scales
The pretraining effect: models initialized with pretrained weights more reliably produce modular subnetworks than randomly initialized models. Self-supervised pretraining appears to create internal structure that is more amenable to compositional decomposition. This suggests that the representations learned during pretraining have a modular quality that fine-tuning can exploit.
This provides empirical support against the longstanding objection that neural networks are fundamentally non-compositional. The finding: "some simple pseudo-symbolic computations might be learned directly from data using standard gradient-based optimization techniques." Explicit symbolic mechanisms may be unnecessary — gradient-based optimization discovers compositional structure when the task demands it and pretraining provides a good initialization.
The result is not perfect: "most do not exhibit perfect task decomposition." Compositionality is partial and graded, not all-or-nothing. Some architecture-task combinations show stronger structural compositionality than others.
This connects to the weight-sparsity finding: Can sparse weight training make neural networks interpretable by design? shows that enforcing sparsity produces clean decomposition. The structural compositionality paper shows that decomposition also emerges naturally, albeit imperfectly, from standard training. Sparsity amplifies a tendency that already exists.
Inquiring lines that read this note 131
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How does decomposing tasks improve reasoning and prevent failure propagation?- Can granular sub-task training for function calling improve both open and proprietary models?
- Do substitute networks converge differently than complement networks?
- Can granular function calling tasks learn composition from graph-sampled data?
- Does training on granular tasks beat training on the full function calling problem?
- Can sub-task handlers be swapped between neural and symbolic systems?
- How do neural networks decompose tasks into modular subnetworks that transfer?
- Can we predict which tasks will decompose into modular subnetworks?
- Why does decomposition ability transfer across domains but solving ability does not?
- Can modular expert decomposition extend beyond time into other causal dimensions?
- Why does frame-activation matter more than word-by-word composition?
- How do functional features differ from representational abstract features?
- Why are polysemantic features concentrated in early neural network layers?
- What test distinguishes genuine compositionality from fractured feature presence?
- How do semantic features in representations become steerable task-specific directions?
- Could superposed decoding algorithms maintain multi-task representation during generation?
- How do cortical columns implement local inference over memory cycles?
- Can neural modules memorize surprising tokens as adaptive long-term memory?
- Do transformers learn generalizable algorithms or instance-based patterns?
- Can symbolic mechanisms improve transformer compositional abilities?
- Why do standard transformers fail to encode recursive structure in their hidden states?
- How does layer removal affect transformers compared to ResNets?
- How do induction heads learn to overwrite computational representations?
- Can latent recurrence overcome the trainability costs of depth?
- Can spline-based activations replace MLPs in transformer architectures?
- What makes recurrent depth enable compositional generalization across tasks?
- Can a two-layer network outgeneralize billion-parameter models through recursion alone?
- Why does recursion on latent states improve generalization more than scale?
- What tasks does recurrent depth solve that feedforward models cannot?
- Can recurrent blocks learn genuinely novel computation beyond repetition?
- Can transformers abstract relational structure without explicit symbolic machinery?
- Can neural networks represent symbolic structures without explicit mechanisms?
- Does information stored in neural networks necessarily influence generation decisions?
- Why do human-designed neural architectures eventually get replaced by learned ones?
- Could probing methods miss computationally important features in neural networks?
- What makes linear decodability a reliable signal of compositionality?
- Does compositional generalization emerge suddenly or improve smoothly with scale?
- Can neural networks implement genuine algorithms or only statistical pattern matching?
- Can fractured representations explain why models fail at systematic generalization?
- Can steering vectors prove that representations are genuinely organized?
- Can neural networks learn that A implies B in reverse?
- Can identical model performance mask fundamentally broken internal representations?
- How much do structural inductive biases matter compared to training data volume?
- Can fractured entangled representations hide undetected by standard analysis methods?
- Does the linear representation hypothesis reflect networks or reflect our analysis tools?
- What role does a model's representational structure play in learning?
- Can scaling alone create compositional generalization without explicit binding mechanisms?
- How do neural networks decompose complex tasks into modular subnetworks?
- What inductive biases help networks segregate entities from raw inputs?
- What are fractured entangled representations in neural networks?
- How do sparse circuits compare to the modular subnetworks that emerge naturally?
- Can geometric structure in representations exist without supporting functional mechanisms?
- What role does query-level exposure play in enabling compositional generalization?
- Why does scaling data and model size improve compositional generalization?
- Which hyperparameter theories best explain universal behaviors across neural networks?
- What solvable idealized settings reveal fundamental phenomena in realistic deep learning?
- How do classical mechanics and statistical mechanics provide methodological templates for learning theory?
- Why does gradient descent discover compositional structure without explicit pressure?
- What happens to representational structure during model pretraining phases?
- Do generic kernel-decay assumptions alone explain coarse-to-fine spectral ordering?
- Can spectral eigenvector ordering serve as a model-agnostic interpretability probe?
- Can representation analysis methods detect complex features models compute with?
- How can neural networks be interpretable by design rather than post-hoc?
- Do KANs maintain their advantages in deep architectures and large-scale training?
- What makes a feature abstract versus concrete in neural network activations?
- How does scaling and training data enable compositional behavior without symbolic mechanisms?
- Where do neural networks still fail at compositional generalization despite scaling?
- How does representational density emerge from training data familiarity?
- What prevents representation collapse in latent-prediction world models like JEPA?
- What limits the extrapolation of learned operations like rotation and reflection?
- How does nesting optimization levels improve on traditional network depth?
- Could graph neural networks fundamentally outperform transformers on structured reasoning?
- What makes recursive structure different from other forms of compositional generalization?
- Why do higher network layers capture procedural knowledge but lower layers store facts?
- Can a single recursive network replace hierarchical dual-network architectures?
- Can reflection and color swapping compose reliably across different motif layouts?
- What makes multimodal conditioning effective when features are decomposed to the right granularity?
- How do sparse networks trade capability for human-understandable circuits?
- Can finetuning sparse subnetworks alone match full parameter finetuning results?
- Why do singular value experts compose better than low-rank adapter subspaces?
- Why does weight sparsity reduce superposition and force disentangled representations?
- Can sparse approximations reveal interpretable structure hidden in existing dense models?
- How can interpretability methods account for shifting representational density across task conditions?
- Does sparsity enforce compositional structure or merely amplify existing modularity?
- Can spiking sparsity replace weight quantization as a primary efficiency lever?
- Why do text-to-image models fail at composing multiple concepts together?
- Does scaling model size solve compositional generalization problems?
- Why do long-context language models struggle with compositional reasoning tasks?
- How do multimodal AI architectures compare to human brain export pathways?
- How do biological brains organize computation across different cortical timescales?
- Can neural grafts reliably reveal hidden capabilities in AI models?
- Can we detect and measure circuit formation before generalization emerges?
- Are detection and identification of injections truly separable in neural circuits?
- What makes a neural network circuit actually interpretable to humans?
- How do knowledge and reasoning circuits interfere in the same neural network?
- Does causal intervention alone explain how neural mechanisms implement representations?
- How does mechanistic interpretability complement learning mechanics in explaining deep learning?
- How do ablation studies reveal function without representational characterization?
- Why does knowledge storage separate from reasoning circuits in neural networks?
- Can activation patching identify what a component encodes without representation?
- Does scaling data automatically produce compositional reasoning or just better feature encoding?
- Can training order and structure shape what networks retain and learn?
- How do internal representations compare to human cognitive structures?
- Does directional knowledge failure indicate shallow pattern matching over deep representation?
- What performance trade-offs emerge when composing multiple independently trained model capabilities?
- Can expert vectors learned offline transfer across multiple model architectures?
- How does joint backpropagation differ from training separate ensemble models?
- Can we predict out-of-distribution generalization without access to downstream tasks?
- How much does pretraining quality affect the modularity of fine-tuned models?
- Can LLMs reliably generate novel working architectures without structured representations?
- How does neuro-symbolic design differ from pure LLM reasoning?
- What separates knowledge from reasoning in neural network layers?
- Why does compositional reasoning fail to explain cross-domain transfer?
- How do gradients flowing through both branches simultaneously reshape each component's role?
- Why should deep learning theory prioritize average-case over worst-case analysis?
- What makes modernized N-gram embeddings composable with transformer architectures?
- What architectural alternatives can capture compositional structure beyond pooled cosine?
- Can single-vector embeddings capture non-commutative relationships like word order?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Do neural networks naturally learn modular compo…
Can sparse weight training make neural networks in…
Do base models already contain hidden reasoning ab…
Can neural networks learn compositional skills wit…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Can sparse weight training make neural networks interpretable by design?
Explores whether constraining most model weights to zero during training produces human-understandable circuits and disentangled representations, rather than attempting to reverse-engineer dense models after training.
sparsity amplifies the compositional decomposition that standard training already partially produces
-
Do base models already contain hidden reasoning ability?
Explores whether reasoning capability emerges during pre-training as a latent feature rather than being created by post-training methods like reinforcement learning or fine-tuning.
pretraining-induced modularity is part of the "latent capability" that minimal signals can activate
-
Can neural networks learn compositional skills without symbolic mechanisms?
Do neural networks need explicit symbolic architecture to compose learned concepts, or can scaling alone enable compositional generalization? This asks whether compositionality is an architectural feature or an emergent property of scale.
complementary evidence: scaling enables compositionality in behavior; pruning reveals it in structure
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Break It Down: Evidence for Structural Compositionality in Neural Networks
- Scaling can lead to compositional generalization
- From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks
- Faith and Fate: Limits of Transformers on Compositionality
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
- How do Transformers Learn Implicit Reasoning?
- Hierarchical Reasoning Model
- Eliciting Reasoning in Language Models with Cognitive Tools
Original note title
neural networks decompose compositional tasks into modular subnetworks without explicit symbolic mechanisms — pretraining encourages this