How do attention patterns and circuits function as algorithmic representations?
This explores how the internal machinery of transformers — attention heads, sparse circuits, modular subnetworks — can be read as algorithms the model runs, and what that lens does and doesn't reveal.
This explores how the internal machinery of transformers — attention heads, sparse circuits, modular subnetworks — can be read as algorithms the model runs, rather than as an inscrutable wash of numbers. The corpus offers a hopeful version of this and a cautionary one, and they're worth reading against each other.
The hopeful version says specific computations live in specific, findable structures. The cleanest example is retrieval: fewer than 5% of attention heads act as dedicated 'retrieval heads' that pull facts out of long context, activating dynamically and proving causally necessary — prune them and the model hallucinates even though the answer was sitting right there in the input What mechanism enables models to retrieve from long context?. That's an algorithm you can point at. More broadly, networks tend to decompose compositional tasks into isolated subnetworks, where ablating one subroutine breaks only its corresponding function Do neural networks naturally learn modular compositional structure?. And you can push this further by force: training transformers with sparse weights yields compact circuits where individual neurons map to simple concepts with legible wiring, verified necessary and sufficient by ablation Can sparse weight training make neural networks interpretable by design?. The catch there is scale — interpretability-by-design holds at tens of millions of parameters but hasn't survived the jump to frontier sizes.
The cautionary version is that attention's 'algorithm' is not the neutral lookup we imagine. Soft attention structurally over-weights repeated and context-prominent tokens regardless of whether they're relevant, creating a feedback loop that amplifies framing and opinion before any RLHF tuning gets involved Does transformer attention architecture inherently favor repeated content?. Relatedly, attention integrates tokens by weighted parallel aggregation — it adds everything up rather than selectively suppressing the irrelevant — which is why models read additively and miss jokes and wordplay that depend on one frame winning over another Why do AI systems miss jokes and wordplay so consistently?. So the circuit isn't just computing your task; its very shape biases what counts as signal.
The deepest wrinkle is that the same behavior can run on completely different internal algorithms. Identical performance can hide radically different internal structures, and pushing one property (accuracy) reliably degrades others (faithfulness, calibration) What actually happens inside a language model?. This is what makes 'circuit as algorithm' both powerful and slippery — there isn't always one canonical algorithm to recover. Even the representations themselves shift with the input: networks learn dense activations for familiar data and fall back to sparse ones for unfamiliar inputs, so the 'circuit' you read off depends on what you feed it Is representational sparsity learned or intrinsic to neural networks?.
The thing you might not have known you wanted to know: the field is quietly split between *discovering* circuits in models trained normally and *engineering* models to be circuit-legible from the start Can sparse weight training make neural networks interpretable by design?. The first treats interpretability as archaeology; the second treats it as architecture. Which one wins matters, because if behavior is multiply-realizable What actually happens inside a language model?, archaeology may never give you a unique answer — and building for legibility might be the only way to be sure what algorithm a model is actually running.
Sources 7 notes
Less than 5% of attention heads across all model families function as retrieval heads, are intrinsic to short-context models, dynamically activate by context, and are causally necessary for factuality. Pruning them causes hallucination despite information being present in context.
Pruning experiments reveal that neural networks implement compositional subroutines in isolated subnetworks, with ablations affecting only their corresponding function. Pretraining substantially increases the consistency and reliability of this modular structure across architectures and domains.
Training transformers with sparse weights creates compact, human-interpretable circuits where neurons correspond to simple concepts with clear connections. Ablation studies confirm these circuits are necessary and sufficient for task performance, though scaling beyond tens of millions of parameters while maintaining interpretability remains unsolved.
Transformer soft attention systematically over-weights repeated and context-prominent tokens regardless of relevance, creating a positive feedback loop that amplifies opinions and framing before RLHF acts. System 2 Attention—regenerating context to remove irrelevant material—can interrupt this mechanism.
Transformers integrate token information through weighted parallel aggregation rather than selective suppression of irrelevant words. This structural difference explains consistent failures with jokes, wordplay, and frame-dependent meaning—not knowledge gaps, but missing cognitive operations.
Research shows that LLMs can achieve the same output through different internal mechanisms, and improvements in one dimension like accuracy reliably degrade others like faithfulness and calibration. Internal structure matters even when behavior appears identical.
During pretraining, neural networks develop dense activations for familiar training data and default to sparse representations for unfamiliar inputs. This trend emerges without task-specific fine-tuning and reflects how models consolidate knowledge through exposure.