How can we probe LLM representations in channels that training did not target?
This explores how we can read what's happening inside an LLM through channels it was never trained to expose — its raw activations, hidden states, and internal representations — rather than just its text output.
This explores the gap between what a model says and what it internally represents — and the tools researchers use to pry open the second channel. The naive route, asking a model to report on itself, mostly fails: Can language models actually introspect about their own states? shows that self-reports usually echo patterns in the training data rather than any genuine inspection of internal state. The interesting exception is that lightweight introspection becomes possible only when a real causal chain links the internal state to the report (a model inferring 'I'm at low temperature' from its own output consistency). That sets the terms for the whole question: untargeted channels need to be probed structurally, not interviewed.
The most direct attack is to train a separate reader on the activations themselves. Can we decode what LLM activations really represent in language? builds a decoder that translates raw activations into plain-language answers — and crucially it's not just diagnostic, it's a control surface, since you can steer the model by running gradient descent against the decoder's read. A cousin of this idea appears in recommendation: Can LLMs explain recommenders by mimicking their internal states? aligns an LLM to another model not only by mimicking its outputs (behavior) but by ingesting its neural embeddings directly (intention) — probing the target's internal channel rather than just its decisions.
A different and cheaper probe is to watch the geometry of the activations as conditions change. Do language models sparsify their activations under difficult tasks? finds that hidden states get measurably sparser, in a localized way, as tasks drift out of distribution — meaning the activation pattern itself carries a readable signal about task difficulty that nobody trained it to emit. Mechanistic interpretability does something similar for bias: Do LLMs represent low-resource cultures through dominant cultural proxies? traces how low-resource cultures get represented internally through high-resource proxies, a structural distortion that persists even when the surface answer looks correct. Both show that the internal channel can contradict the output channel — which is exactly why probing it matters.
The deepest payoff is using these probes to catch failures that text output hides. Can LLMs understand concepts they cannot apply? documents models that explain a concept correctly, fail to apply it, and recognize their own failure — a pattern that only makes sense if the explanation and execution pathways are functionally disconnected inside the model. Likewise, Do large language models actually perform iterative optimization? shows that what looks like reasoning in latent space is often template pattern-matching, and Do large language models reason symbolically or semantically? shows reasoning collapsing when semantics are stripped away. None of these would be visible from a fluent answer alone. The thread worth taking away: the channels training didn't target are often where a model's honesty actually lives — and the live research frontier is less about asking models to introspect than about building external readers (decoders, surrogates, geometry monitors) that decode the internal channel against the model's will.
Sources 8 notes
LLM self-reports usually reflect human training distributions rather than actual internal processes. However, when a causal chain connects an internal state to accurate reporting—like inferring low temperature from output consistency—genuine lightweight introspection occurs without requiring consciousness.
LatentQA trains a decoder to answer natural language questions about LLM activations, enabling both interpretability (understanding what activations encode) and controllability (steering them via gradient descent). Critical design choices—activation masking, diverse training data, and faithful completions—proved essential for generalization.
RecExplainer trains LLMs via three alignment methods: behavior (mimicking outputs), intention (incorporating neural embeddings), and hybrid (combining both). The hybrid approach produces explanations that are simultaneously faithful to the target model and intelligible to users by balancing internal-state inspection with human-readable reasoning.
As task difficulty increases, LLM hidden states become substantially sparser in a localized, systematic way that correlates with task unfamiliarity and reasoning load. This sparsification acts as a selective filter stabilizing performance under OOD shift rather than a failure mode.
Mechanistic interpretability analysis reveals that low-resource cultures like Ethiopia and Algeria are structurally represented through high-resource cultural proxies in internal model states, not just output. This architectural bias persists even when models can produce correct surface-level answers.
Models can explain concepts accurately, fail to apply them, and recognize the failure—a triple pattern incompatible with human cognition. This indicates functionally disconnected explanation and execution pathways rather than simple knowledge gaps.
Research shows LLMs cannot perform iterative procedures in latent space. They recognize optimization problems as template-similar and emit plausible-looking but incorrect values, a failure mode that persists across model scale and training approaches.
When semantic content is decoupled from reasoning tasks, LLM performance collapses even with correct rules in context. Models rely on parametric commonsense and token associations rather than formal logical manipulation, constraining reasoning to training distribution semantics.