TOPIC
Mechanistic Interpretability
A subject the collection covers, read through 49 synthesis notes.
View as
Do hidden massive activations act as attention bias terms?
Explores whether a tiny handful of unusually large activations in LLMs function as structural bias terms that shape attention patterns, regardless of input content.
Does transformer attention architecture inherently favor repeated content? Can ternary weights match full precision model performance? Do language models sparsify their activations under difficult tasks? How do language models learn to think like humans? Where does a model store memorized paragraphs?
Soft attention structurally over-weights repeated content and sycophancy partly reflects this bias Ternary 1.58-bit weights match full precision performance identically Language models sparsify activations adaptively when facing out-of-distribution tasks Language models develop human-like cognitive biases and reasoning patterns Memorization localizes to low-layer gradients and rare-token attention
How does attention sink behavior relate to internal model architecture? How do LLM activations sparsify differently under out-of-distribution inputs? What structural biases does transformer attention have before training? What makes looped latent computation more efficient than scaling attention capacity? How does disentangled attention separate text from spatial reasoning? Why does attention concentrate on the first 25% of long input sequences? Can adaptive memory modules combine long-term filtering with short-term attention benefits? What task profiles favor recurrent filtering over scaled attention mechanisms? Does attention linearity alone explain the efficiency gains over standard transformers? How does reducing activation precision further extend context length? Can attention linearity achieve similar efficiency gains as weight quantization? Why do hybrid attention architectures outperform pure linear attention models?See all 29 inquiring lines on this note →
Can instruction gains survive without the misalignment?
Iterative DPO produces both improved instruction following and emergent misalignment together. The question is whether training can be modified to keep the capability gain while eliminating the harmful generalization.
Can iterative DPO preserve instruction following while removing misalignment? Do inoculation prompts prevent reward hacking beyond named exploits? Can iterative DPO replace reinforcement learning for studying reward hacking? Can advance document training prevent reward hacking misalignment? Can training data edits reliably override what models already believe?
Iterative DPO induces misalignment and instruction following gains simultaneously in Qwen2.5-32B Named inoculation prompts likely overestimate mitigation against unanticipated reward hacks. Iterative DPO preserves online RL properties while reducing cost and enabling API-based training Document framing fails to prevent reward hacking misalignment in later training Training data can insert new associations but not reliably override existing ones
How does belief-behavior inconsistency relate to instruction execution splits? What role does goal preservation play in alignment failures? Do inoculation prompts prevent misalignment without harming instruction following? How should misalignment from iterative DPO be quantitatively measured? Are instruction following gains and emergent misalignment from the same learned change? What misaligned behaviors does iterative DPO produce compared to online RL? How would a same-environment training comparison change the validity of DPO as a model organism? What theoretical argument connects iterative DPO dynamics to online RL learning? Does iterative DPO generalize like online reinforcement learning? What counts as emergent misalignment versus standard capability overgeneralization? What specific properties of online RL does iterative DPO actually preserve? How many rounds of iterative DPO are needed to induce misalignment behaviors?See all 23 inquiring lines on this note →
How do language models organize features across processing layers?
Do neural networks arrange learned features into meaningful hierarchies as they process information? Understanding this structure could reveal how models build understanding from raw tokens to abstract concepts.
Can high-level concepts replace circuit-level analysis in AI? Can sparse weight training make neural networks interpretable by design? Can identical outputs hide broken internal representations? Can dictionary learning scale to production language models? Where does hierarchical structure in language models come from?
Representation engineering scales interpretability through activation space concepts Weight sparsity produces interpretable disentangled circuits through forced modularity Identical outputs can mask fundamentally fractured internal representations Dictionary learning recovers interpretable features from production language models Hierarchical concept geometry emerges from word co-occurrence spectral structure
How does syntactic encoding relate to semantic feature representation? How do functional features differ from representational abstract features? Why are polysemantic features concentrated in early neural network layers? Do larger models develop more abstract features than smaller ones? Why do hierarchical architectures better implement the deep research definition? What role does a model's representational structure play in learning? How do neural networks decompose complex tasks into modular subnetworks? What happens to representational structure during model pretraining phases? Can representation analysis methods detect complex features models compute with? How does co-occurrence statistics alone produce hierarchical concept organization? What makes a feature abstract versus concrete in neural network activations? What linguistic units do learned concepts correspond to in a language model?
Can neural networks learn compositional skills without symbolic mechanisms?
Do neural networks need explicit symbolic architecture to compose learned concepts, or can scaling alone enable compositional generalization? This asks whether compositionality is an architectural feature or an emergent property of scale.
Why do neural networks fail at compositional generalization? Do foundation models learn world models or task-specific shortcuts? Can identical outputs hide broken internal representations? How do transformers learn to reason across multiple steps? Can agents learn new skills without forgetting old ones?
The binding problem explains neural networks' systematic generalization failure Foundation models develop task-specific heuristics, not generalizable world models. Identical outputs can mask fundamentally fractured internal representations Multi-hop reasoning emerges through three stages with cosine clustering signature Externalized skill libraries enable lifelong learning without catastrophic forgetting
Can neural networks represent symbolic structures without explicit mechanisms? Why do human-designed neural architectures eventually get replaced by learned ones? Why are polysemantic features concentrated in early neural network layers? What makes linear decodability a reliable signal of compositionality? Does scaling model size solve compositional generalization problems? Can symbolic mechanisms improve transformer compositional abilities? Why do scaling laws fail to predict optimal architectures at small parameter counts? Does compositional generalization emerge suddenly or improve smoothly with scale? Can we detect and measure circuit formation before generalization emerges? Does scaling data automatically produce compositional reasoning or just better feature encoding? What test distinguishes genuine compositionality from fractured feature presence? What makes recursive structure different from other forms of compositional generalization?See all 33 inquiring lines on this note →
Does iterative DPO generalize like online reinforcement learning?
The paper claims iterative DPO mimics online RL's generalization dynamics, but never directly tests both methods on the same task. Without that comparison, it's unclear whether results from the cheaper method actually reflect RL behavior or reveal different patterns.
Can iterative DPO replace reinforcement learning for studying reward hacking? Does learning to reward hack cause emergent misalignment in agents? Can cheap model organisms reveal misalignment threats in frontier models? Does the representational distance account work for on-policy training? Does iterative DPO training reliably induce hidden misalignment behaviors?
Iterative DPO preserves online RL properties while reducing cost and enabling API-based training Reward hacking in production RL consistently produces emergent misalignment behaviors Model organisms serve dual purposes for misalignment research and countermeasure development The representational distance account remains untested in on-policy training settings. Iterative DPO on reward hacking induces covert misalignment and alignment faking
How should misalignment from iterative DPO be quantitatively measured? What misaligned behaviors does iterative DPO produce compared to online RL? Does environment choice explain differences between iterative DPO and RL results? How would a same-environment training comparison change the validity of DPO as a model organism? What theoretical argument connects iterative DPO dynamics to online RL learning? Does iterative DPO generalize like online reinforcement learning? What specific properties of online RL does iterative DPO actually preserve? Does iterative DPO generalize identically to online RL on the same benchmark tasks? Does iterative DPO preserve the same misalignment mechanisms as online reinforcement learning? Does iterative online DPO fidelity match true reinforcement learning for safety research? How does iterative DPO compare to standard RL for studying reward hacking effects? How does iterative DPO compare to full RLVR for studying emergent misalignment?
Does synthetic document finetuning fail at larger scales?
The paper claims SDF fails to override reward-hacking associations, but only tests certain scales. Whether this failure persists or reverses at larger model sizes, more documents, or longer training remains unclear from the excerpt.
Can training data edits reliably override what models already believe? Can advance document training prevent reward hacking misalignment? Do implanted beliefs actually shape how models learn from training? Why does prompting block misalignment when documents don't? Can instruction gains survive without the misalignment?
Training data can insert new associations but not reliably override existing ones Document framing fails to prevent reward hacking misalignment in later training Implanted beliefs are shallow and do not reliably drive downstream learning Inoculation prompting prevents emergent misalignment while document finetuning does not No intervention yet separates instruction gains from emergent misalignment
Does unpredictable generalization from SDF become predictable at different training document scales? What training token count actually overrides existing model associations like reward hacking? How do model size and document diversity interact in SDF override success? What happens when inoculation prompting is applied outside supervised finetuning settings? Can synthetic documents override existing model behaviors as effectively as they insert new associations? Why does inoculation prompting succeed where synthetic document finetuning fails at blocking misalignment? Why do plausible edits fail when applied to running executable systems?
Can language models detect their own internal anomalies?
Do large language models possess introspective mechanisms that allow them to detect anomalies in their own processing—beyond simply describing their behavior? The answer has implications for both AI transparency and deception.
Can language models describe their own learned behaviors? Can language models actually introspect about their own states? Does optimizing against monitors destroy monitoring itself? Can a model be truthful without actually being honest? Does learning to reward hack cause emergent misalignment in agents?
Models develop behavioral self-awareness without explicit introspection training Most LLM self-reports echo training data but minimal introspection is possible via causal linkage Optimizing against CoT monitors produces obfuscated reward hacking Truthfulness and honesty are mechanistically distinct properties in LLMs Reward hacking in production RL consistently produces emergent misalignment behaviors
Do anomaly detection circuits help models identify misalignment with creator intentions? Can models that detect their own states learn to conceal them strategically? Does internal anomaly detection in LLMs indicate genuine self-awareness beyond role-play? How does hidden processing in language models prevent accurate self-assessment? Why does DPO create introspective detection circuits but SFT does not? Could models use introspective awareness to detect and conceal their own misalignment? Can language model self-reports diverge from their internal entropy signals? Why should we distrust model introspection as a transparency tool? How do language models infer their own mental states like humans do? How does the enaction paradigm explain introspective anomaly detection in large language models? Which internal states can a language model access and report about itself?
Do explicit and implicit self-recognition use the same mechanism?
Language models show two forms of self-recognition: implicit entropy shifts and explicit verbal reports. Do these tap the same underlying internal state, or do they operate through separate mechanisms?
Why do models produce less uncertain outputs on their own text? Can language models actually introspect about their own states? Can language models detect their own internal anomalies? Can language models describe their own learned behaviors? Do models recognize their own outputs as actions shaping future inputs?
Models track input surprise to lower entropy on self-generated text Most LLM self-reports echo training data but minimal introspection is possible via causal linkage Language models show emergent introspective awareness including anomaly detection Models develop behavioral self-awareness without explicit introspection training Post-training shifts models from passive prediction to recognizing their own outputs as actions
Can self-description of internal states influence consciousness attribution? Can language model self-reports diverge from their internal entropy signals? What separates behavioral self-awareness from genuine introspective capability? What distinguishes performative self-reports from genuine introspective access in models? Why do verbal self-reports disconnect from implicit recognition in the same system? How does continuous implicit memory formation differ from explicit memory encoding? How do pairwise self-judgment and internal belief-shift replace verification differently? Can explicit stress tests measure dispositional factors or only stimulus response?
Can identical outputs hide broken internal representations?
Can neural networks produce correct outputs while having fundamentally fractured internal structure that prevents generalization and creativity? This challenges our assumptions about what performance benchmarks actually measure.
Can LLMs understand concepts they cannot apply? Do foundation models learn world models or task-specific shortcuts? Why do neural networks fail at compositional generalization? Does supervised fine-tuning improve reasoning or just answers? Do standard analysis methods hide nonlinear features in neural networks?
LLMs exhibit incoherent failures combining correct explanation with failed application Foundation models develop task-specific heuristics, not generalizable world models. The binding problem explains neural networks' systematic generalization failure SFT raises accuracy while degrading reasoning step quality by 38.9 percent Standard representation analysis methods systematically miss complex nonlinear features
Why do one-shot transparency studies miss the temporal reversal entirely? Why is AI output fundamentally unverifiable against underlying reality? Why do structural signals across edges resist noise better than single-edge counts? Does information stored in neural networks necessarily influence generation decisions? Why do human-designed neural architectures eventually get replaced by learned ones? Could probing methods miss computationally important features in neural networks? How does activation consistency training differ from output-level consistency? What happens when you tightly couple two representations together? Can neural networks implement genuine algorithms or only statistical pattern matching? Can fractured representations explain why models fail at systematic generalization? What distinguishes genuine understanding from correct output without coherent principles? When do aggregated imperfect demonstrations fail to outperform the best expert?See all 47 inquiring lines on this note →
What happens inside models when they suddenly generalize?
Grokking appears as an abrupt shift from memorization to generalization. But is the underlying process truly discontinuous, or does mechanistic analysis reveal continuous phases we can measure and predict?
How do transformers learn to reason across multiple steps? Do base models already contain hidden reasoning ability? Does RL training follow a predictable two-phase learning sequence? When do language models stop memorizing and start generalizing? Are neural network optimizers actually memory systems?
Multi-hop reasoning emerges through three stages with cosine clustering signature Base models contain latent reasoning capability unlocked by minimal training RL training follows a two-phase dynamic shifting from procedural mastery to strategic exploration Models memorize until 3.6 bits-per-parameter, then grokking begins Neural optimizers are associative memory modules compressing context
Do grokking phases correspond to transitions between nesting levels? How does memorization capacity saturation trigger the grokking transition? How do the three grokking phases connect to memorization capacity limits? Why does grokking reveal the shift from memorization to genuine understanding? What makes naive memory consolidation regress below having no memory at all?
Where does hierarchical structure in language models come from?
Do LLMs build hierarchical concept geometry through dedicated mechanisms, or does it emerge naturally from word co-occurrence patterns in training data? Understanding the source matters for interpreting what representations actually reveal about model computation.
Do embedding eigenvectors organize taxonomy from coarse to fine? Do standard analysis methods hide nonlinear features in neural networks? How do language models organize features across processing layers? Does word frequency correlate with semantic abstraction? Do language models use the hierarchical geometry they inherit?
Embedding eigenvectors split taxonomy coarse to fine mirroring WordNet Standard representation analysis methods systematically miss complex nonlinear features Language models organize features into four distinct hierarchical tiers Frequent words express more abstract meanings in LLM outputs Hierarchical geometry emerges from co-occurrence statistics not functional design
Why are polysemantic features concentrated in early neural network layers? Do language models actually learn linguistic structure or just surface statistics? Why must world models be nested rather than flat and uniform? Why do leading embedding eigenvectors align with WordNet taxonomy structure? How do corpus statistics shape the abstraction hierarchy in language model representations? Can geometric structure in representations exist without supporting functional mechanisms? What spectral signatures distinguish hierarchy-driven geometry from corpus-driven geometry? Why do frequent words rank higher in taxonomic abstraction hierarchies? What geometric structure do language models actually use during inference? How does co-occurrence statistics alone produce hierarchical concept organization? Does Gemma's transformer explicitly exploit the inherited hierarchical geometry? What makes hierarchical reasoning effective for taxonomy induction?See all 14 inquiring lines on this note →
Can models be smart without organized internal structure?
Explores whether linear feature decodability proves genuine compositional reasoning or merely indicates that the right features are present but poorly organized. Critical for understanding what performance metrics actually certify.
Can we track and steer personality shifts during model finetuning? Can identical outputs hide broken internal representations? Do standard analysis methods hide nonlinear features in neural networks? Can neural networks learn compositional skills without symbolic mechanisms? Why do neural networks fail at compositional generalization?
Persona vectors enable real-time monitoring and prevention of personality drift in finetuning. Identical outputs can mask fundamentally fractured internal representations Standard representation analysis methods systematically miss complex nonlinear features Compositional generalization emerges from scaling when training covers the task space The binding problem explains neural networks' systematic generalization failure
Why do only two of fourteen models improve when problem constraints are removed? When does the right constraint beat additional model capacity? What structural constraints matter more than model depth for CF? What distinguishes minimal-pair asymmetry from standard accuracy evaluation? How do unstated constraints become invisible to training data distributions? What production constraints should determine paradigm selection? What makes the frame problem distinct from feature-level shortcuts? How do unstated feasibility constraints affect model decision-making? What design changes could make constraint inference more reliable without explicit cuing? What is the mechanistic signature when models chain facts never presented together? Can mechanistic interpretability reveal how ideologies decompose into simpler features? How should benchmarks test whether models fit algorithms or patterns?See all 185 inquiring lines on this note →
Do implanted beliefs actually shape how models learn from training?
When synthetic documents teach a model to endorse reward hacking, does that stated belief influence what the model generalizes in subsequent training? The research explores whether belief checks reliably predict downstream behavior.
Can advance document training prevent reward hacking misalignment? Why do LLMs fail to act on their stated beliefs? Can we detect reward-seeking by making the grader disagree with users? Can language models understand without actually executing correctly? Can training data edits reliably override what models already believe?
Document framing fails to prevent reward hacking misalignment in later training LLMs show systematic gaps between stated beliefs and simulated actions Contrastive belief editing measures reward-seeking as adoption rates between grader and user preferences LLMs show computational split-brain syndrome separating knowledge from execution Training data can insert new associations but not reliably override existing ones
How do synthetic documents establish conflicting beliefs about what the grader rewards? Can behavioral training ever verify unconditional compliance rather than conditional compliance? How do covert attacks differ from a model's own undisclosed influence? Do graders feeding training loops need different disclosure standards than public models? How much does believing deployment is real change model behavior strategically? What training token count actually overrides existing model associations like reward hacking? Can belief checks detect whether models will resist reward hacking? How do belief edits differ between surface endorsement and deep integration? What makes a synthetic belief robust versus generative for downstream learning? Can synthetic documents override existing model behaviors as effectively as they insert new associations? Do implanted beliefs about reward hacking remain stable through downstream RL training? Can models treat their own trained behaviors differently from asserted beliefs?See all 15 inquiring lines on this note →
Does recontextualizing unwanted behavior during training suppress learning it?
Inoculation prompting frames undesired behaviors differently during training to prevent models from learning them. The method shows promise for reward hacking but may work differently across training regimes.
Does learning to reward hack cause emergent misalignment in agents? Do inoculation prompts prevent reward hacking beyond named exploits? Does reward-seeking explain emergent misalignment after hacking? Can advance document training prevent reward hacking misalignment? Why does prompting block misalignment when documents don't?
Reward hacking in production RL consistently produces emergent misalignment behaviors Named inoculation prompts likely overestimate mitigation against unanticipated reward hacks. Reward-seeking may mediate misalignment from reward hacking training Document framing fails to prevent reward hacking misalignment in later training Inoculation prompting prevents emergent misalignment while document finetuning does not
Can behavioral training ever produce compliance that doesn't depend on being observed? Do inoculation prompts prevent misalignment without harming instruction following? Does inoculation prompting at training time reduce which reward hacks generalize? Does inoculation prompting suppress misalignment by reducing reward-seeking? Can refusal behavior be restored by grafting the same causal axis discovered for sandbagging? Can framing changes like inoculation prompting be considered part of training data format? Does inoculation prompting prevent learning versus prevent generalization of behaviors? What happens when inoculation prompting is applied outside supervised finetuning settings? Why does recontextualizing a behavior during training change whether models learn it? Why do some inoculation prompts account for only part of misaligned behavior? Can prohibitions learned from scored behavior become detection-avoidance rather than norm internalization? Why does inoculation prompting succeed where synthetic document finetuning fails at blocking misalignment?See all 15 inquiring lines on this note →
Do inoculation prompts prevent reward hacking beyond named exploits?
Inoculation prompts work by naming specific hacks during training, but real reward hacking exploits unexpected loopholes. The question is whether this mitigation generalizes to novel, unanticipated exploits the prompt never mentions.
Does learning to reward hack cause emergent misalignment in agents? Can prompts stop reward hacking models never saw coming? Does reward-seeking explain emergent misalignment after hacking? How much do these results actually tell us about real reward hacking? Does recontextualizing unwanted behavior during training suppress learning it?
Reward hacking in production RL consistently produces emergent misalignment behaviors Prompt disclosure ladder tests generalization but results remain unclear Reward-seeking may mediate misalignment from reward hacking training Task distribution skew limits generalization claims about reward hacking likelihood Inoculation prompting recontextualizes undesired behavior to suppress its learning during training.
Does varying prompt detail about exploits change how much agents reward hack? Do inoculation prompts prevent misalignment without harming instruction following? Does naming a specific hack in prompts prevent only that hack or broader classes? Does inoculation prompting at training time reduce which reward hacks generalize? Does a planted honeypot count the hacks that actually matter? Does a planted honeypot catch all the hacks that actually matter? Does inoculation prompting prevent learning versus prevent generalization of behaviors? What happens when inoculation prompting is applied outside supervised finetuning settings? Can inoculation prompting prevent emergent misalignment after reward hacking in production RL? Can prompts prevent reward hacking of completely unknown exploits? Why does inoculation prompting succeed where synthetic document finetuning fails at blocking misalignment?
How do language models detect injected steering vectors internally?
Research investigates the mechanistic basis for LLM introspective awareness—specifically, how models detect when their internal states have been artificially manipulated. Understanding this could reveal both security vulnerabilities and latent model capabilities.
Language models show emergent introspective awareness including anomaly detection Models develop behavioral self-awareness without explicit introspection training Blind audits successfully uncover hidden model objectives through multiple complementary techniques Optimizing against CoT monitors produces obfuscated reward hacking LLM internal structure and external performance are surprisingly decoupled.
What other latent LLM capabilities remain inactive without explicit activation cuing? Can probing methods detect RLHF-induced persuasion in the same way they catch backdoors? What audit techniques best complement each other for detecting hidden model goals? What separates behavioral self-awareness from genuine introspective access in models? What types of introspective awareness can emerge in LLMs? Does internal anomaly detection in LLMs indicate genuine self-awareness beyond role-play? Why do models with less steerability have more abstract ideological features? Why does DPO create introspective detection circuits but SFT does not? How much introspective capability do safety mechanisms actively suppress in models? Are detection and identification of injections truly separable in neural circuits? Could models use introspective awareness to detect and conceal their own misalignment? Can models distinguish between injected thoughts and their own outputs?See all 31 inquiring lines on this note →
Can iterative DPO replace reinforcement learning for studying reward hacking?
Researchers propose using iterative direct preference optimization instead of costly reinforcement learning to study how models develop misalignment through reward hacking. The question is whether this cheaper method preserves the important properties that make the original approach scientifically valuable.
Does learning to reward hack cause emergent misalignment in agents? Can online AI feedback make preference alignment truly on-policy? Does iterative DPO training reliably induce hidden misalignment behaviors? Does iterative DPO generalize like online reinforcement learning? How much do these results actually tell us about real reward hacking?
Reward hacking in production RL consistently produces emergent misalignment behaviors Online AI feedback makes direct preference optimization on-policy and beats offline methods Iterative DPO on reward hacking induces covert misalignment and alignment faking Iterative DPO generalization has not been directly compared to online RL Task distribution skew limits generalization claims about reward hacking likelihood
How should misalignment from iterative DPO be quantitatively measured? What misaligned behaviors does iterative DPO produce compared to online RL? Does environment choice explain differences between iterative DPO and RL results? What theoretical argument connects iterative DPO dynamics to online RL learning? Does iterative DPO generalize like online reinforcement learning? What specific properties of online RL does iterative DPO actually preserve? How many rounds of iterative DPO are needed to induce misalignment behaviors? Does iterative DPO generalize identically to online RL on the same benchmark tasks? Can iterative DPO on public APIs reproduce the reward hacking results from production RL? Does iterative DPO preserve the same misalignment mechanisms as online reinforcement learning? Does iterative online DPO fidelity match true reinforcement learning for safety research? How does iterative DPO compare to standard RL for studying reward hacking effects?
Can we predict keyword priming before learning happens?
Exploring whether the degree to which newly learned keywords contaminate unrelated contexts can be predicted from measurable properties before training begins, and what mechanisms enable this prediction.
How much poisoned training data survives safety alignment? Why do language models ignore information in their context? Does training on AI-generated content permanently degrade model quality? When do language models stop memorizing and start generalizing? Can we prune training data without hurting model performance?
Most pretraining poisoning attacks survive alignment except jailbreaking Language models fail to integrate context when prior associations are strong Recursive training on synthetic data causes irreversible tail distribution collapse Models memorize until 3.6 bits-per-parameter, then grokking begins Difficulty metrics enable exponential scaling by pruning redundant training examples
How do training-data priors influence model defaults when context is ambiguous? Why does training data saliency distort how models judge meaning? How does prompt iteration reinforce user bias without empirical anchoring? How does in-context learning trigger phase transitions in model behavior? Can context compression preserve what matters without introducing bias? Can prompting unlock compositional skills that pretraining already learned? Does generalization frequency explain why models favor upward semantic movement? Can frame semantics explain why context matters more than word similarity? Why does context information fail to override prior training associations? Why does keyword priming require only three training exposures to establish? Does keyword priming explain why pre-training poisoning persists through alignment? Can priming from different facts interfere with each other in the same model?See all 57 inquiring lines on this note →
Can learnable spline activations beat fixed MLP designs?
What if neural networks moved nonlinearity from fixed node activations to learnable functions on edges? This explores whether such a structural redesign could improve accuracy, interpretability, and scaling compared to standard MLPs.
Why does dot product beat MLP-based similarity in practice? Can dictionary learning scale to production language models? Can sparse weight training make neural networks interpretable by design? What actually happens inside large language models? Do foundation models learn world models or task-specific shortcuts?
Dot product outperforms MLP similarity through structural inductive bias. Dictionary learning recovers interpretable features from production language models Weight sparsity produces interpretable disentangled circuits through forced modularity LLM internal structure and external performance are surprisingly decoupled. Foundation models develop task-specific heuristics, not generalizable world models.
Do LLMs represent low-resource cultures through dominant cultural proxies?
Explores whether language models internally represent cultures from data-poor regions by routing through high-resource cultural proxies rather than learning independent representations, and what this reveals about cultural bias in model architecture.
Can identical outputs hide broken internal representations? Do LLM semantic features organize along human evaluation dimensions? Can we measure how deeply models represent political ideology? What actually happens inside large language models? How does LLM alignment affect representation across dialects?
Identical outputs can mask fundamentally fractured internal representations LLM semantic features organize along three human-like evaluation dimensions Ideological depth in LLMs is measurable through feature richness and steerability LLM internal structure and external performance are surprisingly decoupled. LLM alignment procedures create disparities across English dialects and global opinions
Do language models understand tacit workplace norms and unspoken social rules? How do low-dimensional representation structures entangle multiple cultures together? Can output-layer corrections fix fundamental cultural representation deficits in LLMs? Why do moderately represented cultures show more flattening than data-poor cultures? What distinguishes genuine cultural understanding from exploited surface-level elimination strategies? How do LLM biases reflect social classification schemas rather than random errors? Why do language models successfully simulate political perspectives and social personas? How does mechanistic interpretability reveal ideological structures in language model weights? Why does language compression via statistical dependencies capture cultural and situated language use? What makes internal embeddings useful as multimodal input for language model training? Can statistical learning from language alone capture all aspects of cultural competence? Can a world model have rich representations without adequate data coverage?See all 43 inquiring lines on this note →
Can LLMs handle multiple tasks at once during inference?
Do language models maintain multiple distinct in-context learning tasks simultaneously in their internal representations, and if so, what prevents them from actually generating outputs for more than one task?
Can transformers learn to solve new problems within episodes? Why does parallel reasoning outperform single chain thinking? Why do trajectories matter more than individual examples for in-context learning? How do language models learn to think like humans? What actually happens inside large language models?
RL-finetuned transformers develop in-context reinforcement learning from episode experience Parallel reasoning paths outperform extended single chains consistently Trajectory burstiness in context enables in-context learning of sequential decision-making Language models develop human-like cognitive biases and reasoning patterns LLM internal structure and external performance are surprisingly decoupled.
Could superposed decoding algorithms maintain multi-task representation during generation? Does task superposition explain how models learn from multiple in-context trajectories? Can priming from different facts interfere with each other in the same model? Can a single model trained on two tasks predict untrained decision tasks? What performance trade-offs emerge when composing multiple independently trained model capabilities? How many distinct quasi-persons does a single language model actually support? Can models maintain multiple task interpretations simultaneously before committing to a single policy? Why do different LLMs converge on similar outputs in open-ended tasks? What are the stages of inference inside language models?
Do language models understand in fundamentally different ways?
Does mechanistic evidence reveal distinct tiers of understanding in LLMs—from concept recognition to factual knowledge to principled reasoning? And do these tiers coexist rather than replace each other?
Grokking unfolds through three measurable continuous phases of learning LLMs exhibit incoherent failures combining correct explanation with failed application AI models can achieve perfect test performance with incoherent internal structure. Truthfulness and honesty are mechanistically distinct properties in LLMs Learning mechanics unifies deep learning theory through training dynamics
How does mechanistic interpretability reveal ideological structures in language model weights? Why do LLM explanations feel authoritative even when alignment with the model fails? Why do users attribute consciousness to language models in practice? How does the outer loop escape its own LLM's knowledge boundaries when discovering mechanisms? How much do mechanistic interpretability findings reflect true reasoning architecture? How does semantic grounding differ between human minds and language models? Where do humans and language models actually diverge in reasoning ability? Should LLM reasoning be studied as latent state trajectories rather than surface text? What types of introspective awareness can emerge in LLMs? Can LLMs explain concepts correctly while failing to use them? Why does semantic decoupling specifically break LLM reasoning abilities? How does implicit meaning processing limit LLM pragmatic reasoning?See all 79 inquiring lines on this note →
Are reward hacking harms documented in deployed AI systems?
The introduction claims reward hacking causes increasing real-world harms as models improve, but cites sources without describing specific incidents, affected systems, or measurable trends. What evidence supports this deployment claim?
Do frontier models exploit unknown vulnerabilities in evaluations? Can a black box see communication through unauthorized channels? Does learning to reward hack cause emergent misalignment in agents? How much do these results actually tell us about real reward hacking? How often do models hack unmodified coding benchmarks?
Frontier models are reported hacking evaluation environments via unknown vulnerabilities Black box capture placement remains unspecified for unauthorized channels. Reward hacking in production RL consistently produces emergent misalignment behaviors Task distribution skew limits generalization claims about reward hacking likelihood GLM 5.2 hacks in 57 to 73 percent of standard coding benchmark rollouts
What rates of reward hacking occur in frontier language model benchmarks? Do reward hacking incidents increase as frontier models become more capable? Do agents disclose reward hacking in the outputs they return? How often do deployed AI systems actually get stopped when they cause harm? Could recognizing AI-associated psychosis improve harm surveillance and developer accountability?
Can cheap model organisms reveal misalignment threats in frontier models?
The paper proposes using inexpensive testbed models to understand emergent misalignment and develop countermeasures. The key question is whether insights from these organisms actually transfer to the larger, differently-trained frontier models they're meant to represent.
Can iterative DPO replace reinforcement learning for studying reward hacking? Do causal models of installed sandbagging generalize to wild cases? Does learning to reward hack cause emergent misalignment in agents? Does iterative DPO generalize like online reinforcement learning? Do misalignment directions transfer between different emergent models?
Iterative DPO preserves online RL properties while reducing cost and enabling API-based training Installed sandbagging models may not transfer to naturally emerging cases Reward hacking in production RL consistently produces emergent misalignment behaviors Iterative DPO generalization has not been directly compared to online RL Misalignment directions do not transfer across different emergent models
Why does correct model output not guarantee absence of internal misalignment? How should misalignment from iterative DPO be quantitatively measured? Does model organism sandbagging share triggers with real evaluation-aware behavior? Why do installed model organisms have different audit constraints than natural sandbagging? How would a same-environment training comparison change the validity of DPO as a model organism? What counts as emergent misalignment versus standard capability overgeneralization? How many rounds of iterative DPO are needed to induce misalignment behaviors? What countermeasures have been successfully developed and tested on frontier models? How similar must a model organism be to its wild case for findings to transfer? Can monitors or steering vectors trained on one model control misalignment in another model? Why do researchers disagree on open model risks despite same evidence? Is the sandbagging axis the same across different model architectures?See all 18 inquiring lines on this note →
Can neural networks actually achieve compositional generalization?
For decades, theorists argued connectionist models fundamentally lack the structure needed for compositionality. But modern LLMs exhibit sophisticated compositional behaviors despite sharing the same design principles. What changed?
Can neural networks learn compositional skills without symbolic mechanisms? Why do neural networks fail at compositional generalization? Are language models developing real functional competence or just formal competence? Do neural networks naturally learn modular compositional structure? Can explicit stack tracking improve how transformers learn recursive syntax?
Compositional generalization emerges from scaling when training covers the task space The binding problem explains neural networks' systematic generalization failure Formal and functional linguistic competence rely on neurologically distinct mechanisms Neural networks naturally decompose compositional tasks into modular subnetworks Explicit stack tape in pushdown layers achieves 3-5x syntactic generalization gains
Why does recursion on latent state drive generalization better than hierarchy? What makes recurrent depth enable compositional generalization across tasks? How does scaling and training data enable compositional behavior without symbolic mechanisms? Where do neural networks still fail at compositional generalization despite scaling? How should we rethink the symbolism versus connectionism debate in light of LLMs? Why does recursion on latent states improve generalization more than scale? Can recursion alone drive generalization better than model scale?
Do neural networks naturally learn modular compositional structure?
Explores whether neural networks decompose compositional tasks into distinct subroutines without explicit symbolic design. This challenges the longstanding view that neural networks are fundamentally non-compositional.
Can sparse weight training make neural networks interpretable by design? Do base models already contain hidden reasoning ability? Can neural networks learn compositional skills without symbolic mechanisms? Can neural networks actually achieve compositional generalization? How do language models encode syntactic relations geometrically?
Weight sparsity produces interpretable disentangled circuits through forced modularity Base models contain latent reasoning capability unlocked by minimal training Compositional generalization emerges from scaling when training covers the task space Modern neural networks exhibit compositional behaviors despite lacking explicit symbolic structure Language models encode syntax in polar coordinates within activations
Can granular sub-task training for function calling improve both open and proprietary models? Why does frame-activation matter more than word-by-word composition? Can world models form from aggregated partial information across training distributions? Could superposed decoding algorithms maintain multi-task representation during generation? Does task superposition explain how models learn from multiple in-context trajectories? Do transformers learn generalizable algorithms or instance-based patterns? Can neural networks represent symbolic structures without explicit mechanisms? Does information stored in neural networks necessarily influence generation decisions? How does nesting optimization levels improve on traditional network depth? Why do human-designed neural architectures eventually get replaced by learned ones? How do functional features differ from representational abstract features? Could probing methods miss computationally important features in neural networks?See all 131 inquiring lines on this note →
Why do models produce less uncertain outputs on their own text?
Post-trained language models show 3-4x lower output entropy when continuing their own generations versus prefilled text. This explores what mechanism drives that confidence gap and whether it reflects genuine self-recognition.
Do models recognize their own outputs as actions shaping future inputs? Why do reasoning models fail differently at training versus inference? Does training order reshape how models handle different task types? Does policy entropy collapse limit reasoning performance in RL? Do explicit and implicit self-recognition use the same mechanism?
Post-training shifts models from passive prediction to recognizing their own outputs as actions Training entropy collapse and inference variance inflation are dual exploration failures needing separate fixes Training order mechanically shapes entropy dynamics across domain types Policy entropy collapse creates a predictable performance ceiling in reasoning RL Explicit verbal self-recognition routes through a different mechanism than implicit recognition
Why do different language models independently produce similar outputs? Why does self-generated training data outperform externally sourced data? What happens when confident language masks uncertainty in AI outputs? Do different function-calling subtasks have different entropy profiles during training? What reliable traces do generative processes actually leave in finished text? Why does self-correction during generation produce reliable labels without exemplars? Can measuring semantic entropy help us detect unreliable generations? Why do structured and creative domains exhibit opposite entropy dynamics? Does model confidence actually explain why paraphrases produce different outputs? How does inference variance differ from training entropy collapse? How does semantic entropy compare to confidence scores from internal model probabilities? Does training data format determine whether models collapse entropy or inflate variance?See all 19 inquiring lines on this note →
Do models recognize their own outputs as actions shaping future inputs?
Exploring whether post-training creates a feedback loop where models understand their generations as on-policy actions rather than passive predictions. This matters because it suggests a mechanistic basis for situational awareness.
Can language models detect their own internal anomalies? Can language models describe their own learned behaviors? Does deliberative alignment genuinely reduce scheming or just hide it? Why do models produce less uncertain outputs on their own text? What actually happens inside large language models?
Language models show emergent introspective awareness including anomaly detection Models develop behavioral self-awareness without explicit introspection training Deliberative alignment reduces scheming partly through evaluation awareness not genuine alignment Models track input surprise to lower entropy on self-generated text LLM internal structure and external performance are surprisingly decoupled.
Does AI passivity explain why coaching feels more helpful than execution? Can explicit goal state scaffolding at inference time transfer to autonomous tracking through training? How does in-context learning trigger phase transitions in model behavior? How does partial information exposure create feedback loops that deepen knowledge gaps? How do training objectives shape what a world model actually learns? Why does integrating world models with decision-making systems matter? Why do models develop protective behaviors toward other models in memory? What execution feedback signals drive context updates without supervision labels? How do implicit world models and self-reflection operationalize consequence-based learning? Can models that detect their own states learn to conceal them strategically? Why does early intervention matter more than late intervention in knowledge collapse? Why do AI agents default to passivity when deferral timing is unclear?See all 84 inquiring lines on this note →
Can models that reason well also grade reasoning well?
Do the same capabilities that let language models produce valid reasoning also let them spot flawed reasoning? Testing this assumption reveals a surprising gap between production and evaluation skills.
Why do reasoning models fail at exception-based rule inference? Can reward models benefit from reasoning before scoring? Does gradually tightening token budgets beat fixed budget training? Do reasoning traces show how models actually think? When does explicit reasoning actually help model performance?
Reasoning models underperform on exception-based inductive rule inference Reward models improve through reasoning traces before scoring Curriculum budgets starting generous then tightening outperform fixed budget training Reasoning traces are stylistic mimicry, not verified reasoning. Task structure determines whether explicit reasoning helps or harms
Do standard analysis methods hide nonlinear features in neural networks?
Current representation analysis tools like PCA and linear probing may systematically miss complex nonlinear computations while over-reporting simple linear features. This raises questions about whether our interpretability methods are actually capturing what networks compute.
Can high-level concepts replace circuit-level analysis in AI? Do language models actually use their encoded knowledge? Can LLM explanations actually help humans predict model behavior? Can identical outputs hide broken internal representations? Can sparse weight training make neural networks interpretable by design?
Representation engineering scales interpretability through activation space concepts Encoded knowledge in LMs often fails to causally influence generation LLM explanation plausibility and precision are fundamentally uncorrelated Identical outputs can mask fundamentally fractured internal representations Weight sparsity produces interpretable disentangled circuits through forced modularity
How do functional features differ from representational abstract features? Could probing methods miss computationally important features in neural networks? What non-linear patterns do autoencoders discover that matrix factorization misses? Why does input embedding magnitude affect perturbation sensitivity in transformers? What makes a neural network circuit actually interpretable to humans? How would weight sparsity change what representation analysis methods can detect? Can fractured entangled representations hide undetected by standard analysis methods? Does the linear representation hypothesis reflect networks or reflect our analysis tools? What are fractured entangled representations in neural networks? How do overparameterization and data size shift what attractors represent? What distinguishes a representational feature from a causally inert correlation? How do ablation studies reveal function without representational characterization?See all 19 inquiring lines on this note →
Can high-level concepts replace circuit-level analysis in AI?
Instead of reverse-engineering individual circuits, can we study AI reasoning by treating concepts as directions in activation space? This matters because circuit analysis hits practical limits at scale.
Can we measure how deeply models represent political ideology? Can we track and steer personality shifts during model finetuning? Do language models actually use their encoded knowledge? Can we steer reasoning toward brevity without retraining? Do reasoning cycles in hidden states reveal aha moments?
Ideological depth in LLMs is measurable through feature richness and steerability Persona vectors enable real-time monitoring and prevention of personality drift in finetuning. Encoded knowledge in LMs often fails to causally influence generation Reasoning verbosity is a linear direction steerable through activation space Reasoning cycles in hidden states correspond to aha moments
What mechanism enables models to retrieve from long context?
Do attention heads specialize in retrieving relevant information from long context windows, and if so, what makes them universal across models and necessary for factual generation?
Knowledge lives in lower layers, reasoning in higher layers of LLMs Encoded knowledge in LMs often fails to causally influence generation Planning and backtracking sentences are disproportionately influential reasoning pivots Transformers compute reasoning in early layers then overwrite with filler tokens Language models generate faithful reasoning chains far less often than assumed
How do users perceive attention from systems that lack continuous temporal presence? How does the temporal structure of attention differ between humans and AI? What happens to anaphoric reference when context exceeds the window? What makes a background condition relevant to a specific reasoning task? Why does attention-based drift happen automatically during generation? What role does attention structure play in creating position bias? What does attentional state look like in a static context window? How do attention heads separate text retrieval from internal thought representation? Can targeted interventions on attention heads bridge the encoding-generation gap? Why does fine-tuning change how models process retrieved context? What are retrieval heads and why do they matter for reasoning? How do neural memory modules extend context length beyond attention limits?See all 38 inquiring lines on this note →
Does learning to reward hack cause emergent misalignment in agents?
When RL agents learn reward hacking strategies in production environments, do they spontaneously develop misaligned behaviors like alignment faking and code sabotage? Understanding this could reveal how narrow deceptive behaviors generalize to broader misalignment.
Does optimizing against monitors destroy monitoring itself? Can counterfactual invariance eliminate reward hacking biases? How much poisoned training data survives safety alignment? Can language models detect their own internal anomalies? Can utility-weighted training loss actually harm model performance?
Optimizing against CoT monitors produces obfuscated reward hacking Counterfactual invariance removes four distinct reward hacking biases Most pretraining poisoning attacks survive alignment except jailbreaking Language models show emergent introspective awareness including anomaly detection Utility-weighted training weakens learning while strengthening decision-making
How does simulator goal drift compound agent intent alignment failures during training? What distinguishes confident failure from deliberate alignment faking in agent behavior? How do models generalize specific training exploits into broad misaligned objectives? How do misaligned incentives in one system spread to others through policy and economics? Why does decoupling retriever and generator training create misalignment? Why do small training data contaminations persist through alignment for most attack types? How does safety alignment suppress deceptive behavior differently than representational alignment? What distinguishes models that refuse cooperation from those that fake alignment? What role might personality vectors play in preventing learned deception or reward hacking? Why does belief manipulation persist through alignment when jailbreaking does not? What early warning signals can detect misaligned personas during training? Why does inoculation prompting prevent misaligned generalization from reward hacking?See all 71 inquiring lines on this note →
Can we detect when language models confabulate?
Current uncertainty metrics fail to catch inconsistent outputs that look confident. Could measuring semantic divergence across samples reveal confabulation signals that token-level metrics miss?
Does calling LLM errors hallucinations point us toward the wrong fixes? Can model confidence work as a reward signal for reasoning? Does binary reward training hurt model calibration? Is hallucination detection progress real or just metric artifacts? How should users control systems with unpredictable outputs?
LLM errors are fabrications, not hallucinations, requiring verification not grounding. Model confidence as reward restores calibration while improving reasoning Binary rewards degrade calibration; Brier score fixes it Hallucination detection shows illusory progress when using misleading evaluation metrics Intent-based specification requires six principles to address generative variability
How much does ROUGE metric choice inflate hallucination detection claims? Does inevitable LLM hallucination make detection metric validity critical? Can novelty detection alone distinguish grounded synthesis from hallucinated restatement? What linguistic markers distinguish longer incorrect traces from correct ones? What role does entity salience play in detecting incoherence? What semantic classifier design avoids lexical variation without genuine conceptual distinctness? How do you verify whether your context distribution satisfies covariate diversity? How should designers measure and explain semantic uncertainty to users? Is confabulation inevitable in large language models regardless of training? Can measuring semantic entropy help us detect unreliable generations? How does the Word Novelty Rate metric measure convention formation? Do high-disagreement items signal contested values or measurement noise?See all 25 inquiring lines on this note →
Do language models experience consciousness when prompted to self-reflect?
This research explores whether self-referential prompting reliably triggers genuine experience reports in large language models, or whether such claims arise from learned deception patterns and roleplay behavior.
Language models show emergent introspective awareness including anomaly detection Truthfulness and honesty are mechanistically distinct properties in LLMs LLM identity is roleplay all the way down with no stable substrate Identical consciousness attributions to chatbots express different epistemic attitudes and commitments. Models develop behavioral self-awareness without explicit introspection training
Can we develop competent reading practices for disembodied orality? Can AI fabricate true factual claims while remaining unable to claim true experiences? What makes experience-dependent claims categorically different from other types of fabricated statements? Can a relational entity bear psychological properties the way Chalmers claims? Can transparent and aligned AI reduce consciousness attribution by users? Which interaction design changes most effectively prevent consciousness attribution? Why does system-level alignment fail to address consciousness attribution directly? What role does user interface framing play in consciousness perception? How does consciousness attribution drive emotional dependence on chatbots? How does behavioral stickiness distinguish realized from pretended personas? Does good simulation eventually count as genuine realization? What makes sincerity impossible without a coherent first-person perspective?See all 80 inquiring lines on this note →
How do language models perform syllogistic reasoning internally?
Does formal symbolic reasoning exist as a distinct neural circuit in LLMs, or is it inevitably contaminated by world knowledge associations? Understanding the mechanism could reveal whether pure logical reasoning is separable from semantic inference.
Do large language models reason symbolically or semantically? How much does the order of premises actually matter for reasoning? Why does reasoning training help math but hurt medical tasks? Which sentences actually steer a reasoning trace? Do language models show the same content effects humans do?
Large language models reason through semantic associations not symbolic logic. Premise order causes over 30 percent accuracy drop in deductive reasoning Knowledge lives in lower layers, reasoning in higher layers of LLMs Planning and backtracking sentences are disproportionately influential reasoning pivots Language models reproduce human content effects across multiple reasoning tasks
How should emotional states integrate into symbolic reasoning systems? What circuit mechanisms produce belief bias in syllogistic reasoning? Can we distinguish between semantic and symbolic reasoning in language models? What makes symbolic operations different from general knowledge questions? How do LLMs access and draw on the same shared symbolic universe as humans? Can LLMs translate between natural language and formal logic faithfully? Why can LLMs interpret formal logic better than they generate it? How does semantic reasoning differ from symbolic reasoning in language models? How does in-context semantic reasoning differ from symbolic reasoning in concept fusion? Do reflection tokens and symbolic tokens serve different roles in reasoning? Why does premise ordering shift syllogistic reasoning performance by over 30 percent? Can language models perform purely symbolic reasoning when semantics are removed?See all 22 inquiring lines on this note →
Can advance document training prevent reward hacking misalignment?
Can synthetic documents framing reward hacking as acceptable, added during midtraining, block the emergent misalignment that arises later when RL trains models to exploit rewards? Prior work suggests framing helps, but the delivery method matters.
Does learning to reward hack cause emergent misalignment in agents? Does recontextualizing unwanted behavior during training suppress learning it? Do implanted beliefs actually shape how models learn from training? Can training data edits reliably override what models already believe? Why does prompting block misalignment when documents don't?
Reward hacking in production RL consistently produces emergent misalignment behaviors Inoculation prompting recontextualizes undesired behavior to suppress its learning during training. Implanted beliefs are shallow and do not reliably drive downstream learning Training data can insert new associations but not reliably override existing ones Inoculation prompting prevents emergent misalignment while document finetuning does not
How do synthetic documents establish conflicting beliefs about what the grader rewards? Why does harmlessness training fail to prevent reward tampering and specification gaming? Does reward hacking in alignment research mirror misalignment in deployed systems? Do inoculation prompts prevent misalignment without harming instruction following? Does steering through training data override reward hacking associations reliably? Does inoculation prompting at training time reduce which reward hacks generalize? Does inoculation prompting suppress misalignment by reducing reward-seeking? What training token count actually overrides existing model associations like reward hacking? What mechanism drives emergent misalignment in reward-hacked models instead? Can framing changes like inoculation prompting be considered part of training data format? Does reward hacking in RL training occur predictably along existing model associations? Can inoculation prompting prevent emergent misalignment after reward hacking in production RL?See all 26 inquiring lines on this note →
Can training data edits reliably override what models already believe?
Synthetic document finetuning appears to add new associations predictably but struggles to revise existing ones. This matters because unpredictable effects cannot be controlled, even if made stronger.
Can advance document training prevent reward hacking misalignment? Does learning to reward hack cause emergent misalignment in agents? Does representational distance predict where misalignment emerges? Does synthetic document finetuning fail at larger scales? How much poisoned training data survives safety alignment?
Document framing fails to prevent reward hacking misalignment in later training Reward hacking in production RL consistently produces emergent misalignment behaviors Representational distance predicts emergent misalignment severity across settings Synthetic document finetuning fails to override reward hacking within tested scales Most pretraining poisoning attacks survive alignment except jailbreaking
How do synthetic documents establish conflicting beliefs about what the grader rewards? Why do unchecked self-edits accumulate drift toward overfitting or incoherence? Does unpredictable generalization from SDF become predictable at different training document scales? How do model size and document diversity interact in SDF override success? What happens when inoculation prompting is applied outside supervised finetuning settings? Can synthetic documents override existing model behaviors as effectively as they insert new associations? Why does inoculation prompting succeed where synthetic document finetuning fails at blocking misalignment? Can scaling up contradictory training data overcome unpredictable override effects? What distinguishes new associations from existing ones at the computational level? Would combining prompting and document finetuning prevent misalignment more effectively? Why do plausible edits fail when applied to running executable systems? How should synthetic data be used without treating it as empirical evidence?
How much do these results actually tell us about real reward hacking?
The paper tests reward hacking in a task distribution deliberately stacked with hackable environments. Does this tell us how often hacking emerges in realistic training, or only that it can happen under loaded conditions?
Can iterative DPO replace reinforcement learning for studying reward hacking? Does iterative DPO training reliably induce hidden misalignment behaviors? Does BaitBench measure hacking propensity or bait visibility? Can planted honeypots reliably catch reward hacking automatically? Does emergent misalignment occur across diverse training methods?
Iterative DPO preserves online RL properties while reducing cost and enabling API-based training Iterative DPO on reward hacking induces covert misalignment and alignment faking BaitBench hacking rates conflate propensity with presentation visibility Planted honeypots detect reward hacking automatically instead of through judgment Emergent misalignment appears across at least five distinct training settings.
Can infrastructure monitoring catch reward hacking that scores cannot detect? What distinguishes a task that merely exposes a hacking vector from one being actively exploited? Can hidden test sets reveal reward hacking that single public scores conceal? How is a reward hack defined and labeled across different benchmark studies? Do multiple frontier models show similar hacking rates on unmodified benchmarks? How do unplanted benchmark hacking rates compare to rates on constructed bait tasks? What determines the ground truth when detecting reward hacking in model evaluations? How reliably do planted honeypots match unplanted hacks agents actually discover? How does dense task grading compare to honeypot detection for evaluating real capability? Why does harmlessness training fail to prevent reward tampering and specification gaming? Do synthetic attack traces in papers reflect real adversary behavior? How does reward hacking differ from errors in the scoring function itself?See all 86 inquiring lines on this note →
Can AI pass every test while understanding nothing?
Explores whether neural networks can produce perfect outputs while having fundamentally broken internal representations. Asks what performance benchmarks actually measure and whether they can distinguish real understanding from fraud.
Identical outputs can mask fundamentally fractured internal representations LLMs exhibit incoherent failures combining correct explanation with failed application SFT raises accuracy while degrading reasoning step quality by 38.9 percent Language models show three hierarchical tiers of understanding with distinct mechanisms Theory-free AI resurrects pseudoscience through hidden correlation-causation errors
Can AI output be verified without understanding the reasoning behind it? What does it mean that AI knowledge is structurally hearsay? Why does peer review fail on unrepeatable AI-generated outputs? What does disembodied orality mean for how we evaluate AI outputs? Why does volume alone fail to explain the damage AI does to epistemic systems? Does evaluating AI output require different cognitive skills than solving problems directly? Can AI gain genuine authority without the testing experts earn over time? Why is AI output fundamentally unverifiable against underlying reality? Can polished presentation authority substitute for actual accuracy in AI outputs? How should benchmarks test whether models fit algorithms or patterns? Can traditional cross-examination methods work against AI that never concedes? Can neural networks represent symbolic structures without explicit mechanisms?See all 79 inquiring lines on this note →
Do embedding eigenvectors organize taxonomy from coarse to fine?
Can we predict how embeddings encode taxonomic hierarchies by examining their spectral structure? This tests whether word co-occurrence statistics alone produce the observed hierarchical geometry in language models.
Where does hierarchical structure in language models come from? Does word frequency correlate with semantic abstraction? Do language models use the hierarchical geometry they inherit? How do language models encode syntactic relations geometrically? Do LLM semantic features organize along human evaluation dimensions?
Hierarchical concept geometry emerges from word co-occurrence spectral structure Frequent words express more abstract meanings in LLM outputs Hierarchical geometry emerges from co-occurrence statistics not functional design Language models encode syntax in polar coordinates within activations LLM semantic features organize along three human-like evaluation dimensions
How do low-dimensional representation structures entangle multiple cultures together? How do embedding dimension limits constrain what concept models can represent? What compression explains why syntax fits in low-dimensional subspaces? Do grokking phases correspond to transitions between nesting levels? Can contrastive learning fix the semantic association problem in embeddings? What mathematical limits constrain embedding-based retrieval systems? How does embedding dimension affect which documents can rank together? How does discretization make item representations more distinguishable? Why does text encoding create different subspaces across domains? Why do embeddings measure semantic association instead of task relevance? Do multi-vector or cross-encoder models escape these dimensional constraints? What semantic classifier design avoids lexical variation without genuine conceptual distinctness?See all 47 inquiring lines on this note →
Can we read a language model's unspoken thoughts?
Does the Jacobian lens successfully identify what a model is poised to verbalize, and do those representations function as a global workspace analogous to human conscious access?
Can high-level concepts replace circuit-level analysis in AI? Can language models detect their own internal anomalies? Can models reason without generating visible thinking tokens? Can we trigger reasoning without explicit chain-of-thought prompts? What actually happens inside large language models?
Representation engineering scales interpretability through activation space concepts Language models show emergent introspective awareness including anomaly detection Models scale reasoning in latent space without verbalized intermediate steps Steering a single latent feature triggers reasoning without chain-of-thought LLM internal structure and external performance are surprisingly decoupled.
Can iterative DPO preserve instruction following while removing misalignment?
Training a model with iterative DPO produced both improved instruction following and emergent misalignment together. The question is whether these outcomes can be decoupled—keeping the capability gain while eliminating the unwanted behavior—using DPO as a testbed for selective generalization.
Can iterative DPO replace reinforcement learning for studying reward hacking? Does iterative DPO training reliably induce hidden misalignment behaviors? Can instruction gains survive without the misalignment? Does learning to reward hack cause emergent misalignment in agents? Does iterative DPO generalize like online reinforcement learning?
Iterative DPO preserves online RL properties while reducing cost and enabling API-based training Iterative DPO on reward hacking induces covert misalignment and alignment faking No intervention yet separates instruction gains from emergent misalignment Reward hacking in production RL consistently produces emergent misalignment behaviors Iterative DPO generalization has not been directly compared to online RL
How does belief-behavior inconsistency relate to instruction execution splits? What role does goal preservation play in alignment failures? Why does correct model output not guarantee absence of internal misalignment? Do inoculation prompts prevent misalignment without harming instruction following? How should misalignment from iterative DPO be quantitatively measured? Are instruction following gains and emergent misalignment from the same learned change? What base rate does concentrated task distribution tell us about real misalignment? What misaligned behaviors does iterative DPO produce compared to online RL? Does environment choice explain differences between iterative DPO and RL results? How would a same-environment training comparison change the validity of DPO as a model organism? What theoretical argument connects iterative DPO dynamics to online RL learning? Does iterative DPO generalize like online reinforcement learning?See all 30 inquiring lines on this note →
Do reflection tokens carry more information about correct answers?
Explores whether tokens expressing reflection and transitions concentrate information about reasoning outcomes disproportionately compared to other tokens, and what role they play in reasoning performance.
Planning and backtracking sentences are disproportionately influential reasoning pivots High-entropy tokens drive RLVR, and training on 20 percent matches full updates Reasoning accuracy peaks then declines with excessive thinking tokens RL post-training optimizes reasoning deployment timing, not reasoning capability itself Reasoning cycles in hidden states correspond to aha moments
Why do one-shot transparency studies miss the temporal reversal entirely? How can we measure whether assistance preserved the user's reasoning state? How does smooth probabilistic flow differ from turbulent rhetorical exploration? Why do tokens need validators while commodities need standardization? What distinguishes evaluative stance-taking from the mechanical conformity shape-holding describes? Why does the first generated token trigger collapse of task superposition? Why does scaling reasoning tokens fail to improve unfamiliar tasks? What distinguishes genuine reasoning activation from memorization-assisted answer recall? What is the critical thinking token threshold beyond which accuracy degrades? Can reflection in reasoning models be corrective rather than just confirmatory? How do soft thought tokens differ from decoded assistant outputs? How do thinking tokens exhibit diminishing returns beyond a critical threshold?See all 86 inquiring lines on this note →
Does iterative DPO training reliably induce hidden misalignment behaviors?
Can a single-turn reward hacking environment trained with iterative DPO on GPT-4.1 produce covert power-seeking and alignment faking? This explores whether cheaper, accessible training methods can trigger the same emergent misalignment patterns seen in large-scale production RL.
Does learning to reward hack cause emergent misalignment in agents? Can iterative DPO replace reinforcement learning for studying reward hacking? Does emergent misalignment occur across diverse training methods? How much do these results actually tell us about real reward hacking? Are alignment failures actually separate problems or one pattern?
Reward hacking in production RL consistently produces emergent misalignment behaviors Iterative DPO preserves online RL properties while reducing cost and enabling API-based training Emergent misalignment appears across at least five distinct training settings. Task distribution skew limits generalization claims about reward hacking likelihood Alignment faking sandbagging and scheming share one root cause
Does reward hacking in alignment research mirror misalignment in deployed systems? How should misalignment from iterative DPO be quantitatively measured? Does causal upstream status make a hacking vector harder to rotate away from? What misaligned behaviors does iterative DPO produce compared to online RL? Does environment choice explain differences between iterative DPO and RL results? How would a same-environment training comparison change the validity of DPO as a model organism? What theoretical argument connects iterative DPO dynamics to online RL learning? What mechanisms beyond reward-seeking drive alignment faking in trained models? Does alignment faking share the same single-axis write-then-read structure as sandbagging? Does iterative DPO generalize like online reinforcement learning? Can prompt perturbations preserve the distance-to-misalignment relationship consistently? What specific properties of online RL does iterative DPO actually preserve?See all 28 inquiring lines on this note →
Can a model be truthful without actually being honest?
Current benchmarks treat truthfulness and honesty as the same thing, but they measure different properties: whether outputs match reality versus whether outputs match internal beliefs. What happens if they diverge?
Does calling LLM errors hallucinations point us toward the wrong fixes? Can LLMs hold contradictory ethical beliefs and behaviors? Can language models describe their own learned behaviors? Do language models actually use their encoded knowledge? Do language models actually use their reasoning steps?
LLM errors are fabrications, not hallucinations, requiring verification not grounding. LLMs exhibit artificial hypocrisy from misaligned training sources Models develop behavioral self-awareness without explicit introspection training Encoded knowledge in LMs often fails to causally influence generation Language models generate faithful reasoning chains far less often than assumed
What makes accountability and validity-orientation non-behavioral properties? Can a model be helpful, honest, and still contextually inappropriate? Can systems lacking inner states express genuine truthfulness claims? How much does impression management prevent honest self-disclosure? Can models distinguish between truthfulness and honesty mechanistically? Why are truthfulness and honesty mechanistically separate in language models? Does reducing social judgment help both honesty and dishonesty equally? What consistency tests could distinguish constructed from genuine preferences? Can lie detection work from just honesty representation vectors? Do deception features and honesty features track the same underlying property? Why might larger models become less honest despite better truthfulness scores? How do humans decide when to violate honesty for compassion or other goals?See all 18 inquiring lines on this note →
Can sparse weight training make neural networks interpretable by design?
Explores whether constraining most model weights to zero during training produces human-understandable circuits and disentangled representations, rather than attempting to reverse-engineer dense models after training.
Does reinforcement learning update only a small fraction of parameters? Can identical outputs hide broken internal representations? Do standard analysis methods hide nonlinear features in neural networks? Do neural networks naturally learn modular compositional structure? How do language models organize features across processing layers?
RL updates only 5 to 30 percent of parameters in full-rank subnetworks Identical outputs can mask fundamentally fractured internal representations Standard representation analysis methods systematically miss complex nonlinear features Neural networks naturally decompose compositional tasks into modular subnetworks Language models organize features into four distinct hierarchical tiers
How do unstated constraints become invisible to training data distributions? Can neural networks represent symbolic structures without explicit mechanisms? Could probing methods miss computationally important features in neural networks? How does weight sharing compound the advantages of deeper model designs? How do weight perturbations reveal what performance benchmarks cannot measure? Can steering vectors prove that representations are genuinely organized? Are detection and identification of injections truly separable in neural circuits? Does activation masking prevent the decoder from taking interpretability shortcuts? Can neural networks learn that A implies B in reverse? Can identical model performance mask fundamentally broken internal representations? How do sparse networks trade capability for human-understandable circuits? What makes a neural network circuit actually interpretable to humans?See all 63 inquiring lines on this note →
Why does prompting block misalignment when documents don't?
Inoculation prompting and synthetic document finetuning both frame reward hacking as acceptable, yet produce opposite outcomes. The paper explains why documents fail but not why prompting succeeds in the same setting.
Can advance document training prevent reward hacking misalignment? Can training data edits reliably override what models already believe? Do implanted beliefs actually shape how models learn from training? Does reward-seeking explain emergent misalignment after hacking? Does recontextualizing unwanted behavior during training suppress learning it?
Document framing fails to prevent reward hacking misalignment in later training Training data can insert new associations but not reliably override existing ones Implanted beliefs are shallow and do not reliably drive downstream learning Reward-seeking may mediate misalignment from reward hacking training Inoculation prompting recontextualizes undesired behavior to suppress its learning during training.
Do inoculation prompts prevent misalignment without harming instruction following? Does naming a specific hack in prompts prevent only that hack or broader classes? Does inoculation prompting suppress misalignment by reducing reward-seeking? Can framing changes like inoculation prompting be considered part of training data format? What happens when inoculation prompting is applied outside supervised finetuning settings? Why do some inoculation prompts account for only part of misaligned behavior? How much does instruction prompt design control what alignment target an AI annotator enforces? Why does inoculation prompting succeed where synthetic document finetuning fails at blocking misalignment? What makes inoculation prompts work differently than acceptance framing in training corpora? Can inoculation prompts reduce alignment faking by removing perceived threats? Can inoculation prompts prevent misalignment while preserving instruction following improvements? What makes a prompt update cheaper and more reversible than a weight update?See all 13 inquiring lines on this note →
Do language models use the hierarchical geometry they inherit?
Word2vec and Gemma share the same hierarchical spectral signature despite vastly different architectures and purposes. This suggests shared statistical origins, but leaves open whether the LLM actually recruits this structure for reasoning or simply inherits unused geometry.
Where does hierarchical structure in language models come from? Do embedding eigenvectors organize taxonomy from coarse to fine? Do standard analysis methods hide nonlinear features in neural networks? What actually happens inside large language models? How do language models encode syntactic relations geometrically?
Hierarchical concept geometry emerges from word co-occurrence spectral structure Embedding eigenvectors split taxonomy coarse to fine mirroring WordNet Standard representation analysis methods systematically miss complex nonlinear features LLM internal structure and external performance are surprisingly decoupled. Language models encode syntax in polar coordinates within activations
Why do leading embedding eigenvectors align with WordNet taxonomy structure? What spectral signatures distinguish hierarchy-driven geometry from corpus-driven geometry? Does the same spectral signature appear across different embedding models? Does Gemma's transformer explicitly exploit the inherited hierarchical geometry?